Member Login
E-mail:
Password:

Reset Password

 

 

Vol 25, No. 2 (January 2008)

[article | discuss (0) | print article]

Designing Websites for ESL Learners: A Usability Testing Study

Min Liu
Tomoko Traphagan
Jin Huh
Young Ihn Koh
Gilok Choi
Allison McGregor
The University of Texas at Austin

Abstract:
The purpose of this paper is to report on a usability study for ESL websites conducted to gain insights from learners of English as a second language (ESL) as they interacted with specific sites. Five carefully selected ESL sites were tested by 10 different users generating a total of fifty testing sessions. Two major research questions guided the study: (a) what are intermediate-level ESL learners' perceptions of the design features for the specific ESL websites? and (b) what design features contribute to learning from the ESL websites? Participant reactions to site specific learning tasks and evaluative questionnaires provided evidence of design features relevant to the ESL context. The findings address information, interface, and interaction design issues as well as highlight the importance of (a) organizing the site meaningfully, (b) recognizing the users' prior knowledge to identify unambiguous labels and categories, (c) making the site easy to navigate, and (d) aligning learning activities or tasks with how ESL learners learn. Results of the study provide implications of the usefulness of ESL websites, the effectiveness of site design features, and the degree to which sites meet ESL learners' needs.

Designing Websites for ESL Learners: A Usability Testing Study

KEYWORDS

English as a Second Language (ESL), Website Design, Usability Testing, User-centered Design, Second Language Learning and Teaching

INTRODUCTION

The world wide web has become an established tool for language learning and teaching (Susser & Robb, 2004; Warschauer, 2005). Given the vast number of websites related to English as a Second Language (ESL) and the proliferation of their use, questions have been raised regarding the effectiveness of this evolving technology for learning English. Specifically, from an evaluation standpoint, Susser and Robb (2004) assert that "the vast quantity and variety of Internet resources for language study is welcome but leaves us with the problem of evaluating the quality of what is available" (p. 279).

Investigating design effectiveness of ESL websites requires establishing good design criteria for the context of second language acquisition (SLA). One window into understanding the design factors is by obtaining information directly from users. Usability testing provides an opportunity for understanding users' experience with a particular website. According to

0x01 graphic

207

Nielsen, Snyder, Molich, and Farrell (2000, p. 7) usability testing "involves watching a person use the site in order to discover the ways the site aids or hinders people from reaching their goals." Usability testing results not only highlight user preferences but also their behaviors while interacting with sites. The objective of this research is to contribute to design guidelines for ESL websites based on such directly observed user preferences and behaviors.

REVIEW OF THE LITERATURE

Websites can be useful resources in fostering English language learning. In particular, their capacity to offer authentic language tasks (e.g., reading authentic materials, participating in online chat, listening to native speaker recordings, and communicating with native speakers) and to boost students' motivation holds promise for facilitating the acquisition of English. Some studies have shown positive effects of using web-based activities on the students' conversational, reading, and writing skills (Gu, 2002; Kung & Chuo, 2002; Lin, 2003). Other studies have reported that web-enhanced activities fostered students' development of in-depth cultural understanding while improving linguistic proficiency, as well as generating increased engagement in the learning activities (Hoecherl-Alden, 2000; Osuna & Meskill, 1998).

Literature on using websites in ESL classrooms, however, also suggests that simply providing technologically enhanced learning tools to students does not itself foster learning but that, instead, positive results require carefully designed materials based on a critical understanding of the relationship between the language-learning processes and technology (Warschauer, Turbee, & Roberts, 1996). According to Plass (1998), design should support the cognitive processes involved in linguistic and pragmatic skill development for SLA. His argument is echoed by Peterson (1998, p. 350) who stated that "any general design rationale for the creation of web-based second language learning environments will be an integrative construct encompassing both a high-level theoretical view of how students learn and lower-level issues of interface design and site construction." Therefore, an effective ESL web learning site should embody not only good design with technological enhancements but also connect to SLA theory.

Good design creates a positive experience for users and can lead to more desirable outcomes (Phyo, 2003). In human-computer interaction literature, much emphasis has been placed on the importance of user-centered design (Norman & Draper, 1986; Norman, 2002). Understanding how users think can help create websites that meet users' goals and objectives (Krug, 2005). Design should address the three essential components of information, interface, and interaction. Information design is "the art of organizing data so that it is easy to find and use" the information on a site (Phyo, 2003, p. 2). Interface design refers to those elements that affect the appearance of a site (e.g., font, font size, colors, spacing, visuals, and icons) and the manipulation of these elements to both enhance the presentation of information and make the site easy to navigate (Phyo, 2003). In other words, information and interface should be purposefully implemented so that they enrich the overall experience of using a site. Accordingly, the result of a well organized site is that users know at all times what they can do in the site, where they are, where they can go, and where they have been (Krug, 2005). Interaction design encompasses the elements that create a meaningful and enjoyable experience (Shedroff, 1994) for the purpose of enhancing the way people learn, work, communicate, and interact (Preece, Rogers, & Sharp, 2002). Clearly, the three design aspects are related to each other and, hence, many design features involve more than one aspect. Together, information, interface and interaction make up the basic tenets of a sound design and should, therefore, be incorporated in well designed ESL sites.

208

Given the scarcity of empirical research addressing such design considerations for ESL sites, additional research is needed on how ESL learners actually use sites as a tool to facilitate their learning. The objective of this ESL websites usability study was to gain evaluative information from ESL learners for the purpose of informing design practices. As stated above, usability testing involves observing target audience participants using sites so researchers can note how the sites help, confuse, or defeat users. Then, design principles drawn from the users' perspective (both impressionistic and behavioral feedback) gained from usability testing should help build effective and attractive websites (Preece et al., 2002). Two primary research questions guided the study:

1. What are intermediate-level ESL learners' perceptions of the design features for specific ESL websites?

2. What design features contribute to learning from the ESL websites?

METHODOLOGY

Collection and Selection of ESL Websites for Usability Testing

This usability research study consisted of two phases: identifying potential ESL websites for usability testing and actual testing of five websites by ESL learners. Initially, a total of 80 websites were collected via solicitation from ESL instructors affiliated with a large university in the Southwest as well as Google searches for ESL-related websites. After reviewing each site's goals, content, target skill levels, and media, a list of 37 websites was generated for potential use in the study (see list of websites in Appendix A). A manageable number of candidate websites is recommended for evaluation (Susser & Robb, 2004), and according to Nielsen (n. d.), five evaluators are sufficient to locate most problems in a usability study. Given these recommendations, our goal was to select five sites from the list of 37 and test each site with 10 different ESL users to ensure both a manageable and rigorous research project.

To reduce the number of websites to five, we first excluded those websites that had broken links, charged a fee to use, or were part of other URLs on our list. Second, we eliminated sites that did not have a main learning component because our usability testing intended to evaluate websites from an ESL perspective in learning. At this point, the research team evaluated and rated each site using a 3-point scale based on how useful the site seemed to be. We retained the websites that received the highest ratings and two types of websites emerged: those that addressed more than one language skill and those whose primary focus was on grammar. Given this finding, we decided to focus our testing on the more comprehensive sites (sites addressing more than one language skill) and excluded those covering only English grammar. This resulted in a total of nine sites. The final selections were based on the type of media used, language skills covered, popularity, and appropriate activities for our intermediate-level participants. The final set of five sites for the usability testing were

1. Stuff for English Learners (http://www.Stuff.co.uk/wicked.htm)

2. English Club (http://www.englishclub.com)

3. Grammar & Writing (http://grammar.ccc.commnet.edu/grammar)

4. ESL Cafe (http://www.eslcafe.com)

5. Study Zone (http://web2.uvcs.uvic.ca/elc/studyzone/index.htm)

Each site will be referred to by the name listed above in later sections of this paper. A brief description of each site follows.

209

Website Descriptions

Stuff for English learners

The Stuff for English Learners (hereafter Stuff) site includes vocabulary, pronunciation, TOEFL, reading, and word games. It uses animation and interactivity supported by Shockwave and Flash technologies. For example, the Phrasal Verb Dispenser has a vocabulary activity for two-part verb combinations in which users first choose a topic (e.g. computers, emotions, etc.) from a side menu. Next, by clicking on arrows to change the verbs, particles, or meanings, users try to find the correct combination. Through another feature called the Phonetic Calculator, learners hear English vowel sounds after clicking on the corresponding symbols based on the International Phonetic Alphabet.

English Club

The English Club site offers over 5,000 pages and provides numerous resources for both learners and teachers. These can be accessed through a tab menu across the top of the home page (Learning Center, ESL Clubhouse, Teacher's Lounge, ESL Resources, and ESL World) or found through the site's search function. Activities for learners include quizzes (some with automatic scoring), timed typing practice with automatic scoring, games, synchronous chatting, as well as English jokes, and articles for reading. The site includes lessons not only for general adult learners (from beginning to advanced levels), but also English for business and young learners.

Grammar & Writing

The Grammar & Writing site focuses on issues pertaining to writing, including the writing process, organization, and common problems for ESL learners such as primer style and sentence variety. The home page is organized around six themes (Word & Sentence Level; Paragraph Level; Essay & Research Paper Level; Ask Grammar, Quizzes, Search Devices; Peripherals & PowerPoint; and GrammarPoll, Guestbook, Awards) with pulldown menus to navigate to the information or activities included in each section. Lessons are mainly presented through text, but there are over 170 interactive quizzes and crossword puzzles with feedback created in JavaScript and CGI.

ESL Cafe

Users enter the ESL Cafe site by clicking on one of the following labels: Stuff for Students, Stuff for Teachers, or Stuff for Everyone. The Stuff for Students section includes Help Center, Hint-of-the-Day, Idioms, Phrasal Verbs, Pronunciation Power, Quizzes, Slang, and Student Forums. Within this section, there are extensive lists of vocabulary (idioms, phrasal verbs, and slang) and over 40 interactive quizzes with automatic feedback/scoring on grammar, vocabulary, culture, reading, and writing.

Study Zone

The Study Zone site is designed for various levels of ESL learners: the lessons and practices are organized by five TOEFL score ranges (200, 330, 410, 490, and 570) with level specific lessons, grammar quizzes, reading, and vocabulary. At the 200 level, the vocabulary quizzes

210

are particularly extensive and guessing game activities are included. At the 490 and 570 levels, various activities such as timed reading, reading comprehension exercises, grammar exercises, crossword puzzles, and writing exercises are offered based on reading passages. The exercises have hints and/or immediate feedback.

Participants and Site Order

The participants in this usability study were intermediate-level ESL learners who volunteered to participate in the project in order to practice their English. Most were affiliated with a large research university in the southwestern part of the US and enrolled in ESL classes at the time of this research. Each participant was asked to test two sites with one hour allocated for each site. A short familiarity survey was given to the participants prior to the testing to determine whether they had had any previous experience using any of the websites being tested. Given their responses on the survey, the participants were randomly assigned to test a site with which they were not familiar. A total of 50 testing sessions were generated from 25 participants. In other words, each of the five sites was tested by 10 different participants. We also counterbalanced the order in which the participants tested the sites to minimize any bias that could possibly be introduced through the site sequence.

Instruments

To address our research questions, participants' information and feedback were collected through: (a) a demographic questionnaire, (c) an observation of how participants actually used the site while performing site related tasks, and 3) a feedback questionnaire evaluating the site. Construction of the instruments closely followed Nielsen et al.'s (2000) methodology and previous research on usability testing (Krug, 2005; Mayhew, 1999; Preece et al., 2002; Rubin, 1994) (see instruments used in Appendixes B and C).

Demographic questionnaire

The purpose of the demographic questionnaire was to collect participant information relevant to the usability testing of ESL websites. The questionnaire included three sections: (a) personal information (e.g. age, gender), (b) level of computer and web experience, and (c) English learning history including experience using the web for English training.

Site-related tasks

Based on Nielson et al.'s (2000) usability testing method, three types of tasks--first impressions, specific tasks, and open-ended tasks--were designed for each of the five websites tested. The research team developed these tasks to simulate activities learners would do in self-study. After each task, a think-aloud elicitation question was used to prompt verbal feedback from the participants. Sample think-aloud questions included "Is it easy/difficult to find phrasal verbs in this site? Why? Do you like to learn this way (by trial and error)?" Each task is described in more detail below.

Task 1: First impressions. In order to elicit participants' general first impression of the sites, free-form exploration was used for the initial task. The participants could look at any part of the website and were then asked to describe their first impression of the site.

211

Task 2: Specific tasks. The aim of the specific tasks was to observe participants' interaction with the site in semistructured tasks. Because the five websites covered different content, specific tasks were uniquely created for each site. For example, the following is a specific task used for English Club:

You want to improve your English speaking skill. Today, you have decided to study words and expressions for telephone English. Review the Telephone Terms section and answer the following questions:

1. Name one word or term that you newly learned from this site.

2. How long did it take to find the Telephone Terms section?

Task 3: Open-ended tasks. We asked open-ended questions which required the participants to browse the site in any way they chose and find information or express their opinions about the effectiveness of the activities. For example, the following are open-ended questions for Stuff:

Listen to a poem. Click Hints on Pronunciation for Foreigners and work on the activity for three minutes.

1. What did you learn from this activity about pronunciation?

2. Do you think this information is useful?

For tasks 2 and 3, the topics were chosen carefully from different skill areas including grammar, listening, writing, reading, speaking, chatting, and games (see sample tasks in Appendix B). One researcher created tasks for one site while other researchers from the team performed and evaluated the tasks. Modifications were made based on the feedback, and then two pilot-testing sessions were held to try out the tasks with two ESL participants from the target population of the study. The objectives of the pilot testing were to verify that (a) the tasks were understandable and doable, (b) the questionnaires made sense to the users, (c) the testing environment was functional, (d) the users understood the pretest introduction and instructions, and finally (e) the time allotted for each test was adequate. After pilot testing, several of the task explanations and survey questions were revised to make them more comprehensible. Also, to prevent user fatigue, the total number of tasks was reduced from five to four and the number of the required written responses was lessened.

Site feedback questionnaire

The purpose of the site feedback questionnaire was to assess the participants' overall satisfaction with their website experience. First, based on a Likert scale of 1 (very unsatisfied) to 5 (very satisfied), each website was evaluated in nine aspects including ease of finding the information/activities, quality of the information/activities, and appearance of the site. In the second part of the questionnaire, participants were asked to rate how useful each site was to improve different English skills (i.e. speaking, listening, reading, writing, grammar, and culture) on a scale of 1 (useless) to 5 (very useful). Finally, the third section included questions such as the likelihood of returning to the site and what they liked most or least about the site (See Appendix C for the questionnaire).

212

Procedure and Testing Facility

The usability testing was conducted with one researcher as a moderator for each participant. To ensure the consistency of the testing procedures, all the researchers followed the same testing protocol. When a participant arrived, the assigned researcher briefly explained the purpose of the test, the procedure, and instructions for each task. The participant then completed the demographic questionnaire. Afterwards, the participant was given the site familiarity survey to identify previous exposure to any of the five selected sites. Based on this information, the researcher randomly assigned a site the participant had never seen or used before. Next, the participant went to the assigned website and completed the specific learning tasks described above. Each participant was observed while he or she navigated the site and performed the given tasks. During the observation, the researcher took notes and captured screen shots of the web page(s) in which the participant seemed to have difficulty in performing the tasks.1 After each of the specific tasks, the researcher asked think-aloud elicitation questions. Finally, the participant was asked to complete the site feedback questionnaire.

The majority of the users, 23 out of 25, did their usability testing in one of the computer labs on the university campus. They used DELL PCs equipped with high-speed Internet connection with Internet Explorer and all plug-ins required to use the sites (i.e., Shockwave, Windows Media Player, and Flash Player). One user participated in this study in a university office with a Gateway PC laptop with the same software setup and Ethernet connection. The final participant completed the study at the home of one of the researchers using a PC desktop with the same software setup and high-speed cable connection.

RESULTS AND DISCUSSION

About the Participants

The ESL learners participating in this study ranged in age from 16 to 40, were intermediate-level in English-language learning, and equally represented males and females. All of them had experience in using computers, had access to the Internet, and daily surfed the web for various purposes. Three fourths of them had experience of using the web for studying English and reported doing so for about an average of 2 hours weekly (See Table 1).

0x01 graphic

213

0x01 graphic

Participants' Responses in the Site Feedback Questionnaire

As shown in Table 2, the compiled ratings from the site feedback questionnaire indicated the highest general satisfaction with English Club followed by ESL Cafe. For these two sites, all items were rated 3 or higher on the 5-point scale (1-very unsatisfactory to 5-very satisfied). Participants reported viewing English Club and ESL Cafe positively in terms of information, interaction, and interface designs, as well as their value in helping them practice language skills. Participants also reported that they would be likely to use these sites again or recommend them to other people.

0x01 graphic

In examining how participants with different demographic characteristics rated their experiences using the five sites, we found that higher ratings were given by (a) females, (b) those with fewer years of English learning, (c) intermediate-level learners (rather than high-intermediate-level learners), (d) novice computer users, and (e) participants with no

214

experience in using the web to study English (see Tables 3 and 4 below). Other demographic differences were not noticeable.

The finding that female participants gave higher ratings than male participants is consistent with other research results indicating females tend to have more positive attitudes toward learning using computers than males (Hao & Liu, 2006). The results also indicated that less experienced English learners and computer users found these ESL sites more helpful than the more experienced users. Presumably, less experienced English learners were aided more or found more value in using the websites. Users with more computer and web experience were perhaps more evaluative in their feedback indicating that those users with more technogy savvy may have different expectations than those with less experience.

0x01 graphic

215

0x01 graphic

Findings From Site Task Performance Data1

Analyses of the participants' reactions to the learning tasks took four steps. First, the observation notes for each testing session were written up and compiled. Second, in order to make sense of the data in a consistent and systematic manner, we compiled observation notes from one site. The observation notes were organized into a table with three columns: (a) the participants' reactions to the tasks as observed in their behaviors or their comments, (b) the researcher's interpretation and description of what he or she observed, and (c) whether what was observed related to the information, interface, or interaction design of the site. As themes related to site design emerged, a template for organizing the data was established. Using this template, we organized the data for the other four sites and summarized the findings for each site. We attempted to extrapolate findings reflecting opinions of most participants and identified design consideration across the sites. Finally, by drawing comparisons across all five sites, themes emerged. The following is a discussion of the results.

Overall site design issues

Organization of information. ESL websites often contain a rich amount of information. For users to access site content easily, organization in well thought out categories is imperative (Silver, 2005). Clearly presenting the site hierarchy or in layers of information is particularly important for the home page in which users attempt to understand how the information is organized in the site (Krug, 2005). For example, most participants favorably perceived the home page of ESL Cafe where the information was presented in three major categories (Stuff for Teachers, Stuff for Students, and Stuff for Everyone) and could identify how to enter the site through the navigation system (see Figure 1).

216

0x01 graphic

Similarly for Study Zone, the participants liked the idea of dividing the content of the website according to users' English levels (see Figure 2).

0x01 graphic

They pointed out, however, that the TOEFL scores on the home page might not be meaningful to those users who have not taken a TOEFL test. Since there was no explicit explanation about the levels of the scores, some users might have a hard time understanding the significance of these numbers. Using more meaningful labels such as beginner, intermediate, and advanced together with (or without) TOEFL scores or providing a short explanation about the TOEFL scores would therefore have improved the usability of the site.

Participants found other aspects of ESL Cafe's organization of information less clear.

217

For example, on the Complete List page of the Idiom section, some participants stated that the alphabetical listing was not helpful and suggested listing the idioms by themes. Additionally, under Hint-of-the-Day, after clicking List All Entries, a long list of date, topic, number, and contributor's name sorted by date was presented (see Figure 3).

0x01 graphic

The participants were overwhelmed by the large amount of information and stated they could not easily find the information they were interested in. Similarly, on the Forum page, postings were listed without categorization, and some participants expressed frustration in how much time it took to find information (see Figure 4). In all these cases, a theme-based categorization would have made the organization of information more transparent and accessible to the users.

0x01 graphic

218

Previous research on the organization of information has highlighted the importance of constructing a website's content in a meaningful and relevant way from the users' perspectives (Preece et al., 2002). Our findings suggest that information on ESL websites can be meaningfully organized by category of user (e.g., teachers or students), skill level (e.g., beginner, intermediate, or advanced), task (e.g., grammar, reading, writing, listening, and speaking), or themes/topics.

Layout. Users have varying degrees of tolerance for visual noise, and some users are easily distracted by a complex layout (Krug, 2005). In general, participants in this study disliked busy layouts, those which included too much information or formats which required a lot of scrolling. Pages in the Grammar & Writing site, for example, tended to be long and wide because there were too many words on a page. The participants needed to scroll down to locate the information they were looking for (see Figure 5). As one user stated, "It was very difficult to find information ...".

0x01 graphic

Although there was an Index button on Grammar & Writing's home page that led to an overview of all the topics covered in the site, it was placed near the bottom of the page. This potentially useful navigation tool, therefore, was not utilized by many participants.

Participants also disliked the long pages in ESL Cafe, which required a lot of scrolling (see Figures 3 and 4 above). In Study Zone, the vocabulary practice page was so long that users had to scroll down repeatedly to view all the questions on the page. Participants suggested that the page could be designed with a different layout such as using pulldown menus or multiple columns that could utilize the space on either side of the pictures. They also pointed out that the timer, which showed the remaining time for reading comprehension, was not visible after scrolling down in the window.

In general, pages crowded with words tended to overwhelm users, and long pages requiring a lot of scrolling made it difficult to find information quickly and easily. The challenge to present a rich amount of information in limited screen space necessitates efficient layout designs. For example, the reading comprehension section in Study Zone provided clearly defined task areas (Krug, 2005) in different frames so users could view the text passages, questions, and feedback to their answers simultaneously without feeling overwhelmed (see Figure 6). The participants particularly liked this feature.

219

0x01 graphic

The participants also reported they could easily find information they were seeking in English Club, even with the large amount of information on the home page, because of the left-hand side navigation menu and a multiple-column layout (see Figure 7).

0x01 graphic

Search function. A search engine is the first place where many web users go to find what they need (Krug, 2005; Silver, 2005). In English Club, all participants found the search engine function on the home page very helpful. However, it took a few participants some time to locate the search function (see Figure 8). The search box could have been displayed more prominently.

220

0x01 graphic

Use of pulldown menus. The use of pulldown menus are an effective way to save space when there is a lot of information to present in a website. However, they have a tradeoff: the items in the pulldown menus are not visible to users until they click on the pulldown menu or submenus in the pulldown menu (Krug, 2005). In the Grammar & Writing site, multiple pulldown menus were used as the main navigation tool, but participants could not see the items in the pulldown menus nor the hierarchy of the information without clicking on each one of the menus (see Figure 9).

0x01 graphic

221

One user mentioned "[pulldown] menus in both the main and other pages are ... very difficult and tedious [to use]." Pulldown menus can be useful, yet they should be used only for certain types of lists, such as alphabetized lists of known items (e.g., the 50 states) (Krug, 2005). Using them as the main navigation device is not effective as shown by the results of this study.

Home buttons and breadcrumbs. Breadcrumbs refer to a form of navigation that indicates the current location within the website by a list of pages from the current location up to the home page. Home buttons and breadcrumbs are among the most powerful navigation tools to indicate to users where they are in the site, especially for hierarchically organized sites (Silver, 2005). Breadcrumbs were not present or conspicuous in most sites we examined. For example, in Grammar & Writing, navigating through content pages relied solely on the pulldown menus, and this made it hard for users to navigate back to the home page or other pages. The lack of site ID linked directly to the home page and the inconspicuous home button, placed at the bottom of each subpage (requiring users to scroll down to see it), increased participants' frustration in navigation (see Figure 10).

0x01 graphic

In fact, two participants failed to find the Home buttons, and said, "I hope there is an icon that directly leads to the Home page somewhere because I had to click on the back button of the browser several times." This difficulty in navigation could have been reduced if the website had contained breadcrumbs.

The Study Zone site also provided go back buttons (to the previous or home page) as a main navigation tool, but these buttons were placed on the bottom of the page, and were hard for users to notice. Similarly, participants recommended a home button on Stuff and ESL Cafe to avoid getting lost. Such findings emphasized the need for using home buttons and breadcrumbs as essential navigation tools for sites containing a large amount of information with a complex hierarchical organization (Krug, 2005; Phyo, 2003).

Labels and word choices. Users decide which part of a website to click on based on the labels they see. The mapping between labels and the expected results from clicking them, therefore, should be clear to the users (Silver, 2005). As mentioned before, the TOEFL score

222

labels found on the home page of Study Zone could have been made more meaningful to users if terms such as beginner, intermediate, and advanced had been used.

In the Idiom section of ESL Cafe, there were hyperlinks to Complete List, Meanings and Examples, and Idiom (see Figure 11).

0x01 graphic

The Idiom hyperlink randomly brings up an idiom from the collection. However, only the word "Idiom" was hyperlinked, not the word "random" in front of it. One participant clicked on the link to Idiom without seeing "random" before it and got very confused to find one idiom presented to him. Also, another participant was very confused when he clicked on "E" on the Complete List page because the pulldown menu showed "F" instead of "E" (See Figure 12). In summary, when labels were not used properly, they misled and confused the participants.

0x01 graphic

In the Grammar & Writing site, labels contained a number of grammatical terms, such as "conjunctions," "determiners," and "interjections," which were daunting to the

223

intermediate-level ESL users. For example, "He [The participant] didn't know the word plural and I [the moderator] had to explain it to him," stated one moderator. Some participants experienced navigation problems simply because they were not familiar with the terms used in the menus and did not know where to click for the information they wanted, while the participants who happened to know such terms easily located their desired information. Although definitions of the terms were given at the beginning of each subpage, they did not appear to help the participants. This was perhaps because the participants were not likely to click on a menu item that they did not understand. Similarly, in using Stuff, some participants had trouble getting to the Phonetic Calculator page because they did not know what "phonetic" meant. Also in the Think of a Number activity in Stuff (see Figure 17 below), four participants failed to complete the task because they did not know the words "multiply" and "digits" used in the instructions.

Poor mapping diverts users' attention from the task to the interface (Silver, 2005), thereby sacrificing learning efficiency. Labels and other language elements used in websites should be selected so that they are easily understandable by users. For ESL websites in particular, users' language proficiency and their prior knowledge should be taken into consideration in deciding which labels to use.

Color and font size. A home page is a preview into a website. Users often determine whether they want to look into the site further based on the feeling they receive from a home page. Choice of colors for a home page is critical in this respect. The yellow-based colorful design of ESL Cafe (see Figure 1 above) was well received by most participants. One participant mentioned, "I like the interface design. They are colorful so that they make me interested in the things inside."

In Stuff, five participants found the home page unattractive, and four of them specifically attributed it to its dark background color, suggesting the need for the use of a lighter color (see Figures 13, 15, and 16 below). Similarly, participants perceived the grey background color of the Study Zone site as gloomy. A dark background color makes pages not only unattractive but also makes the text hard to read (Vest, Crowson, & Pochran, 2005). Hence dark colors should be avoided for web page background.

Participants in general did no like the small font sizes used for some websites. For example, in English Club, a few participants complained that the home page was busy with numerous small-lettered links (see Figures 7 and 8 above). Similarly, in Study Zone, the small font size and narrow spacing in the Reading Comprehension section were mentioned as a significant design problem. Designers of these sites may want to consider increasing the size of the fonts.

Design of learning activities

Activity design and instruction. Activity designs for some sites were not intuitive, and participants had difficulties in figuring out how to use the activities. This was particularly the case with Stuff. While exploring the site freely, four participants commented that they did not know how to do the activities. One participant, for example, said, "even though this website shows a new attempt to teach English using various multimedia tools, it is too difficult and complicated for me to figure out how to use each web page." For example, in Phrasal Verb Drinks Dispenser (see Figure 13), the participants were supposed to change verbs, particles, and meanings until they found the corresponding combination of the three components using the arrows. Two participants simply clicked the buttons on the right to change the content areas and got lost. Another participant kept clicking the external links at the top of the page. Later,

224

this participant could not figure out how to check the right answers. There appeared to be too many operational steps for the users to perform the activity, and the operation was not intuitive enough. Although brief instruction was provided, many participants did not notice it because the instruction appeared below the activity box.

0x01 graphic

In using ESL Cafe, a few participants got lost not knowing what they were supposed to do for the Hint of the Day activity. They explained that they did not know what the page was offering and/or how to use the page, specifically the search box. The title of the page, Hint of the Day, was not descriptive enough of what this section was about, nor did the search box or buttons clearly indicate what one was supposed to do on the page (see Figure 14). A brief description of what to do would be beneficial for this activity.

0x01 graphic

In using Grammar & Writing, participants' comments focused on navigation difficulties and the lack of instructions on how to use the website. For instance, one participant

225

complained, "I don't know how to use ... no introduction or instruction on how to get to the grammar part ... it was not very clear to me."

Ideally, the interface of an activity should be designed intuitively enough for users to easily navigate the site and complete the learning activities without instruction. If necessary, brief instruction should be provided in noticeable places for users to follow.

Animation and audio. One clear advantage of websites over textbooks or workbooks is their capacity to include multimedia such as graphics, animations, videos, and audios. In particular, it is desirable to provide audio in language learning websites since ample listening is considered important for language acquisition. In using ESL Cafe, one participant, who was very eager to learn how to pronounce the idioms presented in the site, said he sometimes had no clue how to pronounce the words without listening to them; there was no audio in ESL Cafe. The Pronunciation Calculator page in Stuff was one example that exploited animation and audio to facilitate language learning (see Figure 15).

0x01 graphic

In this activity, users could click the vowel buttons or combinations of buttons to play the vowels or diphthongs and also activate the animation that showed the lip movements to produce those sounds. Some participants mentioned that the animation was helpful to understand how to pronounce these sounds correctly. Unfortunately, the lips moved too fast and the animation was too small for some users to observe how the lips were moving. Many participants mentioned that the audio in this activity was helpful. They said, "It was helpful because it gives actual pronunciation unlike book-type dictionaries" and "I could hear the difference between two sounds." Also, for Hints on Pronunciation for Foreigners in Stuff (see Figure 16), one participant mentioned, "hearing the words helped me understand the words because I could compare some sounds that were similar to each other."

While animation and audio seem to make activities look fun, users may not necessarily appreciate these features if they are not presented in a clearly defined learning framework. In the animation-based Think of a Number activity in Stuff, six participants said it was fun but that they did not consider it as helpful for learning. One participant said, "This is just for fun, not for learning English. If I have free time, I would do it just for fun, but not for studying." Learning objectives should be defined before considering the inclusion of animation and audio, not vice versa. Also, it is important to consider loading time because a long loading time is likely to discourage users. Some activities in Stuff took a long time to load due to animations and audio.

226

0x01 graphic

Feedback. Complex activities may not be effective for learning because they demand a lot of time from users for relatively little learning. In the Phrasal Verb Drinks Dispenser activity in the Stuff site, participants were to change verbs, particles, and meanings until they arrived at the right combination of the three components (see Figure 13 above). All participants except for one stated that they disliked learning through this lengthy trial and error process. As one participant said, "Learning phrasal verbs this way is not easy, because you have to spend a lot of time, and it is boring. It would be better if they give you answers, after a couple of tries." Another participant gave up on the task after spending a fairly long time on it and said she did not like this type of learning because finding a right combination was too difficult for her. She added, "If the website provided two of the three components and asked you to find the other one, it would have worked." Still another participant said that one may not learn anything after spending so much time because the activity did not provide answers. Exploratory learning needs to be designed with care. If users have to explore an activity too long without receiving any concrete feedback, they may lose interest in it or become frustrated. For Think of a Number activity in the Stuff site (see Figure 17), very few participants actually finished the task because the exercise required a long chain of responses, and making one mistake in the process resulted in entire failure in completing the task. If hints were provided for getting back to the right track after making a mistake, the activity may have been perceived as more useful by the participants.

0x01 graphic

227

In all the above cases, task complexity could have been adjusted by incorporating some type of feedback. Potentially effective feedback for these activities could include providing hints or answers on demand from users and providing a dictionary tool. Hints for Pronunciation for Foreigners in Stuff provided hints (see Figure 16 above), which alleviated some participants' frustration when they could not find answers easily, although the participants still performed poorly due to the genre of the passage (poem) and the reading speed. Similarly, the Listen to Poetry activity in English Club allowed the participants to read the script along with the audio clip (see Figure 18), which was perceived as helpful. The participants, however, expressed that they would like to have an option of choosing whether they wanted to listen to the audio clip before actually looking at the script. When providing hints or answers, it is important to consider the ways for users to control the amount of help they receive, depending on their language ability.

0x01 graphic

Many participants liked the Help Center in ESL Cafe in which learners could post questions and receive answers. One participant explained that he was not good enough to communicate with his teachers because of his pronunciation, but, if he used this online function, it would be much easier for him to ask his questions. A few participants, however, commented that they did not like to wait to receive answers via posting and preferred instead asking somebody they knew (e.g., their teachers) or finding answers promptly by using search tools. A few others expressed their desire to have quiz functions; one participant mentioned that information in ESL Cafe was good but not exciting and suggested including quizzes so that users could check their learning themselves. For Study Zone, participants stated that it would be more helpful if the website provided explanations for wrong answers.

When providing feedback, it is important to do so in a way not to discourage learners. One participant was annoyed with the feedback given in Phrasal Verb Drinks Dispenser in the Stuff site. Whenever she made a mistake, the program showed a pink object falling from the top with a dull colliding sound and displayed the word NO!, which she considered to be unpleasant (see Figure 19). Feedback should be provided to encourage learners to continue learning, and feedback to wrong answers should focus on the learners' future improvement rather than their current mistake.

228

0x01 graphic

SUMMARY AND CONCLUSIONS

Because learners' ultimate goal in using ESL websites is to acquire knowledge and skills in English, their cognitive capacity should be oriented towards learning tasks rather than figuring out how to use the sites. In this respect, the usability of ESL websites is particularly important, and good design of a website increases its usability.

The findings of this usability study as revealed in the site feedback questionnaire (see Table 2 above) showed that overall the participants viewed the design of some sites (i.e., English Club and ESL Cafe) more favorably than the others (i.e., Stuff and Grammar & Writing). The English Club site received the highest ratings of all five sites in terms of its information, interface, and interaction designs, and participants had the fewest problems in navigating the site while performing their assigned tasks as indicated in the site specific tasks discussion above. Such results indicate that English Club and ESL Cafe met ESL learners' needs to a large extent. The results also showed that the participants were computer literate (see Table 3 above). All had web surfing experience and access to the Internet, and many of them used the web as a vehicle to learn English. This indicates a trend that as the web becomes a common tool at home and in classrooms, more ESL learners will rely on websites to develop their language skills. The design of ESL websites will become a significant factor to ensure their success and desirable learning outcomes. We have also found that more experienced computer and web users tended to have more critical views in evaluating websites (see Table 4 above). Creators of websites need to pay more attention to design practices as web technology matures and more people become experienced users.

As we observed our participants using the five sites, many usability issues surfaced. These issues, manifested less in some sites (e.g., English Club) but more in others (e.g., Stuff and Grammar & Writing), highlighted the importance of three design aspects: information, interface, and interaction. The insights we gained point to design features that are important to include in ESL sites to facilitate learning. Three main issues are important to consider for designing ESL sites.

First, ESL sites tend to carry a large amount of content because the development of language skills and knowledge requires exposure to a great amount of information. This complicates the design of content organization and navigation in multiple ways. In order for users to locate desired information or exercises, content needs to be presented in an organized structure with meaningful categories recognizable to the users. Although one may be tempted to squeeze as much information as possible into a limited space, long pages crowded with a lot of information written in small fonts with narrow line spacing overwhelm users, and excessive scrolling hinders the visibility of information. Using common and established navigational devices such as search functions, links to the home page, and breadcrumbs will help users locate information quickly and enable them to identify where they are in the site and to move efficiently through the site.

229

Second, users of ESL websites are trying to understand how to use the site in their second language. Special caution is required in deciding what words to use for labels and instructions. Technical terms should be avoided as much as possible, and labels and other language elements that are meaningful from a learner's perspective should be used. Because users' background knowledge and proficiency are likely to vary significantly, reference tools such as a glossary or links to dictionaries may also be useful.

Third, ESL websites often include learning activities such as quizzes, puzzles, and games. The design of learning activities should be guided by the learning objectives and SLA theory rather than the use of technology. Multimedia and feedback should be provided in ways to maximize learning.

Given the findings of this study, we recommend designers of ESL websites consider the following questions:

1. Is your site organized by level (e.g., beginner, intermediate, and advanced), skills (e.g., reading, writing, grammar, and speaking), themes or topics, and types of tasks (e.g., practice of irregular verbs)?

2. Are the pages easy to view in terms of page length, layout, text density, color, and font size?

3. How many clicks does it take to get to a specific place in your site, and can users find their way back to the main page easily? Should you add search functions, links to the home page, and/or breadcrumbs to your site?

4. What prior knowledge and level of English proficiency are assumed to make use of the labels and categories?

5. Will a learning activity or task on the site support SLA? Are the objectives of an activity and how to accomplish these objectives clearly communicated?

Websites are potentially powerful learning resources to a virtually limitless number of ESL learners. The design of ESL learning sites is a critical factor in how websites enhance their language-learning experience. Increasing the usability of an ESL website can lead to a more positive user experience and ultimately increase learning outcomes.

NOTE

1 After this article was accepted for publication, a number of the pages from the evaluated sites have been changed.

REFERENCES

Gu, P. (2002). Web-based project learning and EFL learners: A Chinese example. Teaching English with Technology: A Journal for Teachers of English, 2 (4). Retrieved October 24, 2005, from http://www.iatefl.org.pl/call/j_article10.htm

Hao, Y. W., & Liu, M. (2006). Students' attitudes toward online interaction. Academic Exchange Quarterly, 10 (4), 74-79.

230

Hoecherl-Alden, G. (2000). Turning professional: Content-based communication and the evolution of a cross-cultural language curriculum. Foreign Language Annals, 33 (6), 614-621.

Krug, S. (2005). Don't make me think: A common sense approach to web usability (2nd ed.). Indianapolis, IN: New Riders Press.

Kung, S. C., & Chuo, T. W. (2002). Students' perceptions of English learning through ESL/EFL websites. TESL-EJ, 6 (1). Retrieved July 10, 2006, from http://www-writing.berkeley.edu/tesl-ej/ej21/a2.html

Lin, A. (2003). An initial study on EFL learners' attitude towards multimedia application in language learning. Teaching English with Technology: A Journal for Teachers of English, 3 (2). Retrieved October 24, 2005, from http://www.iatefl.org.pl/call/j_prev.htm

Mayhew, D. J. (1999). The usability engineering lifecycle: A practitioner's handbook for user interface design. San Francisco: Morgan Kaufmann.

Nielsen, J. (n. d.). How to conduct a heuristic evaluation. Retrieved October 24, 2005, from http://www.useit.com/papers/heuristic_evaluation.html

Nielsen, J., Snyder, C., Molich, R., & Farrell, S. (2000). E-Commerce user experience: Methodology of the study. Retrieved October 24, 2005, from http://www.nngroup.com/reports/ecommerce/methodology.html

Norman, D. A. (2002). The Design of everyday things. New York: Basic Books.

Norman, D. A., & Draper, S. (1986). User centered system design: New perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.

Osuna, M., & Meskill, C. (1998). Using the world wide web to integrate Spanish language and culture: A pilot study. Language Learning & Technology, 1 (2), 71-92. Retrieved October 29, 2007, from http://llt.msu.edu/vol1num2/article4/default.html

Peterson, M. (1998). The virtual learning environment: The design of a website for language learning. Computer Assisted Language Learning, 11 (4), 349-361.

Phyo, A. (2003). Return on design: Smarter web design that works. Indianapolis, IN: New Riders.

Plass, J. L. (1998). Design and evaluation of the user interface of foreign language multimedia software: A cognitive approach. Language Learning & Technology, 2 (1), 35-45. Retrieved October 29, 2007, from http://llt.msu.edu/vol2num1/article2

Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: Beyond human-computer interaction. Hoboken, NJ: John Wiley & Sons.

Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. Hoboken, NJ: John Wiley & Sons.

Shedroff, N. (1994). Information interaction design: A unified field theory of design. Retrieved March 15, 2006, from http://www.nathan.com/thoughts/unified/index.html

Silver, M. (2005). Exploring interface design. Clifton Park, NY: Delmar Learning.

Susser, B., & Robb, T. N. (2004). Evaluation of ESL/EFL instructional web sites. In S. Fotos & C. Browne (Eds.), New perspective on CALL for second language classrooms (3rd ed.) (pp. 279-295). Mahwah, NJ: Lawrence Erlbaum Associates.

Vest, J., Crowson, W., & Pochran, S. (2005). Exploring web design. Clifton Park, NY: Delmar Learning.

Warschauer, M. (2005). Socio-cultural Perspectives in CALL. In J. Egbert & G. M. Petrie (Eds.), CALL research perspectives (pp. 41-51). Mahwah, NJ: Lawrence Erbaum Associates.

Warschauer, M., Turbee, L., & Roberts, B. (1996). Computer learning networks and student empowerment. System, 24 (1), 1-14.

231

APPENDIX A

Websites on ESL

Websites

Instruction/

Resource

Target

audience

Skill area(s)

Media

01

instruction

student

comprehensive (grammar, vocabulary, writing, pronunciation, reading, listening, spelling)

text, audio, quizzes with feedback, graphics

02*

instruction/

resource

student

vocabulary, reading

text, interactive quizzes

03

instruction/

resource

student/

teacher

grammar, vocabulary

text, graphics, interactive quizzes

04

instruction

student

comprehensive (grammar, culture, pronunciation, reading, vocabulary)

text, interactive quizzes

05

instruction

student

comprehensive (vocabulary, grammar, writing, pronunciation, reading, listening), TOEFL test preparation

text, interactive quizzes, audio, graphics

06

instruction/

resource

student/

teacher

comprehensive (speaking, listening, reading, writing, vocabulary, grammar, pronunciation)

text, audio, graphics

07

instruction

student

comprehensive (vocabulary, grammar, reading, writing, listening, spelling, culture)

text, interactive quizzes, audio, graphics

08

resource

student/

teacher

comprehensive

text, audio, video

09

instruction

student

writing, grammar

text, graphics, interactive quizzes

10

instruction/

resources

student/

teacher

listening, writing

text, graphics, audio

11

instruction/

resources

student

comprehensive

text, graphics

12

instruction/

resources

student/

teacher

comprehensive (grammar, vocabulary, idiom, phrasal verbs, words, slang, culture, reading, writing, punctuation)

text, graphics, sound, interactive quizzes

13

instruction

student

comprehensive (reading, writing, grammar, vocabulary, puzzles)

text, graphics, interactive quizzes

14*

instruction

student

listening, grammar, vocabulary

text, audio, interactive quizzes,

232

15

instruction

student/

teacher

grammar, vocabulary, games

text, interactive quizzes

16*

instruction/

resource

student

grammar, reading

text

17*

instruction

student

grammar

text, graphics, interactive quizzes

18

instruction

student

grammar, writing, listening

text, graphics, audio, interactive quizzes

19

instruction/

resources

student

grammar

text, audio, quizzes with feedback

20

instruction/

resources

student

grammar

text

21*

instruction

student

grammar

text, quizzes with feedback

22

instruction

student

grammar

text, quizzes with feedback

23

instruction

student

listening

text, audio

24

instruction

student

pronunciation

text

25

instruction

student

vocabulary

text

26

instruction

student

idioms

text

27

instruction/

resource

student

vocabulary

text

28

instruction

student

slang

text

29

resources

teacher

articles, teaching techniques, lesson plans related to ESL

text

30

instruction/

resources

student/

teacher

grammar, idioms

text, graphics

31

resources

student/

teacher

listening

text, audio

32

resources

student

general information on TOEFL test

text

33

resources/

instruction

student

comprehensive (grammar, vocabulary, reading, writing, pronunciation), exam preparation

text

233

34*

resource

teacher

reading, vocabulary

text, audio

35*

resource/

instruction

student/

teacher

comprehensive

text

36

resources/

instruction

student/

teacher

comprehensive (grammar, vocabulary, reading, writing, listening, games)

text, sound, quizzes with feedback

37*

http://www.bbc.co.uk/worldservice/learningenglish/compuspeak

resource

student

comprehensive

text

* These URLs were no longer available at the time of the submission of this manuscript.

APPENDIX B

Tasks for http://www.stuff.co.uk/wicked.htm

(One of the five sites tested in the study)

0x01 graphic

234

0x01 graphic

235

Appendix C

Survey Instruments used in the Study

0x01 graphic

236

0x01 graphic

237

0x01 graphic

238

0x01 graphic

AUTHORS' BIODATA

Min Liu, Ed.D., is Professor in the Instructional Technology (IT) Program at the University of Texas at Austin. She is also the Coordinator and Graduate Advisor for the IT Program. She has been teaching courses on new media design, production, and research for over 15 years. Her teaching and research interests center on the impact of new media technology on learning and the design of engaging and interactive learning environments for all age groups. She also has strong interests in technology use for second language learning. She publishes widely in leading instructional technology research journals and serves on a number of editorial/manuscript review boards. She is active in professional organizations such as AACE, AERA, and ISTE. She has directed and managed both CD-ROM and web-based development projects, including award winning ones.

Tomoko Traphagan is a research associate in the Division of Instructional Innovation and Assessment at the University of Texas at Austin. She has been involved in the research and evaluation of various instructional technology programs at the university, including webcasting, Second Life, digital media services, course management systems, web-based inquiry-based learning, and usability of instructional websites. Her research interests lie in the impact of technology on students' learning behaviors and performance.

Jin Huh is Assistant Professor in the Department of English at the Korea Military Academy. He is a graduate of the Foreign Language Program, Curriculum and Instruction at the University of Texas at Austin. His areas of interest include teaching foreign language pronunciation, accent modification, multimedia-aided language teaching, and teaching writing.

239

Young-Ihn Koh recently received her Ph.D. degree from the Foreign Language Education Program at the University of Texas at Austin and is now a part-time lecturer at Dongduk Women's University and Hanyang University in Seoul, Korea. Her research interests include teaching and learning with technology, especially the use of computer-mediated communication with young and adult ESL learners, teaching culture, and bilingual children's codeswitching patterns.

Gilok Choi is a Ph.D. student in School of Information at the University of Texas at Austin. Her area of specialization is in human computer interaction (HCI), user interface design, and usability studies and methods.

Alison McGregor is a graduate from the Educational Psychology Department at the University of Texas at Austin. Her research interests include technology in language learning, skill development, and prosodic features of American English. She has worked in the ESL field for over 10 years and has taught English in Taiwan, Japan, and the United States. She specializes in accent modification for nonnative English speakers.

AUTHORS' ADDRESSES

Min Liu

The University of Texas at Austin

Instructional Technology Program

Department of Curriculum and Instruction

1 University Station D5700

Austin, TX 78712

Phone: 512 471 5211

Fax: 512 471 8460

MLiu@mail.utexas.edu

Tomoko Traphagan

The University of Texas at Austin

Division of Instructional Innovation and Assessment, Continuing and Innovative Education

P.O.Box 7246

2616 Wichita Street

Austin, TX 78713-7246

Email: tomoko.traphagan@austin.utexas.edu

240