All three surveys had identical or similar questions regarding online experiences, which were used for comparative analysis. Table 1 provides characteristics of the three subject pools. Invitations were sent to respondents worldwide, including non-native English speakers, via email. An unknown number of invitations were lost by email filters.
Two incentives were provided to organizations and web designers to encourage their participation in the survey. First, IDEA pledged to contribute US$1 to One Laptop Per Child (OLPC) for each survey completed. Second, respondents were offered a summary of the study results if they provided IDEA with their email addresses. IDEA contributed US$813 to OLPC.
Table 1. Respondent pools
| Group | Description | Invitation method | Invited / clean responses |
|---|---|---|---|
| Nonprofit organizations & cities | Executives and communications directors from educational organizations, museums, and associations; usability experts; city webmasters. | Email invitations and discussion list posts | 5200 / 563 |
| Web designers & firms | Solo web designers and principals in design firms. | Email invitations | 4058 / 250 |
| General public | Visitors to WebExhibits museum exhibit on “Daylight Saving Time” | Unobtrusive text link on web site | Approx 1 million / 1675 |
In order to measure the index of effectiveness, a regression analysis was performed to identify key drivers. Cluster analysis was then performed to create groups of high/medium/low complexity and effectiveness. Further, several cross-tabulations were developed to identify the impact of site type, number of pages, number of visitors, number of visits, and so forth, for a variety of survey questions. Logistic and multiple linear regression models were used to identify relationships between questions regarding ability to find information and the presence or absence of an information gap.
During data cleansing, all records that were grossly incomplete were discarded (e.g., abusive records with the value “1” for every field and responses with less than 25% of the questions completed). Further, selected incomplete records were automatically rejected during statistical analysis. The number of records eliminated varied depending upon the type of statistical analysis and variables involved.
This study was designed and implemented by IDEA staff, led by Michael Douma. Primary data analysis and report writing was performed by Sathish Menon of Analytic Dimension, San Diego, California.
Based on the responses from visitors, effectiveness was measured relative to visitor enjoyment, ease of finding information, and ability to maintain orientation on the site (ability to maintain orientation was defined on the survey as visitors’ ability to know “where they are, where they can go next, and which pages are related”). An ineffective site is one where visitors are at best somewhat unsatisfied with respect to enjoyment, where it is not very easy (or worse) to find information, and where visitors occasionally (or worse) get “lost” on the site. An effective site is one where visitors are at least satisfied with respect to enjoyment, can find information at least somewhat easily, and never get lost on the site.
Based on the responses from designers, the effectiveness of a site was estimated using their perception of ease with which visitors can find information on the site, maintain orientation, and enjoy the visiting the site. An ineffective site is one where visitors cannot find information very easily, visitor enjoyment is at best neutral, and visitors are rarely able to maintain orientation on the web site. This is in line with the visitor expectations outlined in the paragraph above. With effective sites, according to designers, visitors can find information very easily or somewhat easily, visitor enjoyment is at least neutral, and visitors are able to maintain orientation almost always or most of the time. By at least one point on a five-point scale, visitors have higher expectations of effectiveness than designers. This suggests that designers should give greater consideration to overall effectiveness, thereby reducing the chance of failure for a user to find the information they seek.
Using responses from organizations, an effective web site was defined using visitor enjoyment, influence of personal navigational aids, and the perceived gap between the information provided by the web site and what visitor wants. An effective site is one where organizations believe that gaps do not really exist, visitors are at least “somewhat satisfied,” and organizations express skepticism about improving visitor experience if a “personal navigation aid” is present. An ineffective site is where organizations admit that that some gaps exist, visitors are at best neutral with respect to enjoying the site, and organizations are somewhat certain that a personal navigation aid would improve visitor experience.
It is interesting to note that the information gap surfaced as a critical factor for organizations but not for visitors. One possible explanation is that the general public views the entire World Wide Web as the information source rather than a particular web site. Hence, if sought-after information is not available from a particular web site, visitors will move on to the next one. This is consistent with the Pew Internet survey that searchers usually end up satisfied and that people tended to use two or three information sources in their quests.
According to organizations, ease with which visitors can find information and orient themselves on the web site (i.e., know where they are, where they can go next, and which pages are related) is positively correlated to visitors’ enjoyment. Also, non-profit organizations that believe a gap exists between the content of their site and visitors’ expectations think that their sites are difficult to navigate, that it is not easy to find information, and that visitors dislike using the web site.
Designers believe the ability to maintain orientation on a web site, good visual design, and the ease with which visitors can find information significantly impacts whether or not visitors enjoy a particular web site.
According to visitors, key contributing factors to enjoyment are the ease with which they can find information, the ability to maintain orientation, and, to a lesser extent, reducing the gap between what they expect and what sites provide.
Thus, all three survey groups believe that the ease with which visitors can find information and the ability to maintain orientation is critical to enjoyment. Both organizations and visitors believe that reducing the gap between what web sites provide and what visitors seek is critical to enjoyment. Although good visual design is not a key driver of enjoyment for visitors, more than 50% of visitors think that good visual design is a “very important” part of web design. (About 90% of visitors believe that good visual design is “somewhat important.”) One possible explanation is that visitors expect good visual design from all web sites, so it is not a differentiating factor but, rather, an essential factor.
Since these variables explain only 25% to 30% of the variance in visitor enjoyment, additional factors need to be considered during site design.
