WO2001057706A2

WO2001057706A2 - Method and apparatus for enhanced path analysis on a world wide web site

Info

Publication number: WO2001057706A2
Application number: PCT/US2001/002491
Authority: WO
Inventors: Arthur Wu; Steven P. Ketchpel; Brian J. Hirschfeld; Joseph Tan; Maya Venkatraman; Mikhail Birman; Gareth Ivatt
Original assignee: Vividence Corporation
Priority date: 2000-02-03
Filing date: 2001-01-25
Publication date: 2001-08-09
Also published as: WO2001057706A3; AU2001231150A1

Abstract

A centralized full-service research bureau assists a customer in designing a usability test for typical tasks faced by a visitor to the customer's web site. The test is administered to a pre-qualified pool of testers. A method and apparatus is provided for tracking the paths taken through the customer's web site by a population of testers in attempting to accomplish a predetermined series of objectives during the course of the usability test, and for performing subsequent analysis of the path information. The path data for individual testers are saved to a database and aggregated across the population of testers. A suite of software tools is provided that retrieves and presents the path data in various formats as report sections in a browser window. The suite of path analysis tools organize the path data in a manner that allows the customer or an analyst to discern patterns in the path data and assess the relevance and effectiveness of the various pages and sections of the target web site to the visitor's objectives, and to evaluate the visitor's experience as they navigate the web site. The disclosed method and software tools may also be used to analyze path data from conventional server log files.

Description

METHOD AND APPARATUS FOR ENHANCED PATH ANALYSIS ON A WORLD WIDE WEB SITE

BACKGROUND OF THE INVENTION

TECHNICAL FIELD

The invention relates to usability testing of a World Wide Web site. More particularly, the invention relates to enhanced path analysis through the use of software tools.

DESCRIPTION OF THE PRIOR ART

In the traditional shrin wrap software world, product usability testing was a business process that was considered nice to have. The software sales cycle relied heavily on product reviews, word-of-mouth, and controlled, static demonstrations (demos). Actual use of a product did not happen until after a user had purchased the product. Costs for customers to switch from a first product to a second product were high, so customer usability was given a low priority.

Today, in the online world, the product is the World Wide Web (Web) site and the Web site is the sales cycle. The usability of a Web site therefore has a direct and critical effect on how much revenue the site can generate. However, usability testing methods have not yet caught up to the requirements of Web site development processes.

Traditional user interface (Ul) tests are labor intensive, cost-prohibitive, take ages to run, and yield subjective, non-analytical results at best. Online product competition is now in hyper-drive, and despite multi-million dollar budgets for a typical product launch, Web product managers still do not have an objective, rigorous way of measuring the customer satisfaction and usability of features on their own Web sites. The situation for competitive intelligence is even worse.

The frustration due to the lack of information is summed up by the product manager of a Top 5 Portal Site:

No one in this space knows what they're doing. The most successful people are also the most superstitious about their methods, since with no hard data, they feel their success - however great - has only been based on luck and gut feel so far. Everyone has an opinion, but no one can tell me why any one product is really winning or losing.

Poor understanding of a Web usability and feature quality measurement system is due to a number of factors:

1. The pace of technological advance. In the arms race of new technology, companies' implementation timetables are driven by the competition, not their Information Technology (IT) staff.

2. The multiplicity of functions. Functions for a corporate information Web site are inappropriate for a pre-teen chat room. Designers have to figure out how each function (email, chat, games, shopping, etc.) works best, taking into account the unique audience and ^• objectives of the particular Web site.

3. A flood of raw data but a lack of information. Web logging efforts provide a multitude of traces of user activity. Yet determining what the Web logging data means is impossible because a most important piece, the user's objective, is missing. The most advanced Web site-level tracking methods available focus on whether a marketer's objective, i.e. purchase, registration, ad clicks, etc., was met. The user's objective is ignored. The implication is that a look-to-book ratio twice as good as the industry average might not be cause to celebrate. Perhaps hundreds of willing buyers didn't purchase an item or service because they didn't understand the return policy.

4. Little in-depth testing. Traditional user testing methods, such as, for example, usability laboratories with one-way mirrors, are expensive, labor intensive, and require long turnaround times. In addition, they provide highly subjective results across an insignificant sample size.

O. Davis and V. Jain, Method and Apparatus for Tracking Client Interaction with a Network Resource and Creating Client Profiles and Resource Database, U.S. Patent No. 5,796,952 (18 August 1998) discloses a method for monitoring client interaction with a resource downloaded from a server in a computer network. The method comprises the steps of using a client to specify an address of a resource located on a first server and downloading a file corresponding to the resource from the first server in response to specification of the address. The method further comprises the steps of using the client to specify an address of a first executable program located on a second server, the address of the first executable program being embedded in the file downloaded from the first server. The first executable program includes a software timer for monitoring the amount of time the client spends interacting with and displaying the file downloaded from the first server. The method further comprises the step of downloading the first executable program from the second server to run on the client so as to determine the amount of time the client interacts with the file downloaded from the first server. The method further comprises the step of using a server to acquire client identifying indicia from the client, and uploading the amount of time determined by the first executable program to a third server. The first executable program may also monitor time, keyboard events, mouse events, and the like, in order to track choices and selections made by a user in the file. It may execute upon the occurrence of a predetermined event, as well as monitor or determine the amount of information downloaded by the client. The monitored information and client identifying indicia is stored on a database in a server for use in analysis and for automatically serving out files assembled according to user interests and preferences

While an objective of the Davis teachings is to provide means for creating a database of user profiles containing details of individual user interaction with and use of network resources and of the amount of time spent by users interacting with and/or using particular resources, as well as details of choices created by individual users within a particular resource, Davis does not disclose or suggest a usability test or pre-qualified testers. Davis does not disclose or suggest that a customer take part in the design phase of a usability test to accomplish a set of customer objectives, which take into account a user's objectives.

S. R. Coffey, D. B. Pinsley, and K. A. Poloniewicz, Computer Use Meter and Analyzer, U.S. Patent No. 5,675,510 (07 October 1997) discloses a system that measures and reports the use of a personal computer by a user through a log file. The log file includes entries corresponding to predetermined events and can report on the applications used and communication functions engaged in by the user. The log files from one or more computers may be assembled and analyzed in order to ascertain computer use habits for computer software, computer hardware and computer communications. The system may also be used to predict computer use trends and to represent computer use history.

The monitoring system of Coffey, et al., provides traces of what users choose, but the disclosure does not teach nor suggest why users navigate down particular paths. Coffey, et al, discloses that their system collects child Window information for commercial online service providers and user applications. Window titles of these applications' child Windows generally hold useful descriptions of the activity at that moment. For example, if a subscriber is using a mail system for a service, then the Window title so indicates. The system records those titles in a log file.

Coffey, et al., further discloses that online marketers try to understand the characteristics of Web traffic and how much time users spend at different sites. Traffic statistics become fundamental inputs for media planning, in a fashion analogous to using TV ratings as a basis for buying or pricing commercial time. However, Coffey, et al. does not disclose nor suggest that users' objectives be considered in any process of the system.

A. N. Goldhaber and G. Fitts, Attention Brokerage, U.S. Patent No. 5,794,210 (11 August 1998) discloses a system that provides for the immediate payment to computer and other users for paying attention to an advertisement or other negatively priced information distributed over a computer network such as the Internet. It is the business of brokering the buying and selling of the attention of users. A further disclosure allows advertisers to detach their messages from program content and explicitly target their audience. A special icon or other symbol displayed on a computer screen may represent compensation and allow users to choose whether they view an ad or other negatively priced information and receive associated compensation. Targeting users may be provided by reference to a database of digitally stored demographic profiles of potential users. Information is routed to users based on demographics, and software agents can be used to actively seek out users on a digital network. Private profiles are maintained for different users and user information is released to advertisers and other marketers only based on user permission. Users may be compensated for allowing their information to be released. Competing advertisers may bid for the attention of users using automatic electronic systems, e.g. an auction protocol and these concepts can be generalized to provide an electronic trading house where buyers and sellers can actively find each other and negotiate transactions. While Goldhaber et al. discloses its four basic principles are attention, interest, sponsorship, and privacy, it gains knowledge about the behavior of users through estimates of past behavior. That is, Goldhaber et al. discloses a mechanism by which advertisers actively compete by bidding for a viewer's attention, wherein the bids are based, in part, on estimates of the viewer's interest and likelihood to buy. The estimates are derived from access to the viewer's electronic profiles detailing preferences and past consuming behavior. Goldhaber et al. does not disclose nor suggest a mechanism for learning from a viewer why a viewer chooses particular actions.

Goldhaber et al. discloses demographic profiles constructed through interest questionnaires that a customer completes when subscribing to the disclosed service and also through electronic tracking of the customer's usage of the service (and other habits). The profiles are dynamic and evolve with the customer's transaction history. A customer can choose to exclude any transaction (e.g. viewing of certain material or purchasing of certain products) from his profile. The profile is also interactive in that a customer edits the profile at any time to add or delete interest features and to delete any transaction records. Goldhaber et al. does not disclose nor suggest a testing mechanism to measure a customer's success in achieving the customer's objectives. Nor do Goldhaber et al. disclose or suggest prompting for context-specific feedback ranging from the aesthetics of the design to the reason a page request is terminated. Furthermore, Goldhaber et al. does not disclose or suggest aggregating statistics across a testing population and presenting the testing results with recommended actions backed up by analysis.

Various methods of compiling web site traffic reports from raw data from web server log files are known. Such reports permit a web site owner to view the total number of hits to a site, the number of page views, the paths taken by individual visitors through the site, the URL's the hits originated from, and so forth. The traffic reports permit a web site owner to assess which pages and sections of a site are more heavily visited than others, and to make limited inferences about the demographics of the visitor population. However, web site traffic analysis does not permit aggregation of path information. Additionally, it is not possible to represent an individual web page as it was served up to the visitor, and the traffic reports have no way of eliciting and reporting feedback from the individual visitor. Furthermore, website traffic analysis reports have no way of taking into account the objectives of the individual user, therefore it is impossible to know why a visitor accessed a certain page, and whether they accomplished what they came for. In view of such limitations, the inferences that may be made based on web site traffic reports are of very limited usefulness

It would be advantageous to provide a method and apparatus that provides Web product managers with quick and detailed feedback on customer satisfaction of their own and competitive products across a large sample size using efficient techniques.

It would be advantageous to provide a method and apparatus that provides a usability test for typical tasks faced by a customer to a Web product manager's site. Such a usability test would measure a customer's success in achieving the customer's objectives and also prompt for context-specific feedback ranging from the aesthetics of the Web page design to the reason a page request is terminated.

It would be advantageous to provide a method and apparatus that aggregates statistics across a testing population and presents data with recommended actions backed up by analysis. It would be a further advantage to provide a method of aggregating path information and to provide a series of software tools that facilitate in-depth analysis of both aggregate and individual path information.

SUMMARY OF THE INVENTION The invention provides a method and apparatus for tracking the paths taken through a World Wide Web site by a population of testers in attempting to accomplish a predetermined series of objectives during the course of a usability test of a targeted World Wide Web Site, and for performing subsequent analysis of the path information. The path data for individual testers are saved to a database and aggregated across the population of testers. A suite of software tools is provided that retrieves and presents the path data in various formats as report sections in a browser window. A report reader, typically the owner of a web site or an analyst may then view the data. The usability test and the path analysis tools are all World Wide Web applications; therefore the usability test is administered in a web environment and the report reader generally accesses the path data from a remote location using a web browser.

The suite of software tools includes:

• A Path Analysis Tool that displays aggregate path information, so that a report reader is able to follow the exit paths taken by various groups of testers as they attempt to accomplish a selected objective.

• A Reverse Path Analysis Tool, very similar to the Path Analysis Tool, that presents the aggregate path data in reverse, so that the report reader is able to follow a path in reverse, starting at the path's terminal node.

• A Tester Detail Tool that allows the report reader to choose an individual tester and follow the path taken by that tester as they attempt to accomplish an objective. Demographic data for the individual testers are also accessible using this tool.

An impact analysis tool that presents the various sections and pages of a web site in a hierarchic directory format. The number of visitors to each section or page and the numbers of those visitors who were successful and unsuccessful at accomplishing a selected objective are displayed.

• A Graphical Path Analysis Tool that graphically represents aggregate Path Information as a tree. Exit Paths followed from a node are represented as branches originating from the node. The Graphical Path Analysis Tool represents path information in a convenient, high- impact graphical format.

The suite of path analysis tools organize the path data in a manner that allows the report reader to discern patterns in the path data and assess the relevance and effectiveness of the various pages and sections of the target web site to a visitor's objectives. In a preferred embodiment, the invented apparatus presents the path data textually in a tabular format. In an alternate, equally preferred embodiment, the apparatus employs a graphical interface. Various interface elements common to the several tools allow the report reader to make qualitative assessments of a visitor's experience as they navigate the target web site in attempting to accomplish their objectives.

BRIEF DESCRIPTION OF THE DRAWNGS

Figure 1 provides a schematic illustration of a method of aggregate path analysis, according to the invention;

Figures 2A - D illustrate a Path Analysis Tool, and the vaπous steps of a method of using the tool, according to the invention;

Figures 3A - B illustrate a Reverse Path Analysis Tool, and the various steps of a method of using the tool, according to the invention;

Figures 4A - C illustrate a Tester Detail Tool, and the various steps of a method of using the tool, according to the invention; Figure 5 provides an illustration of an Impact Analysis Tool, according to the invention;

Figure 6A - C illustrate a Graphical Path Analysis Tool, and the various steps of a method of using the tool, according to the invention;

Figure 7 provides a view of an alternate interface to a suite of Path Analysis Tools, wherein aggregate path data are presented in a graphical format, according to the invention;

Figure 8 illustrates a method of navigating the interface of Figure 7, according to the invention;

Figure 9 shows thumbnail images of web pages representing exit paths from a current page in the interface of Figure 7, according to the invention;

Figures 10a - 3d illustrate the steps of a method of setting a Flag threshold and a Flag Criterion in the interface Figure 7, according to the invention; and

Figures 11a - 11c illustrate the steps of a method of displaying the aggregated responses to a selected question in the interface of Figure 7, according to the invention.

DETAILED DESCRIPTION

The parent application to the current application, A. Wu and S. Ketchpel, Full- Service Research Bureau and Test Center Method and Apparatus, U.S. Patent Application Ser. No. 09/277,574, March 26, 1999 describes a centralized full-service research bureau and test center method and apparatus that assists a customer of the research bureau to design a usability test for typical tasks faced by a visitor to the customer's web site. The usability test measures the visitor's success in achieving a series of tasks or objectives. The usability test incorporates means of gathering various data. Among the data gathered during the course of the test are path data for each one of a population of testers to whom the usability test is administered. The current invention provides a method and apparatus for tracking the paths taken through a

World Wide Web site during the course of the usability test and aggregating the path data across the population of testers for further analysis. A suite of software tools is provided in which a report reader, typically a web site owner or analyst, retrieves the path data, and uses each of the software tools to view the data and to perform various analyses thereupon.

DEFINITIONS

• Path - A succession of web pages or World Wide Web sites visited by a tester as the tester attempts to accomplish a predetermined objective.

• Node - An individual web page or World Wide web site in a path.

• Start page - The starting node of a path. At the outset of the usability test, each of the testers is forced to a start page, specified in the design of the usability test.

The invention provides a suite of tools, whereby the owner of a web site can retrieve and display path data, either in aggregate form or for each individual tester, including:

• a Path Analysis Tool

• a Reverse Path Analysis Tool

• a Tester Detail Tool

• an Impact Analysis Tool • a Graphical Path Analysis Tool

The tools are provided in the form of a report presented in separate sections. Each section of the report provides a separate tool. The various sections of the report are accessible through the Table of Contents frame in a multiframe window displayed in a web browser.

During the course of the usability test, the path information is saved to a database. The path information is explicitly encoded and saved to a path table in the database. Each time a new link is followed by a tester, a new entry is made to the path table showing the full path, terminating on the current page accessed by the tester. For example, given a path A→B→C→D, A, A→B and A→B→C each represent separate paths in addition to A→B→C→D and therefore constitute separate entries into the path table. Database entries for duplicate paths taken subsequently by other testers are resolved to the row of the table bearing the original entry.

Many World Wide Web sites dynamically generate pages based on session ID's that are encoded into the URL. The session ID's generally are not significant determiners of the content presented to a visitor to a web site.

Similarly, arguments in the query string of a URL may not be significant determiners of page content. It is desirable to prevent such non-content bearing portions of URL's from being saved in the path data; therefore the site at such a URL is analyzed, and the non-content bearing portions of the

URL are stripped from the URL using regular-expression matching implemented in stored procedures in the database.

As will be described below, the various tools allow a report reader to view a thumbnail image of a web site so that the they may see the web page as it was when the tester visited it. In order to provide meaningful test data, it is important to capture what the tester actually saw. Because of the dynamic, time-dependent nature of many web sites, if the report reader merely attempts to re-visit the web site at a later time, they may not see what the tester saw. However, live hyperlinks are provided so that the site may be viewed as it currently appears, in addition to the historical information provided by the thumbnail images. PATHANALYSIS TOOL

OVERVIEW

The first tool of the suite is a Path Analysis Tool. The Path Analysis Tool appears as a report section, titled "Path Analysis" displayed in the Table of Contents frame of the report. The Path Analysis Tool presents path data in an aggregated format. Data aggregation enables a report reader to discern meaningful patterns in the path data quickly and easily. Figure 1 provides a schematic illustration of how the Path Analysis tool enables a report reader to view the aggregate path data. In this example, a group of twelve testers all start on the same web page, page A (11). From page A, the group of twelve went to three different pages. Of the twelve, eight testers went to page B (12). The other four testers went to other pages. Following the group that went to page B, it can be seen that five of these eight testers went to page C (13) from page B. The other three testers from page B went to other pages. Following the five testers who went to page G, one tester went to page D (14), and the others to another page. As the report reader follows the paths taken by the various groups of testers, it is possible to make inferences about the design, content and functionality of the pages encountered by the testers. The aggregate path data enables the report reader to examine each intervening node in the paths taken by the various groups of testers to identify possible problems with the design or content or functionality of those pages that prevented the majority of the testers from accomplishing the desired objective, and conversely, to identify features of the page that were particularly conducive to successfully accomplishing an objective.

A list of objectives to be accomplished by the testers, originally specified in the design of the usability test is displayed as a pulldown list. The parent application to the current application, U.S. Patent Application Ser. No. 09/277,574, supra, provides a detailed description of the design of a usability test for a World Wide Web site. Generally, although not always, the objectives consist of questions the tester must answer. A typical objective would be to ask the tester to determine what page of the site a particular sort of information may be found on. Thus, successfully accomplishing an objective is usually defined as providing a correct answer to the question posed by the objective. In the current embodiment of the invention, the list of objectives is presented as a pull-down menu, however other interface elements adapted for the presentation of information in lists would also be suitable.

USING THE PATH ANALYSIS TOOL

Figures 2A through 2D illustrate typical steps taken by a report reader as they interact with a preferred embodiment of the invented Path Analysis tool.

Step 1

As shown in Figure 2A, the Path Analysis Section is accessed by selecting the corresponding hyperlink 23 in the Table of Contents frame 21 , whereupon the Path Analysis section is presented in the report reader's browser as a multi-frame window 20. The report reader begins interaction with the Path Analysis Section by selecting an objective from a pulldown menu of objectives 22.

Step Two

After selecting an objective, the report reader is now presented a tabular listing of the pages to which testers went from the Start page. Referring to Figure 2B, the main content frame provides the following interface elements:

• A pulldown menu of objectives 22, previously described, with an objective, in this example, objective #4 - "What is LandsEnd. corn's return policy," - selected.

• A History box 24 - The history box provides a history of the path a reader is following consisting of each of the nodes visited by the reader to arrive at the current node. The current node is listed at the end of the path; nodes to the left of the current node are those pages visited by the testers prior to arriving at the current node. In the current example, the path has only one node 25, start, denoting the start of the test.

• Current Page Section 26 - An image of the current page 27 as it appeared to the testers at the time of the visit is provided. The Current page details 28 indicate the total number of testers who visited the page, the number visiting who accomplished the objective, the number visiting who failed to accomplish the objective, indicated by an incorrect answer to the question of objective #4, and the number visiting who abandoned their attempt to accomplish the objective. In the example provided, two hundred thirty testers started the test. Of these, one hundred sixty-four eventually accomplished the objective; fifty-five gave incorrect answers to the question "What is LandsEnd. corn's return policy?" and therefore failed to accomplish the objective; eleven chose to give up before they found an answer.

• Details for exit paths from current page 29 - A complete listing of the pages that testers went to from the current page is presented in table form. In the current example, the start page, titled "Welcome to

Land's End™," is the only exit page shown. Thus, the number of testers following that exit path, provided in the second column of the table, is equal to the total number of testers. Of the number of testers who followed a specific exit path, the number of those testers who eventually accomplish the objective is provided in the third column, in the current example, one hundred sixty-four. To see where testers went from "Welcome to Land's End™," the reader clicks the URL 30 in the corresponding row of the table.

Step Three

Referring to Figure 2C, the exit paths table 29 now shows a list of the pages testers chose to visit from "Welcome to Land's End™." The history has been updated to show the current node, i.e "Welcome to Land's End." (31) In the current page area 26 the current page image 27 is a screen shot of the "Welcome to Land's End" page. Referring now to the exit path table 29, eighty-four testers followed the link to the page titled "To call, fax or e-mail Lands End." Forty-seven of these eighty-four testers eventually accomplished the objective. Proceeding to the next row of the table, it is apparent that forty-five testers followed the link to the page titled "The Company Inside and Out." Of those forty-five, forty-one eventually accomplished the objective. The report reader may wish to find out if there was some information or some links relevant to the objective on the "Company Inside Out" page that contributed to the high success rate, so they would click on the hyper link 32 in that row to display information for that page. As previously described, the page just selected will become the current page. The screen shot of the selected page, now occupying the current page area 26 allows the report reader to inspect that page visually, and the exit path information 29 allows the reader to see what the testers did once they reached this page. Alternatively, as shown in Figure 2D, the report reader could choose to follow the hyperlink in the first row to see if they could find out why so many of the testers who went to this page eventually failed in their attempt to answer the question, and thus accomplish the objective.

While the Path Analysis Tool has been described in relation to a usability test for a World Wide Web site, the tool has other applications. For example, the Path Analysis Tool may be used either by itself or as one of a suite of tools to analyze data from server log files to generate enhanced web site traffic reports.

REVERSE PATH ANALYSIS TOOL

The Reverse Path Analysis tool illustrated in Figure 3A, also provided as a report section titled "Reverse Path," presents aggregate path information starting at the pages where testers ended the tests. In the same manner as the Path Analysis Tool previously described, the Reverse Path Section is accessed from a clickable hyperlink 51 in the Table of Contents frame 21. The interface elements are analogous to those presented in the foregoing description of the Path Analysis Tool, and thus will be only briefly described herein. The pulldown list of objectives 22 is as previously described. The starting screen displays a list of pages 42 at which testers chose to answer the question posed in the objective or gave up on the test, in other words, at the termination of a path. The history box 24 lists the path history in reverse order to the path history of the previously described Path Analysis Tool. As with the Path Analysis Tool, an image of the current page 27 is displayed and details 28 related to the current page are displayed.

Rather than displaying details for exit paths from the current page, the Reverse Path Analysis Tool displays details for entry paths to the current page. When a report reader chooses one of the pages presented in the start screen, a list of pages is presented from which the testers came to the current page. When the report reader chooses one of these pages, the next screen shows a list of pages 42 from which testers came to the selected page. In this manner, as shown in Figure 3B, the report reader is able to work backward and follow tester groups back to the original Start page. In the current embodiment of the invention, the Path Analysis Tool and The

Reverse Path Analysis Tool resemble each other closely. To assist the report reader in distinguishing between the two sections, the interface of the Reverse Path Analysis Tool is configured as a mirror image of the Path

Analysis Tool.

While the Reverse Path Analysis Tool has been described in relation to a usability test for a World Wide Web site, the tool has other applications. For example, the Reverse Path Analysis Tool may be used either by itself or as one of a suite of tools to analyze data from server log files to generate enhanced web site traffic reports. TESTER DETAIL TOOL

A further tool provided by the suite is the Tester Detail Tool. The Tester Detail Tool allows a report reader to follow the path taken by an individual tester and thus perform a page level analysis for each tester. As with the other tools, the Tester Detail Tool appears as a report section, titled "Tester Details," and is accessed by means of a clickable hyperlink in the Table of Contents frame of the report window.

USING THE TESTER DETAIL TOOL

Step One

As shown in Figure 4A, the Tester Detail Section is accessed by selecting the corresponding hyperlink 51 in the Table of Contents frame, whereupon the report reader drills down a level in the hierarchy of the Table of Contents to a listing of the individual testers, each tester being denoted by a unique numerical identifier. After selecting a tester, the report reader selects an objective from the objective pulldown 22. In addition, the report reader may elect to view demographic information for each individual tester.

Step Two

After a tester and an objective are selected, a listing of the web pages 53 the tester visited in succession while attempting to accomplish the selected objective is presented, and the report reader is then able to follow the path the taken by the tester in attempting to accomplish the objective.

Further entries in the table 54 indicate the amount of time a tester spent viewing a page, and the amount of time the tester had to wait as the web page loaded. The amount of time a tester spends at a page allows the report reader to make inferences about the clarity and understandability of a web page, and its relevance to the objective. The wait time is an important indicator of possible performance problems with a World Wide Web site. Often, when a visitor to a web page has to wait an excessive amount of time for a page to load, they become frustrated and move on to another page. Thus, wait time information may provide the report reader with valuable insight into possible performance problems with the web site.

To follow the path taken by a tester, the report reader can click on the hyperlink in each succeeding row of the table or they may back up by clicking the hyperlink in the preceding row. Graphical interface elements comprising "next" (55) and "previous" (56) arrows, activated by clicking with the mouse, provide an alternate means of navigating the path taken by the tester. In the current tool, the current page image 27 occupies the center of the content frame. A preview image 57 of the page at the node previous to the current page is displayed to the left of the current page image. A preview image 58 of the page at the node following the current page is displayed to the left of the current page image 27. As Figures 4B and 4C show, the images change as the report reader navigates backward and forward in the path. In Figure 4B, the report reader moved back one node, so the former Current Page is displayed as the Next Page, and the former Previous Page is displayed as the Current Page. In Figure 4C, the report reader moved forward one node, so the former Current Page is displayed as the Previous Page and the former Next Page is now occupies the Current Page image.

While the Tester Detail Tool has been described in relation to a usability test for a World Wide Web site, the tool has other applications. For example, the Tester Detail Tool may be used either by itself or as one of a suite of tools to analyze data from server log files to generate enhanced web site traffic reports.

IMPACT ANALYSIS TOOL

An additional tool of the suite is an Impact Analysis Tool. The Impact Analysis Tool presents information such as click stream analysis that permits the report reader to make qualitative inferences regarding the testers' experience and how the various pages and sections of a site affected them.

As with the other tools of the suite, in its preferred embodiment, the Impact Analysis Tool is accessible from the Table of Contents frame of the report, and is viewable in a multi-frame window as a further section of the report. In one frame, the organizational scheme of the web site is depicted as a hierarchic directory structure 61. The directory structure permits the site to be presented as a series of regions and sub-regions. The interface is meant to resemble an explorer tool to browse and navigate directory structures in windowed computing environments. Referring to Figure 5, the example site is organized into Service, Sales, Product Information, and Company Information sections. When the Product Information section is selected, the hierarchy of the selected section is displayed such that the various sub directories of the section drop down beneath the section entry. Thus in the example, Product Information is shown to contain Sub Dir 1 , Sub Dir 2, Sub Dir 3 and Sub Dir 4. Sub Dir 4 is shown to contain Page A. In another frame, for the section highlighted the following information is displayed:

Directory or Page Name

Number of Visitors

Percentage Success

Percentage Success of those who did not visit

Number of views per visitor

Time spent on average per visit.

In another, equally preferred embodiment of the invention, the Impact Analysis Tool incorporates click number data.

The information provided by the Impact Analysis Tool enables the report reader to discern various trends and patterns, among them: • For visitors who failed to access a particular section of a site, was their level of dissatisfaction higher than those who did access it?

• High failure rates for regions that are relevant to the objective indicate that those regions may be poorly designed. • High rates of visits to sections unrelated to the objective may indicate that some aspects of the site misdirect visitors.

Another embodiment of the Impact Analysis Tool provides a feature in which the report reader selects a test question and the answers to the test question aggregated across the entire population of testers who visit a particular region are presented in a graphical format, thereby allowing the reader to assess the quality of the testers' experiences as they visited the various URL's attempting to accomplish the objectives of the test.

While the Impact Analysis Tool has been described in relation to a usability test for a World Wide Web site, the tool has other applications. For example, the Impact Analysis Tool may be used either by itself or as one of a suite of tools to analyze data from server log files to generate enhanced web site traffic reports.

GRAPHICAL PATH ANALYSIS TOOL

OVERVIEW

The Path Analysis Tool .previously described, presents path information primarily in the form of textual data in a tabular format. Figures 6a - 6C illustrate a Graphical Path Analysis Tool, in which the report reader is able to view path information presented as tree. The Graphical Path Analysis Tool presents path information in a convenient, high-impact format.

Referring now to Figure 6A, the Graphical Path Analysis Tool is accessed from a clickable hyperlink 70 in the Table of Contents frame, as with all of the previously described tools. The tool provides a pull down list of objectives and a current page area having a current page image and current page details.

The tree 71 occupies the content window. Each of the nodes 72 is represented as an individual graphical element; in the exemplary embodiment, they are depicted as diamonds. The branches 73 originating at the nodes represent exit paths taken by various groups of testers. Each node bears a page indicator 74 bearing the title and/or the URL of the page at that node. The current page 75 is always situated at the center of the content window. A 'Home' button 77 returns the report reader to the start page of a particular path, and a live hyperlink to the actual URL at the current page is provided.

USING THE GRAPHICAL PATH ANALYSIS TOOL The report reader navigates the tree by mouse-clicking the various nodes of the tree; single-clicking a node causes focus to shift to that node, and the page at that node becomes the current page. The tree 71 is redrawn in the window so that the newly selected node is displayed at the center of the tree. An image of the page at the newly selected node is displayed as the current page image, and the path statistics for that node are displayed in the current page statistics. As the newly selected node assumes focus as the current page, the various exit paths from that node are displayed as an array of branches originating from that node.

The thickness of a branch is directly proportional to the total number of testers following the corresponding path, and bands of color in the branch of varying width indicate the relative numbers of testers who successfully accomplished an objective, the number who failed, and the number who abandoned their attempt. In Figure 6A, a thick, multi-colored branch 76a originates from "Welcome to Land's End." The thickness of the branch indicates that a relatively large group of testers followed that path. The branch is composed of multi-colored bands: green, red and brown. Green indicates successful testers; red indicates testers who failed, and brown indicates testers who abandoned their attempt. The widths of the color bands are indicative of the relative number who succeeded, failed and abandoned their attempt. In Figure 6B, the current page 75 is now "Microfleece Bucket Hat." A solid green branch 76b originates from the current page node, indicating that all testers following that path successfully accomplished the objective. In Figure 6C, a red branch 76c, originates from the current page 75, indicating that all of the testers following that path, failed in their attempt to accomplish the objective.

In the default condition, the tree is displayed three nodes deep; however, in the case of a large, complex tree, the report reader may wish to collapse a portion of the tree to simplify the display and conserve space within the content window. Double-clicking a node hides all of the branches originating from that node; to view the branches again, the report reader simply double clicks the node again. A graphical indicator, such as a colored highlight surrounding a node, indicates that the branches originating from that node are hidden.

GRAPHICAL INTERFACE

From the foregoing discussion, it is apparent that the tools are separate components of a suite of software tools for aggregate path analysis. In the previously described embodiments, textual data is displayed to the report reader, primarily in a tabular format. In an alternate embodiment of the invention, the tools employ a graphical interface.

Figures 7 - 11 provide exemplary views of the invented graphical interface. Referring to Figure 7, shown is the initial screen of the Path Analysis Tool. The objective pulldown list 81 is situated in the upper left hand corner of the window 80. The current page area 82 containing the current page image 83 and the current page details 84 is situated at the left center of the window 80. The path history 87 is displayed across the top of the center of the window 80. The page exit details are presented in a graphical format at the right center of the window. Rather than being presented as rows of a table, as in previous embodiments, the exit details are presented as a series of boxes 85, with the data for each exit path appearing within the area enclosed by one box. The arrows 86 linking the current page area 82 with the list of exit paths are of varying thickness, with the thickness of a particular arrow being proportional to the number of testers who followed the corresponding path. Attention is drawn to the flags 88, 89 in the second and fourth boxes of the list of exit paths. Flags, described in detail farther below, are a convenient graphical indicator of the approximate numbers of a group of testers following a particular path who succeed and fail at an objective.

In using the Path Analysis Tool, a report reader clicks one of the boxes containing path information with their mouse to follow the group of testers who took that exit path from the current page. Clicking the box launches an animation sequence in which the selected box slides to the left of the screen and replaces the page occupying the current page area 82. Thus, focus is shifted to the newly selected page as the current page. Simultaneously, a new list of exit pages is presented in the boxes 85 corresponding to the exit paths from the newly selected current page. The number of exit pages determines the number of boxes 85 visible. A scroll bar 90 permits the report reader to display additional boxes when the list of exit pages is too long to be displayed in entirety. Figure 8 illustrates the scrolling functionality. In the list of exit pages displayed in Figure 8, the box that formerly was second from the top of the list in Figure 7, now occupies the top position 1000, and a new box 101 appears at the bottom of the window. It will be apparent that the scroll bars allow either upward or downward navigation in the list of exit pages.

The current embodiment of the invention includes a feature that allows the report reader to display thumbnail images of the exit pages. A "see thumbnails" button (91 a, 91 b) allows the report reader to toggle between two modes: hide thumbnails (Figure 7) and show thumbnails 110 (Figure 9).

As previously indicated, flags 88, 89 are graphical indicators of the relative numbers of testers who are successful and unsuccessful at accomplishing the specified objective. In default mode, a threshold is set, where the threshold is a percentage. If the percentage of a group of testers following a particular exit path who failed to accomplish the objective exceeds the threshold percentage, then a flag appears next to that exit page, as shown in Figure 7. The report reader sets the threshold by clicking the "Select Flag Threshold" button 92. A dialog box (not shown) appears, allowing the report reader to specify a threshold percentage.

Using the "Select Flag Criterion" button, it is also possible to specify a multidimensional trigger for the appearance of a flag. As described in the parent application to the present application (U.S. Patent Application Ser. No. 09/277,574, supra), throughout the course of the usability test, feedback is elicited from the testers in the form of questions. For example, as a tester visits various pages in attempting to accomplish an objective, they may be posed the question "How satisfied were you with this site while completing this objective?" The tester would then rate their level of satisfaction on a numerical scale, 1-7 for example. Using the "Select Flag Criterion" button 93, the report reader is able to use tester responses to these questions to set the Flag Criterion. Continuing with the foregoing example, the report reader clicks the "Select Flag Criterion" button 93. As Figure 10A shows, this triggers the appearance of a dialog box 120 containing a list of questions (121 , Figure 10B) upon which the flag may be based. This list of questions comprises the feedback questions presented to testers in the course of a usability test. The report reader selects one of these questions as the criterion for the flag, for example "How satisfied were you with this site while completing this objective?" After selecting the question, the report reader sets the criterion for the flag by specifying which of the answers are desirable (122, Figure 10C) using a list of the answers and corresponding checkboxes. After setting the criterion, the report reader sets the threshold (123, Figure 10D) by specifying a threshold percentage for the appearance of a flag, for example, "Show a flag if the percentage of testers selecting undesirable answers is greater than 30%." Thus, the flag allows the report reader to make inferences about the quality of the testers' experiences and their states of mind as they navigated the vaπous pages and sections of the World Wide Web site being evaluated.

The foregoing description of the flag threshold and flag criterion is meant to be exemplary only. Other applications of the flag consistent with the spirit and scope of the invention will be readily apparent to those skilled in the art.

The report reader is also able to display graphs relating to success and failure rates for the various exit pages. The "show graphs" button 94 allows the report reader to toggle between a "hide graphs" mode and a "show graphs" mode. In the "show graphs" mode, a mouseover event triggers the appearance of the graph. That is, if the report reader passes over a box 85 with their mouse, a graph corresponding to the exit page appears. A mouseout event causes the graph to disappear. In the default condition, the numbers of testers following the particular path who accomplished the objective, those who failed, and those who abandoned the attempt to accomplish the objective are represented. The "Select question to graph" button 95 allows the report reader to graph responses to the various questions asked of the testers during the course of the usability test. As Figure 11A shows, when the report reader clicks the button 95, a pulldown list of questions 130 appears. The report reader selects the question (131 , Figure 11 B) and a graph of the responses (132, Figure 11 C) aggregated across the group of testers who went to that exit page from the current page is displayed as the report reader's mouse passes over the box 85. As the mouse is taken away from the box, the graph disappears. It is also possible to incorporate a feature in which success or failure at a test objective is correlated with the various answers to a selected test question, and then to present the result graphically. Thus, the provided method and apparatus allow an analyst or web site owner to design and administer a usability test to a pre-qualified group of testers meeting desired demographic constraints. In the course of the usability test, the testers are asked to accomplish a series of tasks, or objectives similar to those usually faced by visitors to a web site. Paths taken by the testers are tracked and the path data is saved to a database. Unlike prior art methods of examining web traffic data from server log files, the aggregated path data allow the web site owner or analyst to study the behavior of groups of testers as they navigate the web site. The report reader is able to follow testers from node to node in a path and see exactly what the visitor to the URL saw. By examining exit path data from a current page, the report reader is able to know what the testers did upon reaching a particular page. Unlike prior art methods of web traffic analysis, which provide no knowledge of the site visitor or of their objectives and motivation, analysis of the objectives and the data concerning success and failure at accomplishing the objectives allow the reader to know why testers chose to take a certain path, and to correlate paths taken with success or failure for a given objective. The invented method and apparatus allow the report reader to evaluate the quality of the testers' experience, and to determine which sections of a web site are conducive to a satisfactory experience for a visitor, and which are lacking in terms of design, content and functionality.

While the disclosed method and apparatus have been described above in conjunction with a usability test for a World Wide Web site, they may also be applied to analysis of data from server log files to generate enhanced web site traffic reports incorporating aggregate path data. The suite of Path Analysis Tools may be deployed independently, either singly or as a suite of software tools.

Although the invention is described herein with reference to a variety of preferred embodiments, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the Claims included below.

Claims

CLAIMSWhat is claimed is:

1. A method of enhanced path analysis, comprising the steps of: providing path information for each of a population of visitors to a

World Wide Web site, said path information comprising a succession of URL's at said World Wide Web site visited by said visitor; aggregating said path information across said population of visitors; viewing said path information using a software tool, wherein said aggregate path information is presented as a path taken by any of an individual visitor and a group of visitors within said population, and wherein a node in one of said paths taken comprises a visited URL; and replaying at least one of said paths taken in a node-by-node fashion.

2. The method of Claim 1 , wherein said population of visitors comprises a population of pre-qualified testers, and wherein said providing step comprises the steps of: administering a usability test to said population of pre-qualified testers, wherein said testers navigate a series of hyperlinks to accomplish a specified objective, and wherein said node further comprises a URL visited by at least one of said testers attempting to accomplish said specified objective; and storing said path information from said usability test in a database, said database having a table for storage of said path information, and wherein said path information is explicitly encoded therein.

3. The method of Claim 2, wherein a new entry is added to said table each time a new link from said series of links is followed by one of said testers, and wherein said entry comprises a full path to said link.

4. The method of Claim 2, wherein said test is a World Wide Web application.

5. The method of Claim 2, wherein said software tool comprises a Path Analysis Tool, wherein aggregate test paths are displayed for analysis.

6. The method of Claim 5, wherein said viewing step comprises the steps of: selecting an objective from a list of objectives; and viewing aggregate test path information displayed for said objective, said aggregate test path information representing an aggregation of all paths followed by said plurality of testers in attempting to fulfill said selected objective, wherein a node represents a point of departure to any one of several alternative subsequent nodes.

7. The method of Claim 6, wherein said replaying step comprises the steps of: selecting a node from said aggregate path, wherein said selected node comprises a current page; viewing an image of said current page as it appeared during said visit; viewing current page details, wherein said current page details comprise the number of testers out of said population of testers visiting said current page; optionally, visiting said current page; viewing exit paths from said current page, wherein said exit paths comprise all URL's visited from said current page, and wherein each one of said exit path URL's comprises an alternative next node in said aggregate path; and selecting one of said alternative next nodes, wherein said selected next node becomes said current page.

8. The method of Claim 7, wherein a first node in said aggregate path comprises a start page, wherein all of said population of testers start said usability test at said start page.

9. The method of Claim 8, further comprising any of the steps of: analyzing which of said nodes in said aggregate path lead said testers to a next node unrelated to said selected objective; analyzing how many of said testers successfully accomplish said selected objective; analyzing how many of said testers fail to accomplish said selected objective; and analyzing how many of said testers abandon an attempt to accomplish said selected objective.

10. The method of Claim 9, said Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein an objective is selected from a list of objectives; a second interface element accessible from within said window, wherein a path history for said current page is displayed; details for exit paths from said current page displayed in a first of said plurality of frames; current page details displayed in a second of said plurality of frames, said current page details further comprising the URL of said page, an image of said page and current page statistics.

1 1. The method of Claim 10, wherein said current page statistics comprise: the total number of testers following a path to said current page; the number of said testers reaching said current page who successfully accomplish said selected objective; the number of said testers reaching said current page who fail to accomplish said selected objective; and the number of said testers reaching said page who abandon an attempt to accomplish said selected objective.

12. The method of Claim 2, wherein said software tool is a Reverse Path Analysis tool, wherein reverse aggregate test paths are displayed for analysis.

13. The method of Claim 12, wherein said viewing step further comprises the steps of: selecting an objective from a list of objectives; viewing reverse aggregate test path information displayed for said objective, said reverse aggregate test path information representing an aggregation of all paths followed by said plurality of testers in attempting to fulfill said selected objective, wherein a node represents an arrival point from any one of several alternative preceding nodes.

14. The method of Claim 13, wherein said replaying step comprises the steps of: selecting a node from said reverse aggregate path, wherein said selected node comprises a current page; viewing an image of said current page as it appeared during said visit; viewing current page details, wherein said current page details comprise the number of testers out of said population of testers visiting said current page; optionally, visiting said current page; viewing entrance paths to said current page, wherein said entrance paths comprise all URL's from which said testers came to said current page, and wherein each one of said exit path URL's comprises an alternative previous node in said aggregate path; and selecting one of said alternative previous nodes, wherein said selected previous node becomes said current page.

15. The method of Claim 14, further comprising any of the steps of: analyzing which of said nodes in said aggregate reverse path lead said testers to a next node unrelated to said selected objective; analyzing how many of said testers successfully accomplish said selected objective; analyzing how many of said testers fail to accomplish said selected objective; and analyzing how many of said testers abandon an attempt to accomplish said selected objective.

16. The method of Claim 15, said Reverse Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein an objective is selected from a list of objectives; a second interface element accessible from within said window wherein a reverse path history for a current node is displayed; details for entry paths to said current page displayed in a first of said plurality of frames; details for said current page displayed in a second of said plurality of frames, said current page details further comprising the URL of said page, an image of said page and current page statistics.

17. The method of Claim 16, wherein said current page statistics comprise: the total number of testers following a path to said current page; the number of said testers reaching said current page who successfully accomplish said selected objective; the number of said testers reaching said current page who fail to accomplish said selected objective; and the number of said testers reaching said page who abandon an attempt to accomplish said selected objective.

18. The method of Claim 2, wherein said software tool comprises a Tester Detail Tool, wherein a specific tester from said plurality of testers may be selected and wherein path information corresponding to said tester is viewed.

19. The method of Claim 18, further comprising the steps of: selecting a specific tester from said plurality of testers; selecting a test section; and selecting an action, wherein said action is any of Go to the next, and Go to the previous.

20. The method of Claim 19, wherein said Tester Detail Tool comprises: a window, said window comprising at least two frames; a plurality of navigation controls, any of said navigation controls used by a report reader to move from one node to another node in a selected path; an interface element, wherein a URL for a node being currently viewed is displayed; a current page image; a previous page image; and a following page image.

21. The method of Claim 20, wherein said path information is displayed in a first of said plurality of frames, and wherein said current page image, said previous page image and said following page image are displayed in a second of said plurality of frames.

22. The method of Claim 21 , wherein a first of said plurality of navigation controls is a "next" navigation control, said "next" navigation control used by said report reader to view a page on said target World Wide Web site corresponding to a next node in said selected path, and wherein a second of said plurality of navigation controls is a "previous" navigation control, said "previous" navigation control used by said report reader to return to a last- viewed page corresponding to a node immediately preceding a current node.

23. The method of Claim 2, wherein said software tool comprises an Impact Analysis Tool, and wherein a directory structure representing regions and sub-regions of said World Wide Web site is displayed.

24. The method of Claim 23, further comprising the steps of: posing questions to said testers during said navigation of said links, wherein said questions elicit said testers' level of satisfaction; recording and storing said testers' answers to said questions; viewing site impact data, wherein said impact data comprise: directory name; page name; number of testers to visit each of said regions and sub- regions; percentage of success for testers who visit a specified region; percentage of success for testers who do not visit said specified region; number of times each tester views said specified region; and time spent in said region by a selected tester; and analyzing failure rates for said specified region to assess any of design quality, usability, and relevance of said region to said specific objective.

25. The method of Claim 24, further comprising the steps of: selecting one of said questions; retrieving said stored answers to said selected question; creating and viewing a graph of said answers to said selected question; analyzing said impact of said specified region based on said graph; providing a means for storing and recording information about the number of mouse clicks it requires a tester to accomplish said objective; and retrieving and viewing said mouse click data wherein said usability and tester satisfaction are assessed based on said mouse click information.

26. The method of Claim 25, wherein said Impact Analysis Tool comprises: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein said specific objective is selected from a list of objectives; a second interface element accessible from within said window, wherein said question is selected from a list of questions; a third interface element accessible from within said window, wherein said graph is created and displayed; a fourth interface element accessible from within said window, wherein said mouse click data is displayed; said directory structure displayed in a first of said plurality of frames; said impact data displayed in a second of said plurality of frames.

27. The method of Claim 2, wherein said software tool comprises a Graphical Path Analysis Tool, wherein said path information is displayed as a tree structure, said Graphical Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein said specific objective is selected from a list of objectives; and a tree structure displayed in a one of said frames, said tree structure comprising: graphical interface elements representing nodes in a path; branches originating from said graphical interface elements, wherein said branches represent exit paths from said nodes, wherein thickness of a branch is proportional to the number of testers following a path, and wherein relative proportions of testers succeeding at said objective, failing said objective and abandoning an attempt to achieve said objective are indicated by colored bands of varying width within said branches.

28. The method of Claim 27, further comprising the steps of: selecting a node, wherein said selected nodes assumes focus as a current page, and wherein additional branches are displayed representing exit paths from said current page; optionally, hiding branches originating from a node.

29. The method of Claim 1 , wherein said providing step comprises retrieving path information from a server log file.

30. The method of Claim 29, wherein said population of visitors comprises all of the visitors to said World Wide Web site within the time period covered by said server log file.

31. The method of Claim 29, wherein said software tool comprises a Path Analysis Tool, wherein aggregate test paths are displayed for analysis.

32. The method of Claim 31 , wherein said replaying step comprises the steps of: selecting a node from said aggregate path, wherein said selected node comprises a current page; viewing current page details, wherein said current page details comprise the number of visitors out of said population of visitors visiting said current page; optionally, visiting said current page; viewing exit paths from said current page, wherein said exit paths comprise all URL's visited from said current page, and wherein each one of said exit path URL's comprises an alternative next node in said aggregate path; and selecting one of said alternative next nodes,, wherein said selected next node becomes said current page.

33. The method of Claim 32, said Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein a path history for said current page is displayed; details for exit paths from said current page displayed in a first of said plurality of frames; current page details displayed in a second of said plurality of frames, said current page details further comprising a page identifier of said page, and current page statistics.

34. The method of Claim 34, wherein said current page statistics comprise: the total number of visitors following a path to said current page;

35. The method of Claim 29, wherein said software tool is a Reverse Path Analysis Tool, wherein reverse aggregate test paths are displayed for analysis.

36. The method of Claim 35, wherein said replaying step comprises the steps of: selecting a node from said reverse aggregate path, wherein said selected node comprises a current page; viewing current page details, wherein said current page details comprise the number of visitors out of said population of visitors visiting said current page; optionally, visiting said current page; viewing entrance paths to said current page, wherein said entrance paths comprise all URL's from which said visitors came to said current page, and wherein each one of said entrance path URL's comprises an alternative previous node in said reverse aggregate path; and selecting one of said alternative previous nodes, wherein said selected previous node becomes said current page.

37. The method of Claim 36, said Reverse Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein a reverse path history for a current node is displayed; details for entry paths to said current page displayed in a first of said plurality of frames; details for said current page displayed in a second of said plurality of frames, said current page details further comprising a page identifier of said page and current page statistics.

38. The method of Claim 37, wherein said current page statistics comprise: the total number of visitors following a path to said current page.

39. The method of Claim 29, wherein said software tool comprises a Tester Detail Tool, wherein a specific visitor from said plurality of visitors may be selected and wherein path information corresponding to said visitor is viewed.

40. The method of Claim 39, further comprising the step of: selecting a specific visitor.

41. The method of Claim 40, wherein said Tester Detail Tool comprises: a window, said window comprising at least two frames; a plurality of navigation controls, any of said navigation controls used by a report reader to move from one node to another node in a selected path; an interface element, wherein a URL for a node being currently viewed is displayed; a current page image; a previous page image; and a following page image.

42. The method of Claim 29, wherein said software tool comprises an Impact Analysis Tool, and wherein a directory structure representing regions and sub-regions of said World Wide Web site is displayed.

43. The method of Claim 29, wherein said software tool compπses a Graphical Path Analysis Tool, wherein said path information is displayed as a tree structure, said Graphical Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a tree structure displayed in a one of said frames, said tree structure comprising: graphical interface elements representing nodes in a path; branches originating from said graphical interface elements, wherein said branches represent exit paths from said nodes, wherein thickness of a branch is proportional to the number of testers following a path.

44. The method of Claim 29, further comprising the steps of: selecting a node, wherein said selected nodes assumes focus as a current page, and wherein additional branches are displayed representing exit paths from said current page; optionally, hiding branches originating from a node.

45. An apparatus for evaluating tester path data comprising: a usability test, said usability test being administered to at least one of a plurality of pre-qualified testers, wherein said at least one pre-qualified tester navigates a series of hyperlinks to accomplish a specified objective; a storage and retrieval means to store and retrieve path information for each of said plurality of pre-qualified testers resulting from said administered usability test; whereby said stored path information is aggregated across said plurality of tests and usable in future analysis.

46. The apparatus of Claim 45, wherein said test evaluates the usability of a target World Wide Web site and wherein said test is a World Wide Web application; and wherein said path comprises a series of nodes, each of said nodes comprising a URL visited by said tester attempting to accomplish said specified objective.

47. The apparatus of Claim 46, wherein said storage means comprises a database having a table for storage of said path information therein, and wherein said path information is explicitly coded.

48. The apparatus of Claim 47, further comprising a suite of software tools for said retrieval and said analysis of said path information, wherein said software tools are each World Wide Web applications, said suite comprising: a Path Analysis Tool, wherein aggregate test paths are displayed for analysis; a Reverse Path Analysis Tool, wherein reverse aggregate test paths are displayed for analysis; a Tester Detail Tool, wherein a specific tester from said plurality of testers may be selected and wherein path information corresponding to said tester is viewed; an Impact Analysis Tool, wherein a directory structure representing regions and sub-regions of said target World Wide Web site are displayed; and a Graphical Path Analysis Tool, wherein said path information is displayed as a tree structure.

49. The apparatus of Claim 44, said Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein an objective is selected from a list of objectives; and a second interface element accessible from within said window, wherein a path history for a current node is displayed; details for exit paths from a current page displayed in a first of said plurality of frames, said current page being a web page at a URL corresponding to said current node; details for said current page displayed in a second of said plurality of frames, said current page details comprising a page identifier of said page, an image of said page and current page statistics.

50. The apparatus of Claim 49, wherein said current page statistics comprise: the total number of testers following a path to said current page; the number of said testers reaching said current page who successfully accomplish said selected objective; the number of said testers reaching said current page who fail to accomplish said selected objective; and the number of said testers reaching said page who abandon an attempt to accomplish said selected objective.

51. The apparatus of Claim 49, said Reverse Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a first interface element accessible from within said window, wherein an objective is selected from a list of objectives; and a second interface element accessible from within said window, wherein a reverse path history for a current node is displayed; details for entry paths to a current page displayed in a first of said plurality of frames, said current page being a web page at a URL corresponding to said current node; details for said current page displayed in a second of said plurality of frames, said current page details comprising a page identifier of said page, an image of said page and current page statistics.

52. The apparatus of Claim 51 , wherein said current page statistics comprise: the total number of testers following a path to said current page; the number of said testers reaching said current page who successfully accomplish said selected objective; the number of said testers reaching said current page who fail to accomplish said selected objective; and the number of said testers reaching said page who abandon an attempt to accomplish said selected objective.

53. The apparatus of Claim 48, wherein said Tester Detail Tool comprises: a window, said window comprising at least two frames; a plurality of navigation controls accessible from within said window, any of said navigation controls used by said report reader to move from one node to another node in a selected path; an interface element accessible from within said window, wherein a URL for a node being currently viewed is displayed; a current page image; a previous page image; and a following page image.

54. The apparatus of Claim 53, wherein said path information is displayed in a first of said plurality of frames, and wherein said current page image, said previous page image and said following page image are displayed in a second of said plurality of frames.

55. The apparatus of Claim 54, wherein a first of said plurality of navigation controls is a "next" navigation control, said "next" navigation control used by said report reader to view a page on said target World Wide Web site corresponding to a next node in said selected path, and wherein a second of said plurality of navigation controls is a "previous" navigation control, said "previous" navigation control used by said report reader to return to a last-viewed page corresponding to a node immediately preceding a current node.

56. The apparatus of Claim 48, further comprising: means to pose questions to said testers during said navigation of said links, wherein said questions elicit said testers' level of satisfaction; means for recording and storing said testers' answers to said questions; means for viewing site impact data, wherein said impact data comprise: directory name; page name; number of testers to visit each of said regions and sub-regions; percentage of success for testers who visit a specified region; percentage of success for testers who do not visit said specified region; number of times each tester views said specified region; and time spent in said region by a selected tester; and means for analyzing failure rates for said specified region to assess any of design quality, usability, and relevance of said region to said specific objective.

57. The apparatus of Claim 56, said means for viewing site impact data comprising: means for retrieving said stored answers to said selected question; means for creating and viewing a graph of said answers to said selected question.

58. The apparatus of Claim 57, said Impact Analysis Tool comprising: a window, said window comprising a plurality of frames; an interface element, wherein a specific objective is selected from a list of objectives; a first interface element accessible from within said window, wherein said question is selected from a list of questions; a second interface element accessible from within said window, wherein said graph is created and displayed; a third inter ace element accessible from within said window, wherein said mouse click data is displayed, said directory structure displayed in a first of said plurality of frames and said impact data displayed in a second of said plurality of frames.

59. The apparatus of Claim 49, further comprising a Graphical Path Analysis Tool, wherein said path information is displayed as a tree structure, said Graphical Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a tree structure displayed in a one of said frames, said tree structure comprising: graphical interface elements representing nodes in a path; branches originating from said graphical interface elements, wherein said branches represent exit paths from said nodes, wherein thickness of a branch is proportional to the number of testers following a path.

60. The apparatus of Claim 49, further comprising a flag, said flag comprising an interface element wherein any of relative rates of success and failure in achieving said specified objective and tester satisfaction are represented graphically.

61. The apparatus of Claim 60, wherein a threshold is set, said threshold comprising a threshold percentage of a group of testers following an exit path who fail to accomplish said objective, and wherein exceeding said threshold triggers an appearance of said flag.

62. The apparatus of Claim 61 , wherein a criterion is set, said criterion comprising a question selected from a list of questions posed to said testers during the course of said usability test, said question having a predetermined set of possible answers.

63. The apparatus of Claim 62, wherein said predetermined answers to said criterion question are classed as any of undesirable and desirable by a report reader.

64. The apparatus of Claim 63, wherein a threshold is set, said threshold comprising a threshold percentage of a group of testers following an exit path whose answer to said criterion question fall into said undesirable class, and wherein exceeding said threshold triggers an appearance of said flag.

65. The apparatus of Claim 49, wherein said software tools are World Wide Web applications provided on a World Wide Web site of a full-service research bureau and wherein said software tools are accessed remotely by a report reader using a web browser.

66. The apparatus of Claim 65, wherein said tester path data are presented to said report reader in a report format, and each of said software tools is presented to said report reader as a separate view of said report.

67. The apparatus of Claim 49, wherein thumbnail images of said web pages are viewable within a window, and wherein an interface element is provided, whereby said report reader chooses between a "show thumbnails" mode and a "hide thumbnails" mode.

68. The apparatus of Claim 49, wherein graphs showing analyses of saved answers to test questions posed to said test testers during the course of said usability test, aggregated across said tester population, are viewable within a window, and wherein an interface element is provided, whereby said report reader chooses between a "show graphs" mode and a "hide graphs" mode.

69. A suite of software tools for retrieval and analysis of path information, wherein said software tools are each World Wide Web applications, said suite comprising: a Path Analysis Tool, wherein aggregate test paths are displayed for analysis; a Reverse Path Analysis Tool, wherein reverse aggregate test paths are displayed for analysis; a Tester Detail Tool, wherein a specific tester from a plurality of testers may be selected and wherein path information corresponding to said tester is viewed; and an Impact Analysis Tool, wherein a directory structure representing regions and sub-regions of a target World Wide Web site are displayed; and a Graphical Path Analysis Tool, wherein said test paths are displayed as a tree structure.

70. The suite of software tools of Claim 69, wherein said software tools are World Wide Web applications provided on a World Wide Web site of a full-service research bureau and wherein said software tools are accessed remotely by a report reader using a web browser.

71. The suite of software tools of Claim 70, wherein thumbnail images of web pages visited by said testers are viewable within a window, and wherein an interface element is provided, whereby said report reader chooses between a "show thumbnails" mode and a "hide thumbnails" mode.

72. A Path Analysis Tool, wherein aggregate test paths are displayed for analysis; said Path Analysis Tool comprising: a window; a first interface element accessible from within said window, wherein a path history for a current node is displayed; and a second interface element accessible from within said window, wherein details for exit paths from a current page are displayed, said current page being a web page at a URL corresponding to said current node; a third interface element accessible from within said window, wherein details for said current page are displayed, said current page details comprising a page identifier of said page and current page statistics.

73. The Path Analysis Tool of Claim 72, wherein said current page statistics comprise: a total number of testers following a path to said current page; a page identifier; and average time spent on page by a visitor.

74. The Path Analysis Tool of Claim 73, further comprising an interface element accessible from within said window, wherein an objective is selected from a list of objectives.

75. The Path Analysis Tool of Claim 74, wherein said current page statistics comprise: a total number of testers following a path to said current page; the number of said testers reaching said current page who successfully accomplish said selected objective; the number of said testers reaching said current page who fail to accomplish said selected objective; and the number of said testers reaching said page who abandon an attempt to accomplish said selected objective.

76. The Path Analysis Tool of Claim 75, wherein said Path Analysis Tool is a World Wide Web application provided on a World Wide Web site of a full-service research bureau and wherein said path analysis tool is accessed remotely by a report reader using a web browser.

77. A Reverse Path Analysis Tool, wherein reverse aggregate test paths are displayed for analysis, said reverse path analysis tool comprising: a window; a first interface element accessible from within said window, wherein a reverse path history for a current node is displayed; a second interface element accessible from within said window, wherein details for entry paths to a current page are displayed, said current page being a web page at a URL corresponding to said current node; a third interface element accessible from within said window, wherein details for said current page are displayed, said current page details comprising a page identifier of said page, an image of said page and current page statistics.

78. A reverse path analysis tool according to Claim 77, wherein said current page statistics comprise: a total number of testers following a path to said current page; a page identifier; and average spent on page by a visitor.

79. The Reverse Path Analysis Tool of Claim 77, further comprising: an interface element accessible from within said window, wherein an objective is selected from a list of objectives.

80. The Reverse Path Analysis Tool of Claim 79, wherein said current page statistics comprise: a total number of testers following a path to said current page; the number of said testers reaching said current page who successfully accomplish said selected objective; the number of said testers reaching said current page who fail to accomplish said selected objective; and the number of said testers reaching said page who abandon an attempt to accomplish said selected objective.

81. The Reverse Path Analysis Tool of Claim 78, wherein said Reverse Path Analysis Tool is a World Wide Web application provided on a World Wide Web site of a full-service research bureau and wherein said Reverse Path Analysis Tool is accessed remotely by a report reader using a web browser.

82. A Tester Detail Tool, wherein a specific tester from a plurality of testers may be selected and wherein path information corresponding to said tester is viewed, said tester detail tool comprising: a window; a plurality of navigation controls accessible from within said window, any of said navigation controls used by a report reader to move from one node to another node in a selected path; a first interface element accessible from within said window, wherein a URL for a node being currently viewed is displayed. a second interface element accessible from within said window, wherein said path information is displayed, a third interface element accessible from within said window, wherein a current page image, a previous page image and a following page image are displayed.

83. The Tester Detail Tool of Claim 82, wherein said Tester Detail Tool is a World Wide Web application provided on a World Wide Web site of a full- service research bureau and wherein said Tester Detail Tool is accessed remotely by a report reader using a web browser.

84. An Impact Analysis Tool, wherein a directory structure representing regions and sub-regions of said target World Wide Web site is displayed.

85. The impact analysis tool of Claim 84, wherein said Impact Analysis Tool is a World Wide Web application provided on a World Wide Web site of a full-service research bureau and wherein said impact analysis tool is accessed remotely by a report reader using a web browser.

86. A Graphical Path Analysis Tool, wherein path information is displayed as a tree structure, said Graphical Path Analysis Tool comprising: a window, said window comprising a plurality of frames; a tree structure displayed in a one of said frames, said tree structure comprising: graphical interface elements representing nodes in a path; branches originating from said graphical interface elements, wherein said branches represent exit paths from said nodes, wherein thickness of a branch is proportional to the number of testers following a path.

87. The Graphical Path Analysis tool of Claim 86, wherein said Impact Analysis Tool is a World Wide Web application provided on a World Wide

Web site of a full-service research bureau and wherein said impact analysis tool is accessed remotely by a report reader using a web browser.

METHOD AND APPARATUS FOR ENHANCED PATH ANALYSIS ON A WORLD WIDE WEB SITE

ABSTRACT

A centralized full-service research bureau assists a customer in designing a usability test for typical tasks faced by a visitor to the customer's web site. The test is administered to a pre-qualified pool of testers. A method and apparatus is provided for tracking the paths taken through the customer's web site by a population of testers in attempting to accomplish a predetermined series of objectives during the course of the usability test, and for performing subsequent analysis of the path information. The path data for individual testers are saved to a database and aggregated across the population of testers. A suite of software tools is provided that retrieves and presents the path data in various formats as report sections in a browser window. The suite of path analysis tools organize the path data in a manner that allows the allows the customer or an analyst to discern patterns in the path data and assess the relevance and effectiveness of the various pages and sections of the target web site to the visitor's objectives, and to evaluate the visitor's experience as they navigate the web site. The disclosed method and software tools may also be used to analyze path data from conventional server log files.