WO2001024088A9

WO2001024088A9 - Providing feedback using an interactive computer system

Info

Publication number: WO2001024088A9
Application number: PCT/US2000/025760
Authority: WO
Inventors: David L Ronick; Lee I Newman
Original assignee: Brainstorm Interactive Inc
Priority date: 1999-09-24
Filing date: 2000-09-20
Publication date: 2002-10-03
Also published as: WO2001024088A1; AU4025501A

Abstract

Techniques for providing feedback using an interactive computer network include transmitting an electronic mail message to persons from whom the feedback is requested in response to input from a person requesting the personal feedback. The electronic message requests the persons from whom feedback is requested to access an on-line site on the network and complete an evaluation form with respect to the person requesting the feedback (160). Evaluation forms completed by at least some of the persons from whom the feedback is requested are received in the network. Results based on the completed evaluation forms are compiled in the network (162). Acess to the results is permitted to the person requesting the feedback through the on-line site (164).

Description

PROVIDING FEEDBACK USING AN INTERACTIVE COMPUTER SYSTEM

BACKGROUND

The invention relates to providing feedback using an interactive computer system. In the corporate environment, it is essential for employees and supervisors periodically to receive feedback related to their job performance. Many corporate executives believe that constructive criticism in the form of feedback is an important driver of performance improvement in the workplace. Proper feedback assists employees and supervisors alike in developing a realistic assessment of how their work is viewed by others in the workplace. As a result, the benefits of feedback often include increased productivity as workers often receive helpful suggestions for improving specific job skills. Supervisors and high-level executives also benefit by receiving feedback related to their communication and interpersonal skills. This exchange of information often leads to an increased confidence in management and more efficient leadership within a company.

Thorough feedback helps those outside of the corporate environment as well. Independent professionals such as accountants, lawyers, and consultants also benefit from receiving criticism relating to their work product and work habits. Customer feedback is often a very strong and clear indicator of customer satisfaction. These professionals can use customer input to fine tune interpersonal skills, improve customer service, and increase the productivity and profitability of their businesses. A professional who openly accepts and positively responds to client feedback will most often enjoy higher levels of customer satisfaction.

SUMMARY Techniques for providing feedback using an interactive computer network include transmitting an electronic mail message to persons from whom the feedback is requested in response to input from a person requesting the personal feedback. The electronic message requests the persons from whom feedback is requested to access an on-line site on the network and complete an evaluation form with respect to the person requesting the feedback. Evaluation forms completed by at least some of the persons from whom the feedback is requested are received in the network. Results based on the completed evaluation forms are compiled in the network. Access to the results is permitted to the person requesting the feedback through the on-line site.

Various implementations may include one or more of the following advantages. For example, the system can provide a confidential and anonymous environment in which to solicit and provide feedback. As a result, the discomfort of giving or receiving criticism in a face to face interaction can be eliminated, thus promoting participation in the feedback process. Additional advantages include improved logistics. A website serves as a centralized data collection and processing location. Electronic mail automates the task of coordinating participants. Further advantages can include a minimal effort required by participants to obtain or provide feedback. The structured forms can require very little time to complete, and the results can be presented in a format that is easy to understand. Also, follow-on products and services can be offered in real-time. Other features and advantages will be apparent from the following detailed description, the accompanying drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary interactive computer network.

FIG. 2 shows a computer network supporting service on the World Wide Web. FIG. 3 shows an implementation of an interactive Web site according to the invention.

FIG. 4 illustrates an exemplary on-line registration form.

FIG. 5 illustrates an exemplary form for initiating feedback.

FIG. 6 is a flow chart of a method of requesting feedback according to the invention.

FIGS. 7A, 7B, and 7C illustrate exemplary forms for conducting the feedback process.

FIG. 8 illustrates an exemplary self-review form.

FIG. 9 illustrates an exemplary confidential web page. FIG. 10 illustrates an exemplary evaluation form that can be accessed through the web site.

FIG. 11 is a flow chart of a method of initiating feedback according to the invention. FIG. 12 illustrates an exemplary confirmation web page.

FIG. 13 illustrates an exemplary feedback access form.

FIGS. 14 A, 14B, 14C, and 14D illustrate exemplary feedback provided by the interactive network.

FIG. 15 is a flow chart of a method of initiating feedback according to the invention.

DESCRIPTION

As shown in FIG. 1, an exemplary interactive computer network 10, such as the Internet, includes interconnected computers that support communications among different types of users. Such users can include, for example, businesses, individuals, universities and financial institutions, among others. The network 10 can support various types of communication linlcs, such as telephone 12, paging 14, cellular 16 and cable television 18 links. Terminal equipment used with the network 10 can include, for example, local area networks (LANs) 20, personal computers 22 with modems 24, pocket organizers 26, personal data assistants (PDAs) 28, and content servers 29 of multi-media, audio, video and other information.

Communications over the network 10 can be carried out through different layers such as application layers, transport layers, Internet layers and physical layers. An application layer represents the various tools and information services that are used to access information over the Internet. Such tools can include, for example, World Wide Web (the "Web") service to allow access to Hyper Text Transfer

Protocol (HTTP) documents. FIG. 2 shows an example of an Internet network 30 supporting Web service. A Web browser 32 is connected to a Web server 34 through the network 30. The server 34 can be implemented as a computer and provides content, such as images, text, video, sound and ava applets, to users. An Internet Service Provider (ISP) 36 linlcs the browser 32 to the network 30, and an ISP 38 links the server 34 to the network. Regional service providers 40, 42 and other intermediaries route traffic between the ISPs 36, 38.

FIG. 3 illustrates the home page or other page 44 of a secure, interactive, Web site for conducting private and anonymous reviews, and offering news and information concerning related products and follow-up services. A user can click on one or more hypertext lines using a mouse or other input device to move from one Web page to another. Some hypertext linlcs can serve to orient and inform the user as to the available services. Before a review can be obtained or given, an on-line registration form 47 (FIG. 4) is completed. Once the user is registered, the user's account is activated. A unique identification code and/or password are given to the user to allow the system to store and track information that is specific to that user and to allow the person to gain access to various services and information provided through the Web site.

Once the user is registered, the user can access another Web page 48 (FIG. 5) by clicking on the link 46. The Web page 48 allows a user to initiate the process of requesting reviews by selecting an evaluation form to be used as part of the feedback process. As shown in FIG. 5, the user can click on one of several links, such as the "Managing Yourself link 50, to select the particular type of review and feedback desired. Other reviews include "Managing People" and "Managing Sales," among others. The "Feedback Nault" link 52 provides detailed information about the user's account, including information about the status of pending and completed reviews.

Referring to FIGS. 5 and 6, a user seeking to be reviewed accesses 150 the web page 48 and selects 152 a particular type of review by clicking, for example, on the link 50. Clicking on the link 50 transfers the user to another Web page 54 (FIG. 7A) where the user can select the time frame within which tl e reviewers are requested to complete an evaluation form. A pull-down menu 56 can be provided to facilitate selection of an appropriate time frame. The Web page 54 also allows the user to select whether one or more reviewers will be asked to participate in the feedback process. In other words, the user can request a single individual to provide an evaluation or can request multiple persons to provide evaluations. The selections can be made, for example, by clicking in the appropriate areas of the page. Clicking on a link 58 labeled "NEXT" transfers the user to another page 60 (FIG. 7B). Assuming that the feedback is being requested from multiple persons, the user enters 154 the electronic mail addresses of the persons from whom feedback is requested. Examples of persons who would provide feedback include employees, supervisors, managers, co-workers, clients, vendors, etc. After entering the electronic mail addresses, the user clicks a button 62 on the Web page 60 and is transferred to another Web page 64 (FIG. 7C) where the text of an electronic mail message can be entered. The user can be prompted to confirm the information entered on the pages 54, 60 and 64. In response to the user's confirmation, an electronic mail message automatically is generated by the system and transmitted 156 to those persons selected by the user to be reviewers. The electronic mail message received by the recipients requests that they access the Web site to review the user by completing an on-line evaluation form based on the type of review selected by the user. The electronic mail message can include a hyperlink that transfers the reviewer to a Web page where the evaluation form can be completed and submitted.

After confirming the information entered on the Web pages 54, 60 and 64, the user is transferred to another Web page 66 (FIG. 8) where the user is prompted 158 to perform a self-review by completing the same evaluation form that the reviewers are requested to complete. The evaluation form can include questions directed toward the user's management style, leadership skills, business decisions, etc. As described below, the self-evaluation can be compared by the system to the evaluations completed by persons other than the user as part of the feedback for the user. A partial example of an evaluation form 68 is shown in FIG. 8. To complete the form, the reviewer selects a response to each question, for example, by clicking on the appropriate response. The completed self-evaluation form can be submitted by clicking on the button 70. The system saves the completed self-evaluation form. In some cases, the system can automatically provide feedback to the user based on the user's self-evaluation.

The user can change the responses to the self-evaluation form by accessing the form through the "Feedback Vault" link 52 (FIG. 5). The user also can access a Web page 72 (FIG. 9) through the "Feedback Vault" link 52 to display information regarding the status of a requested review. In the example shown in FIG. 9, the displayed information indicates that the "Managing Yourself review is in progress, that none of the reviewers has responded, and that seven days remain for the reviewers to complete their responses. To complete the evaluation form, each reviewer accesses 160 (FIG. 11) a Web page 74 (FIG. 10) that can include the same questions listed that appeared on the evaluation form completed by the person requesting the review. The reviewers can access the Web page 74 directly, through the home page 44 (FIG. 3) or, as previously mentioned, through a hyperlink provided in the electronic mail messages asking them to participate in the review process. After completing the on-line evaluation form on behalf of the requesting party, a reviewer submits the completed form by clicking on a button 76. In several situations, the system may ask a particular reviewer to reconsider the reviewer's responses to the evaluation form. For example, if all the reviewer's responses indicate a very poor evaluation of the person being reviewed, or if all the responses indicate a very high evaluation of the person being reviewed, the reviewer would be given another opportunity to confirm the accuracy of the responses or to change one or more of the responses. Similarly, if all the responses by a particular reviewer are neutral, the system would remind the reviewer to be honest in the evaluation and would give the reviewer an opportunity to change one or more the responses.

Upon submitting the completed evaluation form, the reviewer is transferred to a Web page 78 (FIG. 12) that thanks the reviewer for participating in the review process and prompts the reviewer to use the review and feedback services provided by the system. For example, in one implementation, a hyperlink 80 is provided to the Web site's home page. The on-line system can provide other prompts to persons who have been requested to complete an evaluation form to encourage them to participate in the use of the system and request evaluations of themselves. Such prompts can take the form of buttons or hyperlinks to allow a person to register as a participant automatically, for example, by clicking on the button or hyperlink and filling out the on-line registration form 47 (FIG. 4). Such prompts can be provided, for example, prior to, during, or upon completion of an evaluation form on behalf of another person. The system can send an electronic mail message to reviewers who have not yet submitted a completed evaluation form. The message reminds those reviewers of the deadline for completing the evaluation form and requests their cooperation.

Evaluation forms completed by the persons from whom the feedback is requested are received in the network. Aggregated results and an analysis of the completed evaluation forms received in the network are compiled 162 (FIG. 11) automatically. The system transmits an electronic mail message to the person requesting the feedback. The electronic mail message can indicate to the user that the review process has been completed and that feedback can be obtained through the "Feedback Vault" link 52 (FIG. 5).

By clicking on the "Feedback Vault" link 52 after completion of the review process, the user is transferred to a Web page 82 (FIG. 13) that allows the user to view the results of the review process and obtain feedback based on the reviewers' evaluations by clicking on a button 90. Prior to reviewing the results, the user can change responses to the previously-completed self-review by clicking on a button 84. Another button 86 can be clicked to add additional reviewers to the process and/or to give reviewers more time to complete the evaluation form. Similarly, clicking a button 88 allows the user to abandon the review and feedback process.

If the user clicks the button 90, the user is transferred to a Web page 92 (FIG. 14A) that includes aggregated results of the completed evaluation forms. Preferably the aggregated results are provided to the user anonymously, in other words, without revealing the identity of persons associated with particular responses to the evaluation form. In other cases, however, the results need not be provided anonymously. The aggregated results illustrated on the Web page 92 and on linked pages 94, 96, 98 (FIGS. 14B through 14D) can include, for example, one or more of the following: (1) average overall responses based on all responses from all reviewers (FIG. 14A); (2) average responses of the reviewers to each question in the evaluation form (FIG. 14B); (3) a comparison between the self evaluation completed by the person requesting the feedback and the evaluations provided by the reviewers (FIG. 14C); (4) an indication of the average responses received with respect to other persons who have requested similar feedback or who have been the subject of similar evaluations; (5) comments explaining the import of responses, including an identification of problem areas based on the responses; and (4) recommended sources or links 100 (FIG. 14D) to information such as seminars, books and articles relating to areas identified for improvement. Instead of completing an on-line evaluation form in response to a request by the person to be reviewed, a person may initiate the review of another individual without being prompted, as shown in FIG. 15. To initiate the review, the reviewer accesses 200 the website and selects 210 an appropriate evaluation form for the review. The user then enters 220 the name and electronic mail addresses of the person to be reviewed and the persons 230 from whom feedback is requested.

Electronic mail messages are automatically sent 240 to persons from whom feedback is requested requesting that they access 250 the website and complete an on-line evaluation form selected by the person initiating the review. As previously explained, completed forms are received in the network which then automatically compiles 260 aggregated results based on the responses to the evaluation form.

An electronic message can be sent 270 to the person who was the subject of the review. The message informs the subject of the review that he has been reviewed but preferably does not name the reviewers or the person who initiated the reviewer. In other cases, for example, with the permission of the reviewers and/or the person initiating the review, their identity can be disclosed. The message also informs the person who is the subject of the review that he can access the aggregated results through the Web site, and in particular, through the "Feedback Vault" link 52.

In general, the aggregated results and other information accessed through the "Feedback Vault" are confidential. Therefore, in some cases, only the person who is the subject of the review is given access to the results. In other cases, for example, when an employer initiated the review process with respect to an employee, the employer also can be given access to the results through the "Feedback Vault" link. When the person who is the subject of the review accesses the "Feedback Vault," the system can prompt the subject of the review to complete an on-line self-evaluation form that is similar to the form completed by the reviewers. The self-evaluation form can be completed, for example, prior to the user's viewing any results or analysis of the review. The aggregated results can be similar to those discussed above.

Various features of the Web site can be implemented in hardware, software, or a combination of hardware and software. For example, some aspects of the system can be implemented in computer programs executing on programmable computers. Each program can be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. Furthermore, each such computer program can be stored on a storage medium, such as read-only- memory (ROM) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage medium is read by the computer to perform the functions described below.

Other implementations are within the scope of the following claims.

Claims

What is claimed is:

1. A method of providing feedback using an interactive computer network, the method comprising: transmitting an electronic mail message to persons from whom the feedback is requested in response to input from a person requesting the personal feedback, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site on the network and complete an evaluation form with respect to the person requesting the feedback; receiving in the network completed evaluation forms by at least some of the persons from whom the feedback is requested; compiling in the network results based on the completed evaluation forms; and permitting access to the results through the on-line site to the person requesting the feedback.

2. The method of claim 1 wherein the evaluation form is selected from among a plurality of on-line evaluation forms in response to input from the person requesting the feedback.

3. The method of claim 1 wherein the results to which the person requesting the feedback is permitted access do not indicate the identity of persons associated with particular responses to the evaluation form.

4. The method of claim 1 wherein a single on-line site is utilized for transmitting the electronic mail message, for receiving the completed evaluation forms, and for compiling results of the completed evaluation forms.

5. The method of claim 1 wherein the on-line site is a web site on the world wide web.

6. The method of claim 1 including receiving an on-line self-evaluation form completed by the person requesting the feedback, wherein the results include a comparison between responses in the completed self-evaluation form and average responses based on the completed evaluation forms of the persons from whom the feedback was requested.

7. The method of claim 1 wherein the results include identification of areas that require improvement based on the completed evaluation forms.

8. The method of claim 1 including receiving an on-line self-evaluation form completed by the person requesting the feedback, wherein the results include identification of areas for which there is a disparity between responses to the self evaluation form completed by the person requesting the feedback and persons from whom the feedback is requested.

9. The method of claim 1 wherein the results include an indication of average responses of other persons who have requested similar feedback.

10. The method of claim 1 wherein the results include recommended sources of information relating to areas identified for improvement.

11. A method of providing feedback using an interactive computer network, the method comprising: receiving in the network a request to initiate an evaluation with respect to a person intended to be the subject of an evaluation; transmitting an electronic mail message to persons from whom feedback is requested in response to the request, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site on the network and complete an evaluation form with respect to the person who is the subject of the evaluation; receiving in the network completed evaluation forms by at least some of the persons from whom the feedback is requested; compiling in the network results based on the completed evaluation forms received in the network; and providing access to the results through the on-line site to the person who is the subject of the evaluation.

12. The method of claim 11 wherein the results to which the person who is the subject of the evaluation is permitted access do not indicate the identity of persons associated with particular responses to the evaluation form.

13. The method of claim 11 wherein the evaluation form is selected from among a plurality of on-line evaluation forms in response to input from a person making the request to initiate the evaluation.

14. The method of claim 11 including receiving an on-line self-evaluation form completed by the person who is the subject of the evaluation, wherein the results include a comparison between responses in the completed self-evaluation form and average responses based on the completed evaluation forms of the persons from whom the feedback was requested.

15. The method of claim 11 wherein the results include identification of areas that require improvement based on the completed evaluation forms.

16. The method of claim 11 including receiving an on-line self-evaluation form completed by the person who is the subject of the evaluation, wherein the results include identification of areas for which there is a disparity between responses to the self-evaluation form completed by the person who is the subject of the evaluation and persons from whom the feedback is requested.

17. The method of claim 11 wherein the results include an indication of average responses of other persons who have been the subject of similar evaluations.

18. The method of claim 11 wherein the results include recommended sources of information relating to areas identified for improvement.

19. A method of providing feedback using an interactive computer network, the method comprising: transmitting an electronic mail message to persons from whom the feedback is requested in response to input from a person requesting the feedback, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site in the network and complete an on-line evaluation form with respect to the person requesting the feedback; and providing a hyperlink through the on-line site while a person from whom feedback is requested completes the on-line evaluation form, wherein, when selected, the hyperlink, allows the person completing the evaluation form to initiate a registration process in the on-line site.

20. An article comprising a computer-readable medium that stores computer-executable instructions for causing a computer system to: transmit an electronic mail message to persons from whom feedback is requested in response to input from a person requesting the personal feedback, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site and complete an on-line evaluation form with respect to the person requesting the feedback; compile results based on on-line evaluation forms completed by at least some of the persons from whom the feedback is requested; and permit access to the results through the on-line site to the person requesting the feedback.

21. The article of claim 20 wherein the instructions cause the evaluation form to be selected from among a plurality of on-line evaluation forms in response to input from the person requesting the feedback.

22. The article of claim 20 wherein the results to which the person requesting the feedback is permitted access do not indicate the identity of persons associated with particular responses to the evaluation form.

23. The article of claim 20 wherein the results include a comparison between responses to an on-line self-evaluation form submitted by the person requesting the feedback and average responses based on the evaluation forms completed by the persons from whom the feedback was requested.

24. The article of claim 20 wherein the results include identification of areas that require improvement based on the completed evaluation forms.

25. The article of claim 20 wherein the results include identification of areas for which there is a disparity between responses to an on-line self-evaluation form completed by the person requesting the feedback and persons from whom the feedback is requested.

26. The article of claim 20 wherein the results include recommended sources of information relating to areas identified for improvement.

27. An article comprising a computer-readable medium that stores computer-executable instructions for causing a computer system to: transmit an electronic mail message to persons from whom feedback is requested in response to a request to initiate an evaluation with respect to a person intended to be the subject of an evaluation, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site and complete an on-line evaluation form with respect to the person who is the subject of the evaluation; compile results based on evaluation forms completed by at least some of the persons from whom the feedback is requested; and provide access to the results through the on-line site to the person who is the subject of the evaluation.

28. The article of claim 27 wherein the results to which the person who is the subject of the evaluation is permitted access do not indicate the identity of persons associated with particular responses to the evaluation form.

29. An article comprising a computer-readable medium that stores computer-executable instructions for causing a computer system to: transmit an electronic mail message to persons from whom feedback is requested in response to input from a person requesting the feedback, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site and complete an on-line evaluation form with respect to the person requesting the feedback; and provide a hyperlink through the on-line site while a person from whom feedback is requested completes the on-line evaluation form, wherein, when selected, the hyperlink, allows the person completing the evaluation form to initiate a registration process with respect to the on-line site.

30. A computer system comprising: a computer network including at least one server; and a plurality of user stations that can access the on-line site through the network, wherein a person using one of the user stations can request feedback by accessing the on-line site, selecting an on-line evaluation form, and identifying persons from whom the feedback is requested, wherein, in response to receiving the request for feedback, the at least one server: transmits an electronic mail message to persons from whom feedback is requested in response to input from a person requesting the personal feedback, wherein the electronic message requests the persons from whom feedback is requested to access an on-line site and complete an on-line evaluation form with respect to the person requesting the feedback; compiles results based on on-line evaluation forms completed by at least some of the persons from whom the feedback is requested; and permits access to the results through the on-line site to the person requesting the feedback.

31. The system of claim 30 wherein the at least one server selects an evaluation form from among a plurality of on-line evaluation forms in response to input from the person requesting the feedback.

32. The system of claim 30 wherein the results to which the person requesting the feedback is permitted access do not indicate the identity of persons associated with particular responses to the evaluation form.

33. The system of claim 30 wherein the results include a comparison between responses to an on-line self-evaluation form submitted by the person requesting the feedback and average responses based on the evaluation forms completed by the persons from whom the feedback was requested.

34. The system of claim 30 wherein the results include identification of areas that require improvement based on the completed evaluation forms.

35. The system of claim 30 wherein the results include identification of areas for which there is a disparity between responses to an on-line self-evaluation form completed by the person requesting the feedback and persons from whom the feedback is requested.

36. The system of claim 30 wherein the results include recommended sources of information relating to areas identified for improvement.