JP2005107715A - User test estimation chart - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an estimation chart allowing the quantitative and qualitative integration estimation of data obtained in a user test executed when developing a product having a user interface to obtain a high-accuracy estimation result in a short time. <P>SOLUTION: This user test estimation chart used when estimating the user interface comprises a quantitative data estimation part and a qualitative data estimation part. The quantitative data estimation part and the qualitative data estimation part are listed in a paralleled list format such that both quantitative data and qualitative data related to operation can be integrated and estimated. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a user test evaluation chart used in a user test for evaluating the development of a product having a user interface.

  When a device or system is used, the device or system is designed to perform a predetermined function by operating according to a predetermined operation procedure. Therefore, it can be said that it is an indispensable condition for the user to input an appropriate process to the user interface in order to achieve the task.

Therefore, when a product having a user interface is developed, a user test is performed as a data collection method for improving the convenience of the product. This user test monitors the subject's operation status (for example, the operation time, consideration time, whether or not erroneous operation or advice is required) when a certain task is given to the subject, and searches for product problems and improvement points. This is a technique.

As a technique related to the evaluation of a conventional user test, there is an operation performance evaluation device that is evaluated by paying attention to the operation time of a subject (for example, see Patent Document 1).
JP 2001-56771 A

  For example, it is important that an evaluation method in the development of a product having a graphic user interface such as a copying machine is highly reliable in evaluation, excellent in reproducibility, and more efficient and highly accurate evaluation results can be obtained. In order to realize such evaluation, an evaluation chart that records items related to the operation of the subject is often used.

  However, since the analysis of the subject protocol of the user test that has been conventionally performed requires a lot of time and effort, there is a drawback that it is difficult to use it on a daily basis in a short development system.

  For example, in a conventional user test, means for recording a subject's situation at the time of the user test by video recording and analyzing this is taken, and protocol analysis can be performed without actually conducting an examination. However, since the protocol analysis results alone do not have enough information such as the operation information of the experienced person, it is necessary to look at the actual inspection or the recorded video, but in practice the video is rarely used by the development staff themselves. In addition, there is a drawback that it is difficult to appropriately reflect the test results in product development.

  In addition, it is difficult to compare and evaluate operations between tasks and between subjects, and in order to compare and evaluate these, a method of comparing and evaluating the entire operation only by comparing limited factors such as task achievement rate, error points, and questionnaire evaluation. There was no information and comparative evaluation of information was complicated.

  In addition, the conventional analysis method of user test results has the logic to persuade the subject by accumulating phenomena such as the task achievement rate and error points of subjects, but the structural viewpoint is sparse and analysis of problems such as error operations The clarification of the cause and the presentation of the improvement plan largely depend on the evaluator's rule of thumb. In addition, there is a drawback that it is difficult to promptly reproduce the problem to the developer by quickly reproducing it at a specific location including the problem and the cause in the operation step of the recorded video.

  In addition, since all of the utterance records during task processing are recorded and processed, it takes labor and time, and it has been difficult to realize accurate analysis easily in a short time.

  The present invention has been made in consideration of the above-mentioned circumstances, and quantitatively and qualitatively integrates and evaluates data obtained in a user test performed during development of a product having a user interface in a short time and with high accuracy. An object of the present invention is to provide an evaluation chart capable of obtaining the evaluation results.

  The user test evaluation chart according to the present invention is a user test evaluation chart used when evaluating a user interface in order to solve the above-described problems. The user test evaluation chart includes a quantitative data evaluation unit and a qualitative data evaluation unit. The quantitative data evaluation unit and the qualitative data evaluation unit are listed in a parallel table format, and it is possible to evaluate both structural quantitative data and qualitative data from a structural multi-viewpoint. It is characterized by having comprised as follows.

  According to the evaluation chart according to the present invention, qualitative data and quantitative data collected in a user test can be integrated and visually evaluated, and an analysis of a user test result with high reliability can be performed efficiently. It is possible to provide a user test evaluation chart to be realized.

  The present invention provides a user test evaluation chart that can be evaluated visually and efficiently by integrating qualitative data and quantitative data. According to this user test evaluation chart, not only quantitative evaluation such as operation time and error occurrence rate is easy, but also qualitative by recording subject's behavior, observer's observation contents and subjective evaluation. Therefore, it is possible to achieve multi-faceted and accurate evaluation.

  In the evaluation chart used in the conventional user test, the quantitative data to be evaluated are the task achievement rate and time of the test subject, but these quantitative data alone can cause an error. It is difficult to identify the problem and extract it directly.

  Therefore, in the user test evaluation chart of the present invention, first, the quantitative data evaluation unit grasps the quantitative data, enriches quantitative information related to the operation of the subject, and collects and evaluates quantitative data related to task achievement.

  On the other hand, for the evaluation of qualitative data, which was used as an index of evaluation in the conventional user test and the data analysis was insufficient, a qualitative data evaluation unit was provided so that sufficient analysis evaluation was possible.

  That is, the evaluation chart of the present invention is composed of a quantitative data evaluation unit and a qualitative data evaluation unit. By tabulating and listing each data recorded in the quantitative data evaluation unit and the qualitative data evaluation unit, the operation process It is possible to integrate and evaluate both data, and to consider the relationship between the structure of the user interface and the user's thought structure, thereby improving user test analysis efficiency and improving evaluation accuracy. is there.

  The configuration of the evaluation chart of the present invention will be specifically described below.

  The evaluation chart of the present invention includes a quantitative data evaluation unit and a qualitative data evaluation unit, and the quantitative data evaluation unit includes a system structure column and a functional structure column.

  System structure data is entered in the system structure field. This system structure data indicates how the subject operated and moved on the system hierarchy, such as whether the point of error operation is an error in the hierarchy in which a function is selected, or a hierarchy in which detailed items are set. This is data for grasping the problems related to the design structure.

  In addition, functional structure data is input in the functional structure column. Functional structure data is a sequential recording of how subjects set up functions necessary or unnecessary to achieve a task, and whether they understand the functions required for a given task. These are data for considering the relationship between the function classification structure and the operation of the subject.

  On the other hand, the qualitative data evaluation unit extracts behaviors related to the sensibility of subjects that are difficult to evaluate only by quantitative data (for example, behaviors related to stray, thought, confirmation, etc. or whether there is conversation or advice), and inputs this data as data. The problem of operation and the cause of error are evaluated by analyzing the function of sensitivity to the operation of the subject. The qualitative data input to the qualitative data evaluation unit is referred to as sensitivity information.

  According to the user test evaluation chart configured as described above, in the operation that results in the error setting by the subject, the degree of understanding of the terms of choices displayed on the user interface, the mental model that the subject has, and the functions on the interface The cause of error is easy to identify.

  In addition, by referring to the sensibility information of the qualitative data evaluation unit at the time of error setting, it is easy to determine whether this error is an error that the subject himself is likely to find, and the operation in performing the task processing It is effective as an index for determining importance.

  In addition, it is possible to use the evaluation chart that lists quantitative data and qualitative data when evaluating test results without necessarily using recorded video, which has been used as an important data analysis means in user tests that have been performed in the past. There is an advantage that it is easy to grasp the outline of the operation.

  FIG. 1 shows an overall configuration diagram of an evaluation chart of the present invention. This evaluation chart is preferably created in a spreadsheet on a personal computer (PC). As shown in FIG. 1, the evaluation chart of the present invention is configured by listing the quantitative data evaluation unit 1 and the qualitative data evaluation unit 2 in a parallel table format. The operation procedure and the operation time of the subject recorded in the above, the presence / absence of the moderator's advice on the operation, and the sensitivity information at the time of the operation of the subject are integrated so that it can be visually judged.

  In the user test evaluation chart shown in FIG. 1, the quantitative data evaluation unit includes a system structure column 3, a function structure column 4, a log column 5, and a time column 6.

  As an example of the user test evaluation chart of the present invention, operation of a printer driver was given to a subject as a task, and a user test was performed in order to evaluate the problem of usability. The printer driver is a user interface whose operation result graphically corresponds to a preview. FIG. 2A and FIG. 2B show a printer driver interface which is a task of the embodiment. The printer driver in FIG. 2 (A) is a window display in a state where the Set-up tab is clicked, and a tab group 21 and a combo box 22 are displayed. The printer driver in FIG. 2 (B) is clicked on the Lay out tab. In the displayed window, a tab group 21, a second tab group 23, a combo box 22, and a radio button / check box 25 are displayed.

  FIG. 3 shows the system structure column 3 of the user test evaluation chart of the embodiment. As shown in FIG. 3, the system structure column 3 includes a plurality of entry columns corresponding to tabs and combo boxes to be operated by the subject, in addition to the number column 31 indicating the operation procedure. In this system structure column, how the subject behaves on the user interface structure is expressed.

  The number column 31 indicates a state in which some operation is continuously performed for a certain function, and the number of steps is entered with a sensitivity action (thinking, hesitation, etc.) and an operation (click) as one step. The numerical value input is not limited to one click, but in the case of the embodiment, it is a part related to the operability of the OS.

  Here, an input indicating that the operation state is described in a certain column of the system structure column is called a check.

  In the user test evaluation chart of the present embodiment, on the right side of the number column 31 in the system structure column 3 shown in FIG. 3, for example, a set up that is a tab belonging to the tab group 21 located at the upper left of the printer driver in FIG. It has a hierarchical display column 32 corresponding to each tab, Print tab, job tab, Layout tab, Quality tab, About tab. For example, when the Set up tab is selected, S is entered in the hierarchical display column 32, and a check 33a is entered in the tab column 33, indicating that the subject is operating in the set up tab display window. ing.

  In a user interface such as the printer driver shown in FIGS. 2A and 2B, as a means for inputting settings, a second tab group 23, a combo box 22, a radio button / check box 25, and the like are provided. Although the function is provided, the user test evaluation chart of this embodiment is provided with each column for entering information relating to the operation of these functions. The relationship between these functions and each column will be described in detail below.

  The user test evaluation chart of the embodiment has a configuration in which operations performed on each option displayed on the printer driver shown in FIG. 2 are described in the system structure column 3 shown in FIG.

  That is, each option (each setting such as paper size and paper type) displayed in the display window of the setup tab of the printer driver shown in FIG. 2A is referred to as a combo box 22, and is shown in FIG. A combo box column 34 corresponding to the combo box 22 of the printer driver is provided in the system structure column 3 shown in FIG. Next to the right side of the combo box column 34, a term selection column 36 for recording the selection of a plurality of options stored in the combo box 22 of FIG.

  That is, for selecting (clicking) the combo box 22, a check 34a is entered in the combo box column 34, and for selecting an option in the combo box 22, the term selection column 36 on the right side of the combo box column 34 is checked. 36a is entered.

  Following the setting of the setup tab, the layout tab is set in the printer driver window shown in FIG. Ls displayed in the hierarchy display column 32 of FIG. 3 indicates that “standard” is selected from the display of the second tab group 23 in the layout tab setting display window of FIG. On the other hand, Lb indicates that a booklet is selected from the second tab group 23 of the layout tab in FIG. That is, the operation hierarchy of the subject can be clearly confirmed by filling in the check 33b in the tab column 33 and the check 37a in the second tab column 37 of FIG.

  As for selection from the second tab group 23 in FIG. 2B, a second tab column 37 is provided on the right side of the tab column 33 in which the tab being operated is displayed in the system structure column 3 in FIG. By checking the tab field 37, it becomes clear which subject is operating which second tab of which dub.

  In the system structure column 3 of FIG. 3, in addition to the combo box column 34 and the second tab column 37 described above, a radio button / check box column 38 for supplying radio button / check box operations is also provided.

  In this way, entry fields corresponding to all setting options that can be selected by the subject are provided. In other words, when the subject performs any operation on the printer driver, the operation is reflected on the evaluation chart and recorded entirely.

  Further, in the system structure column shown in FIG. 3, the operation requirements of each tab are represented by hierarchy, and are laid out in order from a shallow hierarchy to a deep hierarchy, and the operable hierarchy and the inoperable hierarchy are displayed on the table. It is clearly expressed.

  That is, the operable hierarchy (active hierarchy) is expressed in white on the evaluation chart, and the operation status of the subject can be input (description) in this operable hierarchy. On the other hand, the inoperable layer 39 is hatched in gray or the like on the evaluation chart so that the operation status of the subject is not input (description). In FIG. Enclosed with That is, since there is no information to be input (description) for an operation screen (window) that cannot be opened in the display state of an interface in a certain operation state, the evaluation chart is easily hatched by hatching in advance.

  The user test evaluation chart of the present invention having the system structure column configured as described above indicates to which hierarchy the current option operated by the subject belongs (for example, a detailed item or a selection of a large function). Therefore, it is easy to extract the cause of the error occurrence from the options at the time of error setting, and analysis and evaluation can be appropriately performed.

  FIG. 4 shows the function structure column 4. This functional structure column 4 is filled with the history of actual operation of the subject.

  As shown in FIG. 4, the function structure column 4 displays options that can be set, such as the number of output copies and booklets, in a list format in each operation column displayed in the printer driver. For example, when an operation for setting the number of copies is performed in the display windows of FIGS. 2A and 2B, when an operation of clicking an option is performed, the number is entered in the number of copies column of the function structure column. The upper end of the operation record (check 41a) in the copy number column indicates the first operation, and the operation up to the check 41b at the lower end is a continuous operation. In the check 41a, an option related to the number of copies (Number of copies) is set and checked. 41b shows that the task of setting the number of copies has been achieved.

  That is, taking the setting of the number of copies as an example, in order to set the number of copies, Number of copies which is a setting for the number of copies in the combo box 22 is operated from the display window of FIG. (Click) indicates that the task of setting the number of copies has been achieved.

  In FIG. 4, the inoperable area is represented by a dotted line. This area indicates that no selectable options are displayed on the current display screen. It may be hatched in gray for easier viewing.

  On the other hand, in addition to the necessary setting, the subject may perform an operation that is not originally required for the achievement of the task. The operation is analyzed and evaluated in an unnecessary setting column provided in the function structure column 4.

  That is, in the user test evaluation chart of the present invention, an unnecessary setting field is provided next to the necessary setting field, and a record when the subject performs an unnecessary setting is stored. For example, regarding the achievement of the task of this embodiment, there is no need to change the paper orientation setting, but when the subject accidentally performs an operation relating to the paper change, the operation record is stored in the paper orientation column 43 of the unnecessary operation column. To be recorded. In this embodiment, in addition to the paper orientation field 43, a paper size field 44, a paper cassette setting field 45, a paper type setting field 46, and the like are installed in unnecessary setting fields.

  In this function structure field 4, when the function setting is achieved, achievement is input (description), and when the operation is finished with the default setting, the letter D is recorded. If an incorrect setting is made, an NG check is added to the entry column.

  In this way, the settings necessary for accomplishing the task and unnecessary settings are listed in chronological order in parallel, which part of the subject's continuous operation caused an error and smooth operation. Thus, it is possible to efficiently and accurately evaluate whether the task has been achieved.

  That is, by referring to the function structure column 4, it is possible to identify which screen the subject has seen and operated because it is specified which screen the subject has seen. It is also possible to visually determine how the subject has determined the necessary and unnecessary functions. Further, by grouping necessary functions and unnecessary functions on the system and distinguishing them on the evaluation chart with tab colors, it is easy to consider the relationship between functions and operation procedures.

  Moreover, a log is described in the log column 5 shown in FIG. 5, and a test subject's operation is displayed concretely. That is, by referring to this log column 5, the operation location of the subject is clarified and can be easily identified. In addition, the log in the log column 5 can be color-coded so that necessary operations and useless operations (for example, operations that only open a tab) can be visually and clearly displayed. With reference to this log column, the correlation with the system structure column 3, the function structure column 4, and the qualitative data evaluation unit 2 can be considered.

  On the other hand, the time column 6 shown in FIG. 6 describes the duration of the subject's action. The time information described in the time column 6 is information that complements the sensibility data, and is a visual display of the time required for the sensibility behavior of the subject in seconds. Judgment is possible.

  In the user test evaluation chart of this embodiment, the system structure column, the functional structure column, the log column, and the time column constitute a quantitative data evaluation unit.

  Next, visualization of sensitivity information, which is a qualitative data evaluation unit in the evaluation chart of the present embodiment, will be described below with reference to the drawings.

  FIG. 7 shows the qualitative data evaluation unit 2. In this embodiment, for example, at the time of subject's operation evaluation, the operation level is input (description) about the clarification of the purpose of the operation, the type of action, the judgment, the degree of certainty, or the type of advice (help) by the moderator. An input sensitivity label group 71 is provided. By referring to the sensitivity information of the subject described by the evaluator in this sensitivity label group 71, the trend of sensitivity regarding the operation of the subject in a certain operation procedure (for example, the judgment that is determined from the movement to the target, the situation leading to the stack, etc.) The quality of the operation is clarified. In addition, it becomes clear how the subject's operation has changed due to the advice of the moderator, and the influence of the advice on the operation can be evaluated.

  Note that the sensitivity label group 71 was created by using a technique in which the sensitivity state between all clicks of the subject and the sensitivity state at the time of the click was documented (for example, clicked while hesitating) and this sentence was labeled by the KJ method.

  Further, in the qualitative data evaluation unit 2, as shown in FIG. 7, a help column 72 indicating the level of advice given by the moderator, a conversation column 73 for recording the conversation between the subject and the moderator, and a large sensibility change of the subject occurs. It is also possible to provide an impression column 74 or the like for recording an impression of the sustained operation of the qualitative data so that qualitative data can be evaluated more integratedly.

  As described above, in this user test evaluation chart, temporal evaluation can be performed together with the sensitivity information described above. That is, in the time column 6 shown in FIG. 6, the operation time required for the operation is entered in units of seconds. By referring to the sensitivity information of the qualitative data evaluation unit 2 with respect to this operation time, for example, with the moderator Since the time required for the conversation is taken into consideration, it is possible to easily determine the sensibility situation and trend of the subject during the actual operation time.

  Thus, by making it possible to evaluate the operation time quantitatively and qualitatively, for example, the accumulated operation time can be calculated and set as an evaluation target. FIG. 8 shows a graph of the cumulative frequency for each second of operation time. FIG. 8 is a cumulative graph of operation times in the fourth to sixth tasks in which subjects No. 1 and No. 2 were given the same task a plurality of times and the task processing tendency of both subjects was changed. .

  As shown in FIG. 8, it is possible to clearly evaluate how a subject's task achievement time has changed due to several tasks. In this graph, the operation time is accumulated on a horizontal axis at a certain time interval, and the tendency of the number of seconds required for the operation by the subject is graphed, for example, how many operations are performed within 5 seconds.

  As is apparent from FIG. 8, the subject No. 1 has a tendency that the number of operations of 10 seconds or less decreases as the tasks are repeated. On the other hand, subject No. 2 does not show a significant change in the number of operations for 10 seconds or less even when tasks are repeated. The operation for 10 seconds or less is determined as an action for searching for a function by repeatedly opening various tabs and combo boxes without knowing where the next operation is in the interface. When the subject's understanding of the user interface is improved, the number of operations for 10 seconds or less is reduced because the operation behavior due to the search is reduced.

  That is, as test subject No. 1 experiences the fourth to sixth tasks, it is visually determined that the task processing ability has improved and the user interface has been deeply understood. On the other hand, in the case of the test subject No. 2, it becomes clear that there is no significant change in the task processing time due to experience in the fourth to sixth tasks.

すなわち、図８にＡ％として表される指標は、１０秒未満の操作回数を総操作回数で除したものに１００を掛けて指標化したものであり、数値の小さいもの程タスクを迷いなく処理していることを示す。こうした被験者の操作状況の視覚化により、被験者の理解度合が的確に評価可能となる。 In addition to the accumulated graph of time data, FIG. 8 introduces a mathematical expression as shown below, and introduces an index calculated by this mathematical expression to evaluate the operation time more quantitatively. Is also possible.

That is, the index represented as A% in FIG. 8 is obtained by dividing the number of operations less than 10 seconds by the total number of operations and multiplying by 100, and the smaller the numerical value, the easier the task is processed. Indicates that By visualizing the operation status of the subject, the degree of understanding of the subject can be accurately evaluated.

  An example is shown in which a user test evaluation chart having the above-described configuration is created when a user test is performed, and a completed user test evaluation chart is evaluated.

  For example, as shown in FIG. 9, when the subject performs task processing, there is an NG display in the check 91 of the function structure column 4, and it is found that the subject is performing an error operation in the operation related to the paper orientation.

  Next, referring to the qualitative data part 2, it is clear that the purpose of what the next operation should be performed is clarified by the check 92, and the operation (click) is performed with certainty by the check 93.

  When this information is integrated, the subject views the paper orientation setting as a clear purpose of the operation, and the error setting is the one that is set with confidence without any hesitation, so the user can be confident that the operation is correct. Therefore, it is difficult to notice that the setting made is an error setting.

  When the above setting operation and the sensitivity of the subject viewed from the evaluator are integrated and considered, the error is determined to be a highly important error that is difficult for the user to notice.

  In addition, it can be seen from the check 94 that the subject has not been able to select an appropriate item from a plurality of choices in the combo box, and the cause of the error is considered to be the difficulty in understanding the terms constituting the choice.

  As described above, the evaluation chart of the present invention lists the quantitative data evaluation unit and the qualitative data evaluation unit and can evaluate the user test in an integrated manner, so that the operation evaluation in the task processing becomes efficient. Highly accurate evaluation is possible.

The whole block diagram of the user test evaluation chart concerning the present invention. (A) And (B) is a block diagram of each display screen of the printer driver which is the task of the first embodiment. The block diagram of the system structure column of a quantitative data evaluation part. The block diagram of the function structure column of a quantitative data evaluation part. The block diagram of the log column of a quantitative data evaluation part. The block diagram of the time column of a quantitative data evaluation part. Qualitative data evaluation section diagram. The evaluation graph figure of the frequency according to the number of seconds of operation time. Explanatory drawing which shows the example of evaluation of the user test using the user test evaluation chart of this invention.

Explanation of symbols

1 Quantitative Data Evaluation Unit 2 Qualitative Data Evaluation Unit 3 System Structure Field 4 Function Structure Field 5 Log Field 6 Time Field 21 Tab Group 22 Combo Box 23 Second Tab Group 25 Radio Button / Check Box 31 Number Field 32 Hierarchy Display Field 33 Tab Field 34 Combo box field 36 Term selection field 37 Second tab field 38 Radio button / check box field 41 Copies field 41a, 41b Check 43 Paper orientation field 44 Paper size field 45 Paper cassette field 46 Paper type field 71 Sensitivity label group 72 Help field 73 Comment column 74 Impression column 91, 92, 93, 94 Check

Claims (3)

In the user test evaluation chart used when evaluating the user interface, the user test evaluation chart is composed of a quantitative data evaluation unit and a qualitative data evaluation unit, and the quantitative data evaluation unit and the qualitative data evaluation unit are arranged in parallel. A user test evaluation chart characterized in that it is tabulated and configured so that both quantitative data and qualitative data related to operations can be integrated and evaluated. 2. The user according to claim 1, wherein the quantitative data evaluation unit includes a system structure field for displaying an operation position on a system hierarchy and a function structure field for recording a history of setting contents in time series. Test evaluation chart. The user test evaluation chart according to claim 1, wherein the qualitative data evaluation unit includes a sensitivity label for entering sensitivity and behavior characteristics during operation as sensitivity information.

Images