JP2011145996A - Reviewer evaluation apparatus, reviewer evaluation method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a reviewer evaluation apparatus, a reviewer evaluation method and a program for quantitatively evaluating the capability of a reviewer. <P>SOLUTION: The reviewer evaluation apparatus includes: a review history storage means for storing information indicating a place according to a record in a document pointed by a reviewer, for each review of the document; an evaluation value calculation means for increasing weight according to the number of times that the same place is pointed out, based on the information stored in the review history storage means, applying the weight to each record and calculating a rate of the total weight to total records as an evaluation value of the reviewer; and an output means for outputting the evaluation value. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a reviewer evaluation device, a reviewer evaluation method, and a program, and more particularly to a reviewer evaluation device, a reviewer evaluation method, and a program for evaluating a reviewer in document review.

  In the review of documents such as design documents such as software, seniors or supervisors (hereinafter referred to as “advanced persons”), who have more specialized knowledge and experience than review reviews (document creators, etc.). ) Will serve as a reviewer. By receiving reviews from advanced users, it is expected that the quality of documents will be improved, and in turn, the quality of software developed based on the documents will be improved.

  However, simply being an expert cannot be said to be suitable as a reviewer. For example, even an advanced person with abundant technical knowledge may lack the ability to teach others. When reviews are performed by reviewers who are not qualified, it is likely that more time will be spent than necessary to improve document quality.

JP 61-160152 A Japanese Patent Laid-Open No. 11-224187

  However, the review is performed by interactive work between people, and the document to be reviewed is different for each item. Therefore, it has been difficult to quantitatively grasp the reviewer's ability.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a reviewer evaluation apparatus, a reviewer evaluation method, and a program capable of quantitatively evaluating the ability of a reviewer.

  Therefore, in order to solve the above-mentioned problem, the reviewer evaluation apparatus includes a review history storage unit that stores information indicating a point to be pointed out according to an item pointed out by the reviewer for each document, and the review history storage unit. Based on the information stored in the above, the weighting is increased according to the number of times pointed out for the same pointed portion, and the weighting is given to each pointed item, and the ratio of the sum total of the weighting to the total number of the pointed items is Evaluation value calculating means for calculating as an evaluation value, and output means for outputting the evaluation value.

  According to the disclosed technology, the reviewer's ability can be quantitatively evaluated.

It is a figure which shows the structural example of the reviewer evaluation system in embodiment of this invention. It is a figure which shows the hardware structural example of the reviewer evaluation apparatus in embodiment of this invention. It is a figure which shows the function structural example of the reviewer evaluation system in embodiment of this invention. It is a flowchart for demonstrating the whole process sequence of a reviewer evaluation system. It is a figure which shows the structural example of a review history memory | storage part. It is a flowchart for demonstrating the process sequence of the calculation process of a review coefficient. It is a figure which shows the example of the database produced for an analysis of variance. It is a figure which shows the example of an output of the calculation result of learning efficiency, and the result of an analysis of variance. It is a flowchart for demonstrating an example of the process sequence of the input of review history information, and a recording process. It is a figure which shows the example of a display of the initial state of a review screen. It is a figure which shows the example of input of a new indication matter. It is a figure which shows the example of an input of a continuation indication matter. It is a figure which shows the example of a display of a review screen when making a user judge new or neglected pointed out part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of a reviewer evaluation system according to an embodiment of the present invention. In the reviewer evaluation system 1 in FIG. 1, the reviewer evaluation apparatus 10 and one or more reviewer terminals 20 may be a local area network (LAN) or a network 30 such as the Internet (wired or wireless). ) To enable communication.

  The reviewer terminal 20 is a computer such as a PC (Personal Computer) used to input a review result (review content) for a document or the like created by the reviewer. Reviewers are those who perform reviews. Reviewers are those who receive reviews.

  The reviewer evaluation device 10 accumulates the history of review results input via the reviewer terminal 20, and based on the history, the reviewer evaluation device 10 calculates an evaluation value for evaluating the reviewer, such as a PC (Personal Computer). It is a computer.

  FIG. 2 is a diagram illustrating a hardware configuration example of the reviewer evaluation apparatus according to the embodiment of the present invention. The reviewer evaluation apparatus 10 in FIG. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, and an input connected to each other via a bus B. Device 107.

  A program for realizing the processing in the reviewer evaluation apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 on which the program is recorded is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 realizes functions related to the reviewer evaluation device 10 according to a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a GUI (Graphical User Interface) or the like by a program. The input device 107 is a keyboard, a mouse, or the like, and is used for inputting various operation instructions.

  The reviewer terminal 20 may have the same hardware configuration as that in FIG.

  FIG. 3 is a diagram illustrating a functional configuration example of the reviewer evaluation system according to the embodiment of the present invention. In the figure, the reviewer terminal 20 has an input control unit 21. The reviewer evaluation apparatus 10 includes a review coefficient calculation unit 11, a learning efficiency calculation unit 12, a statistical processing unit 13, an output unit 14, a review history storage unit 15, and the like. Each of these units is realized by processing that a program installed in the reviewer terminal 20 or the reviewer evaluation apparatus 10 causes each CPU to execute.

  The input control unit 21 receives input of information (review history information) indicating a review execution result from a user every time a review related to a document (document) such as a software design document or a project plan is performed. The input control unit 21 records the input review history information in the review history storage unit 15. The review history storage unit 15 stores review history information using the auxiliary storage device 102 or the like.

  Based on the review history information recorded (accumulated) in the review history storage unit 15, the learning efficiency calculation unit 12 is an index (hereinafter, “learning efficiency”) indicating the review efficiency as a reviewer evaluation value for each document. Is calculated). The smaller the learning efficiency value, the higher the reviewer's evaluation. When calculating the learning efficiency, the review coefficient calculation unit 11 calculates a value (review coefficient) used as a weighting coefficient for the items to be pointed out in the review. The value of the review factor is the part that was pointed out for the first pointed out point, the pointed out pointed out in the past, or the part that was subject to review at the time of the past review (ie, the existing part) ).

  The statistical processing unit 13 performs predetermined statistical processing on the review history information recorded in the review history storage unit 15. In the present embodiment, analysis of variance is used as statistical processing. With the analysis of variance, the significance of the reviewer factor and the interaction between the reviewer and the combination of the reviewer can be confirmed for the learning efficiency by the learning efficiency calculation unit 12. That is, when a calculation result indicating that the learning efficiency of a reviewer is low is obtained, whether the calculation result is accidental or caused by the reviewer is analyzed. In addition, the appropriateness (compatibility) of the combination of reviewer and reviewer is also analyzed.

  The output unit 14 outputs a processing result by the learning efficiency calculation unit 12, a processing result by the statistical processing unit 13, or a processing result by both. The output destination of the processing result is the display device 106, the auxiliary storage device 102, a printer (not shown), or the like.

  The reviewer evaluation apparatus 10 may have the function of the reviewer terminal 20. That is, the function shown in FIG. 3 may be realized in one computer.

  Hereinafter, a processing procedure in the reviewer evaluation system 1 will be described. FIG. 4 is a flowchart for explaining the entire processing procedure of the reviewer evaluation system.

  In step S <b> 101, the input control unit 21 receives input of review history information for one review, and records the review history information in the review history storage unit 15.

  FIG. 5 is a diagram illustrating a configuration example of the review history storage unit. The recorded contents of the review history storage unit 15 shown in the figure indicate the recording result of the review history information for one document. One record in the review history storage unit 15 corresponds to one indication for the document. Therefore, for example, when there are 10 points to be pointed out in one review, 10 records are recorded in the review history storage unit 15. The number of reviews for one document is not limited to one. For example, when there is an indicated item, a review is performed to confirm whether or not the indicated item is corrected.

  The record in the review history storage unit 15 includes items such as an indication ID, document ID, type, indication location ID, previous indication ID, indication content, number of reviews, review coefficient, reviewer ID, and review ID.

  The indication ID is an identifier of each record (that is, each indication item). The document ID is identification information of a document to be reviewed. The document ID may be, for example, a document name or an ID assigned to a document in a document management database (not shown). The type is information indicating the type of the indication item. In the present embodiment, the indication items are classified as new, continued, or neglected. New indications (new indications) refer to items that have not been reviewed in the past (not subject to review). Therefore, all the points to be pointed out for documents that are reviewed for the first time become new points to be pointed out. Continuation indication item (continuation indication item) refers to an indication item for a part that has been pointed out in the past. In other words, it is a continuous indication for the same part. Items to be left unattended (items to be left unattended) refer to items that have been subject to review at the time of past reviews (ie, existing portions). In other words, the abandoned items are items that have been overlooked until the previous review (items that were not pointed out).

  The indication location ID is information for identifying the indication location. What is necessary is just to employ | adopt an appropriate thing according to the form of the document used as review object as the form of indication part ID. For example, if the document is a paper document, it may indicate a range specified by a page number, a line number, or the like. If the document is an electronic document (electronic data), it may be information indicating a position in the document or a character string itself included in the pointed out position depending on the data format of the electronic data. . Alternatively, it may be information indicating a position in the document or reference information (file name or the like) to a storage basin (file or the like) in which a character string included in the indicated location is recorded. In any case, it is assumed that the position and range of the pointed part in the document can be specified by the pointed part ID.

  The previous indication ID is an indication ID of a record in a past review with respect to the same indication portion with respect to a record whose type is “continuation”. That is, a plurality of records (pointed items) for the same pointed location are associated with the previously pointed out ID.

  The pointed out item is a comment such as a correction instruction for the pointed out point. The number of reviews is the number of reviews for a document when a record is registered. That is, it is determined by the number of reviews the number of reviews registered in the review. The review coefficient is a review coefficient for the pointed item related to the record. The reviewer ID is reviewer identification information (for example, the name of the reviewer). The reviewer ID is reviewer identification information (for example, the name of the reviewer).

  In step S101, the value of each item other than the review coefficient is recorded. Further, the input person of the review history information may be a reviewer or an operator who has received the review history information from the reviewer. Furthermore, the input method of review history information is not limited to a predetermined method.

  Subsequently, the review coefficient calculation unit 11 calculates a review coefficient for each record (pointed item) registered in the review history storage unit 15, and records the calculation result as a review coefficient for each record (S102). Step S102 may be executed synchronously with step S101 (that is, each time review history information is recorded), or synchronously with step S103 and subsequent steps (that is, when learning efficiency is calculated). May be. Note that step S101 and step S103 are not necessarily executed synchronously. Steps S103 and after may be executed after review history information has been accumulated to some extent.

  Details of the review coefficient calculation process will be described. FIG. 6 is a flowchart for explaining the processing procedure of the review coefficient calculation processing. The figure shows a process of calculating a review coefficient for one record in the review history storage unit 15. Therefore, when there are a plurality of records to be processed, the process shown in FIG. Records are processed in the oldest order (that is, the registration order).

  In step S <b> 201, the review coefficient calculation unit 11 confirms the type of record to be processed (hereinafter referred to as “target record”). When the type of the target record is “new”, the review coefficient calculation unit 11 sets the review coefficient of the target record to “1” (S202). When the type of the target record is “Left”, the review coefficient calculation unit 11 sets the number of reviews required until the pointed out part of the target record is pointed out as the review coefficient of the target record (S203). Specifically, for example, when a pointed item occurs for the first time in the second review for a part described in the document from the beginning, the review coefficient for the pointed item is “2”. Note that document data version management may be performed in order to automatically determine the number of reviews required until the pointed-out part of the target record is pointed out. Specifically, for example, for each review, the document data at the time of the review may be stored in association with the number of reviews. By doing so, it is possible to determine how many times the pointed out point was included based on the pointed point ID of the target record. As a result, the number of reviews required to be pointed out can be determined.

  When the type of the target record is “continuation”, the review coefficient calculation unit 11 sets a value obtained by adding 1 to the review coefficient of the record related to the previous indication ID of the target record as the review coefficient of the target record (S204). ).

  As is clear from the processing procedure shown in FIG. 6, the review coefficient is basically increased in accordance with the number of indications for the same indication location. However, for the items pointed out for the first pointed out point (first pointed point), the value of the review coefficient for the point (existing part) that was included in the document data at the past review is included in the document data at the past review. It is made larger than the value of the review coefficient for the part that is not.

  In addition, although the addition value in step S204 of FIG. F3 is set to “1,” the addition value may be set to 2 or more when the reviewer's evaluation is lower when the continuously indicated item occurs. Similarly, in the case where the reviewer's evaluation in the case where a neglected item is generated is lower, the number of reviews + α required to be pointed out may be used as a review coefficient for the neglected item.

  In addition, neglected points may be handled in the same way as new points. Specifically, it is possible to eliminate the distinction between the abandoned indication item and the new indication item. However, by clearly distinguishing the newly pointed out item and the neglected pointed item as in this embodiment, the reviewer can be more appropriately evaluated when the pointed out item is left unattended.

  Returning to FIG. Following the review coefficient calculation process, the learning efficiency calculation unit 12 calculates the learning efficiency based on the review history storage unit 15 for each document to be reviewed (S103). In the present embodiment, it is assumed that the combination of the reviewer and the reviewer is invariant in the review performed once or more for one document. That is, if the document A is reviewed five times, the reviewer of the five reviews is always B, and the review is always C. Therefore, calculating the learning efficiency for each document means that the learning efficiency is calculated for each combination of the document, the reviewer, and the reviewer.

  The learning efficiency for one document is calculated using the following equation (1).

  In equation (1), N is the value of the review coefficient. In other words, the learning efficiency is calculated as the ratio of the sum of the review coefficients to the total number of items indicated. This means that the learning efficiency in the present embodiment is defined as the centroid of the number of reviews in which the indicated item is pointed out. Therefore, the smaller the value of learning efficiency, the higher the percentage of items pointed out upstream in the review process. The higher the percentage of indications upstream in the review process, the more likely it is that an efficient review has been performed. Therefore, the lower the learning efficiency value, the higher the reviewer's evaluation. Note that, as described above, the review coefficient is set to “1” if the pointed out point is a new point (for example, a portion added after the previous review), even if it is a pointed out item downstream. If the pointed-out item is an existing part, the number of neglected reviews is also reflected in the review coefficient. Therefore, the validity of the learning efficiency is appropriately ensured by the processing of FIG.

For example, in the review history storage unit 15 in FIG. 5, the number of items (number of records) with the review coefficient 1 is four. In addition, the number of items for which the review coefficient is 2 is 2. Therefore, the learning efficiency in this case is
(1 × 4 + 2 × 2) ÷ 6 = 1.3 (rounded to one decimal place)
It becomes.

  After step S103 is executed for a plurality of documents and the calculation result of the learning efficiency is accumulated to some extent, the statistical processing unit 13 analyzes the variance of the accumulated learning efficiency automatically or in response to an instruction input by the user. Is executed (S104). Specifically, the statistical processing unit 13 creates a database in the auxiliary storage device 102 in which the reviewer ID and the review ID are independent variables and the learning efficiency for each document is a dependent variable. The statistical processing unit 13 detects the effect on the reviewer ID and the interaction between the reviewer ID and the reviewer ID by analysis of variance based on the database.

  FIG. 7 is a diagram illustrating an example of a database created for analysis of variance. In the database 200 shown in the figure, reviewers 1 to 5 indicate reviewer IDs. Moreover, the reviews 1-5 show review ID. That is, the database 200 (table) in the figure records the learning efficiency for three documents for each combination of reviewer and reviewer. For example, for the combination of the reviewer 1 and the review 1, learning efficiencies of “1.5”, “1.8”, and “1.7” are recorded for three documents. In the example of the figure, the number of samples (the number of learning efficiency) is arranged for all the combinations of the reviewer and the reviewer, but the number of samples may be different for each combination.

  When the analysis of variance is performed based on the database 200, the F value related to the reviewer factor is 2.56. Further, the F value related to the interaction between the reviewer and the reviewer is 1.85. Any F value falls within the significance level P of 5% or more, and it is detected that both the reviewer factor and the interaction are significant (influenced) on the learning efficiency.

  Subsequently, the output unit 14 outputs the learning efficiency calculation result and the ANOVA result to the display device 106 or the like (S105).

  FIG. 8 is a diagram illustrating an output example of the learning efficiency calculation result and the analysis of variance result. In the example of the figure, for each reviewer, the average value of the learning efficiency by the combination with each reviewer is compared and displayed as a graph. The average value of learning efficiency is the average value of learning efficiency for a plurality of documents in the combination. Based on FIG. 7, in the present embodiment, the average value of the learning efficiency for the three documents is obtained. In addition, the results of analysis of variance for learning efficiency (F value and information indicating that the F value falls within the significance level P) are displayed below the graph.

  From the graph in the figure, it can be seen that the learning efficiency of the reviewer 3 is relatively low. Further, it can be seen that the learning efficiency of the combination of the reviewer 3 and the reviewer 4 is remarkably low when viewed relatively. Therefore, in this case, it is possible to consider measures such as improving the review ability of the reviewer 3 or assigning a person other than the reviewer 3 to the reviewer for the reviewer 4 Levy.

  The analysis of variance is for confirming that the learning efficiency calculation result is significant (not accidental). Therefore, if it is not necessary to confirm significance, analysis of variance may not be performed.

  Next, an example of step S101 will be described in detail. FIG. 9 is a flowchart for explaining an example of a processing procedure of review history information input and recording processing. In the figure, it is assumed that the document to be reviewed is electronic data (document data).

  When the reviewer specifies document data to be reviewed and inputs a review screen display instruction on the review terminal 20, the input control unit 21 causes the review screen to be displayed on the display device of the reviewer terminal 20 (S301).

  FIG. 10 is a diagram illustrating a display example of the initial state of the review screen. In the figure, the review screen 300 includes a document display area 310, an indication item display area 320, an indication item button 330, a completion button 340, and the like.

  The document display area 310 is an area where the content (document) of document data to be reviewed is displayed. The indication item display area 320 is an area where the indication item is displayed when the indication item is input. The indication item button 330 is a button for receiving an indication input instruction item. The completion button 340 is a button for accepting a review completion instruction.

  The reviewer performs a review by browsing the document displayed in the document display area 310 of the review screen 300. In the course of the review, when an item to be pointed out occurs, the reviewer selects a point (area) to be the item to be pointed out in the document and presses the pointed item button 330. In response to pressing of the indication item button 330, the input control unit 21 displays a comment input field in the indication item display area 320 (S303). The input control unit 21 stores the indication items (comments) input in the comment input field using the memory device of the reviewer terminal 20 (S304). Note that the indication item input in step S304 is a new indication item.

  FIG. 11 is a diagram illustrating an input example of a new indication item. In the same figure, an example is shown in which a pointed item (“object is ambiguous, specifically describe”) is input in the comment input field 321 displayed by selecting the area 311. In the figure, the comment input field 321 is displayed in the form of a balloon from the area 311 so that the relationship between the area 311 and the comment input field 321 can be easily recognized.

  Subsequently, the input control unit 21 generates an indication location ID indicating a location related to the selected region 311 in the document data (S305). As described above, the indicated location ID may be generated as data suitable for specifying a part of the document data in accordance with the data format of the document data. Subsequently, the input control unit 21 determines whether or not the location (pointed location) related to the region 311 is included in the past version of the document data to be reviewed (S306). The past version refers to document data at the time of review performed on the document data to be reviewed in the past. The past version of the document data is stored in the auxiliary storage device 102 in association with the number of reviews, for example. The determination in step S306 may be performed based on, for example, whether or not a portion (pointed portion) corresponding to the pointed portion ID generated in step S305 is included in the past version document data. In addition, when the review has already been performed a plurality of times, it is determined whether or not a plurality of past versions include the pointed out part.

  If the indicated location is not included in any past version, or if the review is the first time (when there is no past version) (No in S306), the input control unit 21 records a new record (hereinafter, “ The type of “target record” is set to “new” (S307). Subsequently, the input control unit 21 sets the current number of reviews in the target record (S308). The number of reviews this time may be obtained by adding 1 to the number of reviews of the record registered last in the review history storage unit 15 corresponding to the document data to be reviewed. Alternatively, the number of reviews may be separately associated and recorded in the review history storage unit 15, and the current number of reviews may be determined based on the number of reviews.

  Subsequently, the input control unit 21 sets the values of the remaining items excluding the review coefficient in the target record (S309). Specifically, a value obtained by adding 1 to the indication ID of the last record recorded in the review history storage unit 15 is set as the indication ID of the target record. Further, the indication location ID generated in step S305 is set as the indication location ID of the target record. Further, the indication item input in step S304 is set as the indication item of the target record. Further, the reviewer ID of the user performing the review is set as the reviewer ID of the target record. Further, the review ID of the creator of the document data to be reviewed is set as the review ID of the target record. The reviewer ID may be input to the reviewer before the review screen 300 is displayed. Or the login user name to the reviewer terminal 20 may be acquired from OS (Operating System) as reviewer ID. Further, the review ID may be acquired from the document data, or may be input to the reviewer before the review screen 300 is displayed.

  Subsequently, the input control unit 21 registers the target record in the review history storage unit 15 corresponding to the document data to be reviewed (S331).

  On the other hand, in step S306, when the pointed out part is included in any past version (Yes in S306), the input control unit 21 sets the type of the target record to “Leave” (S311). Subsequently, the input control unit 21 sets the current number of reviews in the target record (S312). Subsequently, the input control unit 21 sets the values of the remaining items excluding the review coefficient in the target record (S313). The processing content of step S313 may be the same as that of step S309. Subsequently, the input control unit 21 registers the target record in the review history storage unit 15 corresponding to the document data to be reviewed (S331).

  Further, when any comment input field displayed in the indicated item display area 320 is selected on the review screen 300 and the indicated item button 330 is pressed, the input control unit 21 points out a new comment input column. It is displayed in the item display area 320 (S303). In the case where the review is performed for the second time or later, in the initial item display area 320 of the review screen 300, a comment input column corresponding to the previous items to be displayed is displayed. In the document display area 310, the points pointed out up to the previous time are selected. The input control unit 21 stores the indication items (comments) input in the comment input field using the memory device of the reviewer terminal 20 (S332). Note that the indication items input in step S332 are continuation indication items.

  FIG. 12 is a diagram illustrating an example of inputting a continuation indication item. In the same figure, an example is shown in which a comment item (“Would you like to write a subordinate concept” because the purpose is too abstract?) Is entered in the comment input field 322 displayed by selecting the comment input field 321. ing. In the figure, the comment input field 322 is displayed in a form of a balloon from the comment input field 321 so that the relationship between the comment input field 321 and the comment input field 322 can be easily recognized.

  Subsequently, the input control unit 21 sets the type of the target record to “continuation” (S323). Subsequently, the input control unit 21 sets the indication ID of the record corresponding to the selected comment input field 321 (hereinafter referred to as “parent record”) as the previous indication ID of the target record (S324). Subsequently, the input control unit 21 sets the indication location ID of the parent record as the indication location ID of the target record (S325). Subsequently, the input control unit 21 sets the current number of reviews in the target record (S326). Subsequently, the input control unit 21 sets the values of the remaining items excluding the review coefficient in the target record (S327). Specifically, a value obtained by adding 1 to the indication ID of the last record recorded in the review history storage unit 15 is set as the indication ID of the target record. Further, the indication item input in step S322 is set as the indication item of the target record. Further, the reviewer ID of the user performing the review is set as the reviewer ID of the target record. Further, the review ID of the creator of the document data to be reviewed is set as the review ID of the target record. Subsequently, the input control unit 21 registers the target record in the review history storage unit 15 corresponding to the document data to be reviewed (S331).

  When the completion button 340 is pressed on the review screen 300, the input control unit 21 deletes the review screen 300 and ends the process of FIG.

  Note that a review coefficient may be registered during the processing of FIG. 9, that is, at the time of registration of review history information. In this case, for example, steps S202, 203, or S204 in FIG. 6 may be executed after steps S309, S313, and S327, respectively. When the review coefficient is registered at the time of registering the review history information, step S102 in FIG. 4 may not be executed.

  By the way, in FIG. 9, although the input control part 21 demonstrated the example which judges automatically whether the pointed part is contained in the past version, ie, whether the pointed out matter is new or neglected, Judgment may be left to the user. In this case, the review screen may be as shown in FIG.

  FIG. 13 is a diagram illustrating a display example of a review screen in a case where the user determines whether the indicated part is new or left unattended. In FIG. 13, the same parts as those in FIG.

  The review screen 300a shown in the figure has a new indication item button 331 and a neglect indication item button 332 instead of the indication item button 330. The new indication item button 331 is a button that is pressed when the indication item for the area selected in the document display area 310 corresponds to the new indication item. The neglected indication item button 332 is a button that is pressed when the indication item for the area selected in the document display area 310 corresponds to the neglected indication item.

  The processing procedure when the review screen 300a is employed may be substantially the same as that shown in FIG. However, the branch destination in step S306 may be determined based on whether the pressed button is the new indication item button 331 or the neglect indication item button 332. That is, when the new indication item button 331 is pressed, step S307 and subsequent steps are executed. When the neglect indication button 332 is pressed, step S311 and subsequent steps are executed.

  As described above, according to the present embodiment, it is possible to quantitatively evaluate a reviewer's instruction based on learning efficiency.

  In addition, when calculating the learning efficiency, if an indication is repeated for the same point, the weighting (review coefficient) for the learning efficiency is increased according to the number of points to be pointed out. Regardless, the review coefficient is “1”. That is, the review coefficient is increased according to the number of indications for continuously indicated items, while the review coefficient is set to “1” according to the number of indications for newly indicated items. Therefore, it is possible to avoid the pointed items for the newly added portion from being evaluated in the negative direction, compared with the case of simply evaluating the decreasing tendency of the pointed items according to the number of reviews.

  In addition, even if it is the first pointed out point, if the pointed out point is an existing part, it will be left as a pointed out pointed item, and the method of giving the new pointed item and review coefficient will be changed, so the pointed out pointed out (left out) pointed out It is possible to appropriately reflect the evaluation of the location on the learning efficiency.

  Further, by comparing learning efficiencies according to the combination of the reviewer and the reviewer, it is possible to detect a combination that is not compatible.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
For each document review, review history storage means for storing information indicating the pointed out point according to the items pointed out by the reviewer for the document,
Based on the information stored in the review history storage means, the weighting is increased according to the number of times pointed out for the same pointed portion and the weighting is given to each pointed item, and the ratio of the sum of the weights to the total number of pointed items Evaluation value calculating means for calculating the evaluation value of the reviewer,
A reviewer evaluation apparatus comprising output means for outputting the evaluation value.
(Appendix 2)
The calculation means, for the pointed item for the first pointed out point, the weighting of the pointed item for the point included in the document at the past review is the pointed out for the point not included in the document at the past review The reviewer evaluation apparatus according to appendix 1, wherein the reviewer is set to be larger than the weighting of the matter.
(Appendix 3)
The reviewer evaluation apparatus according to appendix 1 or 2, wherein the output means displays the evaluation value calculated for a document related to the same review for a plurality of reviewers.
(Appendix 4)
The document data of the document is displayed, the indication item is input to the selected location in the document data, and information indicating the selected location is used as information indicating the indication location according to the indication item The reviewer evaluation apparatus according to any one of appendices 1 to 3, further comprising an input control unit that records the review history storage unit.
(Appendix 5)
Computer
For each review of a document, weighting is increased according to the number of times pointed out for the same pointed point based on information stored in the review history storage means for storing information indicating the pointed point in accordance with the items pointed out by the reviewer for the document And assigning the weighting to each indication item, and an evaluation value calculation procedure for calculating a ratio of the total sum of the weighting to the total number of the indication items as an evaluation value of the reviewer,
A reviewer evaluation method for executing an output procedure for outputting the evaluation value.
(Appendix 6)
In the calculation procedure, the weighting of the pointed matter with respect to the point included in the document at the time of the past review is the pointed out point to the point that is not included in the document at the time of the past review. The reviewer evaluation method according to appendix 5, wherein the reviewer is set to be larger than the weight of the item.
(Appendix 7)
The reviewer evaluation method according to appendix 5 or 6, wherein the output procedure displays the evaluation value calculated for a document related to the same review for a plurality of reviewers.
(Appendix 8)
The document data of the document is displayed, the indication item is input to the selected location in the document data, and information indicating the selected location is used as information indicating the indication location according to the indication item The reviewer evaluation method according to any one of appendices 5 to 7, further comprising an input control procedure for recording in the review history storage means.
(Appendix 9)
On the computer,
For each review of a document, weighting is increased according to the number of times pointed out for the same pointed point based on information stored in the review history storage means for storing information indicating the pointed point in accordance with the items pointed out by the reviewer for the document And assigning the weighting to each indication item, and an evaluation value calculation procedure for calculating a ratio of the total sum of the weighting to the total number of the indication items as an evaluation value of the reviewer,
A program for executing an output procedure for outputting the evaluation value.
(Appendix 10)
In the calculation procedure, the weighting of the pointed matter with respect to the point included in the document at the time of the past review is the pointed out point to the point that is not included in the document at the time of the past review. The program according to appendix 9, wherein the program is set to be larger than the weight of the item.
(Appendix 11)
The program according to appendix 9 or 10, wherein the output procedure displays, for a plurality of reviewers, the evaluation values calculated for documents relating to the same review.
(Appendix 12)
The document data of the document is displayed, the indication item is input to the selected location in the document data, and information indicating the selected location is used as information indicating the indication location according to the indication item The program according to any one of appendices 9 to 11, further comprising an input control procedure recorded in the review history storage unit.

DESCRIPTION OF SYMBOLS 1 Reviewer evaluation system 10 Reviewer evaluation apparatus 11 Review coefficient calculation part 12 Learning efficiency calculation part 13 Statistical processing part 14 Output part 15 Review history memory | storage part 20 Reviewer terminal 21 Input control part 30 Network 100 Drive apparatus 101 Recording medium 102 Auxiliary Storage device 103 Memory device 104 CPU
105 interface device 106 display device 107 input device B bus

Claims (6)

For each document review, review history storage means for storing information indicating the pointed out point according to the items pointed out by the reviewer for the document,
Based on the information stored in the review history storage means, the weighting is increased according to the number of times pointed out for the same pointed portion and the weighting is given to each pointed item, and the ratio of the sum of the weights to the total number of pointed items Evaluation value calculating means for calculating the evaluation value of the reviewer,
A reviewer evaluation apparatus comprising output means for outputting the evaluation value.   The calculation means, for the pointed item for the first pointed out point, the weighting of the pointed item for the point included in the document at the past review is the pointed out for the point not included in the document at the past review The reviewer evaluation apparatus according to claim 1, wherein the reviewer is set to be larger than the weighting of items.   The reviewer evaluation apparatus according to claim 1, wherein the output unit displays the evaluation value calculated for documents related to the same review for a plurality of reviewers.   The document data of the document is displayed, the indication item is input to the selected location in the document data, and information indicating the selected location is used as information indicating the indication location according to the indication item The reviewer evaluation apparatus according to any one of claims 1 to 3, further comprising input control means for recording in the review history storage means. Computer
For each review of a document, weighting is increased according to the number of times pointed out for the same pointed point based on information stored in the review history storage means for storing information indicating the pointed point in accordance with the items pointed out by the reviewer for the document And assigning the weighting to each indication item, and an evaluation value calculation procedure for calculating a ratio of the total sum of the weighting to the total number of the indication items as an evaluation value of the reviewer,
A reviewer evaluation method for executing an output procedure for outputting the evaluation value. On the computer,
For each review of a document, weighting is increased according to the number of times pointed out for the same pointed point based on information stored in the review history storage means for storing information indicating the pointed point in accordance with the items pointed out by the reviewer for the document And assigning the weighting to each indication item, and an evaluation value calculation procedure for calculating a ratio of the total sum of the weighting to the total number of the indication items as an evaluation value of the reviewer,
A program for executing an output procedure for outputting the evaluation value.

Images