JP2013101501A - Verification device, control method for verification device, input device, examination system, control program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently confirm determination as to whether a discrimination candidate obtained by recognition of input with handwriting operation matches a verification candidate.SOLUTION: The present invention is provided with: a reliability evaluation unit 15 for determining reliability 4 of a correctness/erroneousness assessment 3a according to whether or not correct response data 8 matches discrimination data 1 if a score 2 for the discrimination data 1 approximates a score 2 for other discrimination data 1; and a correctness/erroneousness assessment processing unit 16 which presents, to the user, the reliability 4 together with the correctness/erroneousness assessment 3a.

Description

  The present invention relates to a collation apparatus that collates an identification candidate that has recognized an input by handwriting operation with a collation target to determine whether or not they match.

  In an electronic device having a display screen with a touch panel, a user can input characters and figures by handwriting operation using an indicator such as a finger or a stylus (hereinafter simply referred to as “indicator”). The above handwritten interface capable of intuitive operation has been widely spread mainly in information terminals such as so-called smartphones and some home game machines.

  In recent years, there has been a movement to apply electronic devices having the above-described interface in the education field. For example, a system has been developed that automates the scoring of the test by allowing the examinee to input the written test answer to the electronic device by handwriting, recognizing the input handwritten character and converting it to text data, etc. Yes.

  In the above system, the accuracy of character recognition becomes a problem. That is, there is a possibility that the system may misrecognize the scoring (correction determination) of the test by misrecognizing handwritten characters. As a direct solution to this problem, improving the recognition accuracy of handwritten characters can be considered. For example, as described in Patent Document 1 below, the accuracy of character recognition is improved by improving the accuracy of recognition processing such as character recognition processing without preparing the entire document for the recognition processing. Efforts are made widely.

JP 2011-048499 A (published March 10, 2011)

  However, it is difficult to eliminate errors in correct / incorrect determination only by improving the accuracy of character recognition. This is because there is a limit in recognizing a large number of handwritten characters of an unspecified number of examinees with high accuracy using a classifier trained based on a limited number of learning data.

  Therefore, a method for preventing errors by allowing a human to visually confirm the result of character recognition or correctness determination is conceivable. However, a method of preventing an error by presenting an identification candidate obtained by character recognition to the examinee and selecting a correct candidate is undesirable because it may also present an answer to the examinee. In addition, the method in which the grader confirms the result of the correct / incorrect determination has a problem that a greater amount of labor is required as the number of examinees and the number of questions increase.

  The present invention has been made in view of the above problems, and its purpose is to provide a user with certainty about determining whether or not an identification candidate that has recognized input by handwriting operation matches a collation target. It is to provide a collation device, a collation device control method, an input device, a test system, a control program, and a recording medium that enable efficient confirmation of the determination by presenting.

In order to solve the above problems, the collation device of the present invention is:
(1) For each of at least one identification candidate obtained by recognizing input by handwriting operation, an evaluation value that evaluates the accuracy of the recognition is calculated, and the first identification that is the identification candidate with the largest evaluation value A collation device that determines whether the first identification candidate matches the collation target by collating the candidate with the collation target,
(2) When the evaluation value of the first identification candidate and the evaluation value of another identification candidate are close to each other, the group of the first identification candidate and the other identification candidate matches the collation target Evaluation means for determining the reliability of the determination according to whether there is a
(3) The present invention is characterized by comprising a presenting means for presenting the reliability determined by the evaluation means to the user together with the result of the determination.

In addition, in order to solve the above-described problem, the control method of the collation device of the present invention is:
(1) For each of at least one identification candidate obtained by recognizing input by handwriting operation, an evaluation value that evaluates the accuracy of the recognition is calculated, and the first identification that is the identification candidate with the largest evaluation value A method for controlling a collation device that determines whether or not the first identification candidate matches the collation target by collating a candidate with a collation target,
(2) When the evaluation value of the first identification candidate and the evaluation value of another identification candidate are close to each other, the group of the first identification candidate and the other identification candidate matches the collation target An evaluation step for determining the reliability of the determination according to whether there is a
(3) The present invention includes a presentation step of presenting the reliability determined in the evaluation step to the user together with the result of the determination.

  According to the above configuration, the collation device of the present invention and the control method of the collation device are based on the evaluation value calculated for each of the identification candidates and the evaluation value of the first identification candidate and the other identification candidates. It can be determined whether or not the evaluation value is approximate. When the evaluation values are approximate, the collation device of the present invention matches the first identification candidate with the collation target according to whether or not there is a match with the collation target among the identification candidates. It is possible to determine the reliability of the determination as to whether or not the image is present and to present the reliability and the determination result to the user.

  Therefore, the user can discover the error with a high probability only by confirming a determination evaluated as having low reliability, for example. That is, the collation apparatus and the control method for the collation apparatus according to the present invention enable the user to efficiently confirm the determination result, and reduce the burden of the work of checking whether there is an erroneous determination.

In the above collation device,
(1) The evaluation unit compares the comparison value calculated based on the evaluation value with a predetermined threshold value, so that the evaluation value of the first identification candidate and the evaluation value of another identification candidate are It is characterized by determining whether or not they are approximate.

  According to the above configuration, the collation device of the present invention compares the comparison value with the threshold value to determine whether or not the evaluation value of the first identification candidate and the evaluation value of the other identification candidates are approximated. Can be determined. Therefore, the evaluation criteria for the reliability of the determination can be changed by the user appropriately adjusting the threshold value. In other words, for example, if a determination error is not permitted, the threshold value is set strict, and if the user's workload is to be reduced even if some error is allowed, the threshold value is set gently. Evaluation criteria can be set.

  Thereby, the collation apparatus of this invention enables a user to confirm a determination result more efficiently, and can further reduce the burden of the operation | work which checks the presence or absence of a misjudgment.

In the above collation device,
(1) It is a difference value between the evaluation value of the first identification candidate and the second largest evaluation value or the third largest evaluation value.

  According to said structure, the collation apparatus of this invention is the difference value of the largest evaluation value and the 2nd largest evaluation value among evaluation values calculated with respect to each of identification candidates, or the largest evaluation. Reliability is evaluated by using a difference value between the value and the third largest evaluation value as a comparison value. Therefore, the collation device of the present invention can evaluate the reliability in consideration of the first identification candidate (first identification candidate) to the third identification candidate.

In addition, the verification device
(1) When the presenting means presents the determination result and the reliability of the determination result to the user, the presenting means further comprises a correcting means for accepting an operation for correcting the determination result from the user. .

  According to said structure, the collation apparatus of this invention receives operation which corrects the result of the shown determination from a user. Therefore, for example, after confirming a determination evaluated as having low reliability, the user can correct this determination if the determination is erroneous. Thereby, the collation apparatus of this invention can correct the result of determination efficiently, and can reduce the burden of the operation | work which corrects the said result.

In the above collation device,
(1) The correction means includes the first identification candidate obtained from a handwriting operation, the collation target collated with the first identification candidate, and whether the first identification candidate matches the collation target. From the storage device that stores a plurality of combinations with the determination results, a combination in which both the first identification candidate and the collation target match the predetermined first identification candidate and the predetermined collation target is extracted. An operation for correcting the result of each determination is received from a user.

  According to said structure, the collation apparatus of this invention is for correcting each determination result of the combination in which both a 1st identification candidate and a collation target correspond to a predetermined 1st identification candidate and a predetermined collation target. Accept user operations.

  Therefore, for example, after confirming a determination evaluated as low reliability, the user can not only correct the result of the determination if the determination is erroneous, but also determine the result of the determination that is likely to be erroneous. Further corrections are possible. Thereby, the collation apparatus of this invention can correct the result of determination more efficiently, and can further reduce the burden of the operation | work which corrects the said result.

The input device of the present invention is
(1) An input surface capable of acquiring a locus of handwriting operation is provided, information can be input based on the locus, and the input information is sent to the verification device.

  According to said structure, the input device of this invention sends out the information input by handwriting operation to the said collation apparatus. Here, as an input device of the present invention, an answering client used by the examinee is given as an example. The information input via the input device of the present invention is, for example, a response to data received from the outside, that is, an answer to problem data.

Moreover, the test system of the present invention includes:
(1) It is characterized by including the collation device and the input device.

  With the above-described configuration, the test system of the present invention makes it possible to efficiently perform determination confirmation, and can reduce the burden of work for checking whether there is an erroneous determination. Thereby, the test system of this invention can support effectively the operation | work accompanied by the said determination based on the input by handwriting operation, such as automatic scoring of a test, for example.

  The collation device or the input device may be realized by a computer. In this case, a control program for causing the computer to implement the verification device or the input device by operating the computer as each means of the verification device or the input device, and a computer-readable recording medium on which the control program is recorded are also included in the present invention. It falls into the category.

  If the evaluation value of the first identification candidate and the evaluation value of the other identification candidate are close, is there a match between the first identification candidate and the group of other identification candidates that matches the collation target? The configuration includes an evaluation unit that determines the reliability of the determination according to whether or not, and a presentation unit that presents the reliability determined by the evaluation unit to the user together with the result of the determination.

  In addition, when the evaluation value of the first identification candidate and the evaluation value of the other identification candidate are approximate, a group that includes the first identification candidate and the other identification candidate and that matches the collation target The configuration includes an evaluation step for determining the reliability of the determination according to whether or not there is, and a presentation step for presenting the reliability determined in the evaluation step to the user together with the result of the determination.

  Therefore, the user can discover the error with a high probability only by confirming a determination evaluated as having low reliability, for example. That is, the collation device of the present invention has an effect of enabling the user to efficiently confirm the determination result and reducing the burden of checking the presence / absence of erroneous determination.

It is a block diagram which shows a partial structure of the scoring server which concerns on one Embodiment of this invention. It is a block diagram which shows the partial structure of the client for an answer which concerns on one Embodiment of this invention. It is the schematic of the structure of the test system which concerns on one Embodiment of this invention containing the said scoring server and the said client for an answer. It is the schematic diagram showing the example of the question data displayed on the display part of the said client for answers, (a) is before the answer by a student, (b) shows the example of a screen after handwritten characters were filled in by the student . It is the schematic diagram which illustrated the format of the data showing the handwritten character transmitted to the said scoring server from the said client for an answer. It is the schematic diagram showing the test result of a certain student displayed on the display part of the said scoring server. It is a schematic diagram which shows the identification result of a handwritten character, (a) is when the value of the score with respect to 1st identification data is remarkably larger than the value of the score with respect to other identification data, (b) is between scores The case where there is no remarkable difference is shown. It is the schematic diagram which showed the example of a screen displayed when correcting a right / wrong judgment, (a) shows before correction of right / wrong judgment, (b) shows after correction. FIG. 10 is a schematic diagram illustrating a result of a problem in which the combination of identification data and correct answer data related to the corrected determination is the same when the teacher corrects an incorrect correct / incorrect determination due to erroneous recognition. FIG. 10 is a schematic diagram illustrating an example of a confirmation dialog displayed by the scoring server when the teacher selects whether to display a list of test results illustrated in FIG. 9. It is a flowchart which shows an example of the process which the said scoring server and the said client for an answer perform. It is a flowchart which shows an example of the process for calculating the reliability which the said scoring server performs. It is a flowchart which shows an example of the process for confirming and correcting the scoring result which the said scoring server performs. It is a flowchart which shows an example of the process for confirming and correcting the similar scoring result which the said scoring server performs.

  Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. In addition, in order to concisely describe the structure that is clearly shown to exist by adding a or b to one symbol, the structure may be collectively referred to in order to simplify the description. (For example, the correctness determination 3a and the correctness determination 3b may be collectively referred to as “correctness determination 3”).

[Outline of Test System 120]
Based on FIG. 3, the structure of the test system 120 which concerns on this Embodiment is demonstrated. FIG. 3 is a schematic diagram of a configuration of the test system 120 including the scoring server 100 and the answering client 110.

  As illustrated, the test system 120 includes a scoring server 100 and an answering client 110. Both perform communication based on a server / client system. In addition, the kind of communication network which relays both is not limited. That is, the communication network only needs to be able to communicate with each other. For example, the communication network may be the Internet or a local area network (LAN).

  If the test system 120 is used, the test which evaluates the proficiency of learning in an educational institution etc. can be implemented. In the present embodiment, it is assumed that a teacher (user) of an elementary school imposes an arithmetic test on a student using the test system 120. However, this is only for convenience of explanation, and the situation where the test system 120 can be applied is not limited to the assumed situation. For example, it can be used for examinations for various qualifications and for entertainment purposes such as games.

[Outline of scoring server 100 and answering client 110]
Based on FIG. 1, the outline | summary of the scoring server 100 which concerns on this Embodiment is demonstrated. FIG. 1 is a block diagram showing a partial configuration of the scoring server 100.

  The scoring server (collation device) 100 evaluates the accuracy of recognition for each of at least one identification data (identification candidate) 1 obtained by recognizing handwritten characters (input by handwriting operation) 5 ( (Evaluation value) 2 is calculated, and the first identification data (first identification candidate, hereinafter referred to as “first identification data”), which is the identification data having the highest score 2, is collated with the correct data (collation target). By this, it is an apparatus which determines whether the said 1st identification data correspond with the said collation object.

  Scoring when it is determined that a match is included and the reliability of the determination is low by comparing the value calculated based on score 2 with a predetermined threshold The server 100 requests the confirmation by presenting the correctness / incorrectness determination 3a to the teacher. When the teacher finds an erroneously determined answer, the teacher can correct the correctness determination 3a to the correctness determination 3b via the input unit 80a. The scoring server 100 stores the corrected correctness determination 3b in the storage device 30. That is, the scoring server 100 can be said to be a device that supports scoring of handwritten tests using the score 2 that evaluates the reliability of character recognition and the correct answer data 8 for the problem.

  The identification data 1 is an identification candidate obtained by character recognition of the scoring server 100. The score 2 refers to a value obtained by evaluating the certainty (accuracy) for the scoring server 100 to identify the handwritten character 5 from the identification data 1. In the following, as shown in FIG. 7 to be described later, the score 2 is expressed as a real value such as “82” or “30”, but is not limited thereto. That is, anything can be used as long as the certainty factor for identifying the handwritten character 5 as the identification data 1 can be expressed in order.

  Based on FIG. 2, an outline of the answering client 110 according to the present embodiment will be described. FIG. 2 is a block diagram showing a partial configuration of the answering client 110.

  The answering client (input device) 110 includes an input surface capable of acquiring a handwriting operation trajectory, and can input a handwritten character (handwriting operation trajectory) 5 based on the trajectory. The handwritten character is input to the scoring server 100. It is a device to send out.

[Outline of operation of test system 120]
Based on FIGS. 4-10, the outline of the flow of the said process performed by the test system 120 (the scoring server 100, the client 110 for an answer) is demonstrated. In the test system 120, processing proceeds in the following four steps.
(1) The scoring server 100 transmits the question data 6 to each of the answering clients 110.
(2) The answering client 110 transmits to the scoring server 100 the handwritten character 5 input by the handwriting operation from the student as an answer to the question data 6.
(3) The scoring server 100 recognizes the handwritten character 5, makes a correct / incorrect determination (answer matching) by collating the recognized character with the correct answer, and presents the result to the teacher.
(4) The scoring server 100 receives an input for correcting the correctness determination 3 a from the teacher, and stores the correctness determination 3 b after correction in the storage device 30.

  FIG. 4 is a schematic diagram illustrating an example of the question data (data received from the outside) 6 displayed on the display unit 70b of the answering client 110. (a) is before the answer by the student, and (b) is the student. The example of a screen after handwritten character 5 is filled in is shown. As shown in the screen example shown in FIG. 4A, the answering client 110 presents the question data 6 received from the scoring server 100 to the student. As shown in FIG. 4B, the student enters the answer (handwritten character 5) by handwriting.

  FIG. 5 is a schematic view illustrating the format of data representing the handwritten character 5 transmitted from the answering client 110 to the scoring server 100. As illustrated in the figure, the answering client 110 is provided on the input surface of an indicator such as a user's finger or stylus (hereinafter simply referred to as “indicator”) that has touched the input surface of the input unit 80b. The two-dimensional coordinates are acquired at predetermined time intervals, and the series of coordinates is used as data representing the handwritten character 5 written for each problem.

  FIG. 6 is a schematic diagram showing the test result of a student (in the figure, a student named “Nihon Taro”) displayed on the display unit 70a of the scoring server 100. As illustrated in the figure, for each question, identification data 1, correct answer data 8, correctness determination 3a, and reliability 4 are presented to the teacher in tabular form. Note that the identification data 1 presented in the table is first identification data. That is, the identification candidate with the highest score 2 is used for answer matching (correction determination).

  Here, the correct answer data 8 is a correct answer corresponding to the question data 6. The right / wrong judgment 3 is information indicating whether or not the student has correctly answered the question. As illustrated in FIG. 6, the scoring server 100 presents the correctness / incorrectness determination 3 to the teacher by displaying an appropriate symbol indicating the correct or incorrect answer. The reliability 4 is information indicating the reliability of the correctness determination 3 a by the scoring server 100. The reliability 4 will be described in detail later.

  FIG. 7 is a schematic diagram showing the identification result of the handwritten character 5. FIG. 7A shows a case where the score 2 value for the first identification data 1 is significantly larger than the score 2 value for the other identification data 1. , (B) shows the case where there is no significant difference between the scores of 2. The scoring server 100 obtains at least one identification data 1 as the identification result of the handwritten character 5. For each identification data 1, a score 2 indicating the accuracy of character recognition is calculated.

  In FIG. 7A, the score 2 when “3” is recognized as the first identification data 1 is “82”, and the score 2 in other cases is “30” or “20”. At this time, the scoring server 100 determines that the value of the score 2 for the first identification data 1 is significantly larger than the value of the score 2 for the other identification data 1. At this time, the scoring server 100 sets the reliability 4 to “high” on the assumption that the correctness determination 3 a has high reliability. In the example in FIG. 7, it is assumed that each score is normalized in the range of 0-100.

  In FIG. 7B, the scoring server 100 determines that the value of the score 2 for the first identification data 1 is not significantly different from the value of the score 2 for the other identification data 1. The correct answer data for the question data 6 is “4”. In this case, the student correctly enters “4”, but the first identification data 1 is recognized as “9” due to the misrecognition of the scoring server 100, and the result is collated with the correct answer data 8. Judgment may have been made. At this time, the scoring server 100 sets the reliability 4 to “low” on the assumption that the accuracy determination is low in reliability.

  FIG. 8 is displayed when the teacher who has confirmed the list display of the test results illustrated in FIG. 6 selects “Problem 2” with the reliability 4 set to “Low” and corrects the correctness determination 3 a. 6A is a schematic diagram showing an example of a screen, where FIG. 5A shows before correction of the correctness determination 3a, and FIG. As illustrated in the figure, the teacher can confirm all combinations of the identification data 1, the correct answer data 8, and the correct / incorrect determination 3a for the problem evaluated as having a low reliability 4 of the correct / incorrect determination 3a. When the right / wrong judgment 3a is found, the right / wrong judgment 3b can be corrected.

  FIG. 9 is a schematic diagram showing a result of a problem in which the combination of the identification data 1 and the correct answer data 8 related to the corrected determination is the same combination when the teacher corrects the incorrect correctness determination 3a. As illustrated in the figure, the scoring server 100 has the above-described combination in which the correctness determination 3a is corrected (in the figure, the combination in which the identification data 1 is “9” and the correct answer data 8 is “4”). Only test results with the same combination are presented to the teacher. This is because there is a strong possibility that the same character is wrong in the right / wrong determination 3a in the same way, and it is considered that the necessity of correcting the right / wrong determination 3a is higher than in other combinations. In this way, by prompting the teacher to confirm the test results of the same combination, the correctness determination 3a can be corrected efficiently.

  FIG. 10 is a schematic diagram illustrating an example of a confirmation dialog displayed by the scoring server 100 when the teacher selects whether to display a list of test results illustrated in FIG. 9. As illustrated in the figure, when a list of test results for the same combination is displayed, the scoring server may confirm with the teacher.

[Configuration of scoring server 100]
Based on FIG. 1, the detail of each structure of the scoring server 100 which concerns on this Embodiment is demonstrated. As illustrated, the scoring server 100 includes a communication unit 20a, a character recognition unit 40, a scoring unit 50, a control unit 11, a reliability evaluation unit (evaluation unit) 15, a correctness determination processing unit (presentation unit) 16, and a correction unit ( Correction means) 17, display control unit 60 a, display unit 70 a, input unit 80 a, and storage device 30.

  Note that parts not directly related to the present embodiment (for example, a part that realizes audio output or a connection interface with an external device) are omitted from the description of the configuration and the block diagram from the viewpoint of ensuring the simplicity of the description. did. However, the scoring server 100 according to the present embodiment may include the omitted configuration in accordance with the actual situation.

  Hereinafter, the communication unit 20a (transmission unit 22a, reception unit 21a), character recognition unit 40 (feature extraction unit 41, identification unit 42, score calculation unit 43), scoring unit 50, control unit 11 (reliability evaluation unit 15, correct / incorrect) The function which each structure plays in order of the determination process part 16, the correction part 17), the display control part 60a, the display part 70a, the input part 80a, and the memory | storage device 30 is demonstrated.

  The communication unit 20a communicates with the answering client 110 through a communication network according to a predetermined communication method. It is only necessary to have an essential function for realizing communication, and a telecommunication line, a communication method, a communication medium, or the like to be used is not limited. The device can be configured with, for example, an Ethernet (registered trademark) adapter. Further, as a communication method or a communication medium, for example, IEEE 802.11 wireless communication, Bluetooth (registered trademark), or the like can be used. The communication unit 20a includes a transmission unit 22a and a reception unit 21a.

  The transmission unit 22 a reads the question data 6 from the storage device 30 and transmits it to the answering client 110.

  The receiving unit 21a receives the handwritten character 5 (see FIG. 5) transmitted from the answering client 110 and outputs it to the character recognizing unit 40. The reception unit 21a may output the handwritten character 5 to the character recognition unit 40 each time the handwritten character 5 is input (that is, for each answer to each question), or the input handwritten character 5 may be stored in the storage device 30. The handwritten characters 5 may be output to the character recognition unit 40 after answers to all or some of the questions are obtained.

  The character recognition unit 40 recognizes the handwritten character 5 input from the receiving unit 21a. The parameters of the discriminator necessary for character recognition are stored in the storage device 30 in advance, and the character recognition unit 40 reads and uses them as appropriate. The character recognition unit 40 includes a feature extraction unit 41, an identification unit 42, and a score calculation unit 43.

  The feature extraction unit 41 extracts the feature amount of the handwritten character 5 according to a predetermined standard, and outputs the extracted feature amount to the identification unit 42. After extracting the feature amount, the feature amount may be converted into data in a format suitable for the identification method used in the identification unit 42 by performing multivariate analysis such as Fisher discriminant analysis. Thereby, the identification accuracy in the identification part 42 can be improved.

  The identification unit 42 identifies which of the characters included in the predetermined character set corresponds to the handwritten character 5 based on the feature amount input from the feature extraction unit 41, and identifies the identified result The data 1 is output to the score calculation unit 43 and the scoring unit 50. The identification method used in the identification unit 42 may be a general method, for example, a support vector machine.

  Note that the identification unit 42 may output the identification data to the score calculation unit 43 and the scoring unit 50 every time the identification data 1 is obtained. Alternatively, every time the identification data 1 is obtained, the identification data 1 is sequentially stored in the storage device 30. After the identification data 1 for all problems is obtained, the identification data 1 is read from the storage device 30 and the identification data 1 is scored. You may output to the part 43 and the scoring part 50. In other words, the input handwritten characters 5 may be processed online, or the input handwritten characters 5 may be temporarily stored in the storage device 30 so that all or part of the handwritten characters 5 are stored. Processing may be carried out in batches after they have been prepared. The same applies to processing in the score calculation unit 43 and the scoring unit 50 described later.

  The score calculation unit 43 calculates a score 2 for the identification data 1 input from the identification unit 42 and outputs the score 2 to the reliability evaluation unit 15. When a support vector machine is used for the identification unit 42 as in the above example, a value obtained by normalizing the distance from the support vector to the discrimination boundary within a predetermined range can be used as the score 2.

  The scoring unit 50 collates the identification data 1 input from the identification unit 42 with the correct answer data 8 read from the storage device 30, and outputs a correctness determination 3 a to the correctness determination processing unit 16. As described above, it is noted that the scoring unit 50 collates the first identification data 1 and the correct answer data 8 to determine the correctness determination 3a.

  The control unit 11 comprehensively controls various functions of the scoring server 100. The control unit 11 includes a reliability evaluation unit 15, a correctness / incorrectness determination processing unit 16, and a correction unit 17.

  The reliability evaluation unit 15 is based on the score 2 input from the score calculation unit 43 and the identification data 1 and the correct answer data 8 input from the scoring unit 50. The reliability 4 indicates the reliability of recognition of the handwritten character 5. And the identification data 1, the score 2, the reliability 4, and the correct answer data 8 are output to the correctness determination processing unit 16. The calculation method of the reliability 4 will be described in detail later.

  The correctness / incorrectness determination processing unit 16 displays the correctness / incorrectness determination 3a input from the scoring unit 50, the identification data 1, the score 2, the reliability 4, and the correct answer data 8 input from the reliability evaluation unit 15 in the display control unit 60a. By outputting, the result of scoring is presented to the teacher (see FIG. 6). At this time, the correctness determination processing unit 16 may cause the teacher to select whether or not to check the scoring result by displaying a selection dialog or the like.

  Further, when the correction data 9 is input from the correction unit 17, the correctness determination processing unit 16 corrects the correctness determination 3a to the correctness determination 3b according to the correction data. When the correction data 9 is not input, the correctness determination 3a is not changed and is set as the correctness determination 3b. The correctness / incorrectness determination processing unit 16 determines correctness / incorrectness when receiving a signal indicating that the teacher has finished checking the scoring result (for example, when the teacher presses a “confirmation end” button illustrated in FIG. 6). 3b is stored in the storage device 30.

  Further, the correctness determination processing unit 16 is illustrated in FIG. 8 when a signal indicating that the teacher performs correctness determination is received (for example, when the teacher presses the “correct” button illustrated in FIG. 6). The judgment correction screen to be displayed is displayed. In addition, the correctness determination processing unit 16 displays a list of similarity determination results (see FIG. 9). At this time, the correctness determination processing unit 16 may cause the teacher to select whether or not to confirm the similar determination result by displaying a selection dialog or the like as illustrated in FIG.

  The correction unit 17 generates correction data 9 for correcting the correctness determination 3 a to the correctness determination 3 b based on the correction input input from the input unit 80 a and outputs the correction data 9 to the correctness determination processing unit 16.

  The display control unit 60a converts the information output by the correctness determination processing unit 16 (for example, the scoring table illustrated in FIG. 6) into display data 7 in a format that can be displayed on the display unit 70a. Here, the display data 7 may be, for example, an image in a bitmap format, an image according to another format, or another data format suitable for display. The display control unit 60a only needs to be capable of converting the format of the information into the display data 7, and may be a general display adapter, for example.

  The display unit 70a is a device that displays video and the like. In this embodiment, a liquid crystal display (LCD) is mainly assumed. However, the type of hardware is not limited as long as the device has a display function (particularly, a flat panel display). For example, an apparatus including a display element such as a plasma display panel (PDP) or an EL (Electroluminescence) display, and a driver circuit that drives the display element based on display data 7 received from the display control unit 60a Etc.

  The input unit 80a receives a correction input from the teacher. In the present embodiment, a touch panel is mainly assumed as the input unit 80a. However, the type of hardware is not limited as long as it has an input surface capable of inputting information to the scoring server 100 by a correction input by the user (that is, the input surface is included in the input unit 80a). The input unit 80a acquires two-dimensional coordinate information on the input surface of the pointing tool that has contacted the input surface at a predetermined time interval, and outputs the series of coordinate data to the correction unit 17 as a correction input.

  In FIG. 1, the input unit 80 a and the display unit 70 a are illustrated separately in order to clearly show the functions of each configuration. However, for example, in the case where the input unit 80a is a touch panel and the display unit 70a is a liquid crystal display, it is desirable that both are configured integrally (see FIG. 3). That is, the input unit 80a includes a data input surface made of a transparent transparent member such as glass formed in a rectangular plate shape, and is integrally formed so as to cover the data display surface of the display unit 70a. Good. Thereby, since the contact position of the pointing tool with respect to the input surface of the input unit 80a matches the position of the graphic displayed on the display surface by the display unit 70a in response to the contact, the user can obtain a natural handwriting feeling. it can.

  The storage device 30 is a non-volatile storage device that stores the problem data 6 and the parameters of the identifier used in the identification unit 42. The storage device 30 can be constituted by, for example, a hard disk, a semiconductor memory, a DVD (Digital Versatile Disk), or the like. In the present embodiment, the storage device 30 is shown in FIG. 1 as a device built in the scoring server 100, but may be an external storage device that is communicably connected to the outside of the scoring server 100. .

[Configuration of answering client 110]
Based on FIG. 2, the detail of each structure of the client 110 for an answer which concerns on this Embodiment is demonstrated. As illustrated, the answering client 110 includes a communication unit 20b (transmission unit 22b, reception unit 21b), a display control unit 60b, a display unit 70b, and an input unit 80b. Similar to the scoring server 100 described above, the answering client 110 according to the present embodiment may include other omitted configurations in accordance with the actual situation.

  The communication unit 20b is a communication unit 20a, the transmission unit 22b is a transmission unit 22a, the reception unit 21b is a reception unit 21a, the display control unit 60b is a display control unit 60a, the display unit 70b is a display unit 70a, and an input. The unit 80b has the same function as the input unit 80a, and a duplicate description is omitted.

  The input unit 80b accepts input of an answer (handwritten character 5) from the student. Similarly to the input unit 80a, two-dimensional coordinate information on the input surface is acquired (see FIG. 5), and is output as the handwritten character 5 to the transmission unit 22b. Moreover, since the input part 80b outputs the handwritten character 5 also to the display control part 60b, the student can visually recognize the locus | trajectory input by handwriting in real time. As described above, it is desirable that the display unit 70b and the input unit 80b are configured integrally.

  The transmission unit 22b transmits the handwritten character 5 input from the input unit 80b to the scoring server 100. The transmission unit 22b may transmit the handwritten character to the scoring server 100 each time the handwritten character 5 is input (that is, for each answer to each question), or the input handwritten character 5 is stored in a storage device (FIG. The handwritten characters 5 may be transmitted to the scoring server 100 after the answers to all or some of the questions are obtained. In the following description, the transmission unit 22b receives a signal from the input unit 80b indicating that the student has finished answering the question (for example, when the student presses the “End Answer” button illustrated in FIG. 4). Etc.), the handwritten character 5 is transmitted to the scoring server 100.

[Calculation method of reliability 4]
Based on FIG. 7, the calculation method of the reliability 4 performed in the reliability evaluation part 15 is demonstrated. First, the reliability evaluation unit 15 normalizes the score 2 so that the score 2 falls within a predetermined value range. For example, in the example illustrated in FIG. 7A, the score 2 is normalized in the range of 0 to 100 as described above. Note that when the score 2 is already within a predetermined range when the score 2 is input from the score calculation unit 43, the normalization process described above can be omitted.

  Further, the normalization range is not limited to 0 to 100 as in the above example. That is, in order to define a predetermined threshold value (δ to be described later) indicating a relative value range for determining whether or not score 2 is close to each other, a range of possible values of score 2 is It is only necessary. Therefore, the normalization range may be appropriately changed according to the character recognition method used by the identification unit 42, the score 2 calculation method used by the score calculation unit 43, and the like.

  Next, the reliability evaluation unit 15 calculates a difference value (comparison value) between the score 2 that takes the maximum value and the score 2 that takes the second largest value. When the difference value exceeds a predetermined threshold value indicating a relative value range (that is, when the scores are not close to each other), the reliability evaluation unit 15 determines whether the result of character recognition is reliable or not. The reliability 4 is set to “high” on the assumption that the determination 3a is also reliable. This calculation is expressed as follows:

When | A1-A2 |> δ, reliability 4 = “high”
Here, A1 is a score 2 that takes the maximum value, A2 is a score 2 that takes the second largest value, and δ is a threshold value that indicates a relative value range. In the above example, the difference value between the score 2 that takes the maximum value and the score 2 that takes the second largest value is calculated, but the score 2 that takes the maximum value and the score 2 that takes the third largest value are calculated. A difference value may be calculated.

  Further, the threshold value δ may be appropriately adjusted according to the properties of the question data 6 and the students. Thereby, the scoring server 100 can check the determination result more efficiently, and can further reduce the burden of the work of checking whether there is an erroneous determination.

  On the other hand, when the difference value is below the threshold value (that is, when the scores are close to each other), the reliability evaluation unit 15 includes the identification data 1 that matches the correct data 8. It is determined whether or not. For example, in the example shown in FIG. 7B, when the correct answer data 8 is “2”, none of the identification data 1 matches the correct answer data. In this case, the reliability evaluation unit 15 sets the reliability 4 to “high”. Even if the score 2 is close to each other and the result of character recognition is unreliable, the scoring unit 50 performed if the higher-level identification data 1 does not contain data that matches the correct answer data 8 This is because the possibility that the correctness determination 3a is incorrect is considered to be small.

  On the other hand, when the above difference value is below the threshold value and the identification data 1 includes a match with the correct answer data 8, the reliability evaluation unit 15 sets the reliability 4 to “low”. To "". In this case, the correctness / incorrectness determination 3a performed by the scoring unit 50 is highly likely to be erroneous, and it is considered necessary to be confirmed by the teacher.

  When the score 2 of the first identification data 1 is smaller than a predetermined value, the reliability evaluation unit 15 sets the reliability 4 to “low” regardless of the relative relationship with the score 2 of the other identification data 1. May be set. In the above example, the reliability 4 is classified into “high” or “low” binary based on a predetermined threshold value. For example, the difference value is treated as an analog value as the reliability 4. May be. Furthermore, the calculation method of the reliability 4 is not limited to the above method. For example, the variance (standard deviation) of score 2 may be calculated, and if this value is greater than a predetermined threshold value, reliability 4 may be set to “high”.

[Processes performed by the scoring server 100 and the answering client 110]
The flow of processing executed by the scoring server 100 and the answering client 110 will be described with reference to FIGS. FIG. 11 is a flowchart showing an example of processing executed by the scoring server 100 and the answering client 110.

  The transmission unit 22a of the scoring server 100 reads the question data 6 from the storage device 30 and transmits it to the answering client 110 (step 1: hereinafter abbreviated as S1). The receiving unit 21b of the answering client 110 receives the question data 6 (S2) and outputs it to the display control unit 60b, thereby presenting the question data 6 to the student. The input unit 80b accepts handwritten input from the student (S3), and outputs the handwritten character 5 to the transmitting unit 22b. When the transmission unit 22b receives a signal indicating that the student has finished answering the question from the input unit 80b (YES in S4), the transmission unit 22b transmits the handwritten character 5 to the scoring server 100 (S5).

  The receiving unit 21a of the scoring server 100 receives the handwritten character 5 from the answering client 110 (S6) and outputs it to the feature extracting unit 41. The feature extraction unit 41 extracts the feature amount of the handwritten character 5 and outputs it to the identification unit 42 (S7). The identification unit 42 performs character recognition of the handwritten character 5 based on the feature amount (S8), and outputs the obtained identification data 1 to the score calculation unit 43 and the scoring unit 50. The score calculation unit 43 calculates a score 2 based on the identification data 1 (S9), and outputs this to the reliability evaluation unit 15. The scoring unit 50 collates the identification data 1 input from the identification unit 42 with the correct answer data 8 read from the storage device 30, and outputs the correctness determination 3a and the identification data 1 to the correctness determination processing unit 16 (S10). ).

  Based on the score 2 input from the score calculation unit 43, the identification data 1 and the correct answer data 8 input from the scoring unit 50, the reliability evaluation unit 15 sets the reliability 4 indicating the reliability of the correctness determination 3a. Calculation is performed, and the identification data 1, score 2, reliability 4, and correct answer data 8 are output to the correctness determination processing unit 16 (S11, evaluation step). The correctness determination processing unit 16 displays the correctness determination 3a input from the scoring unit 50, the identification data 1, the score 2, the reliability 4, and the correct answer data 8 input from the reliability evaluation unit 15 as a display control unit 60a. The result of scoring is presented to the teacher, and a correction input for the incorrect correctness determination 3a is accepted (S12, presentation step).

  FIG. 12 is a flowchart illustrating an example of a process for calculating the reliability 4 performed by the reliability evaluation unit 15.

  The reliability evaluation unit 15 first normalizes the score 2 (S19). Next, the reliability evaluation unit 15 calculates a difference value between the score 2 that takes the maximum value and the score 2 that takes the second largest value or the score 2 that takes the third largest value. By comparing this difference value with a predetermined threshold value, is the score of the first candidate (score 2 having the maximum value) to the score of the third candidate (score 2 having the third largest value) approximated? It is determined whether or not (S20).

  When it is determined that they are approximate (YES in S20), the reliability evaluation unit 15 determines whether or not the identification data 1 includes data that matches the correct data 8 (S21). When it is determined that the matching items are included (YES in S21), the reliability evaluation unit 15 sets the reliability 4 to “low” (S22). On the other hand, when it is determined that they are not approximate (NO in S20), or when it is determined that no matching items are included (NO in S21), the reliability evaluation unit 15 sets the reliability 4 to “high”. Set (S23).

  FIG. 13 is a flowchart illustrating an example of processing for checking and correcting the scoring result, which is executed by the correctness determination processing unit 16.

  The correctness / incorrectness determination processing unit 16 causes the teacher to select whether or not to confirm the result of scoring by the scoring server 100 (S30). When the teacher selects to confirm the scoring result (YES in S30), the correctness determination processing unit 16 displays a list of scoring results as illustrated in FIG. 6 (S31). Correct / incorrect determination processing unit 16 receives a signal indicating that the teacher has finished checking the scoring result when the teacher makes a selection that does not want to check the scoring result (NO in S30) or YES (YES in S32). The process is terminated (END).

  On the other hand, the correctness determination processing unit 16 does not receive a signal indicating that the scoring result has been confirmed (NO in S32) and receives a signal indicating that the teacher performs a correctness determination (YES in S33). The determination correction screen illustrated in FIG. 8 is displayed (S34).

  The correction unit 17 waits for correction input from the teacher (S35). When the correction unit 17 detects a correction input (YES in S35), the correction unit 17 generates correction data 9 for correcting the correctness determination 3a to the correctness determination 3b and outputs the correction data 9 to the correctness determination processing unit 16. The correctness determination processing unit 16 corrects the correctness determination 3a to the correctness determination 3b based on the correction data (S36). Then, the correctness / incorrectness determination processing unit 16 executes a correction process for the similarity determination result (S37).

  FIG. 14 is a flowchart showing an example of processing for confirming and correcting similar scoring results, which is executed by the correctness determination processing unit 16.

  The correctness / incorrectness determination processing unit 16 displays a selection dialog as illustrated in FIG. 10 (S40), and causes the teacher to select whether or not to confirm a similar determination result (S41). When an input for confirming the similar determination result is detected (YES in S41), the correctness / incorrectness determination processing unit 16 displays a list of similar determination results as illustrated in FIG. 9 (S42). The subsequent processing is the same as S32 to S36 described with reference to FIG.

  On the other hand, when an input for confirming a similar determination result is not detected (NO in S41), or when a signal indicating that the teacher has finished confirming the scoring result is received (YES in S32), the process ends. (END).

[Effects produced by scoring server 100]
The user can find an error in the correctness / incorrectness determination 3a with a high probability only by confirming a determination evaluated as having a low reliability 4. That is, the scoring server 100 can efficiently check the correctness / incorrectness determination 3a, and has an effect of reducing the burden of work for checking the presence / absence of an incorrect determination.

[Combination of configurations (technical means) included in each embodiment]
Note that the configurations included in the above-described embodiments can be combined as appropriate. In other words, all the configurations described in the above embodiments can be used in combination with all or a part of the configurations in the other embodiments as well as the embodiment according to the above description. Embodiments are also included in the technical scope of the present invention.

  In other words, the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  For example, although the character recognition unit 40 has been described as being provided in the scoring server 100, it may be provided in the answering client 110. In this case, the answering client 110 transmits the identification data 1 and the score 2 that have been recognized to the scoring server 100 instead of the handwritten character 5.

[Example of software implementation]
Finally, each block of the scoring server 100 and the answering client 110 (test system 120) may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip) or may be a CPU (Central Processing). Unit) may be implemented in software.

  In the latter case, the scoring server 100 and the answering client 110 (test system 120) include a CPU that executes program instructions for realizing each function, a ROM (Read Only Memory) that stores the program, and a RAM that expands the program. (Random Access Memory), a storage device (recording medium) such as a memory for storing the program and various data, and the like. The object of the present invention is to program the program codes (execution format program, intermediate code program, source program) of the control program of the scoring server 100 and the answering client 110 (test system 120), which are software for realizing the above-described functions. The readable recording medium is supplied to the scoring server 100 and the answering client 110 (test system 120), and the computer (or CPU or MPU) reads and executes the program code recorded on the recording medium. Can also be achieved.

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. IC cards (including memory cards) / optical cards, semiconductor memories such as mask ROM / EPROM / EEPROM / flash ROM, PLD (Programmable logic device), FPGA (Field Programmable Gate Array), etc. Logic circuits can be used.

  The scoring server 100 and the answering client 110 (test system 120) may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited as long as it can transmit the program code. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication network, and the like can be used. The transmission medium constituting the communication network may be any medium that can transmit the program code, and is not limited to a specific configuration or type. For example, even with wired lines such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, and ADSL (Asymmetric Digital Subscriber Line) line, infrared rays such as IrDA and remote control, Bluetooth (registered trademark), IEEE 802.11 wireless, HDR ( It can also be used by radio such as High Data Rate (NFC), Near Field Communication (NFC), Digital Living Network Alliance (DLNA), mobile phone network, satellite line, and digital terrestrial network. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be used for an apparatus that supports scoring of a handwritten test, and can be suitably applied to, for example, various qualification tests. It can also be applied to entertainment purposes such as games.

1 Identification data (identification candidate, first identification candidate)
2 score (evaluation value)
3a Judgment (correction result)
3b Judgment (correction result) (correction result)
4 reliability (reliability)
5 Handwritten characters (input by handwriting operation, locus of handwriting operation)
8 Correct answer data (target for verification)
15 Reliability Evaluation Department (Evaluation Method)
16 Correctness determination processing unit (presentation means)
17 Correction part (correction means)
80a input unit 80b input unit (input surface)
100 scoring server (verification device)
110 Answer client (input device)
120 test system

Claims (10)

For each of at least one identification candidate obtained by recognizing input by handwriting operation, an evaluation value that evaluates the accuracy of the recognition is calculated and collated with the first identification candidate having the largest evaluation value A collation device that determines whether or not the first identification candidate matches the collation target by collating with an object,
When the evaluation value of the first identification candidate and the evaluation value of the other identification candidate are approximate, the group consisting of the first identification candidate and the other identification candidate matches the collation target. Depending on whether or not there is an evaluation means for determining the reliability of the determination;
A collation apparatus comprising: a presentation unit that presents the reliability determined by the evaluation unit to the user together with the result of the determination.   The evaluation means approximates the evaluation value of the first identification candidate and the evaluation value of another identification candidate by comparing a comparison value calculated based on the evaluation value with a predetermined threshold value. It is discriminate | determined whether it exists, The collation apparatus of Claim 1 characterized by the above-mentioned.   The collation apparatus according to claim 2, wherein the comparison value is a difference value between the evaluation value of the first identification candidate and the second largest evaluation value or the third largest evaluation value.   The said presentation means is further provided with the correction means which receives operation which corrects the result of the said determination from a user, when showing the determination result and the reliability of the said determination result to a user. 4. The verification device according to any one of items 1 to 3.   The correction means determines the first identification candidate obtained from a handwriting operation, the collation target collated with the first identification candidate, and whether the first identification candidate matches the collation target. From the storage device that stores a plurality of combinations with the results of the above, a combination in which both the first identification candidate and the collation target match the predetermined first identification candidate and the predetermined collation target is extracted, and each of the extracted combinations The collation apparatus according to claim 4, wherein an operation for correcting a determination result is received from a user.   An input surface that can acquire a locus of handwriting operation is provided, information can be input based on the locus, and the input information is sent to the collation device according to any one of claims 1 to 5. Characteristic input device.   A test system comprising: the collation device according to claim 1; and the input device according to claim 6. For each of at least one identification candidate obtained by recognizing input by handwriting operation, an evaluation value that evaluates the accuracy of the recognition is calculated and collated with the first identification candidate having the largest evaluation value A method for controlling a collation apparatus that determines whether or not the first identification candidate matches the collation target by collating with an object,
When the evaluation value of the first identification candidate and the evaluation value of the other identification candidate are approximate, the group consisting of the first identification candidate and the other identification candidate matches the collation target. Depending on whether or not there is an evaluation step for determining the reliability of the determination;
A method for controlling a collation apparatus, comprising: presenting the reliability determined in the evaluation step to the user together with the result of the determination.   A control program for operating at least one of the collation device according to any one of claims 1 to 5 and the input device according to claim 6, wherein the control causes a computer to function as each of the means. program. A computer-readable recording medium on which the control program according to claim 9 is recorded.

Images