JP7524712B2 - Information processing device, control method, and program - Google Patents

Description

The present invention relates to scoring support technology.

Technology that supports the grading of students' test answers has been known for some time. For example, Patent Document 1 discloses a grading system that acquires handwriting information written by students in the answer section of an answer sheet, displays it on the screen of a terminal used by the teacher, and accepts input for a correct/incorrect judgment. Patent Document 2 discloses technology that compares answers written with an electronic pen with the correct answer and judges whether the answer is correct or incorrect.

JP 2014-134854 A JP 2009-3227 A

Marking support is being implemented in which answer sheets with marking marks (marking marks) handwritten by graders to indicate correctness and incorrectness are scanned, and the marks are identified to automatically calculate scores. In this case, a classifier used to identify the marking marks can be obtained by performing machine learning using a large amount of training data including handwritten patterns of marking marks by multiple people and their correct answers. However, classifiers trained using training data of handwritten patterns by an unspecified number of people can be subject to misrecognition due to differences in handwriting habits and ink smudges, and ultimately require manual review, which is a problem of considerable effort.

The present disclosure has been made to solve the above problems, and the main objective of the present disclosure is to provide an information processing device, control method, and program that can easily check and correct errors in scoring.

In one aspect of the present invention, an information processing device has an acquisition unit that acquires answer information representing the answer for each question written on an answer sheet and marking information representing marking marks that are marks for marking the answers, an identification unit that identifies a category to which each of the marking marks belongs based on the similarity with each category, and a display unit that displays, based on the answer information and the marking information, an answer image representing the answer and the marking mark for the answer, and the identification results of the marking marks identified by the identification unit, in a list as an identification result confirmation screen. According to this aspect, a grader can easily check visually whether or not there is a misrecognition of the marking marks.

In one aspect of the information processing device, the display unit displays, for each question, a list of the answer image corresponding to the question and the recognition result. According to this aspect, by looking at the list display, the grader can easily check whether there are any misrecognitions of the marking marks for the multiple questions that make up the answer sheet.

In one aspect of the information processing device, the acquisition unit acquires multiple pieces of answer information by multiple answerers and multiple pieces of marking information, and the display unit displays a list of the answer images corresponding to a specific question and the identification results for the multiple answerers. According to this aspect, the information processing device can display a list of, for example, the answer images of all the answerers in a class corresponding to a specific question and their identification results. Therefore, by looking at the list display, the grader can easily check whether there are any misrecognitions of marking marks for specific questions by multiple answerers.

In one aspect of the information processing device, the display unit further displays a correction screen when a request to correct the identification result is obtained on the identification result confirmation screen, and the information processing device further has a correction unit that corrects the identification result based on input information on the correction screen. According to this aspect, when the grader confirms an erroneous recognition of a marking mark from the list displayed on the identification result confirmation screen, the grader can easily correct the erroneous recognition using the correction screen that is further displayed.

In one aspect of the information processing device, the classification includes a correct answer mark indicating a correct answer, an incorrect answer mark indicating an incorrect answer, and an intermediate mark between correct and incorrect answers for which partial points are awarded, the correction request can be obtained for each answer, and the display unit displays a number for specifying the partial points on the correction screen. According to this aspect, the grader can easily correct misrecognition of partial points using the number displayed on the correction screen.

In one aspect of the information processing device, the display unit displays only numbers smaller than the allocated score for the question as the numbers for specifying the partial score. According to this aspect, the information processing device can prevent the grader from making a mistake of specifying a partial score that is greater than the allocated score.

In one aspect of the information processing device, the display unit displays, as the numbers for specifying the partial score, numbers smaller than the score allocated to the question in a selectable state, and numbers equal to or greater than the score allocated to the question in an unselectable state. According to this aspect, the information processing device prevents the grader from selecting a number greater than the score allocated, thereby preventing mistakes such as specifying a partial score greater than the score allocated.

In one aspect of the information processing device, the display unit also displays an answer image on which the marking marks have been entered on the identification result confirmation screen, and when the correction request is obtained for an answer for which the identification unit was unable to identify the marking marks, a predetermined range including the answer is highlighted in the answer image, and when a marking mark outside the range is specified, the marking mark is associated with the answer, and the identification result of the marking mark by the identification unit is displayed as the identification result corresponding to the answer. According to this aspect, even if the information processing device, for example, writes a marking mark away from the answer and is unable to identify the marking mark, the marking mark can be easily associated with the answer by the grader's specification.

In another aspect of the information processing device, the identification unit identifies the classification to which each of the marking marks belongs based on a classifier trained using multiple combinations of images of marking marks handwritten by multiple people and ground truth data related to the classification of the marking marks as training data, and the classifier is trained using images of marking marks corresponding to the correction request and the classification result corrected by the correction unit as the ground truth data as the training data. According to this aspect, the information processing device is able to perform highly accurate classification using a classifier that reflects the correction content.

In another aspect of the present invention, a control method executed by an information processing device acquires answer information representing the answer for each question entered on an answer sheet and marking information representing marking marks that are marks for marking the answers, identifies the classification to which each of the marking marks belongs based on the similarity with each classification, and based on the answer information and the marking information, displays an answer image representing the answer and the marking mark for the answer, and the recognition result of the marking mark, in a list as a recognition result confirmation screen. By executing this control method, the information processing device can display the recognition result confirmation screen that allows the grader to easily check whether there is any misrecognition of the marking marks.

In yet another aspect of the present invention, a computer is made to function as an acquisition unit that acquires answer information representing an answer written on an answer sheet and marking information representing marking marks that are marks for marking the answer, an identification unit that identifies a category to which each of the marking marks belongs based on the similarity with each category, and a display unit that displays, based on the answer information and the marking information, an answer image representing the answer and the marking mark for the answer, and the identification results of the marking marks identified by the identification unit, in a list as an identification result confirmation screen. By installing this program on a computer and executing it, an information processing device according to the present invention can be configured.

The information processing device according to the present invention makes it easy to check and correct misidentification of marking marks.

1 shows a schematic configuration of a scoring system. 1 is an example of a marked answer sheet. FIG. 2 is a block diagram showing a functional configuration of an information processing device. 13 is a flowchart showing an outline of a display process of a recognition result confirmation screen. FIG. 3 is a diagram showing the result of labeling some of the marking marks entered on the marked answer sheet shown in FIG. 2 . FIG. 13 is a diagram showing marking marks and partial marks associated with the marking marks. 13 is a display example of a recognition result confirmation screen. 13 is a display example of a correction screen for correcting a recognition error. 13 is a configuration example of a scoring system according to a modified example.

Below, we will explain a preferred embodiment of the present invention with reference to the drawings.

[Scoring system configuration]
FIG. 1 is a schematic diagram of a marking system according to the present embodiment. The marking system according to the present embodiment includes an information processing device 1 used by a test scorer such as a teacher, a scanner (optical device) 2, and a marked answer sheet 6 on which the grader has put marks indicating correctness or incorrectness (also called "marking marks") on each answer column of the answered answer sheet. In this embodiment, as an example, the marking marks are classified into a correct answer mark indicating a correct answer, an incorrect answer mark indicating an incorrect answer, and an intermediate mark indicating that a partial score is given between a correct answer and an incorrect answer. The marking system accurately identifies the marking marks written on the marked answer sheet 6 to automatically calculate the score of the answerer.

The information processing device 1 is electrically connected to the scanner 2 via a wired or wireless connection, and when the scanner 2 performs a reading process on the marked answer sheet 6, it receives an image of the marked answer sheet 6 scanned (also called the "marked answer image Im") from the scanner 2. The information processing device 1 then extracts marking marks from the marked answer image Im, and identifies the classification of the extracted marking marks based on the similarity with each classification of marking marks. The information processing device 1 then displays a screen showing the marking mark classification results (also called the "classification result confirmation screen").

Figure 2 is an example of a marked answer sheet 6. As shown in Figure 2, the marked answer sheet 6 has answer columns for writing answers to each question, in addition to columns for writing the student number and name of the student taking the test and questions. Each answer column on the marked answer sheet 6 also contains the answer of the test taker to the corresponding question, and is marked with a mark by the grader.

Note that while FIG. 2 shows an example of an answer sheet for arithmetic, the marked answer sheet 6 in this embodiment may be an answer sheet for writing answers for any subject other than arithmetic. Also, the marked answer sheet 6 may be made up of multiple sheets.

[Configuration of information processing device]
3 is a block diagram showing the functional configuration of the information processing device 1. Functionally, the information processing device 1 includes a display unit 11, an input unit 12, a storage unit 13, an interface 14, and a control unit 15. These elements are connected to each other via a bus line 10.

The display unit 11 displays various information such as a recognition result confirmation screen under the control of the control unit 15. The display unit 11 may be, for example, a display or projector, or an interface for connecting to these devices. The input unit 12 generates information to be input to the information processing device 1. The input unit 12 may be, for example, a mouse, a button, a touch panel, a camera, a voice input device such as a microphone, or an interface for connecting to these devices. The interface 14 is an interface for connecting to the scanner 2 via a wired or wireless connection, and supplies the marked answer sheet image Im received from the scanner 2 to the control unit 15.

The storage unit 13 is composed of various types of memory such as RAM, ROM, and readable/writable non-volatile memory. The storage unit 13 stores programs executed by the control unit 15. The storage unit 13 also functions as a working memory when the control unit 15 executes various processes.

The storage unit 13 also stores classifier information 16, which is information about a classifier for identifying the marking marks and the numbers associated with the marking marks. Here, the numbers associated with the marking marks are, for example, numbers that represent partial scores (also called "partial scores") given as scores when intermediate marks are made. The above-mentioned classifier is a learning model that is trained to output a classification result regarding the classification of the marking marks and the identification of the partial scores when an image showing a marking mark or a partial score is input. This learning model may be, for example, any learning model used in machine learning such as a support vector machine or a neural network. The classifier information 16 is information obtained by learning the above-mentioned learning model. When the above-mentioned learning model is a neural network such as a convolutional neural network, the classifier information 16 includes various parameter information such as a layer structure, a neuron structure of each layer, the number of filters and filter size in each layer, and the weight of each element of each filter.

Here, a supplementary explanation is given regarding the learning of the classifier. The classifier is trained using training data that is, for example, a combination of images of handwritten markings by multiple people and correct answer data indicating the correct classification of the markings. This training data includes a sufficient number of the above combinations for each classification to be identified by the classifier. Furthermore, when there are multiple patterns of markings that fall into the same classification, the classifier is trained using training data that includes combinations of the markings of each expected pattern and their correct answer data. For example, when there are people who write "x" as an incorrect answer mark and people who write "v" (check mark), the classifier is trained using training data that includes combinations of the markings of each pattern and their correct answer data. The training of the classifier may be performed by the information processing device 1, or may be executed by a device other than the information processing device 1.

Furthermore, the memory unit 13 may store information necessary for calculating the score of the answerer of the marked answer sheet 6 scanned as the marked answer sheet image Im. For example, the memory unit 13 may store, for each test conducted, point allocation information for each question to which a mark is to be added and information indicating the arrangement of each question on the answer sheet. The memory unit 13 may also store the calculation rules for partial points for intermediate marks as rule information. The calculation rules for partial points can be arbitrarily set for each question, for example, "a specified ratio of the points allocated," "rounding up to the nearest whole number," "flat 2 points," etc.

The memory unit 13 also stores information necessary for extracting the marking marks, such as color information of the marking marks written on the marked answer sheet 6.

The storage unit 13 may be an external storage device such as a hard disk connected to or built into the information processing device 1, or may be a storage medium such as a flash memory. In another example, the storage unit 13 may be a server device that performs data communication with the information processing device 1. In this case, the storage unit 13 may be composed of multiple server devices.

The control unit 15 is configured with a processor (computer) such as a CPU and a GPU, and controls each component of the information processing device 1. The control unit 15 performs a predetermined process based on a program stored in the storage unit 13. The control unit 15 is an example of an "acquisition unit," an "identification unit," a "display unit," a "correction unit," and a "computer" that executes a program.

In this embodiment, when the control unit 15 receives the marked answer image Im from the scanner 2, it identifies the classification of each of the marking marks included in the marked answer image Im based on the classifier information 16, and adds a label indicating the identified classification to each marking mark. Furthermore, the control unit 15 identifies the partial score associated with the marking mark to which the intermediate mark label has been added, and associates it with the intermediate mark. Then, the control unit 15 generates display information for an identification result confirmation screen that lists images for each question cut out from the marked answer image Im and the identification results of the marking marks included in the images, and causes the display unit 11 to display the identification result confirmation screen.

[Display process of the identification result confirmation screen]
Next, a process related to the display of the identification result confirmation screen will be described.

(1) Overview
4 is a flowchart showing an outline of a display process of the identification result confirmation screen. The information processing device 1 repeatedly executes the process of the flowchart shown in FIG.

First, the information processing device 1 determines whether or not the marked answer image Im has been acquired from the scanner 2 (step S11). If the information processing device 1 determines that the marked answer image Im has been acquired from the scanner 2 (step S11; Yes), it cuts out an image for each question (hereinafter also referred to as a "question image") from the acquired marked answer image Im based on the information indicating the arrangement of each question stored in the memory unit 13 (step S12). The question image includes, for example, an answer column, an answer handwritten by the answerer, a marking mark handwritten by the grader, etc., and is associated with a question number. Note that the question numbers are expressed as "1-1" for main question 1 and "1-2" for main question 1 (2).

On the other hand, if the information processing device 1 has not acquired the marked answer image Im from the scanner 2 (step S11; Yes), it continues to determine whether or not the marked answer image Im has been acquired from the scanner 2. Note that instead of proceeding to step S12 immediately after acquiring the marked answer image Im from the scanner 2, the information processing device 1 may proceed to step S12 after acquiring the marked answer image Im from the scanner 2 and then detecting a user input regarding a display request for the recognition result confirmation screen. Furthermore, the information processing device 1 extracts the marking marks from the acquired marked answer image Im (step S13). This extraction method will be described later.

Next, the information processing device 1 identifies the classification of each of the marking marks extracted in step S13, and assigns a label indicating the identified classification to each marking mark (step S14). In this case, the information processing device 1 inputs an image of the marking mark to a classifier configured based on the classifier information 16, thereby obtaining a label indicating the classification of the marking mark, and associates the obtained label with the target marking mark. In this case, the label is information indicating, for example, a correct mark, an incorrect mark, or an intermediate mark for which partial points are awarded.

Next, the information processing device 1 determines whether or not a partial score has been entered associated with the mark that has been given a label indicating an intermediate mark (step S15). If a partial score has been entered (step S15; Yes), the information processing device 1 identifies the partial score and associates it with the target mark as metadata (step S16). A specific example of a method for identifying a partial score will be described later. On the other hand, if a partial score has not been entered (step S15; No), the information processing device 1 proceeds to step S17.

Then, the information processing device 1 displays an identification result confirmation screen including a list of the question images cut out in step S12 and the identification results of the marking marks arranged by question (step S17). Here, the identification results of the marking marks specifically include a correct answer mark "○", an incorrect answer mark "×", an intermediate mark "△", a specified partial score, etc. An example of this display will be described later with reference to FIG. 7.

(2) Extraction of marking marks
Here, the process of extracting the marking marks from the marked answer sheet image Im, which is executed in step S13 in FIG. 4, will be described.

In the first example, the information processing device 1 extracts marking marks based on color information (pixel values) of each pixel of the marked answer sheet image Im. For example, in this case, the information processing device 1 pre-stores information on the color of the pen (e.g., red) used by the grader to check the marking marks, extracts pixels having color information similar to the stored color, and recognizes adjacent pixels among the extracted pixels as pixels that form the same marking mark. Then, the information processing device 1 cuts out a rectangular area that includes each group of pixels that form the same marking mark, and generates a mask image of the marking mark (an image that displays only the marking mark) from the cut-out image.

In the second example, the information processing device 1 pre-stores a scanned image of the answered answer sheet before the grader checks the marking marks, and calculates the difference in pixel values for each corresponding pixel between the scanned image and the marked answer sheet image Im acquired in step S11. The information processing device 1 then recognizes adjacent pixels among the pixels for which a difference value exists as pixels that form the same marking mark. After that, the information processing device 1 cuts out a rectangular area that includes each group of pixels that form the same marking mark, and generates a mask image of the marking mark from the cut-out image.

(3) Labeling
Next, a specific example of label assignment executed in step S14 of FIG. 4 will be described.

Figure 5 is a diagram showing the result of labeling some of the marking marks entered on the marked answer sheet 6 shown in Figure 2. In Figure 5, of the marking marks entered on the marked answer sheet 6, marking marks 21 to 25 labeled with a correct mark "O" are shown in the upper row, marking marks 31 and 32 labeled with an intermediate mark "△" are shown in the middle row, and marking mark 41 labeled with an incorrect mark "X" is shown in the lower row. In addition, in Figure 5, the question number to which each marking mark is attached is shown for easy understanding of the correspondence with the marked answer sheet 6 shown in Figure 2.

When the information processing device 1 acquires the marked answer sheet image Im shown in FIG. 2 from the scanner 2, it extracts all the marking marks in the marked answer sheet image Im including the marking marks shown in FIG. 5 based on step S13 in FIG. 4. Then, the information processing device 1 inputs the extracted images of the marking marks (mask images) to a classifier configured based on the classifier information 16. In this case, the information processing device 1 preferably normalizes the image of the marking marks to be input to the classifier by enlarging or reducing it so that the vertical and horizontal sizes are uniform, and inputs the normalized image of the marking marks to the classifier. Then, the information processing device 1 assigns a label to the marking marks input to the classifier based on the classification result output by the classifier. Note that, when the classifier is a learning model based on a neural network, the output of the classifier includes confidence information indicating the likelihood of the output classification result.

The information processing device 1 also recognizes the correspondence between the marking marks and the questions from the positions of the marking marks on the marked answer sheet image Im based on information indicating the position of the answer column for each question in the marked answer sheet 6. The question number of the question is associated with the marking mark as metadata. Note that the information indicating the position of the answer column for each question is stored in advance, for example, in the memory unit 13, as information indicating the layout of each question.

(4) Identifying partial points
Next, a specific example of specifying partial points executed in steps S15 and S16 of FIG. 4 will be described.

Figure 6 shows a mark 32 with a label indicating an intermediate mark and partial marks 45 to 48 associated with the mark 32. Although the method of writing partial marks varies depending on the grader, they are written in numbers around a triangle, which is an intermediate mark. Specifically, as shown in FIG. 6(A), partial mark 45 is written inside the mark 32, partial mark 46 is written on the outer upper right of the mark 32, partial mark 47 is written on the outer upper left of the mark 32, and partial mark 48 is written on the outer lower part of the mark 32. In other words, partial marks are considered to be written within a predetermined range from the mark 32, and more specifically, as shown in FIG. 6(B), inside a dashed frame 53, which is a circle of a predetermined radius 52 from the center of gravity 51 of the triangle represented by the mark 32. Here, the triangle is a handwritten symbol, and is not a figure strictly surrounded by three straight lines. The center of gravity of a triangle is the approximate point where the lines connecting each vertex and the midpoint of each opposite side intersect in the handwritten symbol of the triangle.

According to this, when the information processing device 1 assigns a label indicating an intermediate mark, it identifies the target mark and cuts out a group of pixels within a circle of a specified radius from the center of gravity of the mark from the marked answer sheet image Im. The information processing device 1 generates a mask image of the partial marks from the cut-out image. Furthermore, the information processing device 1 inputs the generated mask image of the partial marks to a classifier configured based on the classifier information 16. Then, based on the classification result output by the classifier, the information processing device 1 identifies the number indicated by the partial marks input to the classifier.

Note that partial points are not always entered when entering intermediate marks, so if a mask image of the partial point cannot be generated or if the number indicated by the partial point cannot be identified, the information processing device 1 determines that no partial point has been entered.

(5) Display examples
Fig. 7 is a display example of a recognition result confirmation screen. In Fig. 7, the information processing device 1 displays a marked answer sheet image Im on the recognition result confirmation screen, and also displays a list of question images cut out from the marked answer sheet image Im and the recognition results for each extracted mark, arranged for each question.

Specifically, the information processing device 1 displays a list of question images and the recognition results for each question number associated with the question image and the recognition result, respectively. The recognition result for each extracted mark is indicated by "○", "△", or "×". If the mark mark cannot be extracted or recognized, the information processing device 1 displays the recognition result as "-". In this way, the recognition result confirmation screen displays a list of question images cut out from the marked answer sheet image Im and the recognition results of the mark marks contained in the question images, for each question number. This allows the grader to easily visually check for misrecognition of mark marks.

In addition, if partial points are specified for intermediate marks, the information processing device 1 displays the partial points together with "△" as the identification result. This allows the grader to check not only the marking marks but also the identification result of the partial points entered. Note that, if partial points are not specified for intermediate marks, the information processing device 1 calculates partial points for the points allocated to the question to which the intermediate mark is assigned based on the rule information stored in the memory unit 13, and displays the partial points together with the identification result "△". In this way, since the information processing device 1 automatically calculates the partial points, it becomes possible for the grader to accurately calculate the total score, etc., even if the grader does not understand the partial point calculation rules for intermediate marks for each question.

Furthermore, the information processing device 1 displays the answerer's score (64 points in this case) in a total score column 61 on the identification result confirmation screen based on the point allocation information for the target test and the identification result of the mark. Note that the answerer's score is not limited to the total score for the entire test, and the total score for each section may be calculated and displayed. For example, on the identification result confirmation screen, the information processing device 1 displays the total score for (1) to (5) of main question 1 in a section score column 62, and the total score for (1) and (2) of main question 2 in a section score column 63.

In this case, first, the information processing device 1 acquires the point allocation information of the target test from the storage unit 13, etc., based on a user input specifying the test corresponding to the marked answer sheet image Im. The information processing device 1 may identify the target test using a known image recognition process (e.g., template matching, etc.) based on the arrangement of the answer columns included in the marked answer sheet image Im. The information processing device 1 then calculates the answerer's score based on the identification result of the marking marks for each question and the point allocation for each question. Here, the identification result of the marking marks also includes partial points associated with intermediate marks. If the identification result of the marking marks is changed, the information processing device 1 re-tallys the scores based on the identification result after the change.

(6) Correction of misconceptions
FIG. 8 is a display example of a correction screen for correcting a recognition error. When a recognition error of a marking mark is confirmed on the recognition result confirmation screen, the grader can make a correction. For example, on the recognition result recognition screen shown in FIG. 7, when the recognition result of question number 1-1 is the correct answer mark "○" but the actual mark is the intermediate mark "△", the grader transmits a correction request by clicking or tapping on the recognition result item 66. Upon receiving the correction request, the information processing device 1 displays a pull-down menu 71 for correcting the recognition result, as shown in FIG. 8 (A). The grader can easily correct the recognition error of the recognition result by specifying the correct marking.

When the grader selects the intermediate mark "△" in the pull-down menu 71, the information processing device 1 further displays a pull-down menu 72 for specifying partial points. At this time, the pull-down menu 72 does not display or does not allow selection of numbers greater than the allocated point for the target question. This makes it possible to prevent the grader from making a mistake by selecting a number greater than the allocated point when selecting the partial point associated with the intermediate mark. When no partial point is specified, for example, a hyphen "-" may be selected, or no selection may be made and a click or tap on any other arbitrary point is considered to be the absence of partial point specification.

The method of correcting the misrecognition of the identification result is not limited to the method of displaying a pull-down menu as a correction screen as described above, but may also display a software keyboard 73 as a correction screen as shown in FIG. 8 (B). In this case, when the grader clicks or taps on the misrecognized identification result item, the information processing device 1 displays the software keyboard 73. The grader can easily correct the misrecognition of the identification result by selecting the correct mark. Furthermore, the numbers on the software keyboard 73 are used to select partial points, but numbers above the allocated points are inactive. Note that when the grader selects partial points, the information processing device 1 may automatically specify an intermediate mark as the mark.

When a misrecognition is corrected on the recognition result confirmation screen, the information processing device 1 can perform more accurate recognition by reflecting the correction content in the classifier. For example, an image of a marking mark that corresponds to a correction request, i.e., that corrects the recognition result, and the corrected recognition result as corrective data regarding the classification of the marking mark may be combined and used as learning data for the classifier to learn.

(7) Associating marks and questions outside the scope
The information processing device 1 recognizes the correspondence between the marking marks and the questions from the positions of the marking marks in the marked answer image Im based on the information indicating the positions of the answer columns for each question in the marked answer sheet 6. Therefore, for example, as shown in FIG. 7, when the grader writes the marking mark 26 in a position away from the answer column 67 of the question number 1-5, the information processing device 1 cannot extract or identify the marking mark 26 as the marking mark corresponding to the question number 1-5. Therefore, in the identification result confirmation screen, "-" is displayed in the identification result item of the question number 1-5. In such a case, the grader clicks or taps the identification result item 68. Then, the information processing device 1 displays a dashed frame 69 in the answer column of the question number 1-5 in the marked answer image Im displayed on the left side of the identification result confirmation screen. Then, when the grader specifies the marking mark 26 corresponding to the question number 1-5, the information processing device 1 associates the question number 1-5 with the marking mark 26. At this time, the information processing device 1 emphasizes the dashed frame 69 and the marking marks 26 by adding colors, so that the grader can easily recognize the associated questions and marking marks. Then, the information processing device 1 identifies the associated marking marks 26, and reflects the identification results in the identification result items 68 of the questions No. 1-5.

(8) Extracting features for each grader
The information processing device 1 acquires N-dimensional (N is an integer equal to or greater than 1) features (i.e., feature vectors) by applying an arbitrary feature extraction algorithm to the image of each mark. This feature extraction algorithm may be, for example, an arbitrary algorithm that extracts one or more feature amounts related to shape. The feature extraction algorithm may also be a feature extractor obtained by learning a machine learning learning model such as a neural network. In this case, the information processing device 1 stores information on parameters of the feature extractor obtained by learning in advance in the storage unit 13. This enables the information processing device 1 to perform highly accurate identification based on the feature vectors when identifying marking marks.

Furthermore, the information processing device 1 may extract features from the image of each marking mark for each grader. In this case, it is assumed that the image of each marking mark is associated with grader identification information that identifies the grader. For example, when the information processing device 1 acquires the marked answer sheet image Im from the scanner 2 in step S11 or when it acquires a display request for the identification result confirmation screen, it may also acquire the grader identification information and associate it with the marking mark as metadata. In addition, the grader identification information and the answer sheet to be graded may be stored in advance in association with each other in the storage unit 13, and the grader identification information may be identified from the answer sheet on which the marking marks are written and associated with the marking mark as metadata. According to this, the information processing device 1 can extract features from the image of each marking mark for each grader based on the grader identification information. In addition, the information processing device 1 can create learning data that is a combination of images of handwritten marking marks and correct answer data regarding the classification of the marking marks, i.e., the identification results, for each grader. By learning the features extracted by the classifier and the created training data, the information processing device 1 can perform classification taking into account the handwriting habits of each grader, etc., and can perform classification with higher accuracy.

Specifically, as described in the section "(4) Identifying partial marks," the way in which partial marks are written in conjunction with intermediate marks varies depending on the grader, but the same grader will often have a set position on the triangle that is the intermediate mark where the partial mark will be written. For example, a grader who writes partial marks inside the triangle that is the intermediate mark is unlikely to write partial marks outside the triangle. The information processing device 1 obtains a feature vector for each grader by applying any feature extraction algorithm to images of the marking marks and the partial marks associated therewith written by the same grader. This allows the information processing device 1 to perform identification based on the feature vector for each grader, taking into account the handwriting habits of each grader, such as where in the triangle that is the intermediate mark the partial mark will be written, enabling highly accurate identification.

In addition, in the above example, when identifying partial points, a group of pixels within a circular range of a specified radius from the center of gravity of the target mark is cut out, and a mask image of the partial point is generated from the cut-out image. However, by using the feature vector for each grader as a basis, the information processing device 1 can narrow the search and cut-out range of the pixel group, such as "inside the target mark," enabling more rapid identification.

[Modification]
Next, preferred modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

(Variation 1)
The information processing device 1 may function as a server device that transmits display data relating to the identification result confirmation screen to the terminal.

Figure 9 shows an example of the configuration of a marking system according to a modified example. In the example of Figure 9, the information processing device 1A is connected to a display terminal 7 used by a marker via a communication network 9. The display terminal 7 acquires a marked answer sheet image Im from the scanner 2 and transmits the acquired marked answer sheet image Im to the information processing device 1A.

When the information processing device 1A receives a marked answer image Im from the display terminal 7, it executes the process of the flowchart shown in FIG. 4. In this case, in the display process of the recognition result confirmation screen in step S19, the information processing device 1A generates display data "Dd" for the recognition result confirmation screen and transmits the generated display data Dd to the display terminal 7, thereby displaying the recognition result confirmation screen on the display terminal 7. In this manner, in this modified example, the information processing device 1A functions as a server device that transmits display data Dd for the recognition result confirmation screen. Even in this case, the information processing device 1A displays the recognition result confirmation screen on the display terminal 7 used by the grader, and can preferably present the recognition results of the marking marks and accept corrections.

(Variation 2)
In step S12 of Fig. 4, the information processing device 1 may cut out question images from each of a plurality of marked answer images Im obtained by scanning a plurality of marked answer sheets 6 (for example, for the entire class) on which the same grader has applied marking marks. In addition, in step S13, the information processing device 1 may extract marking marks from each of the marked answer images Im. In this case, the information processing device 1 performs the processes of steps S12, S13 and the following steps on the question images cut out from the plurality of marked answer images Im and the extracted plurality of marking marks. Even in this case, the information processing device 1 can display the marked answer images Im on the recognition result confirmation screen, and can display a list of the question images cut out from the marked answer images Im and the recognition results for each extracted marking mark arranged for each question. Furthermore, the information processing device 1 can display a list of question images and recognition results for each solver for one question number, instead of displaying a list of question images and recognition results for each question number for multiple questions that make up the marked answer image Im. For example, the information processing device 1 can display a list of question images and recognition results for all students in a class for question number 1-1.

(Variation 3)
In this embodiment, the answer sheet is a paper medium, the answerer writes the answer with a pencil or the like, and the grader marks the marking with a red pen or the like. However, the present invention is not limited to this, and the answer sheet may be displayed on a tablet or the like, the answerer writes the answer with a stylus pen or the like, and the grader also marks the marking with a stylus pen or the like. Here, the tablet or the like refers to a plate-shaped computer terminal or hardware. Also, the stylus pen or the like refers to a pen-shaped input device used on the tablet or the like.

In this case, there is no need for the scanner 2 to perform a reading process on the marked answer sheet 6 in step S11 of FIG. 4, and the information processing device 1 acquires information on the strokes of the answer entered by the answerer with a stylus pen or the like as answer stroke information, and acquires information on the strokes of the marking marks entered by the grader with a stylus pen or the like as marking stroke information. Note that the information on the strokes includes position information of the strokes on the tablet where they were entered and the time of entry. For example, the information processing device 1 acquires answer stroke information and marking stroke information from information on the strokes entered on the tablet or the like, based on the time of entry.

Then, in step S12, the information processing device 1 generates an image of the answer corresponding to each question based on the information indicating the position of the answer column for each question in the answer sheet and the answer stroke information. The information processing device also generates an image of a marking mark corresponding to each question based on the information indicating the position of the answer column for each question in the answer sheet and the marking stroke information. The information processing device then generates a question image by combining the image of the answer corresponding to each question with the image of the marking mark. Note that the information indicating the position of the answer column is stored in advance in, for example, the memory unit 13. Also, in this case, unlike when the scanner 2 performs the reading process, the question image does not include an answer column.

Furthermore, in step S14, the information processing device 1 identifies the classification of each of the marking marks based on the generated image of the marking marks, and assigns a label indicating the identified classification to each marking mark. In this case, the information processing device 1 performs the processes from step S14 onward on the generated question image and the marking marks.

Reference Signs List 1: Information processing device 2: Scanner 6: Marked answer sheet 11: Display unit 12: Input unit 13: Memory unit 14: Interface 15: Control unit

Claims (11)

an acquisition unit that acquires answer information that indicates an answer for each question written on the answer sheet and marking information that indicates a mark for marking the answer;
an identification unit that identifies a class to which each of the marking marks belongs based on a similarity to each class;
a display unit that displays, based on the answer information and the marking information, an answer image representing the answer and the marking mark for the answer, and an identification result of the marking mark identified by the identification unit, in a list form as an identification result confirmation screen;
An information processing device having the above configuration. The information processing device according to claim 1, characterized in that the display unit displays, for each question, a list of the answer image corresponding to the question and the recognition result. The acquisition unit acquires a plurality of pieces of answer information by a plurality of solvers and a plurality of pieces of scoring information,
2 . The information processing apparatus according to claim 1 , wherein the display unit displays a list of the answer images corresponding to a specific question and the identification results for a plurality of solvers. When a request to correct the identification result is received on the identification result confirmation screen, the display unit further displays a correction screen,
The information processing device includes:
4. The information processing apparatus according to claim 1, further comprising a correction unit that corrects the identification result based on information input on the correction screen. The classification includes a correct mark indicating a correct answer, an incorrect mark indicating an incorrect answer, and an intermediate mark between the correct answer and the incorrect answer and for which partial points are given;
The correction request may be obtained for each answer,
5. The information processing apparatus according to claim 4, wherein said display unit displays numbers for designating said partial points on said correction screen. The information processing device according to claim 5, wherein the display unit displays only numbers smaller than the points allocated to the question as the numbers for specifying the partial points. The information processing device according to claim 5, wherein the display unit displays, as the numbers for specifying the partial score, numbers smaller than the score allocated to the question in a selectable state, and numbers equal to or greater than the score allocated to the question in a non-selectable state. The display unit is
an answer sheet image, which is an image of an answer sheet on which the mark is written, is also displayed on the identification result confirmation screen;
When the correction request is received for an answer for which the identification unit is unable to identify the mark, a predetermined range including the answer is highlighted in the answer sheet image;
An information processing device as described in any one of claims 4 to 7, characterized in that when a marking mark outside the range is specified, the marking mark is associated with the answer, and the identification result of the marking mark by the identification unit is displayed as an identification result corresponding to the answer. The identification unit is
The method identifies the classification to which each of the marking marks belongs based on a classifier trained using a plurality of combinations of images of handwritten markings by a plurality of people and correct answer data related to the classification of the marking marks as training data,
The information processing device according to any one of claims 4 to 8, characterized in that the classifier is trained using, as the learning data, an image of a mark corresponding to the correction request and the classification result corrected by the correction unit as the corrected data. A control method executed by an information processing device, comprising:
Acquire answer information representing the answers to each question written on the answer sheet and marking information representing marking marks for marking the answers;
identifying a classification to which each of the markings belongs based on a degree of similarity to each classification;
A control method in which, based on the answer information and the marking information, an answer image representing the answer and the marking mark for the answer, and an identification result of the marking mark are displayed side by side as a list as an identification result confirmation screen. an acquisition unit that acquires answer information that indicates an answer for each question written on the answer sheet and marking information that indicates a mark for marking the answer;
an identification unit that identifies a class to which each of the marking marks belongs based on a similarity to each class;
A program that causes a computer to function as a display unit that displays, based on the answer information and the marking information, an answer image representing the answer and the marking mark for the answer, along with the recognition result of the marking mark recognized by the recognition unit, in a list form as a recognition result confirmation screen.