JP2019185754A - Descriptive test scoring program and descriptive test scoring method - Google Patents

Abstract

To provide a descriptive test scoring program and a descriptive test scoring method which improve the accuracy and reliability of grading.SOLUTION: A descriptive test scoring program causes a computer to execute: a morphological analysis step for performing morphological analysis on the text information as the criteria for scoring and the character string information to be evaluated; an alignment step for calculating the correspondence relationship between the morphemes of the character string information to be analyzed in the morphological analysis step and the morphological morphemes of the character string information as a criterion for scoring; a similarity calculation step for calculating the similarity between the character string information as the criteria for scoring and the character string information to be scored based on the correspondence relationship between the morphemes of the character string information to be evaluated and the morphological morphemes of the character string information as the criteria for scoring in the alignment step; and a scoring step for calculating the number of points of the character string information to be the criteria for scoring based on the information on the similarity calculated in the similarity calculation step.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a descriptive test scoring program and a descriptive test scoring method.

  The following non-patent document 1 seems to be the closest prior art. This paper proposes a method to reduce the cost of automatic scoring of descriptive questions scheduled to be introduced in the university entrance examination center examination in 2020. There, considering that the descriptive question is scored by a plurality of graders, pre-scored data is used as teacher data, learning as a binary classification problem of correct and incorrect answers, and as a second grader, It is a proposal that makes use of the learning results. Experiments with answer data of hundreds of people showed that both SVM and CNN (convolutional neural network) can be classified with an accuracy of nearly 90%.

Terada et al., Automatic scoring of description problems using neural networks, Proc. Of the 22nd Annual Conference of the Language Processing Society, pp.370-373 (2016)

  Assuming a university entrance examination center examination, it is possible to assume answer data of just under 550,000 from the number of examinees over the past few years. At this time, the conventional technique has the following problems.

(1) In the case of the method as the second grader, it is necessary to score the answer of less than 550,000 people once by hand, and it is still expensive.

(2) Temporarily, out of the answers of less than 550,000 people, the answers for 10,000 people were scored by hand and learned as graded data, and the results of the learning for the remaining 540,000 answers There is no guarantee of accuracy when applying.

(3) Although 90% accuracy may be high, if the number of data is 550,000, about 50,000 means that the correct answer is incorrect. This is a big problem for students and it cannot be said to be a reliable classification.

(4) Because of the binary classification of the correct answer and the incorrect answer, it is not possible to give partial points that are common to descriptive formula problems. In the above-described prior art, it is possible to change to multi-value classification so that partial points can be given, but naturally the accuracy is lowered.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a descriptive test scoring program and a descriptive test scoring method with improved scoring accuracy and reliability.

  According to one aspect of the present invention, in order to solve the above problem, a descriptive test scoring program is transmitted to a computer by using a morphological analysis for character string information serving as a scoring reference and character string information serving as a scoring target. The morpheme analysis procedure, the morpheme of the character string information to be scored analyzed by the morpheme analysis procedure, the alignment procedure to calculate the correspondence of the morpheme of the character string information to be the reference of scoring, Based on the correspondence between the morpheme of the character string information to be used and the morpheme of the character string information to be used for scoring, information on the similarity between the character string information used as the scoring reference and the character string information used as the scoring target is calculated. A scorer that calculates the score of the character string information to be scored based on the similarity calculation procedure and information on the similarity calculated by the similarity calculation procedure It was assumed to be an execution.

  Further, in the alignment procedure, the correspondence between the morphemes of the character string information to be scored and the character string information to be used as a reference for scoring is calculated so that the information on the similarity calculated by the similarity calculation means is maximized. This is desirable.

Furthermore, it is desirable to correct the character string information that serves as a scoring standard based on the answer information of the test taker.
Further, in the scoring procedure, based on whether the character string information to be scored includes a keyword, a word that is incorrect if included, or a word that is incorrect if included at the end of the sentence. It is desirable to calculate the number of character string information to be scored.
In addition, it is determined that the scoring procedure cannot calculate the score by the automatic scoring possibility judgment procedure and the automatic scoring possibility judgment procedure for judging whether the scoring procedure can calculate the score of the character string information to be scored. If you have a clustering procedure for clustering character string information that is subject to scoring, you will be able to identify key features and NG words that are common to sentences that cannot be automatically scored with the current scoring settings. This is desirable because it is easy to set manually.

  Further, according to another aspect of the present invention, a descriptive test scoring method includes a morphological analysis procedure for performing morphological analysis by a computer on character string information serving as a scoring reference and character string information serving as a scoring target; The alignment procedure for calculating the correspondence between the morpheme of the character string information to be scored analyzed by the morpheme analysis procedure and the morpheme of the character string information to be a reference for scoring, and the character string to be scored calculated by the alignment procedure Similarity in which the computer calculates information related to the similarity between the character string information that serves as the scoring standard and the character string information that serves as the scoring target, based on the correspondence between the morphemes of the information and the morphemes of the character string information that serves as the scoring standard Based on the calculation procedure and the similarity information calculated by the similarity calculation procedure, the score of the character string information to be scored Desirable Assuming and a scoring procedure for calculating by Yuta.

  According to the present invention, it is possible to provide a descriptive test scoring program and a descriptive test scoring method with improved scoring accuracy and reliability.

It is a figure which shows the outline | summary of the automatic scoring method by the example of embodiment of this invention. It is a figure which shows the example of the alignment by a present Example. It is a figure which shows the example of the scoring by a present Example. It is a figure which shows the example of the morphological analysis by a present Example. It is a figure which shows the procedure of the determination of the similarity by a present Example. It is a figure which shows the specific example of calculation of the similarity by a present Example. It is a figure which shows the example of the scoring standard of a description type test. It is a figure which shows the outline | summary of the automatic scoring method of Example 2. FIG. FIG. 10 is a diagram illustrating an example of sequence alignment in the second embodiment.

  Embodiments and examples of the present invention will be described below, but the embodiments of the present invention are not limited to the embodiment examples and examples described below.

  In the prior art, most of the proposals of the method as the second grader are other than the above-mentioned papers, but in the present invention, an automatic scoring system is assumed as the first grader.

  In most cases, scoring standards are set, and the keywords to be used and the number of characters are set for the students who are answerers. In consideration of such circumstances, scoring is performed according to the following procedure (see FIG. 1).

1. Step 1
Setting correct answers: Set the conditions and scoring standards imposed on the answerer. Examples: Mandatory keywords, character limit, conditional key phrases, etc. Conditional key phrases are short sentences that contain important words. By scoring with conditional key phrases, it is determined whether important words and their alignment, natural sentences. In addition, partial points can be given.

2. Step 2
Prepare correct and incorrect document data: Prepare a certain amount of correct and incorrect document data. Of the answers, it is not necessary to prepare all the correct document data.

3. Step 3
Based on the configuration diagram of FIG. 1, input of an answer, collation with a correct / incorrect answer sentence group, and the presence / absence of a condition key phrase are checked to see if the necessary condition of the problem is satisfied. Here, the answer that can be determined to be correct or incorrect can be automatically scored, but if it cannot be determined, it is manually scored. Although it is possible for a person to input an answer to a computer, it is preferable to recognize the student's answer sheet with character recognition software and automatically recognize it from the viewpoint of reducing labor and scoring time. .

4). Step 4
As a result of automatic scoring and manual scoring, results of answers are periodically added to correct and incorrect answer sentences. Based on these, clustering and alignment are performed and added as a new conditional key phrase.

  Hereinafter, examples of the present invention will be described together with explanations of terms.

Alignment is a technique used to determine the similarity of two or more sequences, and is a technique for obtaining an appropriate and optimal correspondence between characters. FIG. 2 is a diagram illustrating the concept of alignment.
In FIG. 3, the calculation method of the score (score) of a present Example is shown. When arranging two sequences, (1) if two characters at position p match (match), (2) if two characters at position p do not match (mismatch), (3) at position p When there is no character in the other array (gap) with respect to the character in the other array, it is determined. As a result of the determination, for example, the score is calculated as +1 point in the case of a match, −1 point in the mismatch, and −1 point in the gap. This is merely an example of scoring, and it is preferable to change the points to be added or subtracted according to the importance of the characters. It is preferable to use an algorithm that aligns so as to maximize the calculated score.

  FIG. 4 is a diagram for explaining an outline of morphological analysis which is a premise for performing alignment. The character string is divided into morphemes (the smallest meaningful unit) based on information of parts of speech (for example, pronouns, case particles, proper nouns, verbs, etc.). Then, based on the morpheme array obtained by dividing the character string into morphemes, the alignment, that is, the morpheme array constituting the character string is compared with the reference morpheme array, and the score (score) of the target character string is calculated.

  The outline of the procedure of this embodiment will be described with reference to FIG. First, a morpheme analysis is performed on a character string as a reference (reference character string) and a character string as a scoring target, and divided into morphemes (502). Alignment (for example, alignment by Needleman-Wunsch algorithm) is performed on the reference character string and the scoring target character string that have been subjected to morphological analysis (503). Then, the aligned reference target character string and the scoring target character string are compared, similarity is determined, and a score (score) is calculated (504). The reference character strings may be aligned so as to maximize the score.

  A specific example in which the score for the character string to be scored is calculated is shown in FIG. Assume that the reference character string is “I am going to Tokyo now” and the scoring target character string is “I am going to Tokyo” (601). The reference character string was decomposed into morphemes, “I am going to Tokyo now”, the character string to be scored was divided into morphemes, and “I am going to Tokyo” (602). When the reference character string and the scoring target character string decomposed into morphemes are aligned based on the part of speech of the morphemes that compose them, the reference character string is “I will go to Tokyo from now on”, the scoring character The column reads “I am going to Tokyo-” (603). Here, roughly speaking, the forms constituting the scoring target character string are rearranged so as to correspond to the part of speech of the morphemes constituting the reference character string. Here, “-” means that there is no morpheme of morpheme type (for example, part of speech) constituting the reference character string in the scoring target character string. Conversely, “-” is also displayed when the morpheme of the morpheme type constituting the scoring target character string does not exist in the reference target character string. When comparing the reference character string and the scoring target character string that are aligned in this way, the reference target character string and the scoring target character string indicate that six of the seven words match and the similarity is 85.7%. I understand. For example, this similarity can be used as the score (score) of the character string to be scored as it is.

  FIG. 7 is an example of a descriptive problem of the university entrance examination center test published on May 16, 2017. In this example, “an example of correct answer” is “to be maintained as a tourist resource by the guidelines for protecting the landscape and used as a tourism resource.” This correct example may be used as a reference character string.

  Moreover, there is a condition for the number of characters such as “what is written within 40 characters” as the “correct answer condition”, and whether or not this character number condition is satisfied may be reflected in the final score calculation. Further, other “correct answer conditions” may be reflected in the final score calculation.

The effects of the present embodiment are as follows.
(1) Can be graded as the first grader.
(2) From the scoring flow, initially, the automatic scoring ratio is expected to be about 30% by simulation, but as the scoring progresses, the condition key phrases will be expanded and the automatic scoring ratio will be 100% I know.
(3) About an automatically graded answer, a highly accurate scoring result can be maintained. In the simulation, both correct and incorrect answers were 100%.
(4) Partial points can be set in the description problem by conditional key phrases and alignment.

1. Automatic scoring system For the proposed automatic scoring system, Section 1.1 describes the system configuration and Section 1.2 and later describe the scoring algorithm.

1.1 System configuration Figure 8 shows the system configuration. Automatic scoring is performed based on scoring settings set in advance by hand. If the correct / incorrect answer cannot be determined by the automatic scoring part, the system shifts to manual scoring. The digitized character string data is input, and the correct / incorrect binary is output as the result. When using data with scoring results as test data, support for improving scoring settings is performed by clustering the answers around manual scoring and visualizing the features.
The following items are set as scoring settings.
・ Character limit (If it is not within the range, it will be considered incorrect)
・ Pre-replacement word (replacement word before scoring to unify notation)
・ NG words (words that are incorrect if included)
NG end word (words that are incorrect if included at the end of the sentence)

1.2 Primary scoring Based on the scoring settings, check whether all the conditions to be satisfied are satisfied. If even one is not satisfied, it is determined that the answer is incorrect and the scoring is terminated. If all are satisfied, the process proceeds to key phrase comparison processing. In the comparison with the scored answer, the scored answer data accumulated based on the result of the incorrect answer is compared with the answer data to be newly scored. Exit. The scored answer is added to the scored answer along with the score.

1.3 Key phrase comparison
The answer sentence and key phrase are compared by the sequence alignment algorithm called Needleman-Wunsch Algorithm. This algorithm obtains the optimal sequence by rearranging the sequences using a score function so that the number of matching characters is maximized and the number of mismatches is minimized in the two sequences. By performing sequence alignment after performing morphological analysis on the answer sentence and each key phrase to be graded, it is possible to extract a common part / non-common part between the answer sentence and the key phrase. This is shown in FIG. This calculates the matching rate between the answer sentence and the key phrase by paying attention to the sequence of words, selects the most similar sentence from several set key phrases, and extracts non-common parts. A comparison can be made.
The extracted non-common part is corrected for notation fluctuation and synonym substitution. In simple character string comparison, even if the same word is used, scoring cannot be performed correctly if the notation is different, such as kanji and hiragana. In addition, words such as “people” and “people” having the same meaning cannot be scored correctly because their notations are different. Therefore, in this system, scoring independent of the notation method is performed by judging whether or not they have the same meaning by paying attention to the reading method and the semantic category of the word.

1.4 Support for improvement of scoring settings If there is test data with scoring results and scoring settings can be improved, the answers to manual scoring are clustered after scoring the test data. Since the features common to sentences that cannot be automatically scored with the current scoring settings are clarified, it is easy to manually set valid key phrases and NG words.

2. Experiments For experiments using the proposed automatic scoring system, section 2.1 describes the experimental settings and section 2.2 describes the results.

2.1 Setting up the experiment The graded answer questions for three subjects for junior high school students will be scored for about 1,200 students each. Thereafter, the scoring setting is improved and the scoring is performed again. The characteristics of the problem are as follows.
・ Language filling within 20 characters / Society Filling within 25 characters / Science Unlimited number of characters Free description format As an evaluation index, calculate the automatic scoring rate and scoring accuracy. The automatic scoring rate represents the proportion of answers that have been subject to automatic scoring in this system, and the scoring accuracy represents the proportion of answers that have been scored correctly among the answers that have been automatically scored.

2.2 Experimental results Tables 1 and 2 show the experimental results.
From Tables 1 and 2, it can be seen that scoring is possible with very high accuracy for all problems. It can also be seen that the automatic scoring rate and scoring accuracy are improved by improving the scoring settings. If you look at each subject, you can see that science has the best results. This is probably because science is close to the problem of answering words, so the range of expression is small and scoring is easy. There is a clear answer, and it is thought that the automatic scoring rate and scoring accuracy will improve as the range of expression becomes smaller due to the limitation on the number of characters and the designation of words used.

3. Conclusion In this example, we proposed a system for automatic scoring of Japanese-style descriptive questions against the background of the introduction of descriptive questions to the university entrance common test. As a result of conducting experiments on the answer data of about 1,200 students in each of the three subjects of the Japanese language social science, it was possible to automatically score about 60% of the answers with an accuracy close to 100%, thus reducing the cost of scoring I was able to confirm that they were connected. Compared to the conventional automatic scoring method, it is considered that there is an advantage in that automatic scoring can be performed with high accuracy without requiring data with scoring results.
As future prospects, it is necessary to verify how effective it is in sentences with a higher degree of freedom and to improve the automatic scoring rate while maintaining 100% scoring accuracy.

  The present invention is industrially applicable as a descriptive test scoring program and a descriptive test scoring method.

501 Setting of scoring target sentence and scoring reference sentence 502 Morphological analysis 503 Alignment 504 Similarity determination 601 Example of scoring target sentence and scoring reference sentence 602 Example of morphological analysis 603 Example of alignment 604 Example of similarity

Claims (6)

On the computer,
Morphological analysis procedure for performing morphological analysis on character string information that is a scoring standard and character string information that is a target of scoring,
An alignment procedure for calculating the correspondence between the morpheme of the character string information to be scored and the morpheme of the character string information to be a reference for scoring analyzed by the morpheme analysis procedure;
Based on the correspondence between the morpheme of the character string information to be scored and the morpheme of the character string information to be a reference for scoring calculated by the alignment procedure, the character string information to be the reference for scoring and the character string to be the target of scoring A similarity calculation procedure for calculating information related to information similarity;
A descriptive test scoring program for executing a scoring procedure for calculating the score of character string information to be scored based on information on similarity calculated by the similarity calculating procedure.   In the alignment procedure, the correspondence between the morphemes of the character string information to be scored and the character string information to be the reference of the scoring is calculated so that the information on the similarity calculated by the similarity calculation means is maximized. The descriptive test scoring program according to claim 1.   The descriptive test scoring program according to claim 1, wherein the character string information serving as a reference for the scoring is corrected based on answer information of a test taker of the descriptive test. A morpheme analysis procedure for performing morphological analysis by a computer on character string information as a reference for scoring and character string information to be scored;
An alignment procedure for calculating the correspondence between the morpheme of the character string information to be scored and the morpheme of the character string information to be a reference for scoring analyzed by the morpheme analysis procedure;
Based on the correspondence between the morpheme of the character string information to be scored and the morpheme of the character string information to be a reference for scoring calculated by the alignment procedure, the character string information to be the reference for scoring and the character string to be the target of scoring Similarity calculation procedure for calculating information related to information similarity by the computer;
A descriptive test scoring method comprising: a scoring procedure for calculating a score of character string information to be scored by the computer based on information on similarity calculated by the similarity calculating procedure.   In the scoring procedure, based on whether a keyword, a word that is incorrect if included, or a word that is incorrect if included at the end of the sentence are included in the character string information to be scored The descriptive test scoring program according to claim 1, wherein the score of character string information to be scored is calculated. An automatic scoring possibility determination procedure for determining whether or not the score of the character string information to be scored can be calculated by the scoring procedure;
The description according to claim 1, further comprising: a clustering procedure for clustering character string information that is a target of the scoring determined that the scoring procedure cannot calculate the score by the automatic scoring possibility judging procedure. Formula exam scoring program.