JP7281905B2 - Document evaluation device, document evaluation method and program - Google Patents

Description

The present invention relates to a document evaluation device and a document evaluation method.

As background art in this technical field, techniques for assisting scoring of descriptive questions, improving scoring efficiency, and auto-scoring techniques are known. For example, in Patent Document 1, "Using a computer, first classify the test taker's answer sentences into two layers according to grammatical rules. Next, verify whether the first layer and the second layer are effectively connected.Finally, the number of required words and phrases and the condition of the limited number of characters are scored, and there are 4 items including basic sentence patterns and auxiliary words. automatically calculate the score by summing the scores of the sentences.” An automatic sentence scoring system is described.

JP 2017-129830 A

In grading descriptive questions, there are various grading standards for each question, and grading must be performed according to the grading standards. In automatic grading, a method of preparing graded answer text data as teacher data for each grading standard and constructing an automatic grading device using machine learning is conceivable. However, there is a problem that preparing training data for each individual scoring standard requires a lot of cost.

In addition, a rule-based method is conceivable in which scoring rules are generated based on grammar such as parsing results for each scoring criterion, and automatic scoring is performed. However, rule generation based on parsing results may require expert knowledge of parsing results. There was a problem.

Furthermore, even if the answers to the same question are written answers, there are various answers, and automatic grading is difficult. In order to efficiently divide the scoring between manual and automatic scoring, such as manually scoring things that are difficult to automatically score, there is a problem that it is necessary to judge not only the automatic scoring results but also their reliability. rice field.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to determine the reliability of automatic scoring based on a scoring pattern and automatic scoring.

The present invention is a document evaluation apparatus comprising a processor for executing a program stored in a memory and a computer having a memory for storing data used by the processor when executing the program, wherein the processor is configured to execute the program The document evaluation device functions as the following functional units by executing , and the text evaluation device includes a syntactic analysis unit that analyzes morphemes and dependency relationships from the input answer sentence for generating a scoring pattern, and a syntax analysis unit that analyzes 3 an information extraction unit that extracts a ternary relation; and a machine learning model that learns the relationship between the ternary relation, the syntactic analysis result, and additional information of the ternary relation, such as tense, modality, and date and time, as training data, an information representation type estimating unit for estimating additional information of the ternary relation from the analysis result of the syntactic analysis unit and the ternary relation extracted by the information extracting unit; and a mutual relationship between the syntactic analysis result and the ternary relation. an inter-information relation estimating unit for estimating the mutual relation of the ternary relation extracted by the information extracting unit from the analysis result of the syntactic analysis unit using a machine learning model trained using relationships as teacher data ; A hierarchical clustering unit that classifies into hierarchical clusters, a cluster sorting unit that sorts the clusters based on the size of the clusters, and a grading pattern generation unit that selects grading conditions suitable for the read grading results as grading patterns. and wherein the hierarchical clustering unit classifies the answer sentences into hierarchical clusters according to the similarity of a feature matrix including a value obtained by multiplying the appearance ratio of a word in the answer by the importance of the word. , the grading pattern generation unit replaces the words or phrases included in each term of the ternary relation with a higher concept using a semantic category dictionary in which superordinate and subordinate relations between words or phrases are recorded in advance, and abstracts them. and the ternary relation extracted by the information extracting unit, the degree of abstraction of each term of the ternary relation, the additional information of the ternary relation analyzed by the information representation type estimating unit, and the inter-information relation Among the conditions that combine the mutual relationships of the three-term relationships estimated by the estimation unit, an answer sentence with a high degree of matching of the scoring results is selected as a scoring pattern, and based on the sorting result of the cluster sorting unit, a predetermined A cluster containing an answer sentence equal to or greater than the threshold is determined as a majority answer sentence, a cluster containing an answer sentence less than the threshold is determined as a minority answer sentence, and the reliability of the scoring pattern generated from the majority answer sentence is evaluated. It is set high, and the reliability of the grading pattern generated from the minority answer is set low .

Therefore, the present invention can provide a method of generating scoring patterns for judging correct and incorrect answers for each scoring criterion and performing automatic scoring based on the scoring patterns. In addition, by classifying the answers into majority answers and minority answers, by classifying answers into easy automatic scoring (majority answers) and difficult automatic scoring (minority answers), It becomes possible to determine the reliability of automatic scoring.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows Example 1 of this invention and shows an example of a structure of a document evaluation apparatus. It is a flowchart which shows Example 1 of this invention and shows an example of the process performed by a document evaluation apparatus. It is a figure which shows Example 1 of this invention and shows an example of parsing. It is a figure which shows Example 1 of this invention and shows an example of a parsing result. It is a figure which shows Example 1 of this invention and shows an example of information extraction. It is a figure which shows Example 1 of this invention and shows an example which abstracted from information extraction. FIG. 10 shows Example 1 of the present invention and shows an example after sorting the results of hierarchical clustering of answer sentences; It is a figure which shows Example 1 of this invention and shows an example of the answer sentence after sorting. It is a figure which shows Example 1 of this invention and shows the example before sorting of hierarchical clustering. It is a figure which shows Example 1 of this invention and shows an example of the answer sentence before sorting. It is a figure which shows Example 1 of this invention and shows an example of answer text grading data. It is a figure which shows Example 1 of this invention and shows an example of scoring conditions. It is a figure which shows Example 1 of this invention and shows the example which expresses an answer sentence by a feature vector. It is a figure which shows Example 1 of this invention and shows an example of a question sentence table. It is a figure which shows Example 1 of this invention and shows an example of an answer sentence table. It is a figure which shows Example 1 of this invention and shows an example of the scoring result table of an answer sentence. FIG. 10 is a diagram showing Embodiment 1 of the present invention and showing an example of a tree structure of upper-lower order relationships in a semantic category dictionary; FIG. 2 shows Example 1 of the present invention and shows an example of a semantic category dictionary; It is a flowchart which shows Example 2 of this invention and shows an example of the process performed by a document evaluation apparatus. It is a figure which shows Example 4 of this invention and shows an example of an answer data input screen. It is a figure which shows Example 4 of this invention and shows an example of graded answer data. FIG. 13 is a diagram showing Example 4 of the present invention and showing an example of ungraded answer data. It is a figure which shows Example 4 of this invention and shows an example of a scoring pattern display screen. It is a figure which shows Example 4 of this invention and shows an example of a scoring pattern edit screen. It is a figure which shows Example 4 of this invention and shows an example of a scoring screen.

Embodiments of the present invention will be described below with reference to the accompanying drawings.

Unlike mark sheet-type tests, written tests require manual grading, which poses problems such as the need for a large number of raters and the number of days required for grading. The document evaluation device of this embodiment supports this scoring.

In the following, the embodiment will be described using English test questions as an example, but the same method can be applied to other languages as well. In practice, processing is performed using a large number of answer sentences, but an example is shown below in order to simply explain each processing method.

FIG. 1 is a block diagram showing a configuration example of the document evaluation apparatus 101 of the first embodiment. The document evaluation apparatus 101 of the first embodiment is composed of a computer including an input device 102, a display device 103, a communication device 104, an arithmetic device 105, a memory 106, and an auxiliary storage device 107, for example.

The input device 102 receives inputs such as commands from the user. For example, it is a keyboard, a mouse, a touch panel, or the like. The input device 102 receives inputs such as commands executed for controlling programs executed by the arithmetic device 105 and controlling devices connected to the document evaluation device 101 .

The display device 103 is configured by a display or the like. The communication device 104, for example, according to a predetermined protocol, transmits the processing content of the document evaluation device 101 to the external device and receives information from the external device.

Arithmetic device 105 includes a processor and executes a program stored in memory 106 . The memory 106 includes ROM, which is a nonvolatile storage element, and RAM, which is a nonvolatile storage element. The ROM stores immutable programs (eg, BIOS) and the like. RAM is a high-speed and volatile storage element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the arithmetic unit 105 and data used when the programs are executed.

The auxiliary storage device 107 is, for example, a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). Store. That is, the program is read from the auxiliary storage device 107, loaded into the memory 106, and executed by the arithmetic device 105. FIG.

The program executed by the arithmetic device 105 is provided to the document evaluation device 101 via removable media (CD-ROM, flash memory, etc.) or a network, and stored in the non-volatile auxiliary storage device 107, which is a non-temporary storage medium. be. Therefore, the document evaluation apparatus 101 preferably has an interface for reading data from removable media.

The document evaluation apparatus 101 is a computer system configured on one physical computer, or on a plurality of computers configured logically or physically, and is processed by separate threads on the same computer. Alternatively, it may be processed on a virtual computer constructed on a plurality of physical computer resources.

The computing device 105 functions as a scoring pattern generation processing unit 201 and a scoring processing unit 202 . For example, the arithmetic unit 105 functions as the grading pattern generation processing unit 201 by executing processing according to the grading pattern generation processing program loaded into the memory 106, and performs processing according to the grading program loaded into the memory 106. , and functions as the scoring processing unit 202 . The same applies to other units included in the arithmetic unit 105 .

The grading pattern generation processing unit 201 generates a grading pattern 215 that serves as a criterion for scoring, using the answer text and its grading data. Also, the grading processing unit 202 uses the grading pattern 215 generated by the grading pattern generation processing unit 201 to score the answer sentence.

The auxiliary storage device 107 holds a grading pattern generating answer text 203 , answer text grading data 213 , a semantic category dictionary 214 , a grading pattern 215 , a grading target answer text 217 and a grading result 218 .

The grading pattern generation answer text 203 is a database in which an answer text is recorded for each question, and includes a question text table 901 and an answer text table 902 used for grading pattern generation. FIG. 9A is a diagram showing an example of a question sentence table 901 that stores question sentences. The question sentence table 901 stores question IDs 9011 and question sentences 9012 .

FIG. 9B is a diagram showing an example of the answer sentence table 902. As shown in FIG. The answer sentence table 902 stores a question sentence ID 9021 for identifying a question sentence, an answer sentence ID 9022 for identifying an answer sentence, and an answer sentence 9023 .

The question sentence ID 9021 of the answer sentence table 902 and the question sentence ID 9011 of the question sentence table 901 correspond to each other. In the second row of the answer sentence table 902, an answer sentence with a question ID of "1" and an answer sentence ID of "2" is stored.

In the examples of the question sentence table 901 and the answer sentence table 902 shown in FIGS. 9A and 9B, both the question sentence and the answer sentence consist of one sentence, but the question sentence and the answer sentence consist of multiple sentences. Sometimes it is. In the first embodiment, an example of one sentence will be described for the sake of simplicity, but the same processing is applied to the case of multiple sentences.

The answer grading data 213 records the grading results for the answer of the grading pattern generation answer 203 . This is the result of grading the answers in advance by a person or a computer.

FIG. 9C is a diagram showing an example of the scoring result table 903. As shown in FIG. The grading result table 903 includes a question ID 9031, an answer ID 9032, a criterion A 9033 storing the grading result according to the grading criterion A, and a criterion A 9034 storing the grading result according to the grading criterion B in one record. A single criterion or a plurality of criteria may be used for grading the answers.

FIG. 9C shows a scoring result table 903 based on the standard A 9033 in the third column and a scoring result table 903 based on the standard B 9034 in the fourth column for each answer. Scoring criteria include, for example, tenses being used correctly, conjugations being used correctly, no spelling errors, answers making sense to questions, articles being correct, and the like.

In addition, scoring may be performed in several stages according to the scoring criteria. For example, the reference A 9033 in FIG. 9C has two stages of 0 and 1 points, and the reference B 9034 has three stages of 0 to 2 points. For example, the higher the number, the higher the score.

In the semantic category dictionary 214 of FIG. 1, hierarchical relationships between entities representing entities and concepts such as objects, organizations, people, place names, and occupations are recorded. FIG. 10A is a diagram showing an example of a tree structure of hierarchical relationships in the semantic category dictionary 214. FIG. FIG. 10B is a diagram showing an example of the semantic category dictionary 214. As shown in FIG.

Nodes 1003 to 1001 in FIG. 10A show hierarchical relationships in a tree structure for concepts related to occupation. The semantic category dictionary 214 in FIG. 10B is obtained by tabulating this. In this way, the concepts and the hierarchical relationship between the concepts are recorded in a table.

The grading pattern 215 in FIG. 1 is obtained by patterning and recording the grading criteria from the viewpoint of language processing, which will be described later in detail.

The grading target answer text 217 in FIG. 1 is a record of the grading target answer text, and the format is the same as in FIGS. 9A and 9B. However, while the answer text recorded in the grading pattern generation answer text 203 is used for the purpose of generating the grading pattern 215, the grading target answer text 217 is graded by the document evaluation apparatus 101 of the first embodiment. It is a record of the answer sentences to be tested.

The grading result 218 records the grading result for the answer text of the grading target answer text 217, and has the same format as in FIG. 9C. However, these are the results of scoring using the document evaluation apparatus 101 of the first embodiment.

In the first embodiment, the information used by the document evaluation apparatus 101 may be represented by any data structure such as a table or list without depending on the data structure.

Note that the document evaluation apparatus 101 may include any of the input device 102, the display device 103, and the communication device 104 that have input and output means. If the document evaluation apparatus 101 does not include the input device 102, for example, the communication device 104 receives inputs such as commands from an external device. When the document evaluation device 101 does not include the display device 103, for example, the communication device 104 transmits the processing result (display screen information) generated by the document evaluation device 101 to the external device.

Each processing unit may perform input/output to other processing units via the memory 106 or the auxiliary storage device 107 . For example, when the module A included in the scoring pattern generation process 201 passes the processing result to another module B, the processing result of the module A is temporarily stored in the memory 106 or the auxiliary storage device 107, and the module B stores the processing result in the memory 106. Alternatively, the output result stored in the auxiliary storage device 107 may be obtained as an input.

FIG. 2 is a flow chart showing an example of processing performed by the document evaluation apparatus 101 according to the first embodiment. The processing executed by the document evaluation apparatus 101 is roughly divided into two parts, grading pattern generation processing 201 and grading processing 202 .

In the following description, it is assumed that a grading question for a question sentence 9012 whose question sentence ID 9011 is "1" shown in FIG. 9A is assumed. If there are multiple question sentences, the same process should be performed for each question sentence.

First, an example of processing performed in the scoring pattern generation processing 201 will be described. In basic analysis 204 , syntax analysis 205 and information extraction 206 apply syntax analysis processing and information extraction processing to the answer sentence of answer sentence 203 for grading pattern generation. The processing of syntax analysis 205 and information extraction 206 will be described below.

The syntactic analysis 205 applies syntactic analysis such as morphological analysis and dependency analysis to each sentence of each answer sentence included in the answer sentence 203 for generating a marking pattern. Note that the morphological analysis and dependency analysis may be performed using known or well-known techniques, and will not be described in detail in this embodiment.

FIG. 3A shows an example of the results of morphological analysis and dependency analysis when the answer sentence 9023 is the sentence "I wanted to become a very kind nurse." FIG. 3B is a diagram showing an example of the syntax analysis result 301. As shown in FIG.

The syntactic analysis result 301 includes an ID 3011 that stores a number identifying each morpheme in the answer sentence 9023, a token 3012 that stores the surface expression of each morpheme, and a part of speech of each morpheme (the symbol is the POS tag definition of Penn Treebank, etc.). 3013, a dependency receiver 3014 indicating the ID of the token of the dependency receiver, a dependency type 3015 indicating the type of dependency relationship, and a position 3016 indicating the character position in the answer sentence 9023 of each token and are included in one record.

In the first embodiment, an example of the dependency type 3015 is shown according to the method of type definition of Stanford Core NLP. For the position, for example, it is preferable to use the start point and end point positions of the token in units of bytes from the beginning of the answer sentence 9023 . Note that anaphora analysis, etc., may be performed to analyze the entity pointed to by the referent, such as "it" and "they".

The information extraction 206 extracts the information of the ternary relation from each sentence of each answer sentence 9023 included in the answer sentence 203 for grading pattern generation.

FIG. 3C is a diagram illustrating an example of information extraction. In the illustrated example, the information extraction 206 extracts a ternary relation 303 consisting of three elements "I", "become", and "very kind nurse" from the answer sentence 302="I wanted to become a very kind nurse." are extracting. In this ternary relation 303, for example, the first term (“I”) and the last term (“very kind nurse”) are terms representing entities such as concepts, and the middle term (“become”) is Represents relationships between entities. Information extraction 206 then extracts, for example, three terms consisting of two entities and relations as a ternary relation.

A position 303 indicates the position of a character from the beginning of the sentence corresponding to each item in the answer sentence 302. For example, the position is expressed in a data format using byte-unit positions as shown in FIG. 3C. It should be noted that the expression of each term does not necessarily have to be expressed by the words in the original sentence, but may be expressed by another symbol representing the entity or relationship.

For example, "very kind nurse" may be abstracted and expressed as "medical personnel". In this case, the information (I, become, medical personnel) is extracted from this example sentence. In this ternary relation 303, for example, as in the above example, there are many cases where the first item is the subject, the second item is the predicate, and the third item is the object and complement. Anything that expresses the relationship between

For the information extraction 206 of the basic analysis 204, for example, a classifier that recognizes and extracts entities in the answer text 302 and a classifier that recognizes relationships between entities may be constructed and used by machine learning. In this case, it is necessary to generate teacher data for learning in advance. Alternatively, a method of generating a rule for extracting the ternary relation 303 using the result of syntactic analysis may be used. Alternatively, a publicly known or well-known technology such as OpenIE may be applied.

Also, in the information extraction 206, a plurality of ternary relations may be extracted from one answer sentence 302 in some cases. For example, if the answer sentence 302 is "I wanted to become a very kind nurse when I was little." There are cases such as extracting. If the ternary relation includes demonstrative terms such as "it" and "they", it is recommended to replace them with the entities indicated by them through anaphora analysis.

The results of the basic analysis 204 are stored in the memory 106, the auxiliary storage device 107, or the like. Next, high-order analysis 207 using the results of basic analysis 204 will be described. The high-order analysis 207 includes two processes of information phenotype estimation 208 and information relationship estimation 209 .

In information phenotype estimation 208, the results of syntactic analysis 205 and information extraction 206 are used to determine the tense of the ternary relation 303 extracted in information extraction 206 and the phenotype (additional information for the ternary relation) such as modality and date/time. to parse

FIG. 4 is a diagram showing an example of abstraction from information extraction. In the illustrated example, the ternary relation (information extraction result 402) extracted from the answer sentence 401 by the information extraction 206, and the information phenotype estimation result 403 (additional information) by the information phenotype estimation 208 of the higher-level analysis 207. An example of analyzing the tense is shown. Note that the three-term relationship (information extraction result 402) extracted by the information extraction 206 in the figure is shown in bold.

Next, referring to the parsing result 301 of FIG. 3A, the two items of the ternary relation (the part indicating the relation between entities such as predicate) "become" modifies "wanted" with the dependency type "xcomp". Therefore, "wanted" is identified as a VBD whose part of speech expresses the past tense from FIG. 3A.

From this, it is specified that the ternary relation (I, become, very kind nurse) (303 in FIG. 3) is expressed in the past tense. In this way, by using the syntactic analysis result 301 in the information phenotype estimation 208 of the high-level analysis 207, it is possible to estimate the tense, modality, date and time, etc. of the ternary relation (303) extracted in the information extraction 206. . This estimation method is determined in advance as a rule based on a ternary relation and a part of speech, a dependency relation, a dependency type, or the like, or by machine learning.

As another example of additional information for such a ternary relation, for example, if it is "I went to the park with him", the ternary relation (I, went to, the park) is extracted and parsed as follows: It is possible to analyze information such as whether the ternary relation was performed under the condition of "with him". Also, if it is "I went to the park yesterday", it can be analyzed that (I, went to, the park) is an event of "yesterday".

As described above, in the information phenotype estimation 208 of the high-level analysis 207, rules for estimating necessary additional information and machine learning methods are set in advance based on the ternary relation, part of speech, dependency relation, and dependency type. Keep

Next, in inter-information relationship estimation 209 of high-order analysis 207, mutual relationships among the three-term relationships extracted in information extraction 206 are estimated. For example, assume that two ternary relations (I, become, very kind nurse) and (I, was, little) are extracted from "I wanted to become a very kind nurse when I was little." At this time, it is specified from the syntactic analysis result that the two ternary relations are connected by when. Relations between such ternary relations are determined in advance by rules based on syntactic analysis results or by machine learning.

Based on the information phenotype estimation result estimated in the information phenotype estimation 208, it is determined that the answer 302 (I, become, very kind nurse) is expressed in the past tense, and the inter-information relationship estimation 209 determines that the time is Assume that (I, was, little). As in the above example, the inter-information relationship estimation 209 estimates a predefined relationship between ternary relationships based on rules, machine learning, or the like. The results of these basic analysis 204 and high-order analysis 207 are used in scoring pattern generation 212 .

Next, hierarchical clustering 210 will be described. Here, the answers are hierarchically classified based on the characteristics of the answer sentences, such as the meaning and grammatical characteristics of the answer sentences. For this purpose, first, each answer sentence of the answer sentences for grading pattern generation 203 is expressed as a feature vector.

For example, TF-IDF can be used as a method for using feature vectors. FIG. 8 is a diagram showing an example of expressing an answer using a feature vector. First, the hierarchical clustering 210 extracts phrases such as words and phrases from the entire answer sentences of the scoring pattern generation answer sentences 203 .

Here, in the hierarchical clustering 210, frequently appearing words and phrases such as "a" and "the" that are not useful for characterizing the answer sentence are generated in advance and deleted from the target of processing. do. Next, the hierarchical clustering 210 calculates the ratio of each word or phrase in the answer sentences to be processed, and calculates a frequency matrix 801 as shown in FIG.

The frequency matrix 801 includes a field for storing the appearance ratio for each answer sentence and a field for word importance 802 in one entry, and each entry indicates a word/phrase (or word l) extracted from each answer sentence. For example, the first column of the frequency matrix 801 indicates that in the answer sentence 1, the words 2 and 3 appear at a rate of 0.2, and the word 5 appears at a rate of 0.6.

In addition, hierarchical clustering 210 computes and stores each term importance in term importance 802 . In the illustrated example, a word with a lower appearance frequency is considered to have a higher degree of importance in characterizing the answer sentence, and is expressed as a logarithm of (number of all answer sentences)/(number of answer sentences in which the word/phrase appears). .

For example, if the number of all answer sentences in the illustrated frequency matrix 801 is 5, word 1 appears in two answer sentences, answer sentence 2 and answer sentence 4, so the word importance 802 is log(5/ 2). As an example of another method, when it is known in advance that words and phrases indicating occupations are likely to appear in answers to question sentences, as in answer sentences 302, the words and phrases indicating occupations can be used. A method such as increasing the importance may be adopted. As a result, the answer text can be characterized based on the contents of the question text.

Hierarchical clustering 210 then multiplies each column of frequency matrix 801 by the corresponding term importance 802 to produce feature matrix 803 as shown. Each column of feature matrix 803 is a feature vector that characterizes answer sentences 1-5. For example, the first column of feature matrix 803 indicates the feature vector of answer sentence 1 .

Hierarchical clustering 210 performs hierarchical categorization (clustering) based on the feature vectors of the feature matrix 803. For hierarchical clustering, a publicly known or well-known technique may be applied. Hierarchical clustering 210 can hierarchically cluster the answer sentences based on the similarity based on the feature vector by the above process.

Next, in cluster sorting 211 shown in FIG. 2, the clusters are sorted based on the number of answer sentences belonging to each cluster for the answer sentences categorized by the hierarchical clustering 210 .

FIG. 5A is a diagram showing an example of the output of cluster sorting. FIG. 5A shows an example of the output structure of cluster sort 211 for clusters C1-C10 of FIG. 6A. Although a large number of answer sentence groups are used in practice, a small number of answer sentence groups are schematically illustrated here in order to simply explain the content of the processing.

FIG. 5A shows the hierarchical structure of the answer sentence group 501 in a tree structure, and clusters are subdivided from the left branch to the right branch. The answer sentences belonging to the terminal clusters are surrounded by dotted-line rectangles. In FIG. 5A, the terminal clusters are given identifiers C1 to C10 for the sake of explanation. As for the upper clusters, for example, an identifier of C23 is assigned to a cluster in the upper hierarchy combining the clusters C2 and C3.

Clusters on the upper side of FIG. 5A are sorted so that the number of answer sentences belonging to them tends to be large, and the number of answer sentences tends to be smaller for clusters on the lower side. In the example of FIG. 5A, clusters C1-C2 contain three answer sentences, clusters C3-C6 contain two answer sentences, and clusters C7-C10 contain one answer sentence.

Depending on the hierarchical structure, the cluster on the upper side (on the side where the X value of cluster CX is small) is not necessarily larger than the cluster on the lower side (on the side where the value of X of cluster CX is large). Examples of sorting methods are described later.

By performing such sorting, majority clusters (for example, C1 to C7) have many majority answers, and minority clusters (for example, C8 to C10) have minority answers and individual answers. more answers. When distinguishing between a majority cluster and a minority cluster, the determination may be determined by a predetermined rule based on the number of elements (answer sentences) included in the cluster. For example, the majority cluster is determined in order from the upper cluster so that the number of answer sentences at a ratio equal to or greater than a predetermined threshold is included in the majority cluster. For example, when the threshold is 15/18, the total number of answer sentences belonging to clusters C1 to C10 is 18, and the number of answer sentences in clusters C1 to C7 is 15, which is greater than or equal to the threshold. is the majority cluster, and clusters C8 to C10 are the minority clusters. When classifying answers into majority answers and minority answers, each answer is judged by whether it belongs to the majority cluster or the minority cluster.

The upper (majority answer) cluster is easier to pattern and generate scoring patterns. Therefore, the reliability of automatic scoring for answer sentences belonging to the majority answer cluster is high, and automatic scoring tends to be easy. On the other hand, the lower cluster (minority answers) has many unique answers that are difficult to pattern, and the reliability of automatic scoring is low. In addition, in the lower cluster, in addition to unique answers, answers with spelling errors and grammatical errors frequently appear.

By performing the sorting as described above, it is possible to extract scoring patterns from the upper cluster, separate majority answers from unique answers, and estimate the reliability of automatic scoring. The reliability is set so that the answer text belonging to the upper (majority answer) cluster is larger than the answer text belonging to the lower (minority answer) cluster.

For example, the reliability of an answer sentence may be determined by the ratio of the number of answer sentences belonging to clusters below the cluster to which the answer sentence belongs. In the cluster sort 211, for example, the reliability of answer sentences belonging to cluster C3 in FIG. Since the total number of answer sentences is 18, it is set to 10/18.

In this way, when the cluster sort 211 sets the reliability to the cluster C of the answer sentences, the answer sentences with high reliability tend to be automatically scored easily, and the answer sentences with low reliability tend to be difficult to automatically score. In other words, it can be expected that an answer sentence with a high degree of reliability is highly accurate in automatic scoring. On the other hand, answers with low reliability tend to include grammatical and spelling errors, use complex content and difficult phrases, and are in the minority, making it difficult to generate marking patterns according to this embodiment. Scoring accuracy tends to be low.

Scoring may require a high degree of accuracy. However, it is difficult to automatically grade any descriptive answer with high accuracy. Therefore, for example, answer texts that are difficult to automatically score must be excluded from automatic scoring, and careful manual scoring will be required to efficiently share automatic scoring and manual scoring. For this reason, it is a task to classify the answer sentences that are easy to score automatically and the answer sentences that are difficult to score automatically.

FIG. 5B is an example in which the answer sentences belonging to the upper cluster are displayed in order. That is, the answer texts belonging to C1, the answer texts belonging to C2, and the answer texts belonging to C3 are displayed in this order. With this display method, similar answer sentences appear in succession, and answer sentences appear in order from the majority answer to the minority answer. Become.

In the present embodiment 1, by classifying the answer texts into the majority answer and the minority answer by the above method, the answer sentences that are easy to automatically score (majority answer sentences) and the answer sentences that are difficult to automatically score (minority answer sentences) (sentence) has the effect of being able to determine the reliability of automatic scoring.

Next, an example of the sorting method performed by the cluster sorting 211 is shown. FIG. 6A is a diagram showing an example of the result (output of hierarchical clustering 210) before sorting. FIG. 6B is a diagram showing an example of answer sentences 610 before sorting.

Take an example of sorting the clusters of FIG. 6A. First, the cluster sort 211 searches for the largest cluster among the terminal clusters (C1, C2, . . . , C10). In the example of FIG. 6A, clusters C1 and C2 are searched. When there are two maximum clusters in this way, priority is given to the cluster whose parent cluster is higher.

The parent cluster is located at the position indicated by arrow 602 in FIG. C3).

The position of this arrow represents the distance between clusters. That is, the position of arrow 603 (length 604 from the end to the arrow position shown in FIG. 6A) represents the distance between cluster C2 and cluster C3. If the position of the arrow is on the left side, it is determined to be the upper rank.

As a result, the cluster C1 is the uppermost. Next, the cluster sort 211 sorts clusters (C2, C3) of arrow 603, which are child clusters other than C1 of arrow 602, which is the parent cluster of cluster C1, in a similar manner.

After these sorts are completed, the cluster sort 211 performs the same processing on the parent cluster (arrow 605) of the cluster of arrow 602 and on the child cluster of arrow 606 as well. This yields a cluster sorting result as shown in FIG. 5A.

Next, in grading pattern generation 212 , a grading pattern 215 is generated using basic analysis 204 , high-order analysis 207 , cluster sorting 211 , answer sentence grading data 213 , and semantic category dictionary 214 .

In the processing in the scoring pattern generation 212, the semantic category dictionary 214 replaces the words and phrases included in each term of the ternary relation with a higher-level concept as necessary. Hereinafter, the process by which the scoring pattern generation 212 replaces each term of a ternary relation with a superordinate concept is called abstraction.

In the explanation below, the superordinate concept of "astronaut", "president", "pianist", "nurse", "scientist", and "doctor" is "occupation", and the superordinate concept of "become" is "be". An example in which the definition of abstraction is set in the semantic category dictionary 214 is shown.

The superordinate/subordinate relation of each term of the ternary relation is defined in the semantic category dictionary 214 described later. In the example of FIG. 4, the scoring pattern generation 212 converts the two items of the ternary relation "become" to "be ”, and the three items “very kind nurse” are abstracted to “occupation” (abstraction result 404).

As "very kind nurse" in this example (abstraction result 404) is abstracted to "occupation", even if the item of the ternary relation is modified by an adjective or the like, the superordinate-lower relation of the concept is maintained. are abstracted in the same way. The definition of this abstraction is determined in advance by rules using syntax analysis or the like.

In the grading pattern generation 212, an answer sentence is selected from the answer sentences 203 for grading pattern generation. Score A pattern that can determine the presence or absence of is searched as a scoring pattern. In the first embodiment, a scoring pattern is retrieved while changing various abstractions, various additional information, and various relations among ternary relations.

This example will be described with reference to FIGS. 4 and 7A and 7B. FIG. 7A is a diagram showing an example of the answer grading data 213. As shown in FIG. FIG. 7B is a diagram showing an example of scoring conditions.

The answer text grading data 213 in FIG. 7A is information set in advance, and includes a number 2131 storing an identifier of the answer text, an answer text 2132 storing the answer text, and a score 2133 storing the grading result of the answer text 2132. , a condition A 2134 for storing the grading results with the condition A as the grading standard, a condition B 2134 for storing the grading results with the condition B as the grading standard, and a condition C 2135 for storing the grading results with the condition C as the grading standard; and a condition D2135 for storing the scoring results with the condition A as the scoring standard are included in one record.

An answer sentence 2132 in which "1" is stored in the score 2133 indicates a correct answer, and an answer sentence 2132 in which "0" is stored in the score 2133 indicates an incorrect answer (wrong answer).

First, in order to generate a grading pattern, the grading pattern generator 212 selects one correct answer sentence from the answer sentence grading data 213 . Here, an example in which the scoring pattern generation 212 selects "I wanted to become a very kind nurse." with number 2131="1" will be described.

First, the result of information extraction 206 from this answer is (I, become, very kind nurse) shown in bold in the ternary relation (information extraction result 402) in FIG. Further, the informational phenotype estimation 208 identifies that this ternary relationship is expressed in the past tense, as shown in the informational phenotype estimation result 403 of FIG.

In the following description, it is assumed that the graded results given in the answer graded data 213 are shown in the graded 2133 column in FIG. 7A.

If there are a plurality of scoring criteria, similar processing may be performed for each scoring criteria.

Condition A (2134) in FIG. 7B abstracts the third term of the ternary relation to “occupation”. Condition A is a scoring condition that the pattern (I, become, occupation) is obtained when the third term of the ternary relation extracted from the sentence is abstracted. As shown in the column of condition A 2134 in FIG. 7A, the answers that match this condition A are the correct answer sentences of numbers 1 and 2, and the answer sentences of numbers 4, 6, and 7 do not satisfy the condition. Therefore, the score is 0.

Condition B in FIG. 7B abstracts the second and third terms of the ternary relation to "be" and "occupation", respectively. Condition B is a scoring condition that the pattern (I, be, occupation) is obtained when the third term of the ternary relation extracted from the answer is abstracted. As shown in the column of condition B 2135 in FIG. 7A, the answer that matches this condition B is that all the answer sentences from number 1 to number 7 are correct, and the answer sentences of number 3 and number 5 also meet the conditions. , 1 point (correct answer).

Condition C in FIG. 7B is similar to Condition B above in that the second and third terms of the ternary relation are abstracted to "be" and "occupation", respectively. Furthermore, in condition C, the result of the information phenotype estimation 208 (estimation result of tense) is used to add a tense condition that the ternary relation is expressed in the past tense. As shown in the column of condition C 2136 in FIG. 7A, the answers that match the condition C are correct except for the numbers 3 and 5, and match the column of the answer grading data 213 .

Condition D in FIG. 7 abstracts the third term to "entity" which is higher than "occupation". The second term of the ternary relation is abstracted to "be" like condition C. Similar to condition C, this also matches the column of the answer grading data 213 .

As described above, the scoring pattern generation 212 includes the ternary relation extracted by the information extraction 206, the abstraction of each term and the degree of abstraction, the additional information of the ternary relation by the information representation type estimation 208, While changing various conditions that combine relationships between three-term relationships by information relationship estimation 209, searching for conditions that have a high degree of agreement with the scores given in the answer sentence scoring data 213 (scoring column in FIG. 7A) and a scoring pattern 215 is obtained.

This is condition C and condition D in the case of FIGS. 7A and 7B. In addition, when a plurality of conditions are obtained, one of the conditions may be selected according to a predetermined rule. As an example of the rule, it is preferable to select a grading pattern with stricter conditions (a narrower range of answer sentences that meet the conditions). In this case, if condition C and condition D are compared, condition C is stricter, so condition C should be selected. For example, "I wanted to be a bird." matches condition D, but does not match condition C.

In addition, in the search of the first embodiment, the conditions are searched based on the answer with the number 2131 of "1". , will receive 1 point.

As described above, condition C is selected as the scoring pattern 215 in the examples of FIGS. 7A and 7B. In the above description, the grading pattern generation 212 searches for conditions with a high degree of matching with the grading of the answer grading data 213, but various methods are conceivable for calculating the degree of matching.

The degree of matching should preferably be set so as to search for conditions that do not give excessive points. For example, there are few cases in which the answer grading data 213 is scored 0, but the grading pattern matches the grading pattern and is graded as 1. The conditions are set so that as many answers with 1 point as possible match the grading pattern.

With this setting, although the score of the answer sentence scoring data 213 is 1, there will be answer sentences that do not match the scoring pattern. For example, in the example of FIG. 7A, if only the condition A2134 could be searched, the answer sentences with number 2131="4", number 2131="6", and number 2131="7" are scored in the answer score data 213. Despite being 1 point, it does not match the scoring pattern of condition A2134. In that case, the grading pattern generator 212 selects the correct answer sentence 2132 that does not match the grading pattern, and adds the grading pattern.

For example, the grading pattern generation 212 searches for the answer sentence with the number 2131 = "4", and based on the number 2131 = "4", the answer sentence 2132 with as many correct answers as possible without excessive grading. Search for matching scoring patterns. If the answer text 2132 selected by the grading pattern generation 212 is over-graded, or if a condition that matches a sufficiently large number of correct answer texts cannot be obtained, another answer text can be selected. good.

By generating the grading pattern 215 as in the first embodiment, there is an effect that it is possible to suppress the rate of erroneously determining that an incorrect answer is correct. It is preferable to preferentially select correct answer sentences on the upper side of the result of cluster sorting 211 (upper side in FIG. 5A) as the answer sentences to be the basis of the conditional search.

This is because the answer sentences on the upper side of the cluster sorting result are the answer sentences of the majority, and the answer sentences of the majority are more likely to have scoring patterns that match many correct answer sentences. As a result, it is possible to reduce the processing time for retrieving scoring patterns, and obtain high-quality scoring patterns that match a large number of correct answer sentences with little over-scoring.

Also, by scoring based on the scoring pattern using the information extraction 206 and the semantic category dictionary 214, etc., there is an effect that scoring can be performed based on the meaning of the answer rather than on grammatical errors. Furthermore, there is an effect that scoring can be performed according to various scoring criteria (viewpoints).

In this way, by repeating answer sentence selection and condition search, it is possible to collect patterns that match many correct answer sentences with little overmarking. This is saved in the scoring pattern 215 as a scoring pattern. There are many scoring patterns.

In the example of FIGS. 7A and 7B, the score is in two stages of 0 point and 1 point, but in the case of three stages or more, a method of collecting scoring patterns for each score is taken as necessary. Similar processing can be performed by the method of

The scoring pattern 215 also stores the reliability of the scoring pattern. The higher the reliability, the more the answer on which the grading pattern is based is the majority (the answer belongs to the upper cluster in FIG. 5A). Furthermore, the reliability may be set to be higher as the number of overmarks is less and the number of correct answer sentences matched is higher. The accuracy of grading based on the grading pattern is high for the majority answer, and thus the reliability of the grading result using the grading pattern can be evaluated by assigning the reliability in this way.

In the above, the generation of the grading pattern 215 that matches the correct answer text has been described, but a grading pattern that matches the incorrect answer text may be generated.

By generating the grading pattern 215 by the method of the first embodiment, the grading pattern 215 can be generated according to the grading criteria based on the meaning of each question. In addition, as shown in the abstraction result 404 of FIG. 4, the scoring pattern 215 is easy to understand even without specialized knowledge of parsing results and the like.

In addition, in the first embodiment, the grading pattern generation 212 automatically searches for a grading pattern centering on the majority answer from the result of the cluster sorting 211 and generates the grading pattern 215. Data 213) is relatively at least capable of generating scoring patterns 215; In addition, since the scoring pattern is searched by combining the analysis results of the basic analysis 204, the analysis results of the high-order analysis 207, and the conditions based on the semantic category dictionary 214, the scoring rules are unlikely to become large or complicated.

Note that the grading pattern generation answer text 203 may include an ungraded answer text, so the grading target answer text 217 may be included in the grading pattern generation answer text 203 .

In the grading process 202 , the generated grading pattern 215 is used to grade the grading target answer sentence 217 . An example of the processing performed in the scoring processing 202 will be described below.

In the grading 216 , a preset score is given to an answer sentence that matches the grading pattern 215 for each answer sentence of the answer sentences 217 to be graded. For example, assume that the pattern of condition C in the example of FIG. 7B is the scoring pattern 215 . At this time, if the graded answer sentence 217 is "I wanted to be a pianist."

Also, when the graded answer sentence 217 is "I wanted to be a bird.", "bird" is not "occupation" of the condition C, so it does not match the scoring pattern 215 and is scored 0 points. Also, based on the reliability assigned to the scoring pattern 215, the reliability of scoring is assigned. As another method of giving reliability to the scoring result, the answer sentences to be graded 217 are also clustered in the same way as the hierarchical clustering 210 and the cluster sorting 211, and the answer sentences belonging to the upper cluster (larger cluster) are clustered. It is also possible to use a method of assigning a high reliability to the answer and assigning a reliability of grading based on the reliability assigned to the answer. Also, the reliability of grading may be assigned based on both the reliability assigned to the grading pattern 215 and the reliability assigned to the answer.

When there are both a correct answer scoring pattern 215 and an incorrect answer scoring pattern 215, the application order of the rules and the final judgment rule are determined. For example, if there are three grading patterns A, B, and C for correct answers, the grading patterns A, B, and C are applied in that order. is determined as the correct answer, and so on.

Answers that do not match any of the scoring patterns 215 are regarded as ungraded, and are manually graded later.

In scoring descriptive questions, by using the document evaluation apparatus 101 according to the first embodiment, a part of the answer is automatically scored, and the rest is manually scored, so that the efficiency of scoring can be improved. For example, all answers are automatically graded by the document evaluation device 101, and those whose reliability of the scoring result is below a certain threshold are manually graded, thereby improving efficiency compared to manually grading everything. can do.

In addition, in scoring descriptive questions, multiple graders may grade in order to ensure the consistency and accuracy of the scoring results. For example, two graders grade, and if the two graders agree, the graded result is adopted, and if they do not agree, the third grader After considering the above, the final scoring will be done.

At this time, grading is performed by the document evaluation apparatus 101 of the first embodiment, and among the grading results, those with high reliability are automatically graded by one of the first two graders. Grading can be made more efficient by replacing with the grading result.

According to the first embodiment, automatic scoring and manual scoring can be efficiently shared based on reliability. In addition, since scoring is performed depending on whether or not the scoring pattern 215 is matched, it is easy to understand the reason for the scoring result by the automatic scoring by referring to the scoring pattern 215 .

The results graded as described above are stored in the graded result 218 together with the reliability.

As described above, according to the first embodiment, it is possible to provide a method of automatically generating the scoring pattern 215 for determining correct and incorrect answers for each scoring criterion and performing automatic scoring based on the scoring pattern 215 . In addition, by classifying the answers into majority answers and minority answers, by classifying answers into easy automatic scoring (majority answers) and difficult automatic scoring (minority answers), It becomes possible to determine the reliability of automatic scoring. In addition, by manually grading minority answer sentences, it is possible to efficiently operate automatic grading and manual grading.

FIG. 11 is a flow chart showing an example of processing performed by the document evaluation apparatus 101 in this embodiment. The difference from Example 1 shown in FIG. 2 is that the hierarchical clustering 1101 uses the results of executing the basic analysis 204 and the high-order analysis 207 .

In the first embodiment, in the hierarchical clustering 210 of answer sentences, an example of characterizing the answer sentences 203 for grading pattern generation by the frequency of occurrence of words and phrases has been described with reference to FIG. In the first embodiment, a list of words and phrases such as "a" and "the" to be excluded from the count is generated. performed clustering.

In the second embodiment, only words and phrases related to information extraction 206 and information phenotypic estimation 208 are counted.

In the case of Example 1, assuming that the count exclusion list is "a" and "the", when counting words and phrases from the answer sentence 401 in FIG. want)”, “to”, “become”, “very”, “kind”, and “nurse” are counted.

On the other hand, in the second embodiment, the words "I", "become", "very", "kind", and "nurse" related to the information extraction result 402 and the information phenotype estimation result 403 A total of 6 words, the word "wanted" (or want), are counted. Note that only the five words of the information extraction result 402 may be used.

In this way, when generating a feature matrix such as the feature matrix 803 shown in FIG. By narrowing down the features, it is possible to exclude words and phrases that are not related to subsequent processing from being counted as features.

For example, in the example of FIG. 4, "to" is excluded. By performing such processing, it is possible to obtain hierarchical clustering results that are more suitable for the scoring pattern, and it is expected that highly accurate automatic scoring results and reliability will be obtained.

In addition to the method of counting only those related to the information extraction result 402 and the information phenotype estimation result 403, as in the first embodiment, all words and phrases other than the exclusion list are counted, but the information extraction result 402 and the information phenotype estimation result 403 may be increased by a predetermined ratio in the value of the word importance level 802 shown in FIG.

As described above, by narrowing down the words and phrases for which the feature amount is to be calculated from the information extraction result 402 and the information phenotype estimation result 403, automatic scoring results and reliability can be improved.

Minor spelling errors may not be deducted in the scoring criteria. In such a case, spelling errors are treated as different words or phrases, so in the first embodiment, a scoring pattern 215 is generated for each spelling error. Also, many answer sentences may be required to cover various spelling errors.

In contrast, in the third embodiment, spelling errors included in the grading pattern generation answer text 203, the answer text grading data 213, and the grading target answer text 217 are corrected in advance. For example, a list of frequently occurring spelling errors is generated and the spelling errors are corrected according to the list. For example, there are cases where "doctor" is mistaken for "docter". In this case, the document evaluation apparatus 101 replaces all "docter" in the grading target answer sentences 217 and the like with "doctor".

Also, the document evaluation apparatus 101 may correct grammatical errors unrelated to the marking pattern 215 in advance. For example, to be precise, if tenses should not be scored for a question that should be answered in the present tense, correct past tense or other tense expressions to present tense. You may do so.

The fourth embodiment shows an example of the user interface of the document evaluation apparatus 101. FIG. FIG. 12A is a diagram showing an example of an answer data input screen 1201. FIG. An answer data input screen 1201 is output to the display device 103 . FIG. 12B is a diagram showing an example of graded answer data. FIG. 12C is a diagram showing an example of ungraded answer data.

In the answer data input screen 1201, an ID for identifying the question is entered in the question ID field 121. FIG. A question text is entered in the question text field 122 . File field 123 specifies the name of the file containing the data. The files are, for example, graded answer data 1202 shown in FIG. 12B and ungraded answer data 1203 shown in FIG. 12C.

When the save button 124 is pressed on the answer data input screen 1201 by operating the input device 102, the document evaluation device 101 saves the graded answer data 1202 and the ungraded answer data 1203 for the data matching the question ID 121 to the database. (not shown).

For example, graded answer data 1202 and ungraded answer data 1203 are used as grading pattern generating answer sentences 203, graded answer data 1202 are graded answer data 213, and ungraded answer data 1203 are graded data. It is saved in the target answer sentence 217 . In addition, question sentences are also stored in each database. For example, the data format is as shown in FIGS. 9A to 9C of the first embodiment. When data is input, a marking pattern 215 is generated by the marking pattern generation processing 201 of the document evaluation apparatus 101 .

FIG. 13 is a diagram showing an example of a grading pattern display screen 1301 generated by the document evaluation apparatus 101. As shown in FIG. A grading pattern display screen 1301 is a confirmation screen for maintenance for the user of the document evaluation apparatus 101 to edit the generated grading pattern 215 .

When the question ID is entered in the question ID 131 of the grading pattern display screen 1301, the question corresponding to the question 132 is displayed. Area 1302 displays the number of graded answer sentences to which a grading pattern has been applied and the accuracy (reliability) of automatic grading based on the grading pattern. In this example, the scoring pattern 215 was applied to 450 out of 800 graded answer sentences, and 4/5 (80%) of them were graded correctly. Thereby, the quality of the entire scoring pattern 215 can be confirmed.

A grading pattern list 133 is displayed below the area 1302 . The scoring pattern list 133 includes an item number 1303, a scoring pattern 1304, a cover count 1305, and an accuracy (reliability) 1306 in one record.

The column of the item number 1303 indicates the serial number of the marking pattern. A column of grading pattern 1304 displays a brief description of grading pattern 215 . A method of generating this explanation will be described in detail later in the description of FIG. A column of the number of covers 1305 indicates the number of answer sentences to which the grading pattern is applied, among the graded answer sentences. A column of accuracy 1306 indicates, by accuracy, the percentage of correctly graded answer texts to which the grading pattern is applied.

A page designation 1307 is a button for moving pages when the number of marking patterns is large and the display is divided into several pages. Although the number of covers and accuracy must be indicated for each scoring standard, they are omitted in the present scoring pattern display screen 1301 to avoid complication. Further, it is preferable to add a function to enable/disable or delete each grading pattern to the main grading pattern display screen 1301 . By invalidating/deleting scoring patterns with low accuracy or scoring patterns with a small number of covers, the quality of scoring by automatic scoring can be improved.

Next, generation and editing screens for individual marking patterns will be described. By pressing the item number 1303 in FIG. 13, the editing screen of the grading pattern may be displayed, and a button for screen transition is added to the grading pattern display screen 1301 in order to generate a new grading pattern. You can keep it.

FIG. 14 is a diagram showing an example of a scoring pattern editing screen. An area 1402 in the upper part of the grading pattern edit screen 1401 displays the number of covers and accuracy of the selected grading pattern. An explanation 1403 displays an explanation of the scoring pattern. This may be generated by the user of the document evaluation apparatus 101 when adding or editing a scoring pattern, or may be automatically generated by a template. In the case of automatic generation, for example, as shown in the figure, it is possible to adopt a method such as describing the ternary relation of the pattern 1404 and the checked tense as they are.

The pattern 1404 displays the ternary relation extracted in the information extraction 206 , and the tense 1405 displays the tense of the ternary relation estimated in the information phenotypic estimation 208 . In this example, the past tense applies. This example is an example of a scoring pattern like the abstraction result 404 shown in FIG.

Although the scoring pattern editing screen 1401 shows a simple case of only the tense 1405, a pattern predetermined as an analysis object in the information phenotype estimation 208 and the inter-information relationship estimation 209 can be displayed as necessary. good too. Also, multiple ternary relationships can be displayed in pattern 1404 if multiple information extraction ternary relationships are associated with scoring pattern 215 . The number of items may be increased as needed for editing the scoring pattern 215 . As a result, the user of the document evaluation apparatus 101 can finely edit the grading pattern and improve the quality of automatic grading.

FIG. 15 is a diagram showing an example of a scoring screen 1501. As shown in FIG. Question sentence ID 131 and question sentence 132 are the same as those on the grading pattern display screen 1301 shown in FIG. Table 151 in FIG. 15 shows the results of automatic scoring of ungraded data.

The column of answer ID 1502 in table 151 stores the ID of each answer. The column of answer sentences 1503 stores answer sentences. The grading result based on the grading standard X is displayed in the column of the grading standard X 1504 . This scoring result is the result of automatic scoring or the result input or edited by the user. If the item is not graded, it is left blank, for example. In the applied pattern column 1505, item numbers 1303 of applied scoring patterns are displayed. When an applied pattern 1505 is pressed, a transition to the grading pattern edit screen (FIG. 14) of the pattern may be made.

A page designation 1506 is displayed below the table 151 . A page designation 1506 is the same as the page designation 1307 in FIG. 13, and is a button for switching pages when the number of answer sentences is divided into several pages and displayed.

Using this grading screen 1501, the automatic grading result is manually confirmed or corrected, and the ungraded answer is graded by the automatic grading. It should be noted that a function such as grading the same or similar answer sentences at the same time when grading one answer sentence may be provided. For example, when the user inputs or edits the grading results for the answer with the answer ID 1502 in FIG. , the same grading result is automatically assigned to the answer.

This makes scoring more efficient. In this grading screen 1501, instead of displaying the answer sentences 1503 in the order of the answer sentence ID 1502, the answer sentences may be displayed in the order of the hierarchical clustering results (FIG. 5). By sorting the display order in this way, the answer sentences 1503 appear in order from the majority answer to the minority answer, and similar answer sentences appear in succession. become.

<Summary>
In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above embodiments are described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, addition, deletion, or replacement of other configurations for a part of the configuration of each embodiment can be applied singly or in combination.

Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing them in an integrated circuit. Further, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

Further, the control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.

101 Document evaluation device 105 Arithmetic device 106 Memory 107 Auxiliary storage device 201 Scoring pattern generation processing 202 Scoring processing 203 Answer text for grading pattern generation 204 Basic analysis 207 Higher order analysis 210 Hierarchical clustering 211 Cluster sorting 212 Scoring pattern generation 213 Answer text grading data 214 semantic category dictionary 215 grading pattern 217 grading target answer text 218 grading result

Claims (11)

A document evaluation apparatus comprising a processor for executing a program stored in a memory and a computer having a memory for storing data used by the processor when executing the program,
The processor functions as the following functional units by executing the program,
The document evaluation device
a syntactic analysis unit that analyzes morphemes and dependency relationships from an input answer text for generating a scoring pattern;
an information extraction unit that extracts a ternary relation from the answer text;
Using a machine learning model that has learned the relationship between the ternary relation, the syntactic analysis result, and the additional information of the ternary relation, such as tense, modality, and date and time, as training data, the analysis result of the syntactic analysis unit and the an information phenotype estimation unit that estimates additional information of the ternary relationship from the ternary relationship extracted by the information extraction unit;
Using a machine learning model that learns the relationship between the syntactic analysis result and the mutual relationship of the ternary relationship as teacher data, the mutual relationship of the ternary relationship extracted by the information extraction unit from the analysis result of the syntactic analysis unit an inter-information relationship estimating unit for estimating;
a hierarchical clustering unit that classifies the answer sentences into hierarchical clusters;
a cluster sorting unit that sorts the clusters based on the size of the clusters;
a grading pattern generation unit that selects a grading condition suitable for the read grading result as a grading pattern ;
The hierarchical clustering unit classifies the answer sentences into hierarchical clusters according to the similarity of a feature matrix including a value obtained by multiplying the appearance ratio of a word in the answer by the importance of the word,
The scoring pattern generation unit
abstracting by replacing the words or phrases included in each term of the ternary relation with a higher concept using a semantic category dictionary in which superordinate and subordinate relationships between words or phrases are recorded in advance;
The ternary relation extracted by the information extracting unit, the degree of abstraction of each term of the ternary relation, the additional information of the ternary relation analyzed by the information phenotype estimating unit, and the inter-information relation estimating unit Among the conditions that combine the mutual relationships of the three-term relationships estimated by, select the answer sentence of the condition with a high degree of matching of the scoring results as a scoring pattern,
Based on the sorting result of the cluster sorting unit, determining a cluster containing an answer sentence equal to or greater than a predetermined threshold as a majority answer sentence, and determining a cluster containing an answer sentence less than the threshold as a minority answer sentence,
A document evaluation apparatus, wherein the reliability of the marking pattern generated from the majority answer is set high, and the reliability of the marking pattern generated from the minority answer is set low. The document evaluation device according to claim 1,
A document evaluation apparatus, further comprising a grading unit that reads a grading target answer text and generates a grading result based on the grading pattern. The document evaluation device according to claim 1,
The information extracting unit uses the semantic category dictionary to detect words or phrases in the answer text, and extracts two words or phrases in the answer text from words or phrases that express the relationship between the two words or phrases. 3. A document evaluation apparatus , which estimates a relationship between documents, and extracts three terms consisting of the two words or phrases and a word or phrase representing the relationship as the ternary relationship. The document evaluation device according to claim 1,
The document evaluation apparatus, wherein the grading pattern generation unit outputs screen data displaying the grading pattern and reliability. A document evaluation method in which a processor that executes a program stored in a memory and a computer that has a memory that stores data used by the processor when executing the program scores answer texts,
a first step in which the calculator analyzes morphemes and dependency relationships from the input answer text for generating scoring patterns;
a second step in which the calculator extracts a ternary relation from the answer text;
The computer uses a machine learning model that has learned the relationship between the ternary relation, the syntactic analysis result, and the tense, modality, and date and time, which are additional information of the ternary relation, as teacher data, to determine the morpheme and the dependency relation. and a third step of estimating additional information of the ternary relation from the extracted ternary relation;
The computer uses a machine learning model that has learned the relationship between the syntactic analysis result and the mutual relationship of the ternary relationship as teacher data, and the mutual relationship of the ternary relationship extracted from the analysis result in the first step. a fourth step of estimating relationships;
a fifth step in which the calculator classifies the answer sentences into hierarchical clusters;
a sixth step in which the calculator sorts the clusters based on the size of the clusters;
a seventh step in which the calculator selects, as a grading pattern, a grading condition that matches the read grading result;
including
In the fifth step,
The computer classifies the answer sentences into hierarchical clusters according to the similarity of a feature matrix containing a value obtained by multiplying the appearance ratio of a word in the answer by the importance of the word,
In the seventh step,
the computer abstracts by replacing the words or phrases included in each term of the ternary relation with a higher concept using a semantic category dictionary in which superordinate and subordinate relationships between words or phrases are recorded in advance;
The computer provides the ternary relation extracted in the second step, the degree of abstraction of each term of the ternary relation, and the additional information of the ternary relation analyzed in the third step, Selecting, as a grading pattern, an answer sentence under a condition with a high degree of matching of the grading results from among the conditions combining the mutual relationships of the ternary relationships estimated in the second step,
The calculator determines clusters containing answer sentences equal to or greater than a predetermined threshold as majority answer sentences, and clusters containing answer sentences below the threshold as minority answer sentences, based on the sorting result in the sixth step. judge,
The document evaluation method, wherein the computer sets a high degree of reliability for the marking pattern generated from the majority answer sentences and sets a low reliability for the marking pattern generated from the minority answer sentences. The document evaluation method according to claim 5,
The document evaluation method, further comprising: an eighth step in which the computer reads the answer sentences to be graded and generates a graded result based on the grading pattern. The document evaluation method according to claim 5,
In the second step,
The calculator uses the semantic category dictionary to detect words or phrases in the answer sentence, and from words or phrases that express the relationship between two words or phrases to the relationship between the two words or phrases in the answer sentence. A document evaluation method, wherein a relationship is estimated, and three terms consisting of the two words or phrases and a word or phrase representing the relationship are extracted as the ternary relationship. The document evaluation method according to claim 5,
In the seventh step,
A document evaluation method, wherein the computer outputs screen data displaying the scoring pattern and reliability. A computer having a processor that executes a program stored in a memory and a memory that stores data used by the processor when executing the program, and a program for scoring answer texts,
a first step of analyzing morphemes and dependency relationships from an input answer text for generating a scoring pattern;
a second step of extracting a ternary relation from the answer text;
Using a machine learning model that has learned the relationship between the ternary relation, the syntactic analysis result, and the tense, modality, and date and time that are additional information of the ternary relation as teacher data, the analysis result of the morpheme and the dependency relation and the , a third step of estimating additional information of the ternary relation from the extracted ternary relation;
Estimate the mutual relationship of the ternary relations extracted from the analysis result in the first step using a machine learning model that has learned the relationship between the syntactic analysis result and the mutual relationship of the ternary relations as teacher data. a fourth step;
a fifth step of classifying the answer sentences into hierarchical clusters;
a sixth step of sorting the clusters based on the size of the clusters;
a seventh step of selecting, as a grading pattern, grading conditions that match the read grading results;
A program for causing the computer to execute
In the fifth step,
classifying the answer sentences into hierarchical clusters according to the similarity of a feature matrix containing a value obtained by multiplying the appearance ratio of a word in the answer by the importance of the word;
In the seventh step,
abstracting by replacing the words or phrases included in each term of the ternary relation with a higher concept using a semantic category dictionary in which superordinate and subordinate relationships between words or phrases are recorded in advance;
The ternary relation extracted in the second step, the degree of abstraction of each term of the ternary relation, the additional information of the ternary relation analyzed in the third step, and the second Selecting, as a grading pattern, an answer sentence under a condition with a high degree of matching of the grading results from among the conditions in which the mutual relationships of the three-term relationships estimated in the step are combined,
Based on the sorting result in the sixth step, determining a cluster containing answer texts equal to or greater than a predetermined threshold as majority answer texts, and determining a cluster containing answer texts less than the threshold as minority answer texts,
A program for setting the reliability of a marking pattern generated from the majority answer sentences to a high level and setting the reliability of a marking pattern generated from the minority answer sentences to a low level. The program according to claim 9,
The program, further comprising: an eighth step of reading a grading target answer text and generating a grading result based on the grading pattern. The program according to claim 9,
In the second step,
Using the semantic category dictionary, detect words or phrases in the answer sentence, and estimate the relationship between the two words or phrases in the answer sentence from the words or phrases representing the relationship between the two words or phrases. , a program for extracting, as the ternary relation, three terms composed of the two words or phrases and a word or phrase representing the relationship.

Images