JP2005331615A - Sentence evaluation device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sentence evaluation device and a sentence evaluation method for properly evaluating a sentence. <P>SOLUTION: An answer reason sentence class information calculation section 15 calculates the degree of similarity of an answer feature set of answer reason sentences 32 to a model reason feature set 121 of a model reason feature class 120 and an irrelevant reason feature set 131 of an irrelevant reason feature class 130 respectively. When the respective degree of similarity is calculated, the answer reason sentence class information calculation section 15 determines the model reason feature class 120 which contains the model reason feature set 121 having the highest degree of similarity as the class to classify the answer reason sentences 32 into. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a sentence evaluation apparatus and a sentence evaluation method for evaluating sentences converted into electronic data.

In recent years, computers and computer networks have come to be used also in the education field with the spread of inexpensive computers with high processing power.
For example, some universities connect to other remote universities via a local area network (LAN), a wide area network (WAN), or a computer network (hereinafter referred to as a network) such as the Internet. An environment where students can take lectures at the other university is realized. In addition, online learning that enables home study by preparing educational content that can be displayed on a computer on a site and accessing the site with a home computer via a network is also actively performed.
In order to manage such e-Learning properly, it is important to properly understand the proficiency level of the learner. One method for determining the proficiency level is online testing using a computer. This test question is mainly prepared by preparing a plurality of options as answers and allowing the learner to select one of them. However, there is a problem that the proficiency level of the learner cannot be properly grasped only by the selective test.

  Also, in some qualification tests, a digital test is performed in which an examinee inputs an answer using a computer connected to a network, and this score is automatically scored using the computer. However, in the case of this digital test as well, as in the case of e-Learning, the test question format is mainly a selection formula. Therefore, in order to measure the examinee's deep knowledge and way of thinking, in addition to the selective test, it is necessary to conduct a test in the form of a sentence that answers in a sentence.

In the sentence format test, unlike the selection type, it is difficult to limit the number of correct answers to one or a small number. This is because, when expressing a certain matter as a sentence, it is possible to express the same matter in various sentences depending on the word or syntax used. Therefore, by preparing a model answer in advance and simply comparing the answer and the model answer with the character strings constituting the sentence, an appropriate answer cannot be evaluated as a correct answer.
If such a written answer can be automatically scored using a computer, the proficiency confirmation test and digital test scoring in e-Learning can all be automated, and e-Learning It is possible to improve the efficiency of the system and save labor costs for scoring.

As an existing technology for automatically scoring written answers, there is an automatic essay evaluation technology. This is a technology that evaluates numerically the appropriateness of an answer sentence that expresses the examinee's thoughts and opinions on questions written in sentences. For example, in a business school entrance examination (Graduate Management Admission Test, GMAT) conducted by Educational Testing Services (ETS), an automatic scoring technique called E-rater has been studied (for example, see Non-Patent Document 1). This technology analyzes the tendency of the types and quantities of letters, words, and sentences that make up sentences, and compares them with optimal values determined in consideration of past tests and expert opinions. Is being evaluated for its appropriateness. In addition, experts classify answers in the same exams that were conducted in the past, assign evaluation points to each category in advance, calculate the relevance between the sentences of the answer and the classification, and evaluate highly relevant sentences The point is the evaluation score of the answer. Also in Japanese, examination of jerator, which is an automatic scoring technique equivalent to E-rater, has been conducted (for example, see Non-Patent Document 2). In the study of jerator, classification is made by newspaper articles that are thought to be closely related to the model answer instead of the past answer.
On the other hand, in Intelligent Essay Assessor (IEA) (for example, see Non-Patent Document 3) and Nagasaka et al. (For example, see Non-Patent Document 4) The model calculates the similarity between the model answer and the answer in consideration of the relationship between words.
The applicant has not yet found prior art documents related to the present invention by the time of filing other than the prior art documents specified by the prior art document information described in this specification.
Burstein, J., Kukich, K., Wolff, S., C., Chodorow, M., Braden-Harder, L. and Harris, MD, Automated Scoring Using a Hybrid Feature Identification Technique, Proceedings of the Thirty-Sixth Annual Meeting of the Association for Computational Linguistics and Seventeenth International Conference on Computational Linguistics, 1998, p.206-210 Tsunenori Ishigaki, Masayuki Kameda, "Automatic Japanese Scoring System by Computer", IEICE Technical Report, 2002, No.TL2002-40, p.43-48 Landauer, TK and Dumais, ST, A Solution to Plato's Problem, The Latent Semantic Analysis Theory of the Acquisition, Induction, and Representation of Knowledge, Psychological Review, 1997, Vol. 104, p. 211-240 Nagasaka Yasutaka, Furuse Katsushige, Ishii Satoshi, Nishikawa Tsutomu, "Development of Automatic Scoring Engine for Description Problem in Web Environment", Advanced Information Support Software Seeds Training Project, 2001

In the conventional automatic text evaluation technology described above, if there are past answers to the same test or similar text data, or classification by experts of these answers and text data, it is not limited to the type of test. It is possible to evaluate the answer text.
However, in the case of an examination such as an essay composed of a sentence expressing an option and a reason sentence indicating the reason for selecting this option, the conventional method uses an answer or example of a sentence indicating the option and a reason sentence. Since structural information of the answer is not used, there are cases where the evaluation of the answer is not properly performed.

For example, in the conventional method, even when the sentence representing the option and the reason sentence do not match, the sentence representing the option and the reason sentence are collectively evaluated. Usually, the reason sentence is often longer than the sentence representing the option. Therefore, the conventional method mainly evaluates whether the content of the reason text is close to the content of the reason text with some model answers, and the appropriateness of the connection between the text indicating the option and the reason text is not evaluated. End up.
Also, in the conventional method, even when a sentence representing an exceptional option such as not correct is selected, the sentence representing the option and the reason sentence are collectively evaluated as described above. In some cases, the answer was highly appreciated.

  Therefore, the present invention has been made to solve the above-described problems, and an object thereof is to provide a sentence evaluation apparatus and a sentence evaluation method capable of appropriately evaluating a sentence.

In order to solve the above-described problem, the sentence evaluation apparatus according to the present invention includes one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating the reason for selecting the selected sentence. In a sentence evaluation apparatus for evaluating a composed answer, a plurality of model answer feature classes including a selected sentence and an example reason feature set including features of an example reason sentence indicating an example reason for selecting the selected sentence are stored. The model answer feature DB, extraction means for extracting features from the selection reason sentence, and similarity between the model reason sentence and the selection reason sentence are calculated based on the model reason feature set and the features extracted by the extraction means. , A calculation means for specifying the model answer feature class including the model reason sentence with the highest similarity, and an evaluation for evaluating the answer based on the model answer feature class specified by the calculation means Characterized by comprising a stage. According to the present invention, when the model answer feature class is specified by the calculation unit, the selected sentence corresponding to the model reason sentence having the highest similarity to the answer selection reason sentence is uniquely determined. Therefore, it is possible to evaluate the appropriateness of the connection between the selection reason sentence and the selected sentence.
The feature is an element that can be used when comparing a plurality of sentences, and means all or a part of elements constituting the sentence such as characters or words constituting the sentence.

In the above sentence evaluation apparatus, the evaluation unit includes a determination unit that determines whether or not the selected sentence included in the answer matches the selected sentence included in the model answer feature class specified by the calculation unit. If it is determined that they do not match, the lowest evaluation may be made on the answer.
Here, the evaluation means, based on the similarity between the characteristics obtained from the model reason feature set included in the model answer feature class specified by the calculation means and the characteristics of the answer reason sentence when it is determined that the determination means match. You may make it evaluate an answer.

  Another embodiment of the sentence evaluation device according to the present invention is an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating the reason for selecting the selected sentence. In the sentence evaluation device for evaluating the selected sentence, a plurality of exemplary answer feature class including an exemplary reason feature set composed of the characteristic of the exemplary reason sentence indicating the exemplary reason for selecting the selected sentence and the exemplary example for selecting the selected sentence Model answer feature DB storing irrelevant reason feature class including irrelevant reason feature set consisting of features of irrelevant reason sentence indicating unreasonable reason, extraction means for extracting feature from selected reason sentence, model reason sentence and irrelevant The similarity between the reason sentence and the selection reason sentence is calculated based on the model reason feature set and the irrelevant reason feature set and the features extracted by the extraction means. Based on the calculation means for identifying the model answer feature class or the irrelevant reason feature class including the model reason sentence or the irrelevant reason sentence having the highest degree, and the answer based on the model answer feature class or the irrelevant reason feature class specified by the calculation means An evaluation means for evaluating is provided.

  In the above sentence evaluation apparatus, the evaluation means may perform the lowest evaluation on the answer when the irrelevant reason feature class is specified by the calculation means.

  In the above sentence evaluation apparatus, a model answer DB storing at least one of a plurality of model reason sentence classes including a selected sentence and a model reason sentence corresponding to the selected sentence, and an irrelevant reason sentence class including an irrelevant reason sentence And a model answer feature DB creating means for creating a model answer feature DB from the model answer DB.

  The sentence evaluation method according to the present invention is a sentence for evaluating an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating the reason for selecting the selected sentence. In the evaluation method, a model answer feature DB is created which stores a plurality of model answer feature classes including a selected sentence and a model reason feature set including features of the model reason sentence indicating a model reason for selecting the selected sentence. The similarity between the creation step, the extraction step of extracting the feature from the selection reason sentence, and the similarity between the example reason sentence and the selection reason sentence is calculated based on the feature extracted from the example reason feature set and the extraction step. The calculation step that identifies the model answer feature class that contains the highest model reason sentence, and the answer is evaluated based on the model answer feature class identified by this calculation step. Characterized by comprising an evaluation step for.

In the above sentence evaluation method, the evaluation step includes a determination step of determining whether or not the selected sentence included in the answer matches the selected sentence included in the model answer feature class specified in the calculation step. If it is determined that they do not agree with each other, the lowest evaluation may be made on the answer.
Here, the evaluation step is based on the similarity between the feature obtained from the model reason feature set included in the model answer feature class specified by the calculation step and the feature of the answer reason sentence when it is determined that they match in the determination step. You may make it evaluate an answer.

  In another form of the sentence evaluation method according to the present invention, an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating the reason for selecting the selected sentence. In the sentence evaluation method for evaluating the selected sentence, a plurality of model answer feature class including a model reason feature set including characteristics of the selected sentence and a model reason sentence indicating a model reason for selecting the selected sentence, and a model for selecting the selected sentence A creation step for creating a model answer feature DB storing an irrelevant reason feature class including an irrelevant reason feature set consisting of features of an irrelevant reason sentence indicating an unreasonable reason; an extraction step for extracting a feature from the selected reason sentence; The similarity between the model reason sentence and the irrelevant reason sentence and the selection reason sentence is extracted by the model reason feature set, the irrelevant reason feature set and the extraction step. A calculation step for identifying a model answer feature class or an irrelevant reason feature class including a model reason sentence or an irrelevant reason sentence having the highest similarity, and a model answer feature class identified by the calculation step Or an evaluation step for evaluating the answer based on the unrelated reason feature class.

  In the above sentence evaluation method, the evaluation step may perform the lowest evaluation on the answer when the irrelevant reason feature class is specified in the calculation step.

  In the above sentence evaluation method, the creation step stores at least one of a plurality of model reason sentence classes including a selected sentence and a model reason sentence corresponding to the selected sentence, and an irrelevant reason sentence class including an irrelevant reason sentence. The model answer feature DB may be created from the model answer DB.

According to the present invention, since the answer is evaluated based on the similarity between the feature of the selection reason sentence and the feature of the model reason sentence, the answer can be evaluated using the structural information of the answer or the model answer. It is possible to appropriately evaluate the answers to the exam, such as an essay that states the choice and the reason for choosing this choice.
Further, according to the present invention, it is determined whether or not the selected sentence included in the answer and the selected sentence included in the identified model answer feature class match, and if not, the lowest evaluation for the answer do. Thereby, it is possible to perform an appropriate evaluation in consideration of the appropriateness of the connection between the option and the reason sentence.
Furthermore, according to the present invention, when an irrelevant reason feature class is specified, the lowest evaluation is given to the answer. Thereby, even when an exceptional option such as not correct is included in the answer, an appropriate evaluation can be performed.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a sentence evaluation apparatus according to the present embodiment.
The sentence evaluation apparatus 1 according to the present embodiment receives a model answer prepared in advance for an arbitrary question from the model answer DB 2 in advance, and a user (hereinafter referred to as an examinee) creates the question. The evaluated answer 3 is evaluated using the model answer received from the model answer DB 2, and the evaluation value 4 obtained by quantifying this evaluation is output to the outside.

FIG. 2 is a schematic diagram for explaining the question.
Here, as shown in FIG. 2, the question is a sentence representing an inquiry to the examinee. From the question sentence 5 and two or more answer selection sentences 6 selected by the examinee for the question sentence 5, Composed. In the case of FIG. 2, two answer selection sentences 6 a and 6 b are provided for the question sentence 5.
The question sentence 5 may include not only sentences but also images and sounds.
In the present embodiment as a whole, the sentence means electronic data composed of a plurality of character codes.

The model answer DB 2 includes a plurality of model reason sentence classes 21 and an unrelated reason sentence class 22. FIG. 3 is a schematic diagram for explaining an exemplary reason sentence class 21 and an unrelated reason sentence class 22.
The quantity of the model reason sentence class 21 corresponds to the quantity of the answer selection sentence 6 for the question sentence 5. For example, as shown in FIG. 2, when two answer selection sentences 6a and 6b are set for the question sentence 5, the model reason sentence class 21 corresponds to the answer selection sentences 6a and 6b in a one-to-one correspondence. Two model reason sentence classes 21a and 21b are set.

The model reason sentence class 21 includes one answer selection sentence 6 for the question sentence 5 and at least one example reason sentence 7 indicating the reason for selecting the answer selection sentence 6.
For example, in the model reason sentence class 21a shown in FIG. 3A, three model reason sentences 7a to 7c are provided for the answer selection sentence 6a. Also, the model reason sentence class 21b shown in FIG. 3B is provided with two example reason sentences 7d to 7e for the selected sentence 6b.
Here, the model reason sentence 7 is prepared by a question creator (hereinafter referred to as a tester) in advance for each answer selection sentence 6 if appropriate as a reason for selecting the answer selection sentence 6, or It is also possible to select an appropriate sentence from sentences collected from questions, books, and www (world wide web).

  The irrelevant reason sentence class 22 has a plurality of irrelevant reason sentences 8 that are irrelevant to the reason for selecting the answer selection sentence 6. For example, in the case of the irrelevant reason sentence class 22 shown in FIG. 3C, three irrelevant reason sentences 8a to 8c are set.

Such an exemplary answer DB 2 may be recorded on a known computer storage device, magnetic disk, or the like.
In the present embodiment, the model answer DB 2 is provided independently of the sentence evaluation apparatus 1. As a result, the degree of freedom such as a place and time for creating the model answer DB 2 is increased, which is very convenient for the examiner.
The model answer DB 2 may be incorporated in the sentence evaluation apparatus 1.

The answer 3 includes an answer selection sentence 31 and an answer reason sentence 32. FIG. 4 is a schematic diagram for explaining the answer 3.
The answer selection sentence 31 corresponds to a selection sentence selected by the examinee from a plurality of answer selection sentences 6 as shown in FIG.
The answer reason sentence 32 corresponds to a sentence describing the reason why the examinee selected the answer selection sentence 31.
For example, the answer 3 shown in FIG. 4 is a test taker's answer to the question 5 shown in FIG. 2. The answer selection sentence 31 selected from the answer selection sentences 6 and the answer describing the reason for selecting the answer selection sentence 31. The reason sentence 32 is comprised.

  The sentence evaluation apparatus 1 includes an exemplary answer feature DB creation unit 11, an exemplary answer feature DB 12, a sentence feature extraction unit 13, a similarity calculation unit 14, an answer reason sentence class information calculation unit 15, an evaluation value DB 16, And an evaluation value calculation unit 17.

The model answer feature DB creation unit 11 creates a model answer feature DB 12 based on the model answer DB 2.
The model answer feature DB 12 includes a plurality of model reason feature classes 120 and an unrelated reason feature class 130. FIG. 5 is a schematic diagram for explaining an exemplary reason feature class 120 and an unrelated reason feature class 130.
Each model reason feature class 120 corresponds to each model reason sentence class 21 of the model answer DB 2 on a one-to-one basis. For example, as shown in FIG. 1, when two model reason sentence classes 21a and 21b are provided in the model answer DB 2, the model answer feature DB 12 has a one-to-one correspondence as shown in FIG. Exemplary reason feature classes 120a, 120b are provided.

The model reason feature class 120 includes an answer selection sentence 6 of the model reason sentence class 21 corresponding to one-to-one, at least one model reason feature set 121, a model average feature set 122, and similarity statistical information 123. Is done.
For example, the model reason feature class 120a illustrated in FIG. 5A corresponds to the model reason sentence class 21a illustrated in FIG. 3A, and the answer selection sentence 6a, the model reason feature sets 121a to 121c, and the model average feature set. 122 and similarity statistical information 123. Similarly, the model reason feature class 120b illustrated in FIG. 5B corresponds to the model reason sentence class 21b illustrated in FIG. 3B, and the answer selection sentence 6b, the model reason feature sets 121d and 121e, and the model average feature. It consists of a set 122 and similarity statistical information 123.

The model reason feature set 121 includes a set of features extracted from each model reason sentence 7 of the corresponding model reason sentence class 21.
Here, the feature can be arbitrarily set as long as it is an element that can be used when comparing a plurality of sentences. For example, all or all of the elements constituting the sentence, such as characters or words constituting the sentence, can be set. Mean part. Such features can be found in words of specific parts of speech such as nouns and verbs that are important when comparing sentences among all words included in sentences, or in a dictionary registered in the sentence evaluation apparatus 1 in advance. It may be a matched word.
For example, the model reason feature sets 121a to 121c shown in FIG. 5A are configured from feature sets extracted from the corresponding model reason sentences 7a to 7c in FIG. 3A. Similarly, the model reason feature sets 121d and 121e shown in FIG. 5B are composed of feature sets extracted from the corresponding model reason sentences 7d and 7e in FIG. 3B.

The model average feature set 122 includes, for example, a set of features overlapping each model reason feature set 121, a set of features of each model reason feature set 121 from which duplication is removed, and a fixed number of times among the features of all model reason feature sets 121. A set of features that characterize the exemplary reason feature set 121 included in the exemplary reason feature class 120, such as a set of features that appear above, a union or a product set of the features of each exemplary reason feature set 121, and the like.
For example, the model average feature set 122a illustrated in FIG. 5A includes a set of features that characterize the model reason feature sets 121a to 121c. Similarly, the model average feature set 122b illustrated in FIG. 5B includes a set of features that characterize the model reason feature sets 121d and 121e.

The similarity statistical information 123 is statistical information based on the similarity regarding the degree of similarity between the model average feature set 122 and each model reason feature set 121, and includes an average of similarity and a standard deviation.
For example, the similarity statistical information 123a illustrated in FIG. 5A includes an average similarity and a standard deviation between the exemplary reason feature sets 121a to 121c and the exemplary average feature set 122a. Similarly, the similarity statistical information 123b illustrated in FIG. 5B includes an average similarity and a standard deviation between the model reason feature sets 121d and 121e and the model average feature set 122b.

  The irrelevant reason feature class 130 has an irrelevant reason feature set 131 composed of a set of features extracted from the irrelevant reason sentence 8 of the irrelevant reason sentence class 22. For example, the irrelevant reason feature sets 131a to 131c of the irrelevant reason feature class 130 shown in FIG. 5C are each composed of a set of features extracted from the corresponding irrelevant reason sentences 8a to 8c of FIG. .

The sentence feature extraction unit 13 receives a sentence that is a set of sentences from the model answer feature DB creation unit 11 or the answer reason sentence class information calculation unit 15, extracts features from the received sentence, and extracts a model answer feature DB creation unit 11 or answer reason sentence class information calculation unit 15.
Note that the sentence feature extraction unit 13 may provide a similar dictionary (thesaurus) that is a database in which words are classified, and output the classification in the thesaurus including each word in the sentence as a feature.

When the similarity calculation unit 14 receives a set of two different features from the model answer feature DB creation unit 11 or the answer reason sentence class information calculation unit 15, the similarity calculation unit 14 calculates a similarity indicating the degree of similarity between these sets, The calculation result is output to the model answer feature DB creation unit 11 or the answer reason sentence class information calculation unit 15.
Here, the similarity takes a value between a preset maximum value and minimum value, and the value increases as the degree of similarity increases.

  The similarity is calculated by measuring the number of matching features between two feature sets, obtaining the percentage of matching features in each of the two feature sets, and calculating the ratio of the larger or smaller value. A method of outputting a value, or the number of features that match between sets of features to be compared by setting the maximum number of matches as the maximum value of similarity and the minimum number of matches as the minimum value of similarity in advance in the similarity calculation unit 14 If the value is larger than the maximum value, the maximum value of the similarity set in advance is output. If the value is smaller than the minimum value, the minimum value of the similarity set in advance is output. In other cases, the number of matching features is output as the similarity. And the like.

  Further, when the feature of the sentence is expressed by a word, not only the method described above but also the concept vector of the word may be calculated as the similarity calculation method. A word concept vector is a vector in which a plurality of attributes common to all words to be considered are prepared in advance, and a value representing the weighting of each attribute is prepared for each word. is there. As a method for automatically creating concept vectors, a method using a Japanese dictionary (Kasasa Kasahara, Kazumitsu Matsuzawa, Tsutomu Ishikawa, “Similarity of Daily Words Using a Japanese Language Dictionary”, IPSJ Journal, 1997 , Vol.38, No.7, p.1272-1284) and methods using a corpus such as newspaper articles (Schutze, H, Dimensions of Meaning, Proceedings of Supercomputing 92, 1992, p.787-796) Has been. Such a concept vector is provided in the similarity calculation unit 14 in advance, and the concept vectors corresponding to each of the two feature sets are added to calculate the similarity using the cosine of the angle formed by the two concept vectors. Also good.

When the answer 3 written by the examinee is input, the answer reason sentence class information calculation unit 15 extracts a set of features (hereinafter referred to as answer feature set) from the answer reason sentence 32 of the answer 3 to the document feature extraction unit 13. Let
Next, the answer reason text class information calculation unit 15 refers to the model answer feature DB 12 and sets the answer feature set extracted from the answer reason text 32 and each model reason feature class 120 of each model reason feature class 120 of the model answer feature DB 12. The similarity calculation unit 14 calculates the similarity between the set 121 and the irrelevant reason feature class 130 with each irrelevant reason feature set 131, and based on the calculation result, which model reason feature class 120 or irrelevant reason the answer reason sentence 32 has Whether to classify into the feature class 130 is determined. When the model reason feature class 120 or the irrelevant reason feature class 130 to be classified is determined, the answer reason sentence class information calculation unit 15 includes a set of features extracted from the answer selection sentence 31 of the answer 3 and the answer reason sentence 32 of the answer 3; Information related to the exemplary reason feature class 120 or the unrelated reason feature class 130 (hereinafter referred to as the determined exemplary reason feature class 120 or the determined unrelated reason feature class 130, respectively) for which the classification such as the name is determined is output to the evaluation value calculation unit 17. .

  The method of determining the classification of the answer feature set extracted from the answer reason document 32 of the answer 3 is extracted from the model reason feature set 121 and the unrelated reason feature set 131 included in each class and the answer reason sentence 32 of the answer 3. If it is a method of determining by comparing with a set of features, various methods can be used freely as appropriate. For example, the similarity between the k model reason feature set 121 and the irrelevant reason feature set 131 of each class and the feature set extracted from the answer reason sentence 32 is calculated, and the model taking the maximum similarity of the calculated values. The class to which the reason feature set 121 or the unrelated reason feature set 131 belongs may be determined as the category to which the answer 3 having the answer reason sentence 32 belongs. Here, k means an arbitrary value.

  The evaluation value DB 16 stores an evaluation value upper limit indicating that the answer is most appropriate and an evaluation value lower limit indicating that the answer is most inappropriate. Such an evaluation value lower limit and an evaluation value upper limit are preset by the user. If the evaluation value upper limit is a value larger than the evaluation value lower limit, the values of the evaluation value upper limit and the evaluation value lower limit can be set freely as appropriate.

  The evaluation value calculation unit 17 includes the answer selection sentence 31 of the answer 3 output from the answer reason document class information calculation part 15, the set of features extracted from the answer reason sentence 32 of the answer 3, the decision example reason feature class 120 or the decision Information on the unrelated reason feature class 130 is received, an evaluation value is calculated with reference to the model answer feature DB 12 and the evaluation value DB 16, and a calculation result is output.

As a method for calculating the evaluation value, first, it is confirmed whether or not the answer selection sentence 31 of the answer 3 and the answer selection sentence 6 of the decision example reason feature class 120 match.
If they do not match, the evaluation value calculation unit 17 outputs an evaluation value lower limit. Even when the answer reason sentence 32 is classified into the irrelevant reason feature class 130, the evaluation value calculation unit 17 outputs the evaluation value lower limit.
When the answer selection sentence 31 of the answer 3 and the answer selection sentence 6 of the decision example reason feature class 120 match, the evaluation value calculation unit 17 includes an answer feature set extracted from the answer reason sentence 32 of the answer 3, The similarity calculation unit 14 is caused to calculate the similarity between the decision model reason feature class 120 and the model average feature set 122. The evaluation value calculation unit 17 determines an evaluation value by comparing the calculated similarity with the similarity statistical information 123 of the determination model reason feature class 120.

  As evaluation value determination methods, the similarity between the answer feature set extracted from the answer reason sentence 32 of the answer 3 and the model average feature set 122 of the decision model reason feature class 120 and the similarity of the decision model reason feature class 120 If the statistical information 123 is used and the evaluation value is set between the upper limit of the evaluation value and the lower limit of the evaluation value, various methods can be used as appropriate. For example, using the average value and standard deviation of similarity in the similarity statistical information 123, the calculated similarity is included between a value obtained by adding a certain number of standard deviations to the average value and a value obtained by subtracting the standard deviation. In this case, the upper limit of the evaluation value may be used. In other cases, the lower limit of the evaluation value may be used. As another evaluation value determination method, if the calculated similarity is equal to or greater than the value obtained by subtracting the standard deviation from the average value of the similarity of the similarity statistical information 123, the evaluation value upper limit is set. The lower limit of the evaluation value may be set.

  Such a sentence evaluation device 1 includes an arithmetic device such as a CPU, a storage device such as a memory and an HDD, an I / F device that transmits and receives information to and from the model answer DB 2 and the terminal device of the examinee, an input device such as a keyboard and a mouse. A computer equipped with a display device such as CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), FED (Field Emission Display), or OLED (Electro Luminescence), and a program installed on this computer When the hardware device is controlled by a program, that is, when hardware resources and software cooperate, the above-described model answer feature DB creation unit 11, model answer feature DB12, sentence feature extraction unit 13 The similarity calculation unit 14, the answer reason sentence class information calculation unit 15, the evaluation value DB 16, and the evaluation value calculation unit 17 are realized.

Next, operation | movement of the text evaluation apparatus 1 concerning this Embodiment is demonstrated with reference to FIGS. FIG. 6 is a flowchart showing the operation of the sentence evaluation apparatus 1 according to the present embodiment, and FIG. 7 is a schematic diagram showing a specific example of the similarity calculation operation by the sentence evaluation apparatus 1 according to the present embodiment.
First, with respect to the question sentence 5 shown in FIG. 2, the model answer DB 2 having the model reason sentence classes 21a and 21b and the unrelated reason sentence class 22 shown in FIG. It inputs into the evaluation apparatus 1 (step S601).

The model answer feature DB creation unit 11 of the sentence evaluation device 1 sends the model feature sentence class 21 and the irrelevant reason sentence class 8 of the model reason sentence class 21 and the irrelevant reason sentence class 22 of the inputted model answer DB 2 to the sentence feature extraction unit 13. Features are extracted from the model reason, and an exemplary reason feature set 121 and an unrelated reason feature set 131 are created (step S602).
For example, the sentence feature extraction unit 13 extracts features from the model reason sentences 7a to 7e of the model reason sentence classes 21a and 21b included in the model answer DB 2 shown in FIGS. 3 (a) and 3 (b). ), Exemplary reason feature sets 121a to 121e of exemplary reason feature classes 120a and 120b included in the exemplary answer feature DB 12 shown in FIG. Similarly, the sentence feature extraction unit 13 extracts features from the irrelevant reason sentence 8 in the irrelevant reason sentence class 22 included in the example answer DB 2 shown in FIG. 3C, and the example answer feature DB 12 shown in FIG. 5C. The irrelevant reason feature set 131 of the irrelevant reason feature class 130 included in is generated.

In the present embodiment, the sentence feature extraction unit 13 extracts only nouns from the model reason sentence 7 or the irrelevant reason sentences as features. Therefore, for example, in the case of the model reason sentence 7a of the model reason sentence class 21a shown in FIG. 3A, the sentence feature extraction unit 13 reads “Japan, mountain, landform, rainy season, typhoon, period, rainfall, Extract the characteristics of “dam, disaster prevention, means”. The extracted feature set becomes the model reason feature set 121a of the model reason feature class 120a shown in FIG.
As in the case of the exemplary reason sentence 7a described above, the sentence feature extracting unit 13 also extracts features composed of nouns for the exemplary reason sentences 7b to 7e and the irrelevant reason sentence 8, and the exemplary reason feature sets 121b to 121e and the irrelevant A reason feature set 131 is created.

When the model reason feature set 121 for each model reason feature class 120 is created, the model answer feature DB creation unit 11 creates a model average feature set 122 for each model reason feature class 120 (step S603).
For example, in the model reason feature class 120a shown in FIG. 5A, the model answer feature DB creation unit 11 selects a set of features existing in the model reason feature sets 121a to 121c as a model average feature set 122a. Create as. Similarly, in the model reason feature class 120b shown in FIG. 5B, the model answer feature DB creation unit 11 sets a model average feature set to a set of two or more features in the model reason feature sets 121d and 121e. Created as 122b.

  When the model average feature set 122 is created, the model answer feature DB creation unit 11 causes the similarity calculation unit 14 to create the similarity statistical information 123 for each model reason feature class 120 (step S604). As an example, a method of creating similarity statistical information 123a, b of the exemplary reason feature classes 120a, 120b shown in FIGS. 5 (a), 5 (b) will be described with reference to FIGS. 7 (a), 7 (b).

First, the case of the exemplary reason feature class 120a shown in FIG.
In the present embodiment, the average of the ratio of the number of features common to two feature sets in each feature set is defined as the similarity. Therefore, the similarity between the model reason feature set 121a and the model average feature set 122a is such that the number of features of the model reason feature set 121a is 10, the number of features in common with the model average feature set 122a is 9, and the similarity of the model average feature set 122a Since the number of features is 9, it is calculated by the following formula (1) and becomes 0.95.
(9/10 + 9/9) /2=0.95 (1)
Similarly, the similarity between the model reason feature set 121b and the model average feature set 122a and the similarity between the model reason feature set 121c and the model average feature set 122a are calculated by the following equations (2) and (3), respectively.
(8/10 + 8/9) /2=0.84 (2)
(2/5 + 2/9) /2=0.31 (3)
The average and standard deviation of the three similarities are the average and standard deviation of the similarity statistical information 123a. Therefore, the average of the similarity statistical information 123a is 0.70, and the standard deviation is 0.28.

Similarly, in the case of the model reason feature class 120b shown in FIG. 5B, the similarity between the model reason feature sets 121d and 121e and the model average feature set 122b is calculated by the following equations (4) and (5), respectively. The
(4/11 + 4/4) /2=0.68 (4)
(4/8 + 4/4) /2=0.75 (5)
From these similarities, the average of the similarity statistical information 123b is 0.72 and the standard deviation is 0.03.

  When the above-described similarity statistical information 123 is created, the model answer feature DB 12 is completed.

When the answer 3 created by the examinee is input to the sentence evaluation device 1 after the completion of the model answer feature DB 12, the answer reason sentence class information calculation unit 15 sends the answer reason sentence 32 of the answer 3 to the sentence feature extraction unit 13. The answer feature set is extracted from (step S605).
For example, when the answer 3 as shown in FIG. 4 is input, the sentence feature extraction unit 13 extracts features composed of nouns from the exemplary reason sentences 7a to 7e and the irrelevant reason sentence 8 in the present embodiment as described above. Therefore, the feature of the noun “dam, nature, destruction” is extracted from the answer reason sentence 32 as well.

When the answer feature set of the answer reason sentence 32 is extracted, the answer reason sentence class information calculation unit 15 calculates the similarity of the answer feature set (step S606) and determines the classification of the answer reason sentence 32 (step S606). S607).
In the present embodiment, the classification is determined using the k-nearest neighbor method which is the simplest classification method. The k-nearest neighbor method is a method in which a maximum of k elements are extracted from each classification, the similarity with the classification target is calculated, and the classification of the element with the highest similarity is determined as the target classification. In the present embodiment, the classification is determined using all elements.

  First, the answer reason sentence class information calculation unit 15, as shown in FIG. 7C, includes model reason feature sets 121 a to 121 e and irrelevant reason feature sets 131 a to 131 c of each model reason feature class 120 and irrelevant reason feature class 130. The similarity between each and the answer feature set of the answer reason sentence 32 is calculated.

As an example, the similarity between the model reason feature set 121a and the answer feature set of the answer reason sentence 32 includes ten feature numbers of the model reason feature set 121a, three answer feature sets of the answer reason sentence 32, and model reason features. Since the number of features common to the set 121a and the answer feature set of the answer reason sentence 32 is one, it is calculated by the following equation (6) and becomes 0.22.
(1/10 + 1/3) /2=0.22 (6)

  In the same way, the answer reason sentence class information calculation unit 15 also calculates the similarity between each of the model reason feature sets 121b to 121e and the irrelevant reason feature sets 131a to 131c and the answer feature set of the answer reason sentence 32. Then, each similarity is 0.22 (exemplary reason feature set 121b), 0.27 (exemplary reason feature set 121c), 0.42 (exemplary reason feature set 121d), and 0.46 (exemplary reason feature set 121e). 0.33 (irrelevant reason feature set 131a), 0 (irrelevant reason feature set 131b), and 0.33 (irrelevant reason feature set 131c).

  When each similarity is calculated, the answer reason sentence class information calculation unit 15 classifies the example reason feature class 120b including the example reason feature set 121e having the highest similarity into a class for classifying the answer reason sentences 32, that is, the decision example reason feature. Class 120b is determined.

  As described above, according to the present embodiment, the answer feature set of the answer reason sentence 32 regarding the reason for selecting the answer selection sentence and the reason example reason feature sets 121a to 121e as examples for selecting the answer selection sentence are similar. Since the degree is calculated, that is, the sentences about the reason for selecting the answer selection sentence are compared, it is possible to evaluate the answer using the structural information of the answer and the model answer, and to take the examination for the alternative question Appropriate evaluation can also be made when evaluating a test answer that states the person's answer and the reason for this answer.

When the class for classifying the answer reason sentence 32 is determined, the answer reason sentence class information calculation unit 15 determines the answer feature set and the classification extracted from the answer selection sentence 31 of the answer 3 and the answer reason sentence 32 of the answer 3. Information on the reason feature class 120 or the unrelated reason feature class 130 is output to the evaluation value calculation unit 17 (step S608).
Therefore, when the model reason feature class 120b is determined as a class for classifying the answer reason text 32, the answer reason text class information calculation unit 15 sets the information regarding the model reason feature class 120b and the answer selection text 31 of the answer 3 as “2”. .. Should not be promoted, "answer selection sentence 6 of model reason feature class 120b is" 2. Should not be promoted "and is a set of features extracted from answer reason sentence 32" dam, nature, “Destruction” is sent to the evaluation value calculation unit 17.

  When each piece of information is received from the answer reason sentence class information calculation unit 15, the evaluation value calculation unit 17 calculates an evaluation value (step S609).

First, the evaluation value calculation unit 17 confirms whether or not the two answer selection sentences of the answer selection sentence 31 of the answer 3 and the answer selection sentence 6 of the decision model reason feature class 120 match.
In the present embodiment, the answer selection sentence 31 of the answer 3 is “2. Should not be promoted” and the answer selection sentence 6 of the model reason feature class 120b is “2. It can be seen that the two answer selection sentences match.

When the two answer selection sentences match, the evaluation value calculation unit 17 gives half of the difference between the evaluation value upper limit and the evaluation value lower limit as a partial evaluation value.
When the two answer selection sentences do not match, the evaluation value calculation unit 17 outputs the evaluation value lower limit to the outside.
For example, if the evaluation value upper limit is set to 10 and the evaluation value lower limit is set to 0 and the evaluation value DB 16 is preset, when the model reason feature class 120b is determined as a class for classifying the answer reason sentence 32, the two answer selection sentences match. Therefore, 5 points are given as partial evaluation points.

  As described above, according to the present embodiment, when the answer selection sentences do not match, the answer 3 is evaluated with the lowest evaluation, so that the suitability of the answer selection sentence 31 and the answer reason sentence 32 is evaluated. As a result, an appropriate evaluation can be performed.

The evaluation value calculation unit 17 also outputs the evaluation value lower limit even when the evaluation value calculation unit 17 is classified into the unrelated reason feature class 130 of the answer reason sentence 32.
As described above, according to the present embodiment, when the irrelevant reason feature class 130 is specified, the lowest evaluation is given to the answer 3, so that it is prepared in advance for descriptions of various sentences concerning the same reason. Since the evaluation value of the answer can be calculated appropriately depending on whether it is representative in the collection of sentences for the reason why it is not correct or not, it is possible to evaluate appropriately as a result It becomes.

Subsequently, when the two answer selection sentences match, the evaluation value calculation unit 17 calculates the similarity between the model average feature set 122 of the decision model reason feature class 120 and the answer feature set of the answer reason sentence 32. To do.
As shown in FIG. 7D, in the case of the decision example reason feature class 120b, the features of the model average feature set 122b are “dam, construction, surrounding, environment”, and the feature of the answer reason sentence 32 is “dam, Since “Nature, destruction” and one of the common features are “dams”, the similarity between the model average feature set 122b and the answer feature set of the answer reason sentence 32 is calculated by the following equation (7). 0.29.
(1/4 + 1/3) /2=0.29 (7)

In the present embodiment, the calculated similarity is compared with the similarity statistical information 123 to determine the evaluation value. However, the partial point is larger than the evaluation value lower limit, and the similarity is based on the average of the similarity statistical information 123. If it is equal to or greater than the value obtained by subtracting the standard deviation, 5 points will be given to the evaluation score.
Therefore, the similarity 0.29 between the model average feature set 122b and the answer feature set of the answer reason sentence 32 is a value obtained by subtracting the standard deviation from the average of the similarity statistical information 123b, that is, 0.72-0.03 = 0. Is smaller than .69, the evaluation value calculation unit 17 outputs the five partial points given first as the evaluation value 4 for the answer 3 (step S610).

As described above, according to the present embodiment, there are several options in the sentence format answer, and even if it is a test like an essay that is correct as long as an appropriate reason can be presented for each option, it corresponds to the option By preparing in advance a sentence relating to the reason, it is possible to automatically and appropriately calculate the evaluation of the answer according to each.
In addition, regarding the description of various sentences concerning the same reason, it is possible to appropriately calculate the evaluation value of the answer depending on whether it is representative or exceptional in the collection of sentences of the reason prepared in advance. It becomes possible.

It is a block diagram which shows the structure of the text evaluation apparatus concerning this Embodiment. It is a schematic diagram for demonstrating a question. It is a schematic diagram for demonstrating (a) exemplary reason sentence class 21a, (b) exemplary reason sentence class 21b, and (c) irrelevant reason sentence class 22. 6 is a schematic diagram for explaining Answer 3; FIG. 4A and 4B are schematic diagrams for explaining an exemplary reason feature class 120a, (b) an exemplary reason feature class 120b, and (c) an irrelevant reason feature class 130. FIG. It is a flowchart which shows operation | movement of the text evaluation apparatus 1 concerning this Embodiment. (A) A figure explaining calculation operation of each numerical value of the model reason feature class 120a, (b) A figure explaining calculation operation of each numerical value of the model reason feature class 120b, (c) Each feature set and answer reason sentence 32, The figure explaining the calculation operation | movement of the degree of similarity, (d) The figure explaining the calculation operation | movement of the similarity degree of the model average feature set 122b and the answer reason sentence 32.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Text evaluation apparatus, 2 ... Model answer DB, 3 ... Answer, 4 ... Evaluation value, 5 ... Question text, 6, 6a, 6b ... Selected text, 7, 7a-7e ... Model reason text, 8, 8a-8c ... unrelated reason sentence, 11 ... model answer feature DB creation unit, 12 ... model answer feature DB, 13 ... sentence feature extraction unit, 14 ... similarity calculation unit, 15 ... answer reason sentence class information calculation unit, 16 ... evaluation value DB , 17 ... evaluation value calculation unit, 21 ... model reason sentence class, 22 ... unrelated reason sentence class, 31 ... answer selection sentence, 32 ... answer reason sentence, 120, 120a, 120b ... model reason feature class, 121, 121a to 121e ... model reason feature set, 122, 122a, 122b ... model average feature set, 123, 123a, 123b ... similarity statistical information, 130 ... unrelated reason feature class, 131 ... unrelated reason feature set.

Claims (12)

In a sentence evaluation device for evaluating an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating a reason for selecting the selected sentence,
A model answer feature DB storing a plurality of model answer feature class including the selected sentence and a model reason feature set consisting of features of the model reason sentence indicating a model reason for selecting the selected sentence;
Extraction means for extracting features from the selection reason sentence;
The similarity between the example reason sentence and the selection reason sentence is calculated based on the example reason feature set and the feature extracted by the extraction means, and the example reason sentence including the example reason sentence having the highest degree of similarity is calculated. A calculation means for identifying the answer feature class;
A sentence evaluation apparatus comprising: evaluation means for evaluating the answer based on the model answer feature class specified by the calculation means. The evaluation means includes
A determination unit that determines whether or not the selected sentence included in the answer matches the selected sentence included in the exemplary answer feature class specified by the calculation unit;
The sentence evaluation apparatus according to claim 1, wherein if the determination unit determines that they do not match, the lowest evaluation is made on the answer. The evaluation means includes
If it is determined that the determination means match, the feature obtained from the model reason feature set included in the model answer feature class specified by the calculation means and the similarity between the feature of the answer reason sentence based on the feature The sentence evaluation apparatus according to claim 2, wherein the answer is evaluated. In a sentence evaluation device for evaluating an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating a reason for selecting the selected sentence,
A plurality of exemplary answer feature class including an exemplary reason feature set consisting of features of the exemplary reason sentence indicating the selected sentence and an exemplary reason for selecting the selected sentence, and a non-exemplary reason for selecting the selected sentence A model answer feature DB storing an irrelevant reason feature class including an irrelevant reason feature set consisting of features of an irrelevant reason sentence;
Extraction means for extracting features from the selection reason sentence;
The similarity between the model reason sentence and the irrelevant reason sentence and the selection reason sentence is calculated based on the model reason feature set, the irrelevant reason feature set and the feature extracted by the extraction unit, and the similarity Calculating means for identifying the model answer feature class or the irrelevant reason feature class that includes the highest model reason sentence or the irrelevant reason sentence;
A sentence evaluation apparatus comprising: evaluation means for evaluating the answer based on the model answer feature class specified by the calculation means or the unrelated reason feature class. The sentence evaluation apparatus according to claim 4, wherein when the irrelevant reason feature class is specified by the calculation means, the evaluation means performs the lowest evaluation on the answer. A model answer DB storing at least one of the plurality of model reason sentence classes including the model sentence and the model reason sentence corresponding to the selected sentence, and the irrelevant reason sentence class including the irrelevant reason sentence;
The sentence evaluation apparatus according to claim 1, further comprising: a model answer feature DB creating unit that creates the model answer feature DB from the model answer DB. In a sentence evaluation method for evaluating an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating a reason for selecting the selected sentence,
A creation step of creating a model answer feature DB that stores a plurality of model answer feature feature classes including the selected sentence and a model reason feature set composed of features of the model reason sentence indicating a model reason for selecting the selected sentence; ,
An extraction step of extracting features from the selection reason sentence;
The similarity between the example reason sentence and the selection reason sentence is calculated based on the example reason feature set and the feature extracted by the extraction step, and the example reason sentence including the example reason sentence having the highest degree of similarity is calculated. A calculation step for identifying an answer feature class;
A sentence evaluation method comprising: an evaluation step for evaluating the answer based on the model answer feature class specified by the calculation step. The evaluation step includes
A determination step of determining whether or not the selected sentence included in the answer matches the selected sentence included in the model answer feature class identified by the calculation step;
The sentence evaluation method according to claim 7, wherein, when it is determined that they do not coincide in the determination step, the lowest evaluation is performed on the answer. The evaluation step includes
When it is determined that they match in the determination step, based on the similarity between the feature obtained from the model reason feature set included in the model answer feature class specified by the calculation step and the feature of the answer reason sentence The sentence evaluation method according to claim 8, wherein the answer is evaluated. In a sentence evaluation method for evaluating an answer composed of one selected sentence arbitrarily selected from a plurality of selected sentences and a selection reason sentence indicating a reason for selecting the selected sentence,
A plurality of exemplary answer feature class including an exemplary reason reason feature set including features of the exemplary reason sentence indicating the selected sentence and an exemplary reason for selecting the selected sentence, and a non-exemplary reason for selecting the selected sentence A creation step for creating an exemplary answer plan feature DB storing an irrelevant reason feature class including an irrelevant reason feature set consisting of features of an irrelevant reason sentence;
An extraction step of extracting features from the selection reason sentence;
The similarity between the model reason sentence and the irrelevant reason sentence and the selection reason sentence is calculated based on the model reason feature set, the irrelevant reason feature set and the feature extracted by the extraction step, and the similarity Calculating the exemplary answer feature class or the irrelevant reason feature class that includes the highest example reason sentence or the irrelevant reason sentence;
A sentence evaluation method comprising: an evaluation step for evaluating the answer based on the model answer feature class or the unrelated reason feature class specified by the calculation step. The sentence evaluation method according to claim 10, wherein the evaluation step performs the lowest evaluation on the answer when the irrelevant reason feature class is specified in the calculation step. The creating step includes
The model answer feature from the model answer DB storing at least one of the selected sentence and a plurality of model reason sentence classes including the model reason sentence corresponding to the selected sentence and the irrelevant reason sentence class including the irrelevant reason sentence. The sentence evaluation apparatus according to claim 7, wherein a DB is created.

Images