JP2010224323A - Thesis content evaluation device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a thesis content evaluation device dispensing with data regulation for giving marks even when changing a problem sentence. <P>SOLUTION: This thesis content evaluation device 10 includes a mutual information volume database 24 stored with a mutual information volume, a reverse appearance frequency database 25 stored with a reverse appearance frequency, a score calculating means 27 for calculating a score expressing a relevancy between each word in the thesis and the problem sentence, by substitution with data extracted from the respective databases 24, 25 in a preliminarily stored mathematical expression, a score table preparing means 28 for preparing a score table, based on a score found in each combination of the words, a content evaluation database 30 stored with a plurality of kinds of reference score tables different respectively in patterns of distribution state, and personal content evaluations corresponding to the reference score tables, and a content evaluation determining means 31 for extracting the reference score table most akin to the score table prepared by the score table preparing means 28, and for determining the content evaluation corresponding to the reference score table as the content evaluation of the thesis. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a paper content evaluation apparatus and a paper content evaluation program, and more specifically, automatically evaluates the content of a paper created by a learner in the same language as the question text based on the question text as the theme of the paper. The present invention relates to a paper content evaluation apparatus and a paper content evaluation program.

  A sentence evaluation scoring device that automatically scores the content of a paper created by a learner based on a certain theme has been proposed (see Patent Document 1). In this sentence evaluation scoring device, the degree of approximation between the question sentence and the essay to be graded is obtained by the following procedure, and the essay is evaluated based on the degree of approximation. That is, a number of documents such as newspaper articles are collected in advance to create a word-document matrix, and a singular value decomposition is performed on the matrix to obtain a matrix representing the relationship between words and documents. Then, by using the matrix, the word vector constituting the question sentence and the word vector constituting the essay are each converted into a document vector. Thereafter, the degree of approximation of both document vectors is obtained by a technique called a cosine scale, and is scored. The degree of approximation becomes higher as each document vector is closer.

JP 2004-151757 A

  However, the sentence evaluation scoring device has a problem that according to a paper showing an evaluation related to the technology, the sentence cannot be scored well unless it is a long problem sentence of about 100 to 200 characters. The sentence evaluation scoring apparatus also has a problem that the scoring result of the essay does not necessarily match the human scoring result. In other words, in the sentence evaluation scoring system, scoring is performed based on the degree of approximation between the question text and the essay to be scored, without any scorer's viewpoint, and scoring close to that of the scorer. I can't. For example, when each word in an essay created by a learner is distinguished as a word having a high degree of relevance to each word in the question sentence, a medium word, a low word, etc., the word having a high relevance and a word having a low relevance However, in the case where the number of intermediate words is small, there is a possibility of giving a negative image to the scorer because the content is not unified and the flow of discussion is poor. However, in contrast to this case, a paper with a small number of highly related words and low words but a large number of medium words has the same degree of approximation as the previous case in the sentence evaluation scoring device. May be determined as a better score than the previous case by scoring. Therefore, if the subjective evaluation criteria of the grader is not reflected in some way as the logic for automatic evaluation of the paper, it may be misaligned with the grader's scoring result. There is a risk of being. Here, in order to put the grader's point of view into the document evaluation scoring apparatus of Patent Document 1, collection of a short essay as a reference and scoring are required for each type of question, so it takes time to construct basic data. It becomes complicated and it becomes difficult to easily change question sentences with different variations.

  The present invention has been devised by paying attention to such problems, and the purpose thereof is not to newly perform scoring data adjustment when changing the question sentence, and the question sentence can be reduced at low cost. An object of the present invention is to provide a paper content evaluation apparatus and a paper content evaluation program that can be set freely and can obtain an evaluation result close to the human content evaluation of a paper evaluator.

(1) In order to achieve the above object, the present invention provides a paper content evaluation apparatus that automatically evaluates the content of a paper created by a learner in the same language as the question text based on the question text as the theme of the paper. ,
A mutual information database in which a mutual information amount serving as an index indicating the possibility of a pair of words appearing in the same document is stored for each combination of words;
A reverse appearance frequency database in which the reverse appearance frequency is stored for each word, the numerical value of which is lower for general words appearing in many documents;
Score calculating means for obtaining a score representing the degree of association between one word in the paper and the question sentence by substituting data extracted from the mutual information database and the reverse appearance frequency database into a mathematical formula stored in advance;
From the score obtained for each word in the paper by the score calculation means, a score table creation means for creating a score table representing the distribution status of the score;
A content evaluation database in which a plurality of types of reference score tables having different patterns of distribution status and human content evaluations corresponding to the respective reference score tables are stored,
A content evaluation determination unit that extracts a reference score table of the content evaluation database close to the score table created by the score table creation unit, and uses a content evaluation corresponding to the reference score table as a content evaluation of the paper , Is adopted.

(2) Further, it includes a paper counting means including a function of counting the number of occurrences of each word in the paper, and the score calculation means includes the mutual information amount, the reverse appearance frequency, the number of occurrences of the word, , A score calculation formula using the number of types of words constituting the question sentence as a parameter is stored in advance, and the score calculation formula includes a pair of target words including one word in the question sentence and one word in the paper. The mutual information extracted from the mutual information database, the reverse appearance frequency extracted from the reverse appearance frequency database for one word in the paper among the pair of target words, and the pair of target words Of these, the number of occurrences counted by the paper counting means for one word in the paper and the number of types of words in the question sentence counted in advance are substituted. It adopts a configuration in which the score is calculated.
It is preferable.

  (3) Further, there is a configuration in which a question sentence counting means for counting the number of types of words constituting the question sentence is provided, and the number of word types counted by the question sentence counting means is substituted into the score calculation formula. It can be employed together.

  (4) Here, it is provided with a stop word database in which stop words that are frequently excluded from various documents and stored as a part of processing are stored, and the question counting unit is stored in the stop word database. A configuration may be adopted in which the number of types of words in the question sentence is counted except for stop words.

  (5) In addition, a stop word database in which stop words to be excluded at the time of partial processing of words that frequently appear in various documents are stored, and the score calculation means uses the stop word database based on the stop word database. It is preferable that the score is calculated for all combinations of one word in the paper except for and one word in the question sentence except for the stop word.

  (6) Furthermore, in the content evaluation determination means, the score table of the article to be content evaluation is compared with each of the reference score tables stored in the content evaluation database, respectively, and a preset pattern recognition method is set. Thus, a configuration is adopted in which the reference score table having a pattern close to the score table created by the score table creating means is selected.

(7) Further, the present invention provides a program for causing a computer to execute a process of automatically evaluating the content of a paper created by a learner in the same language as the question text based on the question text as a theme of the paper. ,
A mutual information database in which a mutual information amount serving as an index indicating the possibility of a pair of words appearing in the same document is stored for each combination of words;
A reverse appearance frequency database in which the reverse appearance frequency is stored for each word, the numerical value of which is lower for general words appearing in many documents;
Score calculating means for obtaining a score representing the degree of association between one word in the paper and the question sentence by substituting data extracted from the mutual information database and the reverse appearance frequency database into a mathematical formula stored in advance;
From the score obtained for each word of the paper by the score calculation means, a score table creation means for creating a score table representing the distribution status of the score;
A content evaluation database in which a plurality of types of reference score tables having different patterns of distribution status and content evaluations of articles corresponding to the respective reference score tables are stored;
Extracting a reference score table of the content evaluation database close to the score table created by the score table creating means, and using the computer as a content evaluation deciding means for making a content evaluation corresponding to the reference score table a content evaluation of the paper The structure is made to function.

  According to the present invention, instead of comprehensively obtaining a single value from the combination of each word constituting the question and each word of the paper created by the learner, the contents are evaluated, and the distribution status of each score is represented. After creating the score table, select a reference score table that is similar to the score table of the paper and the distribution pattern from the reference score tables stored in advance corresponding to human content evaluation, and the reference score The content evaluation linked to the table is the content evaluation of the paper. That is, the content evaluation of a paper is determined by comparing the score pattern itself shown on the score table regardless of the content of the question. For this reason, even if the question sentence changes, it is not necessary to change the data in the content evaluation database, and the content evaluation of the learner's paper is performed with the data as it is. Therefore, it is not necessary to create a reference score table for each question sentence, and the question sentence can be easily changed at a low cost on the questioner side. In addition, when creating a standard score table from existing papers in the same procedure as the score table creating means, at the same time, the scorer personally evaluates the contents of the existing papers, and this person By associating general content evaluation with the reference score table, it is possible to obtain an evaluation result close to the human scoring evaluation of the paper evaluator. Further, according to the present invention, accurate scoring is possible even if the question sentence is composed of a small number of words.

1 is a schematic system configuration diagram of a paper content evaluation apparatus according to the present embodiment. The image figure of a score table. Image of content evaluation database.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a schematic system configuration diagram of a paper content evaluation apparatus according to this embodiment. In this figure, a paper content evaluation apparatus 10 is an apparatus that is used for education of English paper preparation by learners such as students and automatically evaluates the content of English papers created by the learner. This paper content evaluation apparatus 10 is constituted by a computer including an arithmetic processing unit such as a CPU and a storage device such as a memory and a hard disk, and a program for causing the computer to function as each unit to be described later is installed. That is, the paper content evaluation apparatus 10 according to the present embodiment functions as a server connected to a terminal 11 such as a personal computer owned by a learner via a data line such as the Internet line, and a paper created by the learner is created. When received, the content of the paper is automatically evaluated and the evaluation result is transmitted to the learner.

  Specifically, the paper content evaluation apparatus 10 extracts a question sentence database 13 storing a plurality of types of English question sentences as a theme of the paper and the question sentence database 13 in response to a request from the terminal 11. The question sentence extraction means 14 to be transmitted, the question sentence transmission means 16 to transmit the question sentence to the terminal 11, and the learner creates a paper in English based on the question sentences received by the terminal 11 and is input to the terminal 11. The receiving means 17 for receiving the paper, the stop word database 19 in which stop words to be excluded at the time of processing of some words frequently appearing in various documents, and the number of types of words constituting the question sentence The question sentence counting means 21 for counting the number of words, the paper counting means 22 for counting the number of words in the paper, the number of sentences, etc. A mutual information database 24 in which the mutual information amount is stored for each word combination, a reverse appearance frequency database 25 in which a reverse appearance frequency representing the reciprocal of the frequency of words appearing in the document is stored for each word, and a paper Score calculating means 27 for obtaining a score representing the degree of association between each word in the sentence and the set of questions, and a score representing the distribution status of the score from the scores obtained for each word in the paper by the score calculating means 27 A score table creating means 28 for creating a table, a content evaluation database 30 in which various reference score tables having different patterns of score distribution patterns and content evaluations of articles are stored correspondingly, and a score table creating means 28 Content evaluation determination means 31 for determining content evaluation of a paper created by a learner based on the content evaluation database 30 from the generated score table, It is constructed and a evaluation result transmitting means 33 for transmitting to the terminal 11 of the learner content rating determined by the evaluation determination unit 31 via the data line.

  The question sentence database 13 stores a large number of question sentences that describe in English the theme of a paper determined by the questioner. The question text is not particularly limited, but is set for each category such as the genre of the theme in addition to the learner's grade and level. Note that the question sentence database 13 may have a function of sequentially adding new question sentences.

  When a request is received from the learner's terminal 11, the question sentence extraction means 14 extracts a question sentence from a category corresponding to the request at random or according to a certain rule, and extracts the question sentence as a question sentence. Transmission is performed from the transmission means 16 to the terminal 11.

  The stop word database 19 includes words that appear frequently in most documents, for example, articles such as “a” and “an”, prepositions such as “in” and “on”, “it”, “that”, and the like. Frequent words such as pronouns, question words such as “who” and “who”, be verbs such as “is” and “are”, and auxiliary verbs such as “can” and “may” are stored in advance.

  The question counting means 21 counts the number of types of words in the question sentence excluding the stop words stored in the stop word database 19 after identifying the words constituting the question sentence using the space in the question sentence as a clue. It is supposed to be. That is, even when the same word repeatedly appears in the question, the word is counted as “1”.

  The paper counting means 22 has a function of identifying words constituting the paper with a clue to a space in the paper, a function of counting the total number of words in the paper after the word is specified, and a paper after the word is specified. A function for counting the number of appearances of each word in the document, a function for counting the number of types of words in the paper after the word is specified, and a function for counting the number of sentences constituting the paper. Here, the number of appearances of each word in the paper is counted for each word, how many times the same word appears in the paper. In addition, the number of types of words in the paper is counted as “one type” by summing up the same words appearing in the paper in duplicate, similarly to the counting by the question counting means 21. It is obtained by counting words that do not appear as “one type” per word. In addition, the number of sentences composing a paper is obtained by detecting any period, exclamation mark, or question mark in the paper, and counting the entire paper as a single sentence divided by the period. It is done.

  In the mutual information database 24, the mutual information of a pair of words in a considerable number of English words that the learner may use is stored for each combination of words. The mutual information amount is obtained in advance from English words described in many documents of various genres such as newspapers and magazines using an arithmetic device (not shown) such as a computer as follows.

ここで、ｐ（ｗ_ｉ，ｗ_ｊ）、ｐ（ｗ_ｉ）、ｐ（ｗ_ｊ）は、次式（２）、（３）で表される。

上式（２）、（３）において、「Ｎ」は、相互情報量を求めるために使用した全文書数である。また、「ｎ_ｉｊ」は、前記全文書の中で、単語「ｗ_ｉ」と単語「ｗ_ｊ」の組み合わせが存在する文書数である。つまり、この「ｎ_ｉｊ」は、ある一文書内に単語「ｗ_ｉ」と単語「ｗ_ｊ」の組み合わせが複数存在していても、一文書で「１」とカウントされる。
例えば、全文書数Ｎが１０００であるとして、その中で、単語「ｗ_ｉ」と単語「ｗ_ｊ」が共に登場している文書数ｎ_ｉｊが５０であれば、
ｐ（ｗ_ｉ，ｗ_ｊ）＝５０／１０００＝０．０５
となる。
なお、「Ｎ」を全文書中にそれぞれ存在する文の総数とし、「ｎ_ｉｊ」を、全文書中で、単語「ｗ_ｉ」と単語「ｗ_ｊ」の組み合わせが存在する文の数としてもよい。
また、ｐ（ｗ_ｉ）、ｐ（ｗ_ｊ）を求める上式（３）の分母は、全文書中に存在する単語の総数を意味し、「ｆ_ｉ」は、全文書中での単語「ｗ_ｉ」の出現数であり、「ｆ_ｊ」は、全文書中での単語「ｗ_ｊ」の出現数である。 That is, the mutual information MI between a certain word “w _i ” and another word “w _j ” is obtained by the following equation (1). In each of the following mathematical formulas, the logarithm is not limited to a natural logarithm, and various bases can be taken.

Here, p (w _i , w _j ), p (w _i ), and p (w _j ) are expressed by the following equations (2) and (3).

In the above formulas (2) and (3), “N” is the total number of documents used for obtaining the mutual information amount. Also, “n _ij ” is the number of documents in which a combination of the word “w _i ” and the word “w _j ” exists in all the documents. That is, “n _ij ” is counted as “1” in one document even if a plurality of combinations of the word “w _i ” and the word “w _j ” exist in one document.
For example, assuming that the total number N of documents is 1000, if the number of documents n _ij in which the word “w _i ” and the word “w _j ” appear together is 50,
p (w _i , w _j ) = 50/1000 = 0.05
It becomes.
Note that “N” is the total number of sentences existing in all the documents, and “n _ij ” is the number of sentences in which the combination of the word “w _i ” and the word “w _j ” exists in all the documents. Good.
Further, the denominator of the above equation (3) for obtaining p (w _i ) and p (w _j ) means the total number of words existing in all documents, and “f _i ” represents the word “ “w _i ” is the number of appearances, and “f _j ” is the number of occurrences of the word “w _j ” in the entire document.

ここで、「Ｎ」は、相互情報量を求める場合と同様、逆出現頻度を求めるために使用した全文書数であり、「ｎ_ｉ」は、当該全文書の中で、単語「ｗ_ｉ」が含まれている文書数である。つまり、この「ｎ_ｉ」は、ある一文書内に単語「ｗ_ｉ」がいくつ存在していても、一文書で「１」とカウントされる。
上式（４）により、逆出現頻度ｉｄｆ（ｗ_ｉ）は、多くの文書に出現する一般的な単語ほど低い数値となる。 The reverse appearance frequency stored in the reverse appearance frequency database 25 is an arithmetic operation (not shown) of a computer or the like from English words described in many documents of various genres such as newspapers and magazines, similar to the mutual information amount described above. Pre-determined using the device. Specifically, the reverse appearance frequency idf (w _i ) of the word “w _i ” is obtained by the following equation (4).

Here, “N” is the total number of documents used for obtaining the reverse appearance frequency, as in the case of obtaining the mutual information amount, and “n _i ” is the word “w _i ” in all the documents. Is the number of documents that contain That is, this “n _i ” is counted as “1” in one document, regardless of how many words “w _i ” exist in one document.
According to the above equation (4), the reverse appearance frequency idf (w _i ) becomes a lower numerical value for general words appearing in many documents.

上式において、「ＭＩＤＦ（ｗ_ｉ，Ｔ）」は、ストップワードを除く出題文中の単語の集合「Ｔ」と、論文中の単語「ｗ_ｉ」とにおけるスコアである。また、「ＭＩ（ｗ_ｉ，ｗ_ｊ）」は、出題文の集合「Ｔ」中の単語「ｗ_ｊ」と論文中の単語「ｗ_ｉ」における前述の相互情報量であり、「｜Ｔ｜」は、ストップワードを除く出題文中の単語の種類数である。更に、「ｉｄｆ（ｗ_ｉ）」は、単語「ｗ_ｉ」における前述の逆出現頻度であり、「ｅ_ｉ」は、単語「ｗ_ｉ」の論文中の出現数である。 In the score calculation means 27, among the words in the paper created by the learner, all the words except the words (stop words) stored in the stop word database 19 are related to the words in the question sentence excluding the stop words. Each score obtained based on the above is calculated. That is, in the score calculation means 27, the score calculation formula of the following equation (5) using the mutual information amount, the reverse appearance frequency, the number of words appearing in the paper, and the number of types of words in the question sentence as parameters. Are stored in advance, and the score is calculated by substituting each value of the parameter into the score calculation formula.

In the above formula, “MIDF (w _i , T)” is a score in a set of words “T” in the question sentence excluding stop words and a word “w _i ” in the paper. In addition, "MI _(w i, w _j)" is the mutual information of the above-mentioned in the word "w _i" of the set and in the article "T" word "w _j" in the question sentence, "| T | "Is the number of types of words in the question sentence excluding stop words. Furthermore, “idf (w _i )” is the above-described reverse appearance frequency of the word “w _i ”, and “e _i ” is the number of occurrences of the word “w _i ” in the paper.

That is, herein, for each word in the article and the word "w _j" in question statement of interest "w _i", mutual information in each of the combinations "MI (w i, _{w j)"} mutual information amount In addition to being extracted from the database 24, the reverse appearance frequency “idf (w _i )” of the word “w _i ” in the paper is extracted from the reverse appearance frequency database 25. Then, together with each mutual information amount “MI (w _i , w _j )” and reverse appearance frequency “idf (w _i )”, the number of types of words in the question sentence counted by the question sentence counting means 21 “| T |” When, the number of occurrences in the article of paper counting means 22 counted word "w _i" and "e _i", but the substituted into the score computing equation, for every word in the article "w _i" score "MIDF (W _i , T) ”is required. According to the above equation (5), this score is obtained by adding each mutual information amount in all combinations with each word “w _j ” in the question sentence for each word “w _i ” in the paper, One value is obtained for each word “w _i ” in the paper.

  The score table creating means 28 creates a score table as follows.

In this score table, as shown in FIG. 2, the horizontal axis indicates the score, and the vertical axis indicates the counted number (number of words). The score on the horizontal axis is divided every predetermined numerical width from zero, and the numerical width d of each section is set in advance uniformly and uniformly. Here, the numerical value width d is obtained by the following equations (6) to (8) based on many papers prepared in advance for a predetermined question sentence. Here, “L” in equation (6) is a predetermined number of categories (for example, 10 categories) set in advance regardless of the paper, and “V” is expressed by the above equation (5) for each word of each paper. It means a set whose elements are score (MIDF) value “v” for each word in each paper obtained in advance.

  That is, Expression (7) is an expression for calculating the average of all scores “v” for each word obtained from each paper, where “| V |” is the total number of obtained scores. Is a mathematical formula for obtaining the unbiased variance of all scores “v” for each word obtained from each paper.

  As described above, the score obtained by the score calculation means 27 for each word in the paper is reflected in the score table in the score table of the numerical value d determined in advance as described above. That is, a large number of obtained scores are counted for each corresponding category and plotted in a score table like a bar graph, and the graph pattern is stored as a score table for a paper created by the learner. Further, each number counted by the paper counting means 22, that is, the total number of words constituting the paper, the number of types of words in the paper, and the number of sentences constituting the paper are also reflected in the score table (the right end in FIG. 2). portion). Note that any or all of these numbers may not be reflected in the score table.

  The content evaluation database 30 corresponds to the reference score table created as described above and each reference score table for each of a number of papers created by a large number of learners for one or more types of questions. An evaluation value obtained as a result of personal evaluation of each paper by a grader is stored in correspondence. In other words, the content evaluation database 30 stores reference score tables having different patterns of score distribution states corresponding to the evaluation values obtained by humans. Although this evaluation value is not particularly limited, various aspects such as a five-step evaluation from 1 to 5, a score evaluation from 0 to 100 points, a two-step evaluation only for pass and fail, etc. Can be adopted. For example, FIG. 3 shows an image of stored data in the content evaluation database 30 when there are five evaluation values.

  The content evaluation determination unit 31 compares the score table of the paper that is the content evaluation target created by the score table creation unit 28 with all the reference score tables stored in the content evaluation database 30, and the content evaluation target The reference score table of the content evaluation database 30 having a similar score distribution pattern to the score table of the paper to be extracted is extracted, and the evaluation value stored corresponding to the extracted reference score table is the learner's score. The content of the paper is evaluated. Here, the approximate determination of the pattern of each score table is performed using a known pattern recognition method set in advance, such as a support vector machine (SVM), a naive Bayes classifier, a cosine scale, a K neighborhood method, or the like. Here, in addition to extracting the reference score table with the closest score distribution pattern to the score table to be evaluated, multiple reference score tables that are close within a certain range to the score table to be evaluated are extracted. It is also possible to select the evaluation value that exists most frequently among the corresponding evaluation values, and use the evaluation value as the content evaluation of the learner's paper. Also, for each evaluation value, a representative point that is closest to the boundary of each evaluation value divided by the pattern recognition method is set, and a reference score table for each representative point that is closest to the score table to be evaluated is specified, The evaluation value corresponding to the reference score table can be used as the content evaluation of the paper. As described above, various methods using the pattern recognition method can be adopted as long as a reference score table having a pattern similar to the score table of the paper can be extracted.

  The evaluation result transmission means 33 transmits the evaluation value determined by the content evaluation determination means 31 to the learner's terminal 11 as the content evaluation result.

  Next, the flow of the content evaluation of the paper in the paper content evaluation apparatus 10 of the present embodiment will be described.

  First, when a learner requests a question to the paper content evaluation apparatus 10 from his / her terminal 11, an appropriate question sentence is extracted from the question sentence database 13 by the question sentence extraction means 14, and the question sentence transmission means 16 To the learner's terminal 11. The learner creates an English paper while writing English based on his / her thoughts on an English question (for example, “My Family”) as the theme of the paper. After completing the paper creation, the learner inputs the paper into his / her terminal 11 and transmits it to the paper content evaluation apparatus 10. The learner's paper received by the receiving means 14 constitutes the total number of words in the paper, the number of appearances of each word in the paper, the number of types of words in the paper, and the paper by the paper counting means 22. The number of sentences is counted. Before and after this, the question sentence counting means 21 counts the number of types of words excluding the stop words stored in the stop word database 19 among the constituent words. The number of types of words in the question sentence counted here may be stored in advance in the question sentence database 13 corresponding to the question sentence.

  Further, the score calculation means 27 obtains a score for each word in the paper excluding the stop word. At this time, the mutual information amount of the word in the question sentence and the word in the paper is extracted from the mutual information amount database 24, and the reverse appearance frequency of the word in the paper is extracted from the reverse appearance frequency database 25. Then, for the words in the question and the words in the paper, the mutual information amount of the pair of words, the reverse appearance frequency of the word in the paper of the pair of words, and the paper counted in the paper counting means 22 And the number of types of words of the question sentence counted by the question sentence counting means 21 are substituted into the above-described score calculation formula to obtain a score.

  For example, when the question sentence is “The holiday of the family”, the score is obtained for one word in the paper created by the learner, for example, “Mother”. First, the stop word database 19 identifies the words other than the words “holiday” and “family” as stop words in the question. Then, the mutual information amount of each word other than the stop word in the question sentence extracted from the mutual information amount database 24, that is, the mutual information amount of the word “holiday” and the word “Mother”, the word “Family” and the word “ After adding the mutual information of “Mother”, the reverse appearance frequency of the word “Mother” in the paper extracted from the reverse appearance frequency database 25 and the word “Mother” in the paper counted by the paper counting means 22 are added. "And the number of types of words constituting the question counted" 2 "counted by the question counting means 21 are substituted into the above-mentioned score calculation formula, and a score is obtained for the word" Mother "in the paper. It is done. Similarly, for each of the other words in the paper, the scores in the set of words (word “holiday” and word “Mother”) excluding the stop word in the question are obtained. In addition, it is preferable to perform the above-described processing on the uppercase and lowercase information of each word in the question sentence and in the paper as the same character information.

  In this way, a large number of scores are obtained from the paper to be evaluated, and each of the obtained scores is stored in a score table having a predetermined numerical value range d by the score table creating means 28. Voted. The total number of words in the paper counted by the paper counting means 22, the number of types of words in the paper, and the number of sentences constituting the paper are also reflected in the score table.

  The score table created as described above is compared with the reference score table stored in advance in the content evaluation database 30, and the content evaluation pattern close to the content evaluation pattern by the content evaluation determination unit 31 is performed. The reference score table of the database 30 is determined, and the evaluation value stored corresponding to the determined reference score table is used as the evaluation value of the paper created by the learner. For example, when the score table of a paper created by the learner has a pattern as shown in FIG. 2, the pattern recognition process uses the pattern closest to the score table stored in the content evaluation database 30. The lowest reference score table in FIG. 3 is extracted, and the evaluation value “4” stored in correspondence with the reference score table is the evaluation value of the paper. And the said evaluation value is transmitted to the learner's terminal 11 from the evaluation result transmission means 33 as an evaluation result.

  According to the experimental research by the present inventors, when the content of a large number of papers is evaluated by the paper content evaluation apparatus 10 according to the embodiment, a result close to the result of scoring the same paper by many scorers is obtained. It was proved that

  Therefore, according to such an embodiment, it is possible to automatically obtain a content evaluation of a paper that is close to a case where a grader evaluates manually. In addition, a score table is created from the paper created by the learner, and the paper is evaluated only in comparison with the pattern of the standard score table prepared in advance. The standard score table is created by unifying the question sentences. It is not necessary. For this reason, it is not necessary to prepare data corresponding to the question text, such as preparing a large number of model papers for the question text as a set, pre-scoring it, and storing it, and it is easy to add, change, delete, etc. It can be done by simple work.

  The paper content evaluation apparatus 10 of the present invention can be applied not only to content evaluation of English papers but also to content evaluation of papers in the same language with respect to questions in other languages by similar means and procedures. .

  Moreover, in the said embodiment, although the paper content evaluation apparatus 10 is made into the server format, this invention is not limited to this, The computer is the said mutual information amount database, the said reverse appearance frequency database, and the said score calculation. Means, the score table creation means, the content evaluation database, and a program that functions as the content evaluation determination means, and installs the program on a terminal of a user who becomes a thesis learner, the terminal itself Can be made to function as a paper content evaluation apparatus.

  In addition, various configurations in the present invention are not limited to the illustrated configuration examples, and various modifications are possible as long as substantially the same operation is achieved.

DESCRIPTION OF SYMBOLS 10 Paper content evaluation apparatus 19 Stop word database 21 Question sentence count means 22 Paper count means 24 Mutual information database 25 Content evaluation database 27 Score calculation means 28 Score table preparation means 30 Content evaluation database 31 Content evaluation decision means

Claims (7)

In a paper content evaluation device that automatically evaluates the content of a paper created by the learner in the same language as the question text, based on the question text that is the theme of the paper,
A mutual information database in which a mutual information amount serving as an index indicating the possibility of a pair of words appearing in the same document is stored for each combination of words;
A reverse appearance frequency database in which the reverse appearance frequency is stored for each word, the numerical value of which is lower for general words appearing in many documents;
Score calculating means for obtaining a score representing the degree of association between one word in the paper and the question sentence by substituting data extracted from the mutual information database and the reverse appearance frequency database into a mathematical formula stored in advance;
From the score obtained for each word in the paper by the score calculation means, a score table creation means for creating a score table representing the distribution status of the score;
A content evaluation database in which a plurality of types of reference score tables having different patterns of distribution status and human content evaluations corresponding to the respective reference score tables are stored,
A content evaluation determination unit that extracts a reference score table of the content evaluation database close to the score table created by the score table creation unit, and uses a content evaluation corresponding to the reference score table as a content evaluation of the paper An article content evaluation device characterized by A paper counting means including a function of counting the number of occurrences of each word in the paper,
The score calculation means stores in advance a score calculation formula using the mutual information amount, the reverse appearance frequency, the number of appearances of the word, and the number of types of words constituting the question sentence as parameters. In the arithmetic expression, the mutual information extracted from the mutual information database for a pair of target words consisting of one word in the question sentence and one word in the paper, and among the pair of target words in the paper The reverse appearance frequency extracted from the reverse appearance frequency database for one word, the number of appearances counted by the paper counting means for one word in the paper among the pair of target words, and the pre-counted 2. The paper content evaluation apparatus according to claim 1, wherein the score is calculated by substituting the number of types of words in the question sentence.   2. A question sentence counting unit that counts the number of types of words constituting the question sentence, wherein the number of word types counted by the question sentence counting unit is substituted into the score calculation expression. 2. Article content evaluation device described in 2. It has a stop word database in which stop words that are frequently excluded from various documents are stored.
4. The article content evaluation apparatus according to claim 3, wherein the question sentence counting means counts the number of types of words in the question sentence excluding stop words stored in the stop word database. It has a stop word database in which stop words that are frequently excluded from various documents are stored.
The score calculating means calculates the score for all combinations of one word in the paper excluding the stop word and one word in the question sentence excluding the stop word based on the stop word database. The paper content evaluation apparatus according to claim 1, 2, or 3.   In the content evaluation determination means, the score table of the article to be content evaluation is compared with each of the reference score tables stored in the content evaluation database, and the score table is determined by a preset pattern recognition method. 6. The article content evaluation apparatus according to claim 1, wherein the reference score table having a pattern close to the score table created by the creating means is selected. In a program for causing a computer to execute a process of automatically evaluating the content of a paper created by a learner in the same language as the question text, based on the question text as the theme of the paper,
A mutual information database in which a mutual information amount serving as an index indicating the possibility of a pair of words appearing in the same document is stored for each combination of words;
A reverse appearance frequency database in which the reverse appearance frequency is stored for each word, the numerical value of which is lower for general words appearing in many documents;
Score calculating means for obtaining a score representing the degree of association between one word in the paper and the question sentence by substituting data extracted from the mutual information database and the reverse appearance frequency database into a mathematical formula stored in advance;
From the score obtained for each word of the paper by the score calculation means, a score table creation means for creating a score table representing the distribution status of the score;
A content evaluation database in which a plurality of types of reference score tables having different patterns of distribution status and content evaluations of articles corresponding to the respective reference score tables are stored;
Extracting a reference score table of the content evaluation database close to the score table created by the score table creating means, and using the computer as a content evaluation deciding means for making a content evaluation corresponding to the reference score table a content evaluation of the paper Thesis content evaluation program characterized by making

Images