KR101713483B1 - Method for scoring supply type answer sheet, computer program and storage medium for the same - Google Patents

Abstract

The present invention relates to an answer scoring method using a machine learning method and a computer program and a recording medium for the same. By the answer scoring method using a machine learning method according to the present invention, at least two first classification criteria are set that respectively correspond to at least two classification grades depending on the types of multiple input answers - each of the two or more first classification criteria including a feature vector that is a coordinate value which shows a position in a feature space having each feature as its axis - the feature corresponding to a characteristic element determined in advance with respect to the multiple input answers is extracted, the feature vector is calculated that is the coordinate value which shows the position in the feature space having each feature as its axis, the multiple input answers are classified based on the classification grade corresponding to the first classification criterion with the highest inter-vector similarity value by comparison between the respective feature vectors of the multiple input answers and the respective feature vectors of the two or more first classification criteria, at least two second classification criteria are formed by averaging of the respective coordinates of the respective feature vectors of the multiple input answers classified based on the classification grades, the two or more first classification criteria are replaced with the two or more second classification criteria, the multiple input answers are classified based on the classification grade corresponding to the second classification criterion with the highest inter-vector similarity value by comparison between the respective feature vectors of the multiple input answers and the respective feature vectors of the two or more second classification criteria, and scoring is performed for each classification grade.

Description

METHOD FOR SCORING SUPPLY TYPE ANSWER SHEET, COMPUTER PROGRAM AND STORAGE MEDIUM FOR THE SAME,

The present invention relates to an answer scoring method using a machine learning method, a computer program therefor, and a recording medium.

The answer item is a form of the question that the subject does not select the right answer among the answers presented beforehand, but composes the correct answer. Usually, the answer forms are divided into detailed types such as short answer, parentheses, and essay (or narrative). The short answer type is a form in which the answer is written in simple words, phrases, clauses, and sentences, and the bracket type is a form in which the answer is written in parentheses. Although the essay or essay type is very similar to the essay type, essay type emphasizes that the student should write his or her thoughts and arguments logically and convincingly. The essay type is relatively short compared with the essay type, do.

The answer to the question is effective in measuring higher mental ability compared to the optional item because the student writes what he thinks. However, due to problems such as the fairness of grading results and the time and cost of scoring, we are not actively utilizing it in our educational field. In particular, the number and format of answer items are limited in large scale tests such as national level academic achievement evaluation.

Korean Patent Registration No. 10-1275146 (published on June 10, 2013)

If large-scale evaluations are conducted on a computer-based basis, the system should be constructed so that the scoring results can be calculated immediately. In other words, it is required to develop a system that can increase the proportion of answer items in the school site or hundreds to thousands of large scale evaluations by reducing the burden related to scoring in the computer based evaluation system. The present invention provides an answer scoring method using a machine learning method, a computer program therefor, and a recording medium to enable such a large scale evaluation.

A method of scoring a plurality of input answers using a machine learning method according to an embodiment of the present invention includes the steps of: a) determining at least two first classifications corresponding to each of at least two classifications, Wherein each of the at least two first classification criteria includes a feature vector that is a coordinate value indicating a position in a feature space around each feature axis; and b) C) calculating a feature vector, which is a coordinate value indicating a position in a feature space centering on each feature, and d) calculating a feature vector corresponding to each of the plurality of input answers And a feature vector of each of the at least two first classification reference vectors is compared with a feature vector of the first classification reference vector having the highest degree of similarity between the vectors Classifying each of the plurality of input answers into a plurality of input answers classified into classification classes; averaging each of the coordinates of the feature vectors of each of the plurality of input answers classified by classification class to form at least two second classification criteria; , f) changing the at least two first classification criteria to the at least two second classification criteria, g) comparing the feature vector of each of the plurality of input answers and the feature vector of each of the at least two second classification criteria Classifying each of the plurality of input answers into a classification class corresponding to a second classifying criterion having a highest degree of similarity between the vectors, and h) assigning a score to the classification class.

Also, a program according to an embodiment of the present invention is stored in a recording medium so as to execute a method of scoring a plurality of input answers using the above-described machine learning method in a computer.

A recording medium according to an embodiment of the present invention stores a program for executing a method of scoring a plurality of input answers using the machine learning method.

According to an embodiment of the present invention, it is possible to accurately and quickly perform scoring on a large number of answers including a question-answer item composed of a plurality of sentences.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an exemplary view showing the configuration of an answer scoring environment according to an embodiment of the present invention; FIG.
FIG. 2 is an exemplary view showing a configuration of an answer scoring apparatus according to an embodiment of the present invention; FIG.
3 is an exemplary view showing a configuration of a processing unit included in the answer scoring apparatus according to an embodiment of the present invention;
FIG. 4 illustrates an example of a natural language process of a document normalization unit according to an embodiment of the present invention. FIG.
5 illustrates an example of syllable generation and spacing state transitions in an exemplary sentence according to an embodiment of the present invention.
6A to 6D are views showing dependency relationships of sentences according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating a procedure of a method of scoring a plurality of input answers using a non-background learning method, which is a machine learning method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions will not be described in detail if they obscure the subject matter of the present invention.

FIG. 1 is an exemplary view showing the configuration of an answer scoring environment according to an embodiment of the present invention.

As shown in FIG. 1, the answer scoring environment 100 includes a server 110, a plurality of student terminals 120-1,..., 120-n, and a plurality of scorer terminals 130-1,. , 130-n). The server 110, the plurality of student terminals 120-1 to 120-n and the plurality of scorer terminals 130-1 to 130-n can communicate with each other via the network N Can be connected to each other.

The server 110 forms a sorting criterion for scoring the score to be formed, performs language processing on the input answer received from the student terminal 120-n, forms an answer to be scored without linguistic processing, The criterion can be used to classify and score scoring subjects' answers. The method of forming the classification standard, the classification of the scoring subject answers, and the scoring method will be described later. The server 110 may form a classification criterion by using a predetermined number of points to be scored, which are set in advance among the scores to be scored, or may classify the scores based on the grading results of a predetermined number of points to be scored, A reference may be formed. The server 110 can receive the grading result of the grader from the grader terminals 130-1, ..., and 130-n via the network N. [ In addition, the server 110 transmits the scoring result of the grading completed answer to the grader terminal 130-1, ..., 130-n or the student terminal 120-1, ..., 120-n 130-1,..., 130-n for the passive scoring of the scorer, which can not be classified or can not be graded according to the formed classification criteria.

The student terminals 120-1 to 120-n receive answers from a plurality of students corresponding to a specific test and receive responses from the server 110 and the grader terminals 130-1 to 130-n The input answer can be formed in a form that can be processed. In one embodiment, each student participating in the test directly inputs his or her answer to the student terminal 120-1, ..., 120-n, .., 120-n may form an input answer, and the student terminal 120-1, ..., 120-n may write an answer written on each student's test sheet by handwriting such as optical character recognition OCR (Optical Character Recognition) The input answers can be formed after recognition using the recognition method. An administrator can collect answers written on the test paper by collectively inputting them in the student terminals 120-1, ..., 120-n, However, the method of forming the scoring target answer is not limited to such an implementation. In addition, the student terminals 120-1, ..., and 120-n can receive the scoring result of the scoring target answer completed by the server 110. Here, the input answer may include all types of answers that may appear in the answer to the question-and-answer test question, such as at least one word, at least one number, at least one sentence. In the present embodiment, a case where the input answer includes at least one sentence will be mainly described.

The grader terminals 130-1 to 130-n perform grading based on the grading result of the grading object answer completed by the server 110 and the grading score that can not be classified or can not be graded according to the classifying criteria formed in the server 110 The target answer can be received via the network N. [ The grader terminals 130-1 to 130-n can easily perform review by the grader on the grading result of the grading result of the server 110 and can not classify it on the server 110 Or scored by the scorer in order to facilitate the scoring. The grader terminals 130-1, ..., 130-n may also be arranged so that the server 110 can form a classification criterion for an answer to be scored, for a predetermined number of points to be scored, The scorer may receive the classification result from the scorer and transmit the result to the server 110.

In one embodiment, the student terminals 120-1, ..., 120-n and the scorer terminals 130-1, ..., 130-n may be personal computers, tablets, smart A smart phone, a laptop computer, a personal digital assistant (PDA), and the like, but the present invention is not limited to such an implementation.

FIG. 2 is an exemplary view showing the configuration of an answer scoring apparatus according to an embodiment of the present invention, and FIG. 3 is an exemplary view illustrating a configuration of a processing unit included in the answer scoring apparatus according to an embodiment of the present invention.

The answer scoring apparatus 200 may include a processing unit 210, a storage unit 220, a communication unit 230, and a system bus 240. In an embodiment, the processing unit 210, the storage unit 220, and the communication unit 230 may be connected to each other via the system bus 240. The answer scoring device 200 may be included in the server 110 or separately from the server 110, but the present invention is not limited thereto. The processing unit 210 may include a language processing unit 212 and a classification scoring unit 214. The language processing unit 212 includes a document normalization unit 212-1, a morphological analysis unit 212-2, A substitution unit 212-3, a negative expression recognition unit 212-4, a grouping unit 212-5, a replacing unit 212-6, and a dependency analyzing unit 212-7. The scoring unit 214 may include a learning answer generation unit 214-1, a feature extraction unit 214-2, an answer classification unit 214-3, and a scoring unit 214-4.

The processing unit 210 performs natural language processing on a plurality of input answers to form a plurality of scoring target answers, selects a first predetermined number of answers having a frequency of a predetermined threshold value or more among a plurality of scoring target answers, Extracting a feature corresponding to a predetermined characteristic element by using the result of performing the natural language processing in each of the first predetermined number of answers, and extracting a feature for the score according to whether the feature is included in each of the first predetermined number of answers And a classification criterion includes a qualification vector which is a coordinate value indicating a position in a qualitative space centering on each of the qualities, and the second predetermined number of answers excluding the first predetermined number of answers Extracts the qualities for each of the second predetermined number of the answers, compares the qualification vector of each of the second predetermined number of answers with the qualification vector of the classification reference, Each of the second predetermined number of answers may be classified into a cluster corresponding to a classification criterion equal to or greater than the determined value and a score may be assigned to each of the classified clusters according to the classification level.

The processing unit 210 may also be configured to: a) set at least two first classification criteria corresponding to each of at least two classification classes according to the type of input answers, without performing natural language processing for a plurality of input answers; Wherein each of the at least two first classification criteria includes a feature vector that is a coordinate value indicating a position in a feature space around each feature; b) a feature corresponding to a predetermined feature element for a plurality of input answers C) calculating a feature vector which is a coordinate value indicating a position in the feature space about each feature axis; d) calculating a feature vector of each of the plurality of input answers and a feature of each of the at least two first classifiers Classifying the plurality of input answers into classification classes corresponding to the first classification class having the highest similarity among the vectors by comparing the vectors, Performing averaging of each of the coordinates of the feature vectors of each of the plurality of classified input answers to form at least two second classification references, f) changing at least two first classification references to at least two second classification references, g) comparing a feature vector of each of the plurality of input answers with a feature vector of each of the at least two second classifiers to obtain a class classification corresponding to a second classifier having a highest degree of similarity between the vectors, And h) a score for each classification.

Also, the processing unit 210 may set at least two second classification criteria to new at least two first classification criteria before step d), and then repeat steps e) to g) When two first classification criteria and at least two second classification criteria are the same, step h) may be performed.

In addition, the processing unit 210 may use any one of K-means clustering, hierarchical clustering, density-based clustering, and grid-based clustering Step d) to step g) can be carried out.

In addition, the processing unit 210 may perform natural language processing for a plurality of input answers, and extract the qualities using the results of the natural language processing.

In addition, the processing unit 210 compares the feature vector of each of the plurality of input answers with the feature vectors of at least two first classification references, classifies the feature vectors into classification classes corresponding to the classification reference having the highest similarity among the vectors, The classification score of the highest score or the correct answer if there are at least two classification classes.

In addition, the processing unit 210 compares the feature vector of each of the plurality of input answers with the feature vectors of at least two second classification references, classifies the feature vectors into classification classes corresponding to the classification reference having the highest similarity among the vectors, The classification score of the highest score or the correct answer if there are at least two classification classes.

Here, the natural language processing includes a sentence separation step, a spacing correction step, a spelling correction step, an acronym expansion step, a document normalization including a symbol removal step, a morphological analysis on an input answer, A sentence for at least two morphemes contained in the result of attaching a part of speech, a phrase or phrases included in the input answer, a sentence for converting to a predetermined standard expression, and a morpheme or vernacular sentence included in the input answer And dependency analysis to analyze the dependency structure.

In addition, the qualities include morpheme qualities formed by morphological analysis and attaching parts of speech, morphological qualities formed by performing morphological analysis and attaching parts of speech, morpheme qualities formed based on phrases included in input answers, basic phrases formed by performing phrases, (N-gram) qualities including one dependency qualifier, a predetermined number of adjacent morpheme qualities, or a phrase qualification.

Also, at least two classification classes may be classification grades in correct answers and incorrect answers, or three or more classification classes having three or more differential scores.

Table 1 shows an example of extracting the phrases, morpheme, basic phrases, and dependency relations according to the embodiment of the present invention, and expressing the scores to be scored according to the feature vector using the extracted feature sets.

[Table 1]

In addition, the processing unit 210 extracts qualities from each of the first predetermined number of answers before receiving the classification criterion, forms a qualification vector of each of the first predetermined number of answers using the extracted qualities, The feature vector and the formed feature vector may be compared to classify each of the first predetermined number of answers into clusters corresponding to the classification criterion corresponding to the reference feature vector having a similarity degree between the vectors equal to or greater than a predetermined value.

The processing unit 210 may also calculate the average value of the feature vector values of the first predetermined number of answers classified with the value of the reference feature vector and update the calculated average value with a new reference feature vector.

In addition, when receiving the classification criteria, the processing unit 210 receives grading results for the first predetermined number of answers, extracts the qualities included in the first predetermined number of answers, forms a classification criterion, Classifying the answers of the second predetermined number into clusters corresponding to the classification criterion and adding answers of the second predetermined number of answers to be included in each cluster to a first predetermined number of answers, Lt; / RTI >

When receiving the classification criterion, the processing unit 210 calculates the probability that the qualities included in the extracted first predetermined number of answers are included in the classification criterion corresponding to the reference qualities vector, and when the first predetermined number of answers is included The first predetermined number of answers can be classified into clusters corresponding to the classification criterion corresponding to the reference qualities vector.

In addition, when the feature vector is formed, the processing unit 210 may be configured to determine, based on the weight of the frequency at which one of the extracted qualities appears in the specific scoring candidate answer, The quality vector may be formed by considering at least one of the weights for the frequency of occurrence. That is, the processing unit 210 may express the extracted qualities as a weight in consideration of the importance of each qualification, instead of expressing the extracted qualities as a simple frequency in order to improve the accuracy of the grading target answer classification. In general, the qualities that often appear in the scoring subject answers can represent important qualities that represent the scoring subject answers. However, most of the sentence level scoring answers include "- end / ending ending ending", but it may not be an important qualification in scoring. Taking this into account, each of the qualities can be calculated as a term tfidf (term frequency-inverse document frequency). tfidf is a statistical number that indicates how important a word is in a particular document when there is a document set consisting of several documents, which can be calculated by multiplying tf (word frequency) and idf (inverse document frequency). It is possible to calculate tf, df, idf and tfidf using the following equations (1) to (4).

[Equation 1]

&Quot; (2) "

&Quot; (3) "

&Quot; (4) "

In Equation (1), tf is a value expressing how often a particular feature appears in a particular scoring target answer, and the higher the value, the more important it can be judged to be important in the scoring subject answer. df is a measure of how often certain qualities are used throughout the scoring objective. The higher the score, the less likely that qualities are not significant in the scoring objective. idf is the reciprocal of df, The higher the quality, the more likely it is that the qualification can be deemed important to the subject.

Table 2 shows the feature vectors of the scoring target answers shown in Table 1 in terms of tfidf according to the embodiment of the present invention. There is no difference between the two qualitative values, since the words "captured" and "morpheme" and "morpheme / sentence noun" are included in the answer " On the other hand, if tfidf is used, the morpheme qualities " captured / nested nouns "are calculated as shown in Equation (6) and can be expressed as 0.397 and 0.222, respectively. In other words, it can be shown that the two qualities have different differences. Reflecting these differences can affect machine learning methods and improve the accuracy of scoring.

&Quot; (5) "

&Quot; (6) "

[Table 2]

In addition, the processing unit 210 calculates the weight of the frequency with which one of the qualities of the extracted qualities appears in the specific scoring target answer as the number of times that one of the qualities appears in the specific scoring target answer, The weight of the frequency with which one of the qualities appears in the plurality of scoring target answers can be calculated by dividing the total number of the scoring target answers by the number of scoring target answers including any one qualification and taking a log.

When classifying each of the second predetermined number of answers, the processing unit 210 compares the feature vector of each of the second predetermined number of answers with the feature vector of the classification criterion so that a cluster corresponding to the classification criterion with the highest similarity among the vectors Or if there are at least two classification criteria with the highest degree of similarity, it can be classified as a cluster corresponding to the classification score having the highest score or the correct answer.

The language processing unit 212 may perform natural language processing on a plurality of input answers formed by the plurality of student terminals 120-1, ..., and 120-n to form a scoring target answer. In one embodiment, the language processing unit 212 may perform language processing such as document normalization, morphological analysis, parts attaching, illegal expression recognition, phrase matching, swapping, dependency analysis, The language processing performed by the processing unit 212 for the input answer is not limited to such an implementation.

The document normalization unit 212-1 can unify various input answers having the same meaning into one expression. FIG. 4 is a diagram illustrating a natural language process of the document normalization unit 212-1 according to an embodiment of the present invention. The document normalizing unit 212-1 includes a sentence separating step S310, a spacing correction step S320, a spelling correction step S330, an abbreviation expanding step S340, a symbol removing step S350, That is, the natural language processing performed by the document normalization unit 212-1 on the input answer is not limited to this order.

The document normalization unit 212-1 can form an output separated by sentence unit by using an input answer composed of a plurality of sentences in the sentence separating step (S310). The document normalization unit 212-1 can separate a plurality of sentences included in the input answer on a sentence-by-sentence basis based on a sentence ending symbol. In one embodiment, the sentence termination symbol may include an on-point (.), A question mark (?), An exclamation point (!), But is not limited to this implementation. The following Table 3 is an example showing the result of the sentence separating step performed by the document normalizing unit 212-1. The document normalization unit 212-1 receives the input answer of the form "We waved hard, but no one of the athletes turned around" , We could output "We shook our hands hard" and "No one looked at the players," but in different sentences.

[Table 3]

However, if the input answer does not contain a sentence terminator, the sentence can not be separated. There may also be a case where a symbol similar to a sentence ending symbol is used in the input answer, and a sentence ending symbol may not be used only when terminating the sentence. For example, in the sentence "President came to Busan (?)", The sentence should not be separated by using "?", And the sentence "0.002 is a very small number" The document normalization unit 212-1 can separate a plurality of sentences included in the input answer using various machine learning methods including a conditional random field model. A conditional random field model is a model that learns and classifies data so that the probabilities of various qualities can be combined to determine the desired category. That is, the document normalization unit 212-1 extracts all the words in the sentences included in the input answers after removing the sentence end symbols from the plurality of input answers, and the two consecutive words in the extracted word are different sentences Calculate sentence separation probabilities that can be separated. In addition, the document normalization unit 212-1 compares the sentence separation probability with a predetermined reference probability, and when the sentence separation probability is equal to or greater than a predetermined reference probability, separates two consecutive words into different sentences. In one embodiment, the predetermined reference probability may be a value within the range of 80% to 90%, but is not limited thereto. Table 4 is an example showing results of performing the sentence separating step of the document normalizing unit 212-1 using the machine learning method. The document normalization unit 212-1 receives the input answer of the form "We waved hard, but no one of the players turned around" "No", "no one", "no one turns around", "turn around", "waved", "waved", "waved" Man "," no man ", and so on. The document normalization unit 212-1 calculates the probability that each of the separated word units in the corpus stored in the storage unit 220 is divided into different sentences and calculates the probability that the computed probability is " , We can separate "we waved our hands hard" and "but no one looked at the players" in different sentences. As an example, a corpus of Sejong provided by the National Institute of Korean Language can be used as a corpus, but any corpus may be used if the corpus includes corpus information.

[Table 4]

In addition, the document normalization unit 212-1 can correct the spacing of each sentence included in the input answer in the spacing correction step (S320) so as not to affect the scoring. The document normalization unit 212-1 can perform spacing correction using a probability-based spacing correction method. That is, the document normalization unit 212-1 pastes the syllables of the sentences contained in the plurality of input answers, extracts all the syllables of the sentences included in the respective input answers, and consecutive two syllables of the extracted syllables The probability of pasting and the probability of spacing can be calculated, and the two consecutive syllables can be written or spaced according to the probability of being pasted and the probability of being spaced out. When the sentence separation probability is greater than or equal to a predetermined reference probability in the sentence separating step (S310), the document normalizing unit (212-1) pastes the two consecutive words in different sentences The syllables of the sentences included in a plurality of input answers can be pasted. On the other hand, the document normalization unit 212-1 can set the two syllables that are consecutive in default to be pasted when the calculated probability of pasting and the probability of being spaced are the same.

In one embodiment, the document normalization unit 212-1 receives a sentence of "We shook his hand hard" that the input answer includes, and writes all the syllables such as " It can be divided into "us", "ri", "hi", "zeal", "heavily", "Hisson", "hand", "sword", "shake", "heard" have. The document normalization unit 212-1 sets the number of sentences including "us" (a form in which "right" and "le" are pasted) among the sentences included in the corpus stored in the storage unit 220 to " , And the number of sentences including "us" (a form in which "space" and "space" are spaced) is included in the "right" syllable We can calculate the probability of spacing "us" by dividing the number of sentences by the number of sentences, and determine the form with a higher probability among the calculated probabilities as the spacing of "us". Since we are more likely to add "us" than the probability of spacing "us", we can decide to stick with "us".

In addition, the document normalization unit 212-1 sets the number of sentences including "rei" (a form in which " And the number of sentences including "Lee" (a form in which "Lee" and "the" are spaced apart) is calculated by dividing the number of sentences including the syllable "Lee" "We divide by the number of sentences containing syllables and calculate the probability of spacing" Lee ", and we can determine a form with a higher probability among the calculated probabilities as the spacing of" Lee ". Since Lee is likely to write "Lee" rather than the spacing of "Lee", "Lee" can decide to paste.

In addition, the document normalization unit 212-1 determines the number of sentences including the " arrays "(forms in which" a "and" columns "are pasted) among the sentences included in the corpus stored in the storage unit 220 "Is calculated by dividing the number of sentences including syllables by the number of sentences including" column "(a form of" space "between" a "and" column "), The probability that the "odd heat" is spaced by dividing it into the number of sentences including the odd odd odd odd odd and even odd odd odd odd odd odd oddities can be determined. Since there is a high probability that it will be spaced more than "odd" rather than "odd", "odd" can be determined by spacing.

In addition, the document normalization unit 212-1 sets the number of sentences including "eardrum" (a form in which "column" and "seam" are pasted) among the sentences included in the corpus stored in the storage unit 220 as " "The number of sentences containing" heat "(a form of" space "and" space ") is called the" heat "syllable Quot; ebb "is divided by the number of sentences including " ebb ", and a form having a higher probability among the calculated probabilities can be determined as a space of" ebb. Because "enthusiasm" is more likely to be affixed with "zeal" than the possibility of spacing, "zeal" can be decided by pasting.

In addition, the document normalization unit 212-1 may set the number of sentences including "severe" (a form in which " "The number of sentences containing" sincerity "(a form of" spacing "" ss "and" ss ") is called a" ss "syllable Quot ;, and a form having a higher probability among the calculated probabilities can be determined as a "very" spacing form. Since there is a high probability of pasting "severely" rather than "deeply" spacing, you can decide to write "very".

In addition, the document normalization unit 212-1 determines the number of sentences including "Hisson" (a form in which "HI" and "HAND" are pasted) among the sentences included in the corpus stored in the storage unit 220 "The probability of pasting" Hisson "is divided by the number of sentences containing the syllable, and the number of sentences including" Hisson "(the form of spacing" Hei "and" The probability that the word "Hisson" is spaced apart by dividing it by the number of sentences including the sentence is calculated, and a form having a higher probability than the calculated probability can be determined as the spacing form of the syllable. "Hisson" can be determined to be spaced because there is a high probability that "Hisson" will be spaced rather than the probability of pasting "Hisson".

In addition, the document normalization unit 212-1 sets the number of sentences including the "hand" (a form in which "hand" and " Quot; hand " and the number of sentences including "hand" (a form in which "hand" and " "It is possible to divide the number of sentences containing syllables and calculate the probability of spacing" hand ", and the form with a higher probability than the calculated probability can be determined as a space of" hand ". Because you are more likely to write "hand" than "hand", you can decide to write "hand".

In addition, the document normalization unit 212-1 sets the number of sentences including "scribble" (a form in which "e" and "scribbled" are written) among the sentences included in the corpus stored in the storage unit 220 to " "The number of sentences containing" syllable "(the spacing between" e "and" syllable ") is calculated by dividing the number of sentences containing the syllable by the syllable" syllable " Quot ;, and a shape having a higher probability than the calculated probability may be determined as a space of "SIGNS ". Since there is a high probability of spacing "sir" rather than the probability of pasting "sir", it is possible to determine "sir" to be spaced.

In addition, the document normalization unit 212-1 may set the number of sentences including "shake" (a form in which "shake" and " "The number of sentences including" shaken "(a space between" shaku "and" syllable ") is called" shaky "syllable by calculating the probability of writing" shaken "by dividing by the number of sentences containing syllable, Quot; shaking "by dividing the number of sentences including the " shaking " by the number of sentences including the " shaking" You can decide to write "shake" because you will be more likely to write "shake" instead of "shake".

In addition, the document normalization unit 212-1 determines the number of sentences including "heard" (a form in which "s" and "i" are pasted) among the sentences included in the corpus stored in the storage unit 220 "The number of sentences including" heard "(spacing between" s "and" y ") is calculated by dividing the number of sentences including the syllable by the number of" syllables " Quot; heard "by dividing by the number of sentences including " hearing ", and a form having a higher probability among the calculated probabilities can be determined as a form of" heard ". You can decide to write "I heard" because you are more likely to write "hear" than "probability" of spacing.

In addition, the document normalization unit 212-1 determines the number of sentences including "I" (a form in which "I" and " "The number of sentences containing" I "(the spacing between" I "and" I ") is calculated by dividing the number of sentences containing the syllable by" Quot ;, and the type having a higher probability among the calculated probabilities can be determined as a space of "I ". It is more likely to add "I" than "I" to the spacing, so I can decide to write "I".

The document normalization unit 212-1 can determine the spacing of all the syllables by the spacing correction method described above and determine the spacing of the entire sentences as "We waved hard." The document normalization unit 212-1 can correct the spacing such that a substantial morpheme such as a noun is spaced apart from the preceding syllable in a spacing correction step and a morpheme such as an investigation or a mother is pasted with a preceding syllable.

FIG. 5 is an exemplary diagram illustrating syllable generation and spacing state transitions in an exemplary sentence according to an embodiment of the present invention.

As shown in Fig. 5, the example sentence is syllable string S = s ₁ s ₂ ... s _n and the spacing tag column T = t ₁ t ₂ ... t _n , where the spacing can be expressed as a "space" and pasted as "pasted". The beginning of every syllable is denoted by $, and the spacing tag of the start syllable is defined as "a". As an example, the example sentence "We Waved Hands Wary" can be represented by the syllable column "We Waved Harder" and the Spacing Tag column "Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Sticky Since the corpus stored in the storage unit 220 may not include " We shook hands hard ", the probability can be calculated by dividing the entire syllable column by one without calculating the probability. We calculated the probability values P1 (we waved hard), P2 (we waved hard), P3 (we waved eagerly), P4 (we waved very hard) (For example, "We shook his hand hard") having the highest probability value (e.g., P1) among the candidate sentences.

In addition, the document normalization unit 212-1 may correct a portion of the spelling error included in each sentence included in the input answer, which does not affect the answer score, in the spelling correction step (S330). That is, the document normalization unit 212-1 removes all the punctuation marks in each sentence included in the input answers, and each of the phrases extracted in the sentence separating step S310 is stored in the corpus stored in the storage unit 220 If each of the extracted words is included in the corpus, the spelling of the word is not corrected, and if each of the extracted words is not included in the corpus, the corrective word is corrected to the correction candidate word . In this case, when the correction target word is corrected by the correction candidate word, when the plurality of correction candidate words are plural, the editing distance of each of the correction target word and the correction candidate word is compared, and the correction target word is corrected can do. The document normalization unit 212-1 can perform spelling correction using a minimum edit distance algorithm. The document normalization unit 212-1 can perform spelling correction using a minimum edit distance algorithm. The editing distance may indicate the number of times editing (substitution, deletion, insertion, etc.) is performed on the character of the correction target word when the correction target word is corrected to the correction candidate word. For example, if the proofreading clause is "hard" and the proofreading clause is "hard", then "o" should be replaced once with "o" to correct "hard" to "hard" The editing distance may be "1", and in order to correct the proofreading phrase "lull" to the proofreading phrase "hard", it is necessary to replace "d" with "o" and "a" with "l" , The editing distance may be "2", and in order to correct the correction candidate word "very" to "proof", it is necessary to insert " Since the insertion occurs, the editing distance may be "1 ".

This spelling correction step (S330) may be applied in units of words, syllables, and phonemes. For example, 'nol: ng:: ng' is a spelling correction rule applied to the unit of a word, and 'a: t: ss: t → t' is a spelling correction rule applied to a phoneme unit. Here, the spelling correction rule 'w: x: z → y' can represent a spelling correction rule that translates x to y if the left word of x, the syllable or phoneme is w, and the right word, syllable or phoneme is z. When the calibration rule is applied in the unit of the word, the accuracy is high while the range of its application is reduced. On the other hand, when the calibration rule is applied on a phoneme basis, the accuracy of the rule is decreased instead of increasing the range of application. Therefore, it is necessary to calibrate them by appropriately combining them.

In addition to the above-mentioned rule-based spelling correction method, various methods can be used to perform the spelling correction step. First, the word-based spelling correction method is a method of searching and correcting the most similar words in the corpus stored in the storage unit 220 based on the editing distance of the wrong word. However, the word-based spelling correction method may be vulnerable to the coined word if the word is not included in the corpus. Also, it is difficult to include all of the words in corpus because Korean combines various morphemes and generates very diverse vernacular phrases.

Second, morpheme-based spelling correction method is a method of using morpheme analyzer for correct spelling correction. The Korean morphological analysis is very effective, but the morpheme - based spelling correction method is inefficient because it performs morpheme analysis every time the candidate is corrected, and is vulnerable to the case where the morpheme analysis itself is performed erroneously.

Third, the syllable-based spelling correction method is a method of correcting spelling errors based on syllables. The construction costs are not large because they learn all the necessary information in the corpus. However, if the characteristics of the training set and the experiment set are similar, the performance is relatively good. Otherwise, the performance may be low.

Fourth, it is a method of generating and verifying spelling correction candidates on a per - speller basis for frequently occurring spelling errors. This method generates a lot of spelling corrections, which greatly increases the complexity of the calibration. Too many calibration candidates interfere with finding the best calibration candidates, so high performance is not expected unless the surrounding context is taken into account.

It is possible to improve the spelling correction performance by correcting stepwise so as to compensate for the advantages and disadvantages rather than using only one of the various calibration methods described above. Especially, it can show good performance when a spelling - based spelling correction method or a syllable - based spelling correction method is performed first and then a spelling - based spelling correction method is performed. This is because some of the spelling mistakes that have been corrected in the spelling correction unit or the syllable unit spelling correction are corrected correctly in the spelling correction unit.

In one embodiment, the document normalization unit 212-1 may remove the punctuation marks from the sentences included in the input answer, and separate the punctuation marks in units of words. If the separated word phrases are included in the corpus stored in the storage unit 220, the word normalization unit 212-1 does not correct the spelling of the corresponding phrase, and each of the separated phrases is stored in the storage unit 220, It is possible to extract a calibration candidate word having an editing distance within a predetermined value from the corresponding word. For example, the document normalization unit 212-1 determines that the phrase "very strict" is not included in the corpus of the phrase included in the input answer, and edits it from the phrase to be corrected " It is possible to extract "hard" by the candidate word of the distance within 1 distance. The document normalization unit 212-1 can perform spelling correction for correcting the correction target word to the correction candidate word using the extracted correction candidate word.

In addition, the document normalization unit 212-1 may modify the abbreviations included in each sentence included in the input answer to a standard expression having the same concept or meaning in the acronym expansion step (S340). In one embodiment, the document normalization unit 212-1 may convert an abbreviation included in the input answer into a standard expression using a thesaurus stored in the storage unit 220. [ That is, the document normalization unit 212-1 checks whether each of the extracted word phrases is included in a thesaurus stored in the storage unit 220, and if each of the phrases extracted in the sentence separating step S310 is included in the thesaurus If it is not included, and if each of the extracted words is included in the thesaurus without changing the corresponding word, the word can be changed to an abbreviated extended word contained in the thesaurus. A thesaurus may mean a dictionary representing a relationship between different words (synonyms or antonyms). For example, the thesaurus may be registered as a thesaurus of "evaluation person", "KICE", "curriculum evaluation person", "Korean curriculum evaluation source", and the document normalization unit 212-1 confirms the input answer, "," KICE ", and" Curriculum Evaluation Institute ", all of them can be changed to" Korea Curriculum Evaluation Institute ".

In addition, the document normalization unit 212-1 may remove unnecessary symbols in scoring included in the input answer in the symbol removal step (S350). In one embodiment, the document normalization unit 212-1 determines whether or not the sentence code (double quotation mark ("), single quotation mark ('), ontometry (.), Question mark (?), Exclamation mark (! The document normalization unit 212-1 can output the same type of sentence by removing the symbol through the symbol removal step S350 in a sentence including another form. The document normalization unit 212-1 can remove the question mark, the question mark, and the warm-up point, and modify it to "We shook his hand hard. &Quot; However, it is necessary to remove the "A-Dawn", "Dawn", "Dawn", and " A: Dawn of the Stars " If it is normalized to "Dawn star", it can not be distinguished from "In reaction of gas A and B, gas C is produced" and "gas A is produced by reacting gas C and B." Therefore, The removal step (S350) can be set as an option so that the grader can selectively utilize the subject and the items.

The morpheme analysis unit 212-2 divides each sentence included in the input answer into units of words by using a corpus (partly attached corpus) stored in the storage unit 220 and considers the vocabulary of the word and the sex of the part-of-speech And extract morphological analysis results of all possible types of morphemes. The part attaching unit 212-3 determines the morphological analysis result of the most probable form using the probabilistic part-of-speech attaching model among the morpheme analysis results by the morpheme analyzing unit 212-2, The parts of the morphemes can be attached.

Table 5 is an exemplary view showing morphological analysis results of the morpheme analysis unit 212-2 according to one embodiment and results of partly affixing parts of the parts attachment unit 212-3.

[Table 5]

Referring to Table 5, the morphological analysis unit 212-2 extracts all possible morphological analysis results of the sentence in units of words using the corpus-attached corpus when the input answer includes "We shook the hand hard" can do.

For example, the morphological analysis result of the word "we" might be "us", pronoun, "helper", "us", or "helper" Is a noun, "Li" is a noun, and "a" may be a form of a subsidiary.

Also, the morphological analysis result of the word "hand" may be "hand" as a noun, and "e" as a helper.

Also, the morphological analysis result of the phrase "hard" may be "hard" in adverbial form, "eager" in noun, and "hi" in adverbial suffix form.

In addition, the morphological analysis result of the word "shake" may be that "shake" is a verb, "yi" is a prose term, "da" is a termination term

In one embodiment, the part-of-speech attaching section 212-3 extracts the morpheme analysis result having the highest probability of each morpheme analysis result from the part-of-speech corpus using the extracted morpheme analysis result and the part- It can be decided from the morphological analysis result.

For example, in Table 5, the word "we" is the pronoun, "we" is the pronoun, and "the number of occurrences (probabilities) , "Hand" in the case of the word "hand" noun, "e" in the form of a part of the corpus with the help of the form is the most frequently occur, so the morphological analysis can determine the form. In addition, in the case of the phrase "hard", the morphology analysis result is determined as the most frequent occurrence in the corpus with part-of-speech attached to the form of "adversely" adverb, and the word "shake" The morphological analysis result can be determined because the number of occurrences of the verb, "I", and "Da" is the most frequently appearing in the corpus with part - of - speech. Using the above-described method, the part-of-speech attaching unit 212-3 can attach the part of speech of each morpheme using the determined morpheme analysis result.

The illegality expression recognizing unit 212-4 recognizes the illegal adverbs stored in the storage unit 220, the negative auxiliary pronouns ('do not allow / deny', 'do not deny' (Eg, 'Do not ~'), a set of indefinite phrases (such as 'can not be'), negative phrases ('no', 'no', etc.) By using the same negative expression, it is possible to recognize a negative expression included in each sentence included in the input answer and to attach an indefinite tag. In one embodiment, the negative expression recognition unit 212-4 determines whether each sentence included in the input answers includes a predetermined negative expression, and if each sentence does not include a predetermined negative expression, If each sentence contains a predetermined negative expression without attaching an indefinite tag, an indefinite tag can be attached to each sentence. The illegality expression recognizing unit 212-4 can attach the irregular expressions to each of the irregular expressions by dividing them into NOT, CAN NOT, and STRESS. In one embodiment, the negative expression recognition unit 212-4 determines whether each sentence included in the input answers includes a predetermined negative expression, and if each sentence does not include a predetermined negative expression, If each sentence contains a predetermined negative expression without attaching an indefinite tag, an indefinite tag can be attached to each sentence. Table 6 is an example of an input sentence according to an embodiment of the present invention and a result of attachment of the irregular tag according to the present invention.

[Table 6]

Referring to Table 6, since the negative expression recognition unit 212-4 includes a negative expression "not" stored in the storage unit 220 when "not disturbing" is input as the input answer, Quot; NOT "by attaching an amorphous tag NOT, which means " NO ". < / RTI >

In addition, since the negative expression recognition unit 212-4 includes a negative expression "can not be stored" in the storage unit 220 when "can not be eaten" is input as the input answer, the negative expression is removed It can be modified to "CAN NOT" by attaching CAN NOT, which is an indefinite tag which means impossible.

In addition, since the illegitimate expression recognition unit 212-4 includes a negative expression "to" stored in the storage unit 220 when the "did not study" It can be modified to "NOT" by attaching an amorphous tag NOT, which means negation.

In addition, since the illegitimate expression recognition unit 212-4 includes the double negative expression "do not care" stored in the storage unit 220 when "I have to go this time" It removes the negative expression and attaches the tag STRESS which means emphasis, and it can be modified to "STRESS" this time.

The phrase bundle 212-5 can extract phrases in non-relational forms that do not contain phrases in a phrase that is syntactically closely connected to each other, such as noun phrases or verbal phrases, in each sentence included in the input answer. A phrase can also be referred to as a partial parsing. It is possible to reduce the complexity of parsing the result set of the input answer. In one embodiment, the phrase bundling unit 212-5 separates each sentence included in the input answer by a word-by-word method using a probability-based method, and the separated phrases are stored in a corpus (phrase bundling corpus) It is possible to compare the probability of being classified into the same sphere and the probability of being classified as the beginning of a new sphere to determine whether to classify the probability of the higher sphere into the same sphere or the new sphere.

Table 7 is an example showing input answers and phrase results according to an embodiment of the present invention.

[Table 7]

Referring to Table 7, the phrase bundle 212-5 can separate the input answer into a word-by-word (book love, reading, quiz, conference, we, school, reading, culture, The number of times that a predetermined number of consecutive phrases "book reading love quiz contest" is classified into the same phrase in the phrase bundle corpus is calculated, and the number of times classified into different new phrases is calculated, It can be judged. In this case, the "book love reading quiz contest" can be performed because the number of times classified in the same phrase is greater than the number of times that the new phrase is divided into different phrases. The phrase bundle 212-5 can carry out phrases such as " our school ", "reading culture ", and" rooted out "

If we try to parse the phrase "Book Love Reading Quiz Competition," as shown in Table 7, which has led to the rooting of our school's reading culture, we can extract all possible pairs of words from 10 words, , But it is necessary to carry out the analysis of the phrases, but when using the result of the phrase bundle, it is possible to use [Book Love Reading Quiz Competition] (Noun phrase), [Our school] (Noun phrase), [Reading culture] (Noun phrase) Since we only know the syntactic relations that are possible in the four clauses, the complexity of the parsing can be reduced when the clause is executed.

The replacing unit 212-6 can perform a replacement process of rewriting each sentence included in the input answer by using a substitute word and an expression or the like and rewriting it into a simple sentence while maintaining the meaning of the sentence. In one embodiment, the rewriting unit 212-6 may convert a representation included in the input answer to a standard expression using a thesaurus stored in the storage unit 220. [ A thesaurus may refer to a dictionary representing a relationship between different words or expressions (synonyms or antonyms). For example, the thesaurus may be registered with the thesaurus of " I / D ", the word " We + can eat rice + "is inputted as an answer, we can refer to the thesaurus and perform the process of" We + eat rice + ".

The dependency analysis unit 212-7 can perform parsing on each sentence included in the input answer based on the dependency grammar. Dependence grammar is a grammatical representation of the relationship between dependent and governor, where the dominant element is the central element of the dependent language elements, and the dependent element is the meaning of the dominant It means the complementary factor. In one embodiment, the dependency analyzer 212-7 may analyze the dependency of each word in the reverse order of the sentence, and may classify the dependency and dominance based on the probabilities. That is, the dependency analysis unit 212-7 calculates the probability that each of the words extracted from the sentences included in the input answers is classified into the dependency and the dominant, and calculates the probability of being classified as the dependent and the class of the dominant By comparing the probabilities with each other, each of the extracted words can be classified with a higher probability.

For example, when analyzing the dependence relationship of "I ate rice today", it is possible to connect "eat" and "rice" in the order of the reverse order of the sentence as shown in FIG. As shown in FIG. 6B, when analyzing the dependency of the next word "I" in the reverse order, it is determined whether "I" is dependent on "eating" or not (A) The dependency analysis unit 212-7 uses the corpus stored in the storage unit 220 to determine the probability that the "I" and "EAT" are dependency relationships and the probability that "I" and "Bob" To select "I" with a higher probability to have "fed" and dependency (A). As shown in FIG. 6C, when analyzing the dependency of the next word "today" in the reverse order, there is no dependence between "today" and "rice" where crossing occurs in the language structure, The dependency analysis unit 212-7 uses the corpus stored in the storage unit 220 to determine whether the dependency relationship is " Today "and" Eat "can be chosen to have a dependency relationship, and" Today "and" I "have a probability of being a dependency so that" Today "with a higher probability has" I "and Dependency (D) . FIG. 6D shows a form in which the dependency analysis of the input answer sentence "Today I ate rice" is completed.

The classification scoring unit 214 may perform a scoring of the scoring objective answer formed in the language processing unit 212 to form a scoring result. In one embodiment, the classifying and scoring unit 214 includes procedures for generating an answer for a learning, extracting and selecting a qualification of an answer to be scored, generating a learning model, forming a classification standard for an answer to be scored, classifying an answer to be scored, , But the procedure performed by the classification scoring unit 214 for the scoring objective answer is not limited to such an implementation. The classifying and scoring unit 214 may include a learning answer generation unit 214-1, a feature extraction unit 214-2, an answer classification unit 214-3, and a scoring unit 214-4.

The learning-answer generating unit 214-1 may select a first predetermined number of answers having a frequency equal to or higher than a predetermined threshold value among the scoring-subject answers that have been subjected to the language processing, and form the first predetermined number of answers as learning answers. In one embodiment, the learning-answer generating unit 214-1 may select the first to-be-selected answers as the scoring-candidate answers having a frequency of 80% or more of all the scoring-answering answers, , 10) can be selected as the first predetermined number of answers, but the method of selecting the first predetermined number of answers is not limited to this embodiment.

The feature extraction unit 214-2 extracts the characteristic elements from the first predetermined number of answers selected by the learning answer generation unit 214-1 by using the result of the natural language processing performed by the language processing unit 212 The corresponding feature can be extracted. In one embodiment, the language processing unit 212 includes a sentence separating step, a spacing correction step, a spelling correction step, an abbreviation expansion step, a document normalization including a symbol removal step, a morphological analysis of an input answer, The morpheme of the morpheme, the recognition of the negative expression of the input answer, the phrase for at least two phrases included in the input answer, the transposition and the input answer to convert the phrase or phrase included in the input answer into a predetermined standard expression A natural language processing including a dependency analysis for analyzing a dependency structure between a morpheme or a verse can be performed.

In addition, the feature extraction unit 214-2 can extract the morpheme qualities using the morphological analysis of the first predetermined number of answers selected by the answer calculation unit 214-1, and the results of performing the part-of-speech attachment. For example, when the sentence included in one of the answers of the first predetermined number is "We shook his hand hard ", the morpheme analysis unit 212-2 and the parts attaching unit 212-3 The result of morphological analysis and part - affixing is like "our / pronoun + is / auxiliary + hand / noun + / assistant + hard / adverb + shake / verb + , "" / "," / "," / "," / "," / "," A / noun "," hand / noun " .

In addition, the feature extraction unit 214-2 can extract the speech quality based on the phrases included in the first predetermined number of answers selected by the learning answer generation unit 214-1. For example, if the sentence included in one of the first set of answers is "We shook his hand hard," then the spacing units of words, "we". You can extract "hand", "hard", "waved" in a phrase.

In addition, the feature extraction unit 214-2 can extract a basic syllable by performing a syllable of the words included in the first predetermined number of answers selected by the answer-generation unit 214-1. For example, if the sentence included in any one of the answers of the first predetermined number of answers is "book reading love quiz contest has rooted down the reading culture of our school ", the phrase bundling section 212-5 The results are as follows: "[Book Love Reading Quiz Competition] (noun phrase) is like [our school] (noun phrase) [reading culture] (noun phrase) It is possible to extract the basic phrase qualities from the [book reading comprehension quiz competition] (noun phrase), [our school] (noun phrase), reading culture (noun phrase), and roots (verb phrase) have.

The qualification extraction unit 214-2 may perform dependency analysis of the first predetermined number of answers selected by the learning answer generation unit 214-1 to form dependency qualities. For example, when the sentence included in one of the answers of the first predetermined number is "I have eaten rice today", the dependency analysis result of the dependency analyzing unit 212-7 is as shown in FIG. 6D Can be formed into dependency qualities, which are the same, and each dependency [I ate], [ate, cooked] [today, I ate].

The feature extraction unit 214-2 may combine the bi-gram or tri-gram of the morpheme qualities or the bi-gram or the tri-gram of the verbal qualities to form an n-gram feature . In one embodiment, a morpheme or a word bi-gram feature may be formed by combining two adjacent words in a morpheme or a word feature, or a combination of three adjacent words in a morpheme or a word feature, (tri-gram) < / RTI >

In addition, the feature extraction unit 214-2 extracts the qualities using the result of the natural language processing performed by the language processing unit 212 from each of the second predetermined number of answers excluding the first predetermined number of answers Can be extracted.

The answer classifying section 214-3 selects a machine learning method for forming a classification criterion to be used for classifying the scoring target answer using the qualities extracted from the qualification extracting section 214-2 and uses the selected machine learning method And classify the scoring candidate answers into at least two clusters (for example, classify them into correct answers and incorrect answers, or classify them into three or more, such as three points, two points, one point, etc.) And classified as a differential score). In one embodiment, a machine learning method for forming a classification criterion is classified into an unsupervised learning method, a supervised learning method, a semi-supervised learning method, an ensemble learning method Ensemble Learning Method).

In the non-background learning method, clustering the first predetermined number of answers selected by the answer-generating unit 214-1 into at least two clusters, forming a classification criterion, and using the classification criterion (the first predetermined number The second predetermined number of answers can be classified into one of at least two clusters. In one embodiment, the answer classification unit 214-3 uses each of the qualities extracted from the first predetermined number of answers by the feature extraction unit 214-2 to convert each of the first predetermined number of answers into the feature vector, 1, an initial classification criterion corresponding to each of at least two clusters may be set using the qualities extracted from the predetermined number of answers, and the set initial classification criterion may be converted into a reference vector. In addition, the answer classification unit 214-3 calculates the similarity (distance) between the reference vector and the feature vectors corresponding to the first predetermined number of answers, and calculates the similarity (distance) based on the calculated similarity Each can be classified into clusters corresponding to the reference vectors having the highest similarity (closest distance). The answer classification unit 214-3 classifies each of the first predetermined number of answers into one cluster and then calculates a vector average of the coordinates of the feature vector corresponding to each of the first predetermined number of answers sorted by the cluster To reset the reference vector corresponding to each cluster. In addition, the answer classification unit 214-3 can repeat the calculation of the degree of similarity between the feature vectors corresponding to each of the first predetermined number of answers using the set reference vectors and classification into clusters having the highest similarity. The answer classifying section 214-3 can repeatedly perform the similarity degree calculation and the cluster classification until there is no change in the first predetermined number of answers included in each cluster. If there is no change in the answers included in the cluster, Can be set as a final reference vector. In one embodiment, the answer classification unit 214-3 may form clustering and classification criteria using K-average clustering, hierarchical clustering, density-based clustering, and grid-based clustering , But the method is not limited thereto.

Here, the K-average clustering method is a method of grouping the given data into k clusters, and can operate in a manner of minimizing the dispersion of the distance difference with each cluster. The K-means clustering method is a type of unsupervised learning that can be used to label input data that does not have a label. The structure of the clusters formed using the K-average clustering method and the structure of clusters formed using the EM algorithm (Expectation-Maximization Algorithm) can have a similar structure.

The hierarchical clustering method is a method of bottom-up merging clusters having a hierarchical structure, a method of clustering (Agglomerative Method), or a method of recursively dividing by top-down split clustering (Divisive Method).

The density-based clustering method is a method that can expand a cluster based on the density of neighboring data.

The grid-based clustering method divides the space in which the data exist in a finite number of spaces of a grid structure and performs a clustering method on the grid structure.

The answer classifying section 214-3 calculates the similarity (distance) between the feature vector corresponding to each of the set final reference vector and the second predetermined number of answers, and outputs the second predetermined number (distance) Can be classified into clusters corresponding to the reference vectors having the highest similarity (closest distance).

Table 8 shows an answer classification method using the non-background learning method according to an embodiment of the present invention. Assuming that the first predetermined number of answers is four, namely, "captured beautiful," "captured beautiful," "is a person who captures animals," and "cares for animals," the morphological qualities Are like "beautiful", "animal", "captured", "catch", "look at". Among the extracted qualities, "Capture", "Beautiful", and "Animal" can be selected as the qualities for the non-geographic learning method. If each of the first predetermined number of answers is converted into a qualitative vector, 1, 1, 0), "caught the beautiful" → (0,1,0), "is the person who captures animals" → (1,0,1), "cares animals" → ).

[Table 8]

이므로 가장 거리가 가까운 클러스터 A로 분류할 수 있고, "아름이를 잡았다"의 경우 클러스터 A와의 거리는 1이고, 클러스터 B와의 거리는

이므로 가장 거리가 가까운 클러스터 A로 분류할 수 있다. 또한, "동물들을 보살핀다"의 경우 클러스터 A와의 거리는

이고, 클러스터 B와의 거리는 0이므로 가장 거리가 가까운 클러스터 B로 분류할 수 있고, "동물을 생포하는 사람이다"의 경우 클러스터 A와의 거리는

이고, 클러스터 B와의 거리는 1이므로 가장 거리가 가까운 클러스터 B로 분류할 수 있다.In one embodiment, the answer classification unit 214-3 can set the initial reference vector of the cluster A to (1,1,0) and set it to the initial reference vector of the cluster B (0,0,1). The answer classifying section 214-3 classifies the initial reference vectors {(1,1,0), (0,0,1)} of the cluster A and the cluster B and the feature vectors {(1 (Distance) between the coordinates (0, 1, 0), (0, 1, 0) Each of the first predetermined number of answers can be classified into clusters (close in distance). That is, in the case of "captured beautiful ", the distance from the cluster A is 0, and the distance from the cluster B is

The distance to the cluster A is 1, and the distance from the cluster B to the cluster B is 1

, It can be classified into cluster A closest to the distance. Also, in the case of "caring for animals ", the distance from cluster A

, And the distance from the cluster B is 0, so that it can be classified into the cluster B closest to the closest distance. In the case of "the person who captures an animal"

And the distance from the cluster B is 1, so that it can be classified into the cluster B having the closest distance.

In addition, the answer classification unit 214-3 can reset the initial reference vectors by averaging the respective coordinate values of the feature vectors of the first predetermined number of answers classified. In other words, the average of the coordinate values of "captured beautiful" (1,1,0) and "captured beautiful" (0,1,0) classified as cluster A is equal to (0.5, 1, 0) (1, 0, 1) and "cares animals" (0, 0, 1), which are classified as cluster B, (0.5, 0, 1) and can be set back to the reference vector of the cluster B.

In addition, the answer classification unit 214-3 can calculate the similarity between the newly set reference vectors and the feature vectors of the first predetermined number of answers, and classify them into clusters having the highest similarity. The answer sorting section 214-3 can calculate the degree of similarity and classify it into clusters having the highest similarity repeatedly until there is no change in the first predetermined number of answers included in each cluster, The reference vector can be set as the final reference vector. That is, the answer classifying section 214-3 classifies the new reference vectors {(0.5, 1, 0), (0.5, 0, 1)} of the clusters A and B and the feature vectors { , 1, 0), (0, 1, 0), (1,0,1), (0,0,1) (0, 1, 0) and "beautiful" (0, 1, 0) The final reference vector of the cluster A is (0.5, 1, 0) and the final reference vector of the cluster B is (0.5, 0, 1), since the cluster A and the cluster B do not change at all. .

In addition, the answer scoring device 200 performs scoring (not using the natural language processing) on the plurality of input answers received from the student terminal 120-n using the non-background learning method You may.

FIG. 7 is a diagram illustrating a procedure of a method of scoring a plurality of input answers using a non-background learning method, which is a machine learning method according to an embodiment of the present invention.

As shown in FIG. 7, the answer grading device 200 can set at least two first classification criteria corresponding to each of at least two classification classes according to the type of the input answers (S710) A feature corresponding to a predetermined characteristic element for the answer can be extracted (S720). In addition, the answer scoring apparatus 200 calculates a feature vector, which is a coordinate value indicating a position in the feature space, with each feature as an axis (S730), and calculates a feature vector of each of the plurality of input answers and at least two (S740). In step S740, each of the plurality of input answers is classified into classification classes corresponding to the first classification reference having the highest similarity among the vectors. In addition, the answer scoring unit 200 performs an average of each coordinate of the feature vector of each of the plurality of input answers classified by the classification grade to form at least two second classification references (S750) To at least two second classification criteria (S760). In addition, the answer scoring device 200 compares the feature vector of each of the plurality of input answers with the feature vector of each of the at least two second classifiers, and classifies the classification class corresponding to the second classifier having the highest similarity among the vectors (S770), and a score may be assigned to each of the classification scores (S780).

In the map learning method, the grading result of the grader for the first predetermined number of answers selected by the answer creating unit 214-1 may be received and analyzed to form a consistent classification standard. In one embodiment, the answer classifying section 214-3 classifies the probability that a first predetermined number of answers divided into clusters of at least two clusters by the scorer is included in each cluster, The probability of emergence in the first predetermined number of answers of the qualities extracted from the candidate is calculated to form a classification criterion and the second predetermined number of answers is classified into one of the at least two clusters .

Table 9 shows an answer classification method using a map learning method according to an embodiment of the present invention. The first predetermined number of answers are five of "I captured beautiful", "I caught beautiful", "I caught the beautiful", "I am a person who captures animals" and "I care for animals" The clustering results of the scorers were "captured", "" [correct] captured, "" [correct], and "beautiful". , "Animal", "Animal", "Animal", and "Animal" are extracted from the first predetermined number of answers in the case of " The answer classifying section 214-3 classifies the "captured", "beautiful", and "animal" among the extracted features as a map learning method And the number of answers which are the correct answers among the 5 answers of the first scoring target is 3, so that the first predetermined number of answers is the correct answer It is possible to calculate that the probability that the first predetermined number of answers will be included in the cluster having the wrong answer is 0.4 because the probability of inclusion in the cluster is 0.6 and the number of answers with the wrong answer is 2. Further, The conditional probability that the selected qualities will be included in each cluster can be calculated by the following method: Among the selected qualities for the map learning method, "beautiful" is included in all three answers, The probability that an included answer is included in a correct answer cluster can be calculated to be 1, and not included in all two incorrect answer answers, so the probability that an answer containing "beautiful" is included in an incorrect answer cluster can be calculated as 0. , Among the selected qualities for the map learning method, "sappo" is included in two out of three correct answer answers. The probability that an answer containing "caught" will be included in an incorrect answer cluster can be calculated to be 0.5, since it is calculated to be 0.67, and it is included in one of two incorrect answer answers. Since the "animal" among the qualities is not included in all three of the answers, the probability that the answer including the "animal" is included in the correct answer cluster can be calculated as 0, Animal "is included in the wrong answer cluster can be calculated as 1. In one embodiment, the answer classifying section 214-3 classifies the answer classifying section 214-3 into Naive Bayes Classification, Hidden Markov Model, Neural Network, Logistic Regression, kNN a k-nearest neighbor algorithm, and a support vector machine, but the present invention is not limited to this method.

The answer sorting section 214-3 also uses the probability that the first predetermined number of answers will be included in each cluster and the appearance probability in the first predetermined number of answers extracted from the first predetermined number of answers And classify each of the second predetermined number of answers into one of a correct answer and an incorrect answer. As exemplified in Table 7, if any of the second predetermined number of answers is "successful in capturing the beautifulness", the extracted morpheme qualities are "beautiful", "captured", "successful" The probability of correct answer is 0.6 × 1.0 × 0.67 = 0.402, and the probability of incorrect answer is calculated as 0.4 × 0.0 × 0.5 = 0 because it includes only "beautiful" and " can do. Therefore, the answer classifying section 214-3 can classify the answer as "having succeeded in capturing the beautiful" as the correct answer cluster because the probability of correct answers is higher than the probability of being an incorrect answer.

[Table 9]

The gradient map learning method can form a sorting criterion for classifying the grading object answer into any one cluster by using the combination of the above-described map learning method and the non-map learning method. That is, in the rough learning method, the grading result of the grader for the first predetermined number of answers selected by the learning-answer creating unit 214-1 is received, the qualities included in the first predetermined number of answers are extracted, And classifies the second predetermined number of answers into one cluster by using the formed classification criterion, and then assigns the answer having a probability of being included in each cluster to a predetermined value (for example, 90% or more) as a first predetermined number of answers So that the classification standard can be formed again.

Table 10 shows a method of forming a classification criterion using the coherence learning method according to an embodiment of the present invention. The answer classification unit 214-3 can form a primary classification rule using a map learning method. That is, the first predetermined number of answers are four, namely, "captured beautiful," "captured beautiful," "are people who capture animals," and "take care of animals. The clustering result of the scorer is classified into one of "cluster" and "wrong answer", and the clustering result of the scorer "captured the beautiful", "the [correct answer] caught the beautiful", " If the morpheme qualities are extracted from the first predetermined number of answers, it is like "beautiful", "animal", "captured", "person", "grab", " Among the extracted qualities, "Sopo", "Beautiful", and "Animal" can be selected as the primary classification criteria for the learning method. When the estimated probability of correct answer or wrong answer of the second predetermined number of target answers that are not classified is calculated by using the first classification standard, the probability of being classified as the correct answer cluster of " The probability of being classified as a correct answer clusters is 95%, the probability of being classified as a correct answer cluster is "92%," and "the person who takes care of animals" , The probability of being classified as an incorrect cluster is 91% and the probability of being classified as an incorrect cluster is 93%. , "Animal discovery" has a 91% probability of being classified as an incorrect cluster, and 75% of the probability that a cluster of "An estimated 82% probability of being classified as a cluster answered, "We also found traces of a bear between ten" is the estimated probability of being classified as incorrect cluster 59%. "I have caught the bears of beautiful bears," "I caught the bears beautifully," "I captured beautiful bears," "Those who take care of animals," "Those who treat animals, Person ", "person who finds an animal" can be added to the first predetermined number of answers. The answer classifying section 214-3 can form a second classifying criterion from the expanded first predetermined number of answers and calculate a correct answer or a wrong answer probability probability of the undecided classifying answer by applying the second classifying criterion It can be reclassified into one cluster. The answer sorting section 214-3 repeats the process of classifying the classification and reclassifying the undivided scoring target answer until the number of the first predetermined number of answers is converged to a specific value, A more accurate classification criterion can be formed. For example, the "recapture of the beast bear bears" includes only "catch" among the qualities of the first classification criterion, so the probability of correct answer to be classified as the correct answer cluster is 75%, but " Among the qualities of the second classification criterion added to the first predetermined number of answers, "caught beautifully" and "captured beautiful bears" can be used to calculate the answer probability probability of "catch" and " have. Therefore, the probability of estimating the correct answer as "caught a beast bear bears again " is increased to 93%, and it is possible to add" bearish bears bears again "to the first predetermined number of answers.

[Table 10]

The ensemble learning method can represent a method of clustering the scoring candidate answers into any one of the clusters using a combination of the non-background learning method, the guidance learning method, and the edge guidance learning method. Since the ensemble learning method uses mixed results of several machine learning methods, more reliable classification results can be obtained than when using one specific machine learning method.

The scoring unit 214-4 may score a second predetermined number of answers classified into the respective clusters according to the classification level. In one embodiment, if the second predetermined number of answers is classified into two different clusters, the scoring unit 214-4 may assign scores to correct answers and wrong answers for each cluster, If the answer is classified into four different clusters, scores of three or more differential scores such as 3, 2, 1, and 0 may be assigned to each cluster.

The storage unit 220 stores information on a plurality of scores to be scored, a classification criterion formed by the processing unit 210, a quality vector, a score extracted from each of the first and second predetermined number of answers, Can be stored. In addition, the storage unit 220 may include a corpus (including a partly attached corpus, a phrase bundle corpus, etc.) for performing a natural language process of the language processing unit 212, a thesaurus, a negative expression recognition unit 212-4 Negative indefinite phrase, negative indefinite phrase, indefinite phrase, indefinite phrase, double indefinite expression, etc. As an example, the storage unit 220 may be a ROM (Read Only Memory), a RAM (Random Access Memory), a CD (Compact Disc) -ROM, a magnetic tape, a floppy disc, Optical data storage, or carrier waves (e.g., transmission over the Internet), but are not limited to such implementations.

The communication unit 230 includes a plurality of student terminals 120-1 to 120-n and a plurality of scorer terminals 130-1 to 130-n and various signals for scoring an answer to be scored Lt; / RTI >

Although the method has been described through particular embodiments, the method may also be implemented as computer readable code on a computer readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission over the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the above embodiments can be easily deduced by programmers of the present invention.

Although the present invention has been described in connection with certain embodiments thereof, it should be understood that various changes and modifications may be made therein without departing from the spirit and scope of the invention as will be apparent to those skilled in the art to which the invention pertains. something to do. It is also contemplated that such variations and modifications are within the scope of the claims appended hereto.

100: Answer scoring environment 110: Server
120-1, ... , 120-n: a student terminal 130-1, ... , 130-n: the scorer terminal
N: Network 200: Answer scoring device
210: processing section 220:
230: communication unit 240: system bus
212: language processing unit 214: classification scoring unit
212-1: Document normalization unit 212-2: Morphological analysis unit
212-3: Part attachment part 212-4: Negative expression recognition part
212-5: Saddle section 212-6: Replacement section
212-7: Dependence analysis unit 214-1:
214-2: qualification extraction unit 214-3: answer classification unit
214-4: scoring unit

Claims (11)

A method for scoring a plurality of consecutive input answers,
a) extracting a feature corresponding to a predetermined characteristic element with respect to the plurality of consecutive input answers;
b) setting at least two first classification criteria based on the extracted qualities to classify the types of consecutive input answers, wherein each of the first classification criteria comprises at least one of: A feature vector which is a coordinate value indicating a position of the feature vector,
c) forming a feature vector, which is a coordinate value indicating a position in a feature space around each feature using the extracted feature,
d) Wherein the feature vector of each of the plurality of consecutive input answers is compared with the feature vector of each of the first classifiers so as to classify the feature vectors of the plurality of consecutive answers as a classification class corresponding to the first classifier having the highest similarity among the vectors, Classifying the input answers;
e) Performing averaging of each coordinate of each of the feature vectors according to a grade and forming a second classification reference for each class, wherein each of the second classification references is characterized by a feature A feature vector which is a coordinate value indicating a position in space,
f) a feature vector of each of the plurality of consecutive input answers and a feature vector of each of the second classifiers is compared with each other, and a class classification corresponding to a second classifier having a highest degree of similarity between the vectors, Classifying each of the answer input answers;
g) assigning a score according to the grade classified in step f)
The method comprising the steps of: The method according to claim 1,
Forming a new second classification criterion by applying a plurality of consecutive input answers sorted by the grade in step f) to step e); and performing the step f) with the new second classification criterion And sequentially classifying each of the plurality of consecutive input answers,
Performing the step of assigning a score according to the classified class in step g) when the second classifying criterion of step f) is the same as the second classifying criterion newly formed by applying the step e)
A method of scoring multiple input answers. 3. The method of claim 2,
The method of any one of the preceding claims, wherein the step (d) to step (b) is performed using any one of K-means Clustering, Hierarchical Clustering, Density-Based Clustering or Grid-Based Clustering. g), < / RTI >
A method of scoring multiple input answers. The method according to claim 1,
The step a)
Performing natural language processing on the plurality of consecutive input answers;
Extracting the qualities using the results of the natural language processing;
The method comprising the steps of: 5. The method of claim 4,
Wherein the performing of the natural language processing comprises:
A document normalization process including a sentence separation step, a spacing correction step, a spelling correction step, an abbreviation expansion step, and a symbol removal step for the plurality of consecutive input answers; Analyzing morphemes for the plurality of consecutive input answers; A step of attaching parts of the morpheme analysis result; Recognizing a negative expression of the plurality of consecutive input answers; Processing a bundle for at least two morphemes included in the result of attaching a part of speech; A replacing step of converting a phrase or phrase included in the plurality of consecutive input answers into a predetermined standard expression; And a dependency analysis process for analyzing a dependency structure between a morpheme or a word phrase included in the plurality of consecutive input answers. 6. The method of claim 5,
The above-
Analyzing the morpheme, attaching the part-of-speech, morphological qualities formed by performing the morpheme analysis, and processing the phrases formed on the basis of the phrases included in the plurality of consecutive input answers, (N-gram) qualities formed by including a predetermined basic phrase qualities, a dependency qualities formed by performing the dependency relationship analysis process, a predetermined number of adjacent morpheme qualities or the phrase qualities A number of written answer input scoring methods. 2. The method of claim 1, wherein step d)
A feature vector of each of the plurality of consecutive input answers and a feature vector of the at least two first classification references are compared to classify classifications corresponding to classification classes having the highest similarity among the vectors, And classifying the classification score into a classification score having the highest score when at least two grades exist. The method of claim 1, wherein step (f)
A feature vector of each of the plurality of consecutive input answers and a feature vector of the at least two second classification references are compared to classify classifications corresponding to classification classes having the highest degree of similarity between vectors, And classifying the classification score into a classification score having the highest score when at least two grades exist. The method according to claim 1,
Wherein the at least two classification classes comprise:
A multiple-entry input answer scoring method that is a classification score of correct answers and incorrect answers, or three or more classification grades with three or more differential scores. A program stored on a recording medium for causing a computer to execute a plurality of recorded answer score scoring methods according to any one of claims 1 to 9. 9. A recording medium for storing a program for causing a computer to execute a plurality of consecutive answer answer scoring methods according to any one of claims 1 to 9.

Images