JP7104965B2 - Program and clustering device - Google Patents

Description

The present invention relates to a program, an information storage medium and a clustering device.

In order to nurture and measure the thinking ability of examinees (answers), the need to impose not only multiple-choice questions but also descriptive questions has become socially recognized. When imposing descriptive questions, highly reliable grading in a short period of time is required. If handwriting recognition technology is used, it becomes possible to perform scoring support that assists scoring by humans and automatic scoring. Prototypes of learning systems that recognize handwritten answers have been published in arithmetic and mathematics, where descriptive answers are highly necessary (for example, Non-Patent Documents 1 to 3).

Masato Suzuki et al., "Development of Basic Mathematics Learning Support System Based on Analysis of Handwritten Mathematical Formulas", The Institute of Electronics, Information and Communication Engineers, IEICE, ET2009-129, pp.147-152 (2010) Satoshi Chiba et al., "Prototype of Mathematics e-Learning System on Tablet PC Using Recognition of Handwritten Numerical Expressions", Information Processing Society of Japan Research Report, Vol.2014-CE-127, No.10, p.1-5 (2014) Wataru Konishi, et al., "Arithmetic/Mathematical Automatic Scoring System Using Recognition of Handwritten Formulas", Information Processing Society of Japan Research Report, Computer and Education Research Group Report, 2016-CE-133 (7), 1-7 (2016)

In human grading, when answers are randomly presented to the grader, the grading often fluctuates. In large-scale tests, multiple people score, and if the score difference exceeds a certain level, it becomes necessary for a third party to score, which increases the cost and time of using a third party. On the other hand, in automatic scoring, it is essential to have an environment in which examinees can check the scoring results and point out scoring errors. A realistic method would be to reject things and pass them on to human scoring. Therefore, it is important to suppress variations in scoring by graders, regardless of scoring support or automatic scoring.

The present invention has been made in view of the problems described above, and its object is to provide a program and information system capable of improving the efficiency of grading by reducing grading errors and grading variations of graders. An object of the present invention is to provide a storage medium and a clustering device.

(1) The present invention is a program for classifying a plurality of patterns of handwritten mathematical formulas that have been input by handwriting. a feature extracting unit for extracting at least two features including the direction feature among features relating to the number of appearances and features relating to the position of appearance of symbols; More specifically, the present invention relates to a program that causes a computer to function as a classifier that classifies a plurality of handwritten mathematical formula patterns into a plurality of groups according to similarity. The present invention also relates to a computer-readable information storage medium storing a program for causing a computer to function as each of the above sections. The present invention also relates to a clustering device including the above units.

According to the present invention, from a handwritten mathematical expression pattern, a plurality of features, including a direction feature, a feature relating to the number of occurrences of symbols, a feature relating to the number of occurrences of positional relationships between symbols, and a feature relating to the appearance position of symbols, the direction At least two features including features are extracted, and a plurality of handwritten mathematical formula patterns are classified into a plurality of groups according to similarity based on the extracted at least two features, thereby classifying the handwritten mathematical formula patterns into similar ones with accuracy. They can be grouped together (clustered) well, and scoring errors and variations in scoring by graders can be reduced, and the efficiency of scoring can be improved.

(2) In the program, information storage medium, and clustering device according to the present invention, the feature extraction section may extract features relating to the number of symbols appearing at each position as the features relating to symbol appearance positions.

According to the present invention, handwritten mathematical formula patterns can be clustered with high accuracy.

(3) Further, in the program, information storage medium, and clustering device according to the present invention, the feature extracting section may extract, as features relating to symbol appearance positions, features relating to the number of occurrences of the positional relationship for each position.

According to the present invention, handwritten mathematical formula patterns can be clustered with high accuracy.

(4) Further, in the program, information storage medium, and clustering device according to the present invention, the feature extraction unit recognizes a mathematical formula from the handwritten mathematical formula pattern by using a mathematical formula recognition engine, and based on the recognition result, identifies a feature related to the number of occurrences of symbols. , a feature relating to the number of appearances of the positional relationship between symbols, and a feature relating to the appearance positions of the symbols may be extracted.

According to the present invention, handwritten mathematical formula patterns can be clustered with high accuracy.

An example of the functional block diagram of the clustering apparatus of this embodiment. The figure which shows an example which nonlinearly normalizes the off-line pattern in a circumscribed rectangle, and extracts an eight-direction feature. FIG. 4 is a diagram showing an example of symbol set features extracted from online patterns; FIG. 4 is a diagram showing an example of relational set features extracted from online patterns; FIG. 10 is a diagram showing an example of position set features extracted from online patterns; FIG. 10 is a diagram showing an example of position set features extracted from online patterns; The figure which shows an example of a 3x3 Gaussian filter. FIG. 4 is a diagram showing an example of position-dependent symbol set features and position-dependent relationship set features extracted from online patterns; FIG. 10 is a diagram showing an example of mathematical expression candidate features extracted from an online pattern; FIG. 4 is a diagram showing an example of symbol set features extracted from an offline pattern; FIG. 4 is a diagram for explaining extraction of relational set features from offline patterns; FIG. 4 is a diagram for explaining extraction of relational set features from offline patterns; FIG. 4 is a diagram showing an example of relational set features extracted from offline patterns; FIG. 4 is a diagram for explaining extraction of position set features from offline patterns;

The present embodiment will be described below. It should be noted that the embodiments described below do not unduly limit the content of the present invention described in the claims. Moreover, not all the configurations described in the present embodiment are essential constituent elements of the present invention.

1. Configuration FIG. 1 shows an example of a functional block diagram of a clustering device according to this embodiment. Note that the clustering apparatus of this embodiment may have a configuration in which some of the constituent elements (each part) in FIG. 1 are omitted.

The input unit 160 is for the user to input handwritten mathematical formula patterns (answers to handwritten mathematical formulas) with a writing medium (pen, fingertip, etc.). online method). The input unit 160 detects coordinate data representing the position (writing point) of the writing medium from when the writing medium touches the writing surface until it leaves the writing surface at regular time intervals, and detects the detected coordinate data string (coordinate point series, online pattern ) is output to the processing unit 100 as stroke data. A vector from the end point of a stroke to the start point of the next stroke is called an off-stroke (stroke vector), and a continuous series of strokes and off-strokes is called a stroke string. In the off-line method, the input unit 160 reads a handwritten mathematical formula pattern (referred to as an off-line pattern) written on paper or the like as a black-and-white image or a grayscale image using a scanner or the like.

The storage unit 170 stores programs and various data for causing the computer to function as each part of the processing unit 100, and functions as a work area for the processing unit 100, and its function can be realized by a hard disk, RAM, or the like.

The display unit 190 outputs an image generated by the processing unit 100, and its function can be realized by a display such as a touch panel, LCD, or CRT that also functions as the input unit 160. FIG.

The processing unit 100 (processor) performs processing such as feature extraction processing, classification processing, and display control based on data (coordinate data, image data) from the input unit 160, programs, and the like. The processing unit 100 performs various processes using the main storage unit in the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs. The processing unit 100 includes a feature extraction unit 110 , a classification unit 112 and a display control unit 120 .

The feature extraction unit 110 extracts, from a handwritten mathematical expression pattern input by handwriting, a directional feature, a feature relating to the number of occurrences of symbols (how many of which symbols appear), and the number of occurrences of positional relationships between symbols (which positional relationships At least two features, including the directional feature, are extracted from the features relating to the appearance position of the symbol (in which position the symbol appears). Further, the feature extracting section 110 may extract features relating to the number of appearances of symbols at each position (how many symbols appear at which positions) as features relating to the appearance positions of symbols. Features relating to the number of occurrences of positional relationships (how many of which positional relationships appear at which positions) may be extracted. Further, the feature extracting unit 110 recognizes a mathematical formula from the handwritten mathematical formula pattern by means of a mathematical formula recognition engine, and based on the recognition result, features relating to the number of occurrences of symbols, features relating to the number of occurrences of positional relationships between symbols, and You may extract the feature regarding an appearance position. Symbols are numbers, letters (English letters, Greek letters), operators (arithmetic operators, logical operators, set operators, relational operators), symbols (addition symbols, operation symbols, multiplication symbols) that appear in formulas. , integral symbols, limit symbols (lim), special symbols (∞, etc.), functions (log, sin, cos, tan, etc.), parentheses (large, medium, small).

The classification unit 112 classifies the plurality of handwritten mathematical formula patterns into a plurality of groups according to the degree of similarity based on the plurality of features extracted from each of the plurality of handwritten mathematical formula patterns.

The display control unit 120 controls the display unit 190 to display a plurality of handwritten mathematical formula patterns classified into a plurality of groups by the classification unit 112 for each group.

2. The method of the present embodiment In the method of the present embodiment, low-level features (orientation features), symbol set features (features related to the number of occurrences of symbols), relation set features (features related to the number of occurrences of positional relationships features) and position set features (features related to the appearance position of symbols), at least two features including a direction feature are extracted, and based on the plurality of extracted features, a plurality of patterns of handwritten mathematical formulas are divided into a plurality of similar patterns. classified (clustered) into groups (clusters) of In the case of the online method, it is also possible to extract position-dependent symbol set features (features relating to the number of occurrences of symbols for each position) and position-dependent relationship set features (features relating to the number of occurrences of positional relationships for each position). Low-level features are features obtained without symbol recognition or extraction; It is a feature obtained by performing extraction and extraction. An example of extracting from one to a maximum of six features will be described below.

2.1. Low-level features (orientation features)
As low-level features, the whole handwritten mathematical formula pattern (online pattern, offline pattern) is divided into M×N sections, and 4 or 8 direction features are extracted in each section. At this time, each section may be blurred by applying a Gaussian filter in order to increase resistance to positional deviation.

Specifically, first, the circumscribing rectangle of the input handwritten mathematical expression pattern is extracted. In the case of an online pattern, it can be extracted from the maximum value and minimum value of the X-coordinate and Y-coordinate of the writing point. Off-line patterns can be extracted from the maximum and minimum values projected onto the X-axis and Y-axis. Next, non-linear normalization is performed. Non-linear normalization is a process for uniforming the two-dimensional spatial density of a handwritten mathematical expression pattern by compressing areas with low density and expanding areas with high density.

Next, the directional features are extracted. In the case of online patterns, feature points such as endpoints and bending points are extracted from the stroke string, and the position and direction of each feature point (4, 8, 16, or more directions are quantized). ) are extracted. For offline patterns, directional features are extracted using methods such as those described in Ning Sun, Masato Abe, Yoshiaki Nemoto: Highly Accurate Handwriting Using Improved Directional Element Features and Subspace Method Character Recognition System, Transactions of the Institute of Electronics, Information and Communication Engineers, D-2, 78 (6), pp922-930 (1995). FIG. 2 shows an example of non-linear normalization of offline patterns in a circumscribing rectangle and extraction of directional features in eight directions.

The directional features are then partitioned into M×N partitions and Gaussian filtered. For example, the Gaussian filter G(X, μ, σ) is represented by the following equation (1), where μ is the average, σ is the standard deviation, and t is the mask width in the X direction.

方向特徴など、シンボルの認識や抽出に影響されない低レベル特徴の利点は、数式認識やシンボル認識などの失敗に影響されないことである。しかし、数式の書き方の違いには影響を受ける。そこで、本実施形態の手法では、シンボルの認識や抽出を伴う特徴抽出（シンボル集合特徴、関係集合特徴、位置集合特徴の抽出）と併用する。

An advantage of low-level features that are immune to symbol recognition and extraction, such as directional features, is that they are immune to failures such as formula recognition and symbol recognition. However, it is affected by the difference in how to write formulas. Therefore, the technique of this embodiment is used together with feature extraction (extraction of symbol set features, relation set features, and position set features) involving symbol recognition and extraction.

2.2. Feature Extraction from Online Patterns For online patterns, a formula recognition engine recognizes formulas, and based on the recognition results, extracts symbol set features, relation set features, and position set features. In mathematical expression recognition, symbol segmentation, symbol recognition, and structural (syntactic) analysis are performed.

2.2.1. Symbol set features (features related to the number of occurrences of symbols)
A symbol set feature (BoS: Bag of Symbols) is a feature vector (symbol appearance vector) in which the number of occurrences of each symbol that appears in the result of formula recognition is arranged. Here, there are 101 types of symbols, and the symbol appearance vector is a 101-dimensional vector.

FIG. 3 shows an example of a symbol set feature. FIG. 3 shows the symbol set feature (symbol appearance vector) extracted from the recognition result of the handwritten mathematical expression pattern "1/2+2/3". The symbol appearance vector shown in FIG. 3 (the second row of the table) has one symbol "1", two symbols "2", one symbol "3", and one symbol "+" in the formula. , indicating that the symbol "fraction bar" appears twice. The symbol set features are represented by the same feature vector if the symbols appearing and the number thereof are the same even if they are different mathematical expressions. For example, the formulas "1/2+2/3" and "2/3+1/2" are represented by the same feature vector.

When multiple recognition results (recognition candidates) are output from the formula recognition engine, symbols are counted not only from the first recognition result (the recognition result with the highest score), but also from the second and subsequent recognition results. , may be added to calculate the symbol appearance vector. Also, a weight of 1 is given to the result counted from the first recognition result, 1/2 to the result counted from the second recognition result, and 1/4 to the result counted from the third recognition result. can be added with Also, the probability of being each symbol (probability of recognition) may be added.

2.2.2. Relation set features (features related to the number of occurrences of positional relationships between symbols)
A bag of relations (BoR) feature is a feature vector (relation appearance vector) in which the number of appearances for each positional relationship between symbols appearing in the result of formula recognition is arranged. Here, six positional relations (horizontal, superscript, subscript, upper, lower, inside) are considered as positional relations between symbols. , the positional relationship between symbols is not limited to this. In order to count this positional relationship, a symbol relationship tree representing the relationship between symbols is constructed from the recognition result of the handwritten mathematical formula pattern, and the number of appearances for each positional relationship is counted from the constructed symbol relationship tree.

FIG. 4 shows an example of relationship set features. FIG. 4 shows a symbol relation tree constructed from the recognition results of the handwritten mathematical expression pattern "1/2+2/3" and relation set features (relation appearance vectors) obtained from the symbol relation tree. The relation occurrence vector shown in FIG. 4 indicates that two positional relations "horizontal", two positional relations "upper", and two positional relations "lower" appear in the formula. Relational set features are represented by the same feature vector even if they are different mathematical formulas if the positional relationships that appear and the number of them are the same. For example, the formulas "1/2+2/3" and "4/3-3/2" are represented by the same feature vector.

As with the symbol set feature, when multiple recognition results are output from the formula recognition engine, the positional relationships are counted not only from the first recognition result, but also from the second and subsequent recognition results, and these are added. A relationship occurrence vector may be calculated. Also, a weight of 1 is given to the result counted from the first recognition result, 1/2 to the result counted from the second recognition result, and 1/4 to the result counted from the third recognition result. can be added with Also, the probability of being each symbol may be added.

2.2.3. Position set features (features related to symbol occurrence positions)
A position set feature (BoP: Bag of Positions) is a feature vector representing the appearance position of a symbol appearing in the result of formula recognition. In order to obtain the appearance positions of symbols, first, the symbol relation tree constructed from the recognition results is divided into M×N sections. Each partition contains at most one symbol or no symbols. The number of divisions (M×N) is determined from the size of the symbol relation tree, the number of upper and lower relations (upper, lower, superscript, subscript) and the number of left and right relations (horizontal, superscript, subscript). The number of divisions may be obtained from the symbol relation tree of the correct formula. Then, of the i-row, j-column partitions P _ij , 1 is given to partitions containing symbols, and 0 is given to partitions not containing symbols, and is represented by a matrix P=(P _ij ). Since the number of M×N divisions of the symbol relation tree varies from formula to formula, to match the largest number of divisions, it is aligned to the upper left and given 0s to the right and bottom sections that do not contain symbols. Then, the elements (0 or 1) of the partition P _ij are connected row by row and further column by column to form a feature vector. This makes it possible to extract feature vectors of the same size from a plurality of formulas having symbol relation trees of different sizes.

5 and 6 show examples of location set features. FIG. 5 shows the symbol relation tree constructed from the recognition result “x ² /2+2/3”, the symbol relation tree divided into M×N, and the position set features obtained from the divided symbol relation tree. 6 shows the symbol relation tree constructed from the recognition result "1/2+2/3", the symbol relation tree divided into M×N, and the position set features obtained from the divided symbol relation tree. . In the example shown in FIG. 5, there are three horizontal relationships (two horizontal, one superscript) and three vertical relationships (two vertically parallel upper and lower, one superscript). The tree is divided into 4x4 partitions. In the example shown in FIG. 6, since there are two horizontal relations (two horizontals) and two vertical relations (two vertically parallel upper and lower), the symbol relation tree is divided into 3×3 sections. Although it is divided, in order to be able to compare with the example shown in FIG. 5, the 4×4 partitions are aligned to the upper left and 0 is given to the blank partitions. The position set _{feature ( before application} of the _{Gaussian filter} ) _shown in _FIG . The position set _{features (} before _application of the _{Gaussian filter} ) _shown in _FIG . Indicates that the symbol appears.

In addition, in order to increase resistance to positional deviation, a 3×3 Gaussian filter shown in FIG. 7 may be applied to each section. In this case, no value is distributed to partitions that have no neighbors. The elements resulting from applying the Gaussian filter to each section are connected for each row and further for each column to form a feature vector. 5 and 6 show the location set features after application of the Gaussian filter.

The position set feature may be combined with the symbol set feature to extract the symbol set feature from each partition. This feature is called the Position based Bag of Symbols (Pb_BoS) feature. A position-dependent symbol set feature is a feature relating to the number of occurrences of symbols for each position (partition). Also, the relationship set feature may be combined with the position set feature to extract the relationship set feature from each partition. This feature is called a Position based Bag of Relations feature (Pb_BoR). The positional dependency set feature is a feature relating to the number of occurrences of positional relationships for each position. The symbol set feature and the relation set feature are extracted from the entire mathematical expression, while the position dependent symbol set feature and the position dependent relation set feature are extracted from each division of the symbol relation tree. One symbol is included in each partition, and several positional relationships are included in each partition. As with the location set feature, each interval may be Gaussian filtered and blurred to increase tolerance to misalignment. As a result, feature values occur in multiple symbols as a result of Gaussian filter blurring, even though each partition contains only one symbol.

FIG. 8 shows an example of a position dependent symbol set feature and a position dependent relation set feature. FIG. 8 shows a symbol relation tree constructed from the recognition result "1/2+2/3", a symbol relation tree divided into M×N (here, 3×3), and a Gaussian filter for each divided section. and a position dependent symbol set feature (Pb_BoS) consisting of the symbol occurrence vectors of each parcel P _ij and a position dependent relationship set feature (Pb_BoR) consisting of the relation occurrence vectors of each parcel P _ij .

In addition to these features, features can also be extracted from candidate recognition results. This is called a formula candidate feature. First, a plurality of recognition candidates are obtained by recognizing a mathematical expression from a handwritten mathematical expression pattern. Divide the symbol relation tree of all recognition candidates into M×N partitions. Recognition candidates with a small number of divisions are aligned to the upper left in accordance with the maximum number of divisions, as shown in FIG. A Gaussian filter may or may not be applied to each partition, but since the Gaussian filter is applied in the position set feature, the position dependent symbol set feature, and the position dependent relationship set feature described above, it is not applied here. Then, the indices of the symbols appearing in each of the maximum M×N intervals are obtained. A feature vector is obtained by concatenating these for each row and further for each column.

FIG. 9 shows an example of mathematical expression candidate features. FIG. 9 shows a symbol relation tree constructed from the recognition result “x ² /2+2/3” and divided into M×N (here, 4×4), the index of the symbol in each section, and the concatenation of the tree. The formula candidate features obtained by

2.3. Feature Extraction from Offline Patterns In the case of offline patterns, it is somewhat more difficult to extract symbols, etc. than online patterns. explain. Of course, feature extraction can also be performed using mathematical expression recognition.

2.3.1. Symbol set features (features related to the number of occurrences of symbols)
When extracting symbol set features, connected components (connected components of black pixels if the background is white pixels) are extracted from the offline pattern, symbol recognition (single character recognition) is applied to the extracted connected components, It outputs recognition results (high-ranking recognition candidates) and their probability values (recognition scores). Then, the probability values for each symbol are arranged to form a symbol appearance vector (101-dimensional vector). In the off-line method, symbols may be separated. For example, the symbol "=". In order to prevent this, a rule is introduced to integrate and re-recognize separately recognized symbols. In the case of "=", if there are two consecutive "-", it is re-recognized as "=", and if the probability is higher than the threshold, each connected element is further connected and re-recognized as "=" do.

As shown in the following equation (2), the symbol set feature S is obtained by summing the symbol appearance vectors S(C) obtained from the recognition results for each connected element C.

図１０に、オフラインパターンから抽出したシンボル集合特徴の一例を示す。図１０には、手書き数式パターン「＝ａ／８」の各連結要素に対する認識結果（認識スコア）から得たシンボル出現ベクトルと、これらシンボル出現ベクトルの和をとって求めたシンボル集合特徴が示されている。図１０に示すシンボル集合特徴は、数式中に、シンボル「ａ」が「０．９」個、シンボル「＝」が「０．８」個、シンボル「fraction bar（分数罫）」が「０．７」個、シンボル「８」が「０．９」個出現していることを示しており、実際のシンボルの出現個数と概ね一致している。この例では、確率値を有限個の候補で打ち切っているが、確率値以外にも順位をもとに、第１位には１、第２位には１／２、第３位には１／４などの重みをつけて加算することもできる。このとき、第１位の結果のみを用いることも可能である。

FIG. 10 shows an example of symbol set features extracted from offline patterns. FIG. 10 shows symbol appearance vectors obtained from the recognition results (recognition scores) for each connected element of the handwritten mathematical expression pattern “=a/8”, and symbol set features obtained by summing these symbol appearance vectors. ing. The symbol set feature shown in FIG. 10 is such that the symbol "a" is "0.9", the symbol "=" is "0.8", and the symbol "fraction bar" is "0.9". 7" and "0.9" symbols of "8" appear, which roughly match the actual number of symbol appearances. In this example, the probability value is truncated by a finite number of candidates, but in addition to the probability value, based on the ranking, 1 for the first place, 1/2 for the second place, and 1 for the third place. It is also possible to add a weight such as /4. At this time, it is also possible to use only the result of the first place.

2.3.2. Relation set features (features related to the number of occurrences of positional relationships between symbols)
When extracting relational set features, connected components are extracted from the offline pattern, and regions around the extracted connected components (symbols) are divided into 9 regions as shown in FIG. 11 and numbered. For example, if there is another connected element in the region '0' (for example, in the case of the symbol '√'), the positional relationship 'inside' in the online method exists. Also, if there is another connected element in the area "1", the positional relationship "upper" exists, if there is another connected element in the area "5", the positional relationship "lower" exists, and the other connected element exists in the area "2". If there is a connected element of , there is a positional relationship "superscript", if there is another connected element in the region "4", there is a positional relation "subscript", and if there is another connected element in the region "3" or "7" If so, the positional relationship "horizontal" exists. The positional relationship between a connected element and surrounding connected elements is obtained by a method such as projection. As shown in FIG. 12, find the bounding rectangle of each connected element, and if there is no other connected element that obstructs between the two bounding rectangles (for example, a line connecting the midpoints of the bounding rectangles of the two connected elements) , provided that the bounding rectangles of other connected components do not intersect), they give either of the relationships of FIG. 11 as neighboring connected components. For example, in FIG. 12, connected components adjacent to the connected component corresponding to the symbol “x” are the connected component corresponding to the symbol “=”, the connected component corresponding to the symbol “+”, and the symbol “fraction bar”. It is the corresponding connected element. Then, the number of appearances (the number of other connected elements) for each positional relationship (region number) is arranged to form a relationship appearance vector.

As shown in the following equation (3), the relationship set feature R is obtained by summing the relationship appearance vectors R(C) of the respective connected components C. In this example, the number of other connected components having any of the nine relationships is used as the relationship set feature, but the feature value may reflect the distance between connected components or the size of the connected components.

図１３に、オフラインパターンから抽出した関係集合特徴の一例を示す。図１３には、手書き数式パターン「＝ａ／８」の各連結要素の関係出現ベクトルと、これら関係出現ベクトルの和をとって求めた関係集合特徴が示されている。例えば、シンボル「a」に相当する連結要素の関係出現ベクトルは、領域「５」の位置関係にある他の連結要素が１個、領域「６」の位置関係にある他の連結要素が１個存在することを示している。また、図１
３に示す関係集合特徴は、数式中に、領域「１」、「５」の位置関係がそれぞれ２つ、領域「２」、「３」、「４」、「６」、「７」、「８」の位置関係がそれぞれ１つ出現していることを示している。

FIG. 13 shows an example of relation set features extracted from offline patterns. FIG. 13 shows the relationship appearance vectors of each connected element of the handwritten mathematical formula pattern “=a/8” and the relationship set feature obtained by taking the sum of these relationship appearance vectors. For example, the relation appearance vector of the connected component corresponding to the symbol "a" has one other connected component with the positional relationship of region "5" and one other connected component with the positional relationship of region "6". exists. Also, Figure 1
The relationship set feature shown in 3 is such that, in the formula, there are two positional relationships between regions "1" and "5", and regions "2", "3", "4", "6", "7", and "8" positional relationship appears one each.

2.3.3. Position set features (features related to symbol occurrence positions)
When extracting the position set features, the off-line pattern is divided into M×N to find the occurrence positions of the symbols. The number of divisions is determined by the number of vertical relationships (relationships between areas 8, 1, 2, 6, 4, 5, and 6 in FIG. 11) and the number of horizontal relationships (areas 2, 3, 4, and 8, 7, 6 relationships). The number of divisions may be obtained from a mathematical formula with a correct answer. In the example shown in FIG. 14, the left and right relationships are "=" and "fraction bar", "x" and "+", and "+" and "1"("+" and "5" in the numerator). Also, since there are many cases where a space is inserted after the equal sign, it is divided into 5 horizontally, and since there are two vertical relationships, the denominator and the "fraction bar" and the numerator and the "fraction bar", the vertical relationship is 3. It is divided into 5×3 partitions.

Then, connected components are extracted from the offline pattern, and the contribution P(G, C) of each connected component C to the section G is obtained by the following equation (4).

それぞれの区画にガウスフィルタを適用し、次式（５）に示すように、連結要素で和をとって、連結要素の各区画への寄与を並べて特徴ベクトルとする。なお、式（５）において、Ｇ_ｎｍ（１≦ｎ≦Ｎ，１≦ｍ≦Ｍ）は、ｎ行ｍ列の区画を示す。

A Gaussian filter is applied to each section, and as shown in the following equation (5), the sum is taken for the connected components, and the contribution of the connected components to each section is arranged to form a feature vector. In equation (5), G _nm (1≦n≦N, 1≦m≦M) indicates a section of n rows and m columns.

以上述べたように、オフラインパターンから抽出する特徴は、オフラインの手書き数式パターンをシンボル関係木として認識しなくても抽出することができる。つまり、オフラインパターンから抽出する特徴は、オンラインパターンから抽出した特徴とは若干異なっている。オフラインパターン用の上記特徴は、オンラインパターンから抽出することもできる。この場合、オンラインパターンから時系列情報を除いて画像に変換し、その画像から特徴抽出を行う。

As described above, features extracted from offline patterns can be extracted without recognizing offline handwritten mathematical formula patterns as symbol relation trees. That is, features extracted from offline patterns are slightly different from features extracted from online patterns. The above features for offline patterns can also be extracted from online patterns. In this case, the time-series information is removed from the online pattern, converted into an image, and features are extracted from the image.

2.4. Clustering In the clustering of handwritten mathematical formula patterns, first, multiple features extracted from each handwritten mathematical formula pattern (low-level features (orientation features) that are not affected by symbol recognition and extraction, symbol set obtained through symbol recognition and extraction, The distance (similarity) between the handwritten mathematical expression patterns is obtained based on the feature, the relationship set feature, the position set feature (or instead of the position set feature, the position dependent set feature, the position dependent relationship set feature, etc.). A first method (weighted sum method) is a method of obtaining a weighted sum of distances between features. This is to treat them fairly, since each feature has a different meaning and variance. The distance Dis _W (HME ₁ , HME ₂ ) between the two handwritten mathematical formula patterns HME ₁ and HME ₂ by the weighted sum method is represented by the following equation (6).

ここで、ｆ_ｉは、手書き数式パターンからｉ番目の特徴ベクトルを抽出する関数であり、Ｉは、特徴の総数である。例えば、Ｉ＝４とした場合、１番目の特徴を方向特徴、２番目の特徴をシンボル集合特徴、３番目の特徴を関係集合特徴、４番目の特徴を位置集合特徴とすることができる。ｄ（ν_１，ν_２）は、ユークリッド距離やマハラノビス距離などの、距離関数による２つのベクトルν_１とν_２の距離である。α_ｉは、ｉ番目の特徴間の距離に与える重みであり、全ての重みの合計が１になるように、列挙法で決定する。

where f _i is a function that extracts the i-th feature vector from the handwritten math pattern, and I is the total number of features. For example, if I=4, the first feature can be a direction feature, the second feature a symbol set feature, the third feature a relation set feature, and the fourth feature a location set feature. d(ν ₁ , ν ₂ ) is the distance between the two vectors ν ₁ and ν ₂ according to a distance function, such as Euclidean distance or Mahalanobis distance. α _i is a weight given to the distance between i-th features, and is determined by an enumeration method so that the sum of all weights is one.

A second method (normalization method) is a method of normalizing by dividing the distance between individual features by the average of the distances. The distance Dis _N (HME ₁ , HME ₂ ) between the two handwritten mathematical formula patterns HME ₁ and HME ₂ by the normalization method is represented by the following equation (7).

ここで、μ_ｉは、任意の２つの手書き数式パターンについてそれらのｉ番目の特徴ベクトル間の距離を求め、その平均をとったものである。

Here, μ _i is the average of the distances between the i-th feature vectors of any two handwritten mathematical formula patterns.

The third method (normalized weighted sum method) is a method that integrates the weighted sum method and the normalization method. In this method, the weighted sum method is applied after dividing the i-th feature vector by its average, in order to prevent the distances between individual features from becoming large and small in the weighted sum method. The distance Dis _NW (HME ₁ , HME ₂ ) between the two handwritten mathematical formula patterns HME ₁ , HME ₂ by the normalized weighted sum method is represented by the following equation (8).

これらの方法により、２つの手書き数式パターン間の距離を、クラスタリングの対象とする手書き数式パターンの全ての組み合わせについて求め、類似する手書き数式パターン毎にクラスタにクラスタリングする。クラスタリングには、Ｋ－ｍｅａｎｓ法、階層化クラスタリング、ＤＢＳＣＡＮなどの方法が利用できる。以上の手法により、複数の手書き数式パターンを似通ったものごとに精度良くまとめることができ、採点者の採点誤りや採点のばらつきを低減して採点の効率を高めることができる。

By these methods, the distance between two handwritten mathematical formula patterns is obtained for all combinations of handwritten mathematical formula patterns to be clustered, and similar handwritten mathematical formula patterns are clustered into clusters. Methods such as the K-means method, hierarchical clustering, and DBSCAN can be used for clustering. By the above method, a plurality of handwritten mathematical formula patterns can be grouped into similar ones with high accuracy, and scoring errors and variations in scoring by the graders can be reduced, and the efficiency of scoring can be improved.

In addition, all answers (handwritten formula patterns) may be clustered and sent to manual grading (grading by a grader), or all answers may be recognized by a formula recognition engine and whether or not the recognition results are reliable. are automatically scored for the recognition results determined to be reliable (determine whether the recognition results are correct or incorrect), and for the recognition results determined to be unreliable, the corresponding recognition results are processed. Answers may be clustered and sent for manual grading. In addition, all answers are clustered, all answers are recognized by the formula recognition engine for each cluster, and for clusters that include even one correct answer (or more than a certain ratio), the answers included in the cluster are manually graded. For clusters that contain at least one correct answer (or less than a certain percentage) and whose distance from the cluster containing the correct answer is equal to or greater than a certain value, the answers included in the cluster may be determined as incorrect answers. Note that the distance between clusters can be easily calculated by finding the centroids of the clusters and using the weighted Euclidean distance between the centroids.

3. Evaluation Experiment An experiment was conducted to evaluate the clustering method of this embodiment. As an evaluation index for clustering, Purity shown in the following formula (9) (a value obtained by adding the number of elements included in each cluster and originally belonging to the same class over all clusters and dividing the sum by the total number of elements) was used. Considering the scoring for each cluster, it is better if there are few differences in each cluster (that is, the Purity is close to 1).

ここで、Ｗ＝{Ｗ_１，Ｗ_２，…，Ｗ_Ｋ}は、クラスタの集合であり、Ｃ＝{Ｃ_１，Ｃ_２，…，Ｃ_Ｊ}は、クラスの集合であり、Ｋは、クラスタの数であり、Ｊは、クラスの数であり、Ｎは、全要素数である。

where W={W ₁ , W ₂ , . . . , W _K } is a set of _clusters , C={C ₁ , C ₂ , . is the number of clusters, J is the number of classes, and N is the total number of elements.

3.1. Experiments on the online method In this experiment, 21 subjects were given 50 types of formulas, each with three interfaces (no guideline, horizontal guideline in the center of the frame, horizontal guideline at the top, center, and bottom of the frame). We collected online handwritten mathematics patterns (a total of 3,150 patterns belonging to 50 classes) that were transcribed by the participants (with guidelines). There are 101 types of symbols appearing in the pattern. The collected data set is called Dset_50.

As a clustering method, the k-means method based on the Euclidean distance is basically used. ) and k-means++ method (distribute the initial centroid as much as possible. The method is according to D. Arthur and S. Vassilvitskii, “k-means++: The Advantages of Careful Seeding,” in Proceedings of The eighteenth annual ACM-SIAM symposium on Discrete algorithms, New Orleans, 2007.) and the ideal method (selecting the initial centroid from the correct class) were tried. Although the ideal method cannot be used in practice, it is possible to obtain an upper bound for clustering.

In this experiment, the number of clusters was the number of classes, and Purity shown in Equation (9) was used as an evaluation index. As features to be used for clustering, the above individual features and their integration were evaluated. Since feature integration requires learning of weights (α _i ), the database is divided into five, learning with four (learning set), and evaluating with the remaining one (test pattern). Experiments were performed 5 times, and the average value of the results was obtained (5-fold cross-validation). Table 1 shows the evaluation results of clustering performance for all individual features and their integration.

個々の特徴では、方向特徴（実験番号１）は安定している。特に、ランダムに初期化するｋ－ｍｅａｎｓ法で性能が一番高い。シンボル集合特徴（実験番号２）は、僅差ではあるが、ｋ－ｍｅａｎｓ＋＋法と理想法で最良の性能を示した。関係集合特徴（実験番号３２）、位置集合特徴（実験番号４）、位置依存シンボル集合特徴（実験番号５）、位置依存関係集合特徴（実験番号６）、数式候補特徴（実験番号７）は、単独では高い性能を示さなかった。なお、数式認識特徴では、認識候補のうち第１位の候補から抽出した特徴を用いた。

For individual features, the directional feature (experiment number 1) is stable. In particular, the k-means method, which initializes randomly, has the highest performance. The symbol set feature (experiment number 2) showed the best performance with the k-means++ method and the ideal method, albeit by a small margin. The relationship set feature (experiment number 32), position set feature (experiment number 4), position dependent symbol set feature (experiment number 5), position dependent relation set feature (experiment number 6), and formula candidate feature (experiment number 7) are: It did not show high performance alone. Note that the feature extracted from the first-ranked candidate among the recognition candidates was used as the mathematical formula recognition feature.

Experiments with feature integration (weighted sum method, normalized method, normalized weighted sum method) showed better performance than individual features, with the weighted sum method being particularly superior. First, it was found that the combination of low-level directional features and symbol set features, relation set features, and position set features, which are features obtained by applying formula recognition, is effective (experiment numbers 8 and 9). against experiment number 10 and later). Next, in an experiment in which position-dependent symbol set features and position-dependent relation set features were integrated instead of position set features, results differed depending on the clustering method, but it was found to be equally effective. (Experiment No. 11 versus Experiment No. 10). Furthermore, adding the formula candidate feature slightly degraded performance (experiment number 12 vs. experiment number 11), and combining the direction feature and the formula candidate feature resulted in a further decrease in performance (experiment number 13). . From this, there is a high possibility that the combination of the symbol set feature, the relation set feature, and the position set feature is superior to the formula candidate feature. Further, in an experiment (experiment number 14) in which a direction feature, a symbol set feature, a relation set feature, and a position set feature, a position-dependent symbol set feature, and a position-dependent relationship set feature were integrated, the position set feature or the position-dependent Integrating either the symbol set feature or the location dependency set feature shows an effect, but both integration (integration of the location set feature and the location dependent symbol set feature, integration of the location set feature and the location dependency set feature) shows an effect. found to be ineffective.

It was confirmed that feature integration is effective, but on the other hand, the larger the number of features to be extracted, the greater the computational complexity. Therefore, when the amount of calculation is limited, a method of integrating the direction feature, which is a low-level feature, with any one of the symbol set feature, relation set feature, and position set feature obtained through symbol recognition and extraction is also considered. be done. Table 2 shows the evaluation results in this case.

シンボル集合特徴、関係集合特徴、位置集合特徴のいずれか１つよりも、これらに方向特徴を統合した方が効果的であること、及び、方向特徴単独よりも、方向特徴とシンボル集合特徴とを統合した場合が優位であることが分かった。

It is more effective to integrate the direction feature into any one of the symbol set feature, the relationship set feature, and the position set feature. It was found that the integrated case was superior.

3.2. Experiment on offline method In this experiment, 23 subjects were asked to write down the correct answers and two patterns of incorrect answers to 23 questions, respectively. patterns) were collected. The collected data set is called Dset_22Qs. In this experiment, feature extraction was performed for each problem, and a clustering experiment was performed. Therefore, there will be one class of correct answers and two classes of incorrect answers for each question. The evaluation results are shown as the average clustering performance for answers to 22 questions. In this case, the number of classes is three. In addition to Dset_22Qs, an off-line version of Dset_50 (connecting writing points in time series with straight lines and fleshing out with a constant thickness) was also used. The number of classes in this case is fifty.

First, we evaluated the clustering performance for each feature. As a clustering method, k-means++ was used. The value of parameter k in equation (9) was set to the number of classes. That is, k=3 for Dset_22Qs and k=50 for Dset_50. Dset — 22Qs is tested for each problem, and the Purity is obtained for each, so the average and standard deviation thereof were obtained. On the other hand, since Dset_50 performs experiments collectively, the Purity for the entire data can be obtained. Table 3 shows the evaluation results of clustering performance for each feature.

個々の特徴では、シンボル集合特徴（実験番号２）が最も高い性能を示した。一方、方向特徴（実験番号１）、関係集合特徴（実験番号３）、位置集合特徴（実験番号４、５）はシンボル集合特徴に及ばなかった。

Among the individual features, the symbol set feature (experiment number 2) showed the highest performance. On the other hand, the orientation feature (experiment number 1), the relationship set feature (experiment number 3), and the position set feature (experiment numbers 4 and 5) did not reach the symbol set feature.

Next, we evaluated the clustering performance when the features were integrated. For the weighted sum method and the normalized weighted sum method, it is necessary to set optimum weights, so five-fold cross-validation was performed. However, since Dset — 22Qs performs clustering for each problem, the number of data for each problem is small, and the optimum weight values are different, so it is not suitable for cross-validation. Therefore, the weighted sum method and the normalized weighted sum method were not tested for Dset — 22Qs. Table 4 shows evaluation results of clustering performance for feature integration.

加重和法で統合した場合（実験番号６～９）は、シンボル集合特徴と統合することで、関係集合特徴や位置集合特徴の単独の場合よりも性能は向上したが、シンボル集合特徴単独からの性能向上は見られなかった。正規化法の場合は、統合の効果は見られなかった。
正規化加重和では、若干ではあるが、シンボル集合特徴単独より高い性能が見られた。

When integrated by the weighted sum method (Experiment Nos. 6 to 9), the performance was improved by integrating with the symbol set feature than when the relationship set feature and the position set feature were used alone. No performance improvement was observed. No consolidation effect was observed for the normalization method.
Normalized weighted sums performed slightly better than symbol set features alone.

As in the case of the online method, when the amount of calculation is limited, one of the orientation feature, which is a low-level feature, and the symbol set feature, relation set feature, or position set feature obtained through symbol recognition and extraction. A method of integrating two is also conceivable. Table 5 shows the evaluation results in this case.

オンライン方式の場合と同様に、シンボル集合特徴、関係集合特徴、位置集合特徴のいずれか１つよりも、これらに方向特徴を統合した方が効果的であること、及び、方向特徴単独よりも、方向特徴とシンボル集合特徴とを統合した場合が優位であることが分かった。また、方向特徴とシンボル集合特徴とを統合した場合、表４の実験番号１１に迫る性能となることが分かった。

As in the case of the online method, it is more effective to integrate the directional feature into any one of the symbol set feature, the relationship set feature, and the position set feature, and than the directional feature alone, It is found that the case of integrating the directional feature and the symbol set feature is superior. Also, it was found that the performance approached Experiment No. 11 in Table 4 when the direction feature and the symbol set feature were integrated.

Since the off-line method employs extraction based on connected components for symbol extraction, it is possible that the off-line method does not perform as well as the on-line method. By extracting symbols, relationships, and positions using a recent deep neural network, it is expected that individual performance will improve and the effect of integrating features will approach that of online methods. In that case, it can be expected that the present invention will be even more effective in the off-line system.

It should be noted that the present invention is not limited to the above-described embodiments, and various modifications are possible. The present invention includes configurations that are substantially the same as the configurations described in the embodiments (for example, configurations that have the same function, method, and result, or configurations that have the same purpose and effect). Moreover, the present invention includes configurations obtained by replacing non-essential portions of the configurations described in the embodiments. In addition, the present invention includes a configuration that achieves the same effects or achieves the same purpose as the configurations described in the embodiments. In addition, the present invention includes configurations obtained by adding known techniques to the configurations described in the embodiments.

DESCRIPTION OF SYMBOLS 100... Processing part 110... Feature extraction part 112... Classification part 120... Display control part 160... Input part 170... Storage part 190... Display part

Claims (7)

A program for classifying a plurality of handwritten mathematical formula patterns input by handwriting,
From the handwritten mathematical expression pattern, among the directional feature , the feature related to the number of occurrences of symbols, the feature related to the number of occurrences of the positional relationship between symbols, and the feature related to the appearance position of symbols, a feature extractor for extracting at least two features , including
The computer is caused to function as a classifying unit that classifies the plurality of handwritten mathematical formula patterns into a plurality of groups according to the degree of similarity based on the at least two features extracted from each of the plurality of handwritten mathematical formula patterns. program. A program for classifying a plurality of handwritten mathematical formula patterns input by handwriting,
At least two features including the directional feature are extracted from the pattern of handwritten mathematical formulas, among a directional feature, a feature relating to the number of occurrences of symbols, a feature relating to the number of occurrences of positional relationships between symbols, and a feature relating to the appearance positions of symbols. a feature extraction unit that
causing a computer to function as a classifying unit that classifies a plurality of handwritten mathematical formula patterns into a plurality of groups according to similarity based on the at least two features extracted from each of the plurality of handwritten mathematical formula patterns;
The feature extraction unit is
A program for extracting features relating to the number of appearances of the positional relationship for each position as features relating to symbol appearance positions. A program for classifying a plurality of handwritten mathematical formula patterns input by handwriting,
At least two features including the directional feature are extracted from the pattern of handwritten mathematical formulas, among a directional feature, a feature relating to the number of occurrences of symbols, a feature relating to the number of occurrences of positional relationships between symbols, and a feature relating to the appearance positions of symbols. a feature extraction unit that
causing a computer to function as a classifying unit that classifies a plurality of handwritten mathematical formula patterns into a plurality of groups according to similarity based on the at least two features extracted from each of the plurality of handwritten mathematical formula patterns;
The feature extraction unit is
A mathematical expression recognition engine recognizes a mathematical expression from the handwritten mathematical expression pattern, and based on the recognition results, extracts features relating to the number of occurrences of symbols, features relating to the number of occurrences of positional relationships between symbols, and features relating to the appearance positions of symbols. A program characterized by In any one of claims 1 to 3 ,
The feature extraction unit is
A program for extracting a feature of the number of symbols appearing at each position as a feature of a symbol appearing position. A clustering device for classifying a plurality of patterns of handwritten mathematical formulas input by handwriting,
From the handwritten mathematical expression pattern, among the directional feature , the feature related to the number of occurrences of symbols, the feature related to the number of occurrences of the positional relationship between symbols, and the feature related to the appearance position of symbols, a feature extractor for extracting at least two features , including
a clustering unit that classifies the plurality of handwritten mathematical formula patterns into a plurality of groups according to the degree of similarity based on the at least two features extracted from each of the plurality of handwritten mathematical formula patterns. . A clustering device for classifying a plurality of patterns of handwritten mathematical formulas input by handwriting,
At least two features including the directional feature are extracted from the pattern of handwritten mathematical formulas, among a directional feature, a feature relating to the number of occurrences of symbols, a feature relating to the number of occurrences of positional relationships between symbols, and a feature relating to the appearance positions of symbols. a feature extraction unit that
a classification unit that classifies the plurality of handwritten mathematical formula patterns into a plurality of groups according to the degree of similarity based on the at least two features extracted from each of the plurality of handwritten mathematical formula patterns;
The feature extraction unit is
A clustering device that extracts, as features relating to symbol appearance positions, features relating to the number of occurrences of the positional relationship for each position. A clustering device for classifying a plurality of patterns of handwritten mathematical formulas input by handwriting,
At least two features including the directional feature are extracted from the pattern of handwritten mathematical formulas, among a directional feature, a feature relating to the number of occurrences of symbols, a feature relating to the number of occurrences of positional relationships between symbols, and a feature relating to the appearance positions of symbols. a feature extraction unit that
a classification unit that classifies the plurality of handwritten mathematical formula patterns into a plurality of groups according to the degree of similarity based on the at least two features extracted from each of the plurality of handwritten mathematical formula patterns;
The feature extraction unit is
A mathematical expression recognition engine recognizes a mathematical expression from the handwritten mathematical expression pattern, and based on the recognition results, extracts features relating to the number of occurrences of symbols, features relating to the number of occurrences of positional relationships between symbols, and features relating to the appearance positions of symbols. A clustering device characterized by:

Images