JP2021114091A - Program and clustering device - Google Patents

Abstract

To provide a program and a clustering device which reduce score errors and variations in scores of a scorer, and enhance score efficiency.SOLUTION: In a clustering device, a processing part includes: a scale image generation part for reducing one handwriting pattern image and generating a plurality of scale images; a feature extraction part for giving each of the images to a learning model, and extracting a symbol position feature concerning appearance positions for each kind of symbols for each of a plurality of scale images; a feature vector generation part for dividing the symbol position feature into the plurality of kinds, and determining appearance probability for each of kinds of the symbols for each of the divisions and generating a feature vector; an integral part for taking a maximum value of appearance probability of the same symbol in the same part of the feature vector generated from each of the plurality of scale images, and generating an integral feature vector; and a classification part for classifying a plurality of handwriting patterns into a plurality of groups based on the integral feature vector generated from each of the plurality of handwriting pattern images.SELECTED DRAWING: Figure 1

Description

The present invention relates to a program and a clustering device.

There is a social recognition that it is necessary to impose not only multiple-choice questions but also descriptive questions in order to develop and measure the thinking ability of examinees (answerers). When imposing descriptive questions, it is required to give a reliable score in a short period of time. By using handwriting recognition technology, it is possible to perform scoring support and automatic scoring that support human scoring. Prototypes of learning systems that employ recognition of handwritten answers have been published in arithmetic and mathematics, where there is a high need for descriptive answers (for example, Non-Patent Documents 1 to 3).

Masato Suzuki et al., "Development of Basic Mathematics Learning Support System Based on Handwritten Mathematical Analysis", Institute of Electronics, Information and Communication Engineers, Academic Techniques, ET2009-129, p.147-152 (2010) Satoshi Chiba et al., "Prototype of Mathematics e-Learning System on Tablet PC Using Handwritten Formula Recognition", IPSJ Research Report, Vol.2014-CE-127, No.10, p.1-5 (2014) Wataru Konishi et al., "Automatic Mathematics and Mathematics Scoring System Using Handwritten Formula Recognition", IPSJ Research Report, Computer and Education Study Group Report, 2016-CE-133 (7), 1-7 (2016)

When scoring by a person, if the answer is randomly presented to the grader, the scoring often fluctuates (varies). In a large-scale examination, multiple people score, and if the score difference is above a certain level, a third party needs to score, which increases the cost and time of using the third party. On the other hand, in automatic scoring, it is indispensable to have an environment in which examinees can check the scoring results and point out scoring errors. It is realistic to reject things and use them for scoring by people. Therefore, it is important to suppress the variation in scoring by graders regardless of whether it is scoring support or automatic scoring.

The present invention has been made in view of the above problems, and an object of the present invention is a program and clustering capable of reducing scoring errors and scoring variations of graders and improving scoring efficiency. To provide the equipment.

(1) The present invention is a program for classifying a plurality of handwritten patterns input by handwriting, and reduces an image of one handwritten pattern to at least one step to reduce the image of one handwritten pattern to at least one original image before reduction and at least one. A scale image generator that generates a plurality of scale images including a reduced image and each of the plurality of scale images are given to a learning model using a convolutional neural network, and each of the plurality of scale images is given for each symbol type. A feature extraction unit that extracts symbol position features related to the appearance position of the image, and a plurality of types of divisions are performed on the symbol position features, and each division obtained by each of the plurality of types of divisions is used for each symbol type. An integrated feature vector is generated by taking the maximum value of the appearance probability of the same symbol at the same location of the feature vector generated from each of the plurality of scale images and the feature vector generation unit that obtains the appearance probability and generates the feature vector. The present invention relates to an integration unit and a program characterized by operating a computer as a classification unit that classifies a plurality of the handwriting patterns into a plurality of groups based on the integrated feature vector generated from each of the images of the plurality of handwriting patterns. .. The present invention also relates to a computer-readable information storage medium, which stores a program for operating a computer as each of the above parts. The present invention also relates to a clustering apparatus including each of the above parts.

According to the present invention, by giving each of a plurality of scale images to a learning model and extracting symbol position features for each of the plurality of scale images, the classes (types) and positions of symbols having different sizes can be extracted. Can be done. In addition, multiple types of divisions are performed on the symbol position features, and the appearance probability of each symbol type is obtained for each division obtained by each of the multiple types of divisions to generate a feature vector of the symbol. The symbol can be detected regardless of the magnitude of the misalignment. Further, by generating a feature vector by performing a plurality of types of partitioning for each of the plurality of symbol position features extracted from each of the plurality of scale images, the sizes of the position features can be made uniform and integrated. By classifying multiple handwriting patterns into multiple groups based on the integrated feature vector obtained by integrating the feature vectors generated from each of the multiple scale images, the handwriting patterns are accurately grouped (clustered) for each similar one. ), And it is possible to improve the efficiency of scoring by reducing scoring errors and scoring variations of graders.

(2) Further, in the program, the information storage medium, and the clustering apparatus according to the present invention, each of the plurality of scale images is given to the learning model, a feature map is acquired for each of the plurality of scale images, and the feature map is obtained. A vector obtained by multiplying the symbol position feature obtained by inputting to the classifier and the feature map is input to the classifier to generate a first appearance probability vector indicating the appearance probability for each symbol type. , The first integrated appearance probability vector is generated by taking the maximum value of the appearance probability of the same symbol at the same position of the first appearance probability vector generated from each of the plurality of scale images, and the first integrated appearance probability vector is combined with the first integrated appearance probability vector. The computer may further function as a learning unit for learning the learning model based on the teacher data indicating the presence or absence of the appearance of each symbol type (the learning unit may be further included).

According to the present invention, attention is paid to learning a learning model using a first appearance probability vector generated from a vector obtained by multiplying a symbol position feature obtained by inputting a feature map into a classifier and a feature map. It is possible to weight the appearance of symbols to be learned and learn the learning model appropriately.

(3) Further, in the program, the information storage medium, and the clustering apparatus according to the present invention, the learning unit gives each of the plurality of scale images to the learning model, and acquires a feature map for each of the plurality of scale images. , The vector taking the maximum value of the feature map is input to the classifier to generate a second appearance probability vector indicating the appearance probability for each symbol type, and the second appearance probability vector generated from each of the plurality of scale images is generated. The second integrated appearance probability vector is generated by taking the maximum value of the appearance probability of the same symbol at the same place of the appearance probability vector, and the first integrated appearance probability vector and the second integrated appearance probability vector are averaged to form the third integrated. An appearance probability vector may be generated, and the learning model may be trained based on the third integrated appearance probability vector and the teacher data.

According to the present invention, the learning model is learned more appropriately by training the learning model using the second appearance probability vector and the first appearance probability vector generated from the vector that takes the maximum value of the feature map. Can be done.

The figure which shows an example of the functional block diagram of the clustering apparatus of this embodiment. The figure which shows the flow of the process of generating an integrated feature vector from a handwritten mathematical expression pattern image. The figure which shows an example of the structure of a deep convolutional neural network and a classifier. The figure which shows the symbol position feature schematically. The figure which shows an example of the hierarchical space pooling schematically. The figure which shows the flow of a learning process. The figure which shows the flow of the process of the global attention pooling. The figure which superposed the symbol position feature extracted from the handwritten mathematical expression pattern image on the handwritten mathematical expression pattern image.

Hereinafter, this embodiment will be described. The present embodiment described below does not unreasonably limit the content of the present invention described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present invention.

1. 1. Configuration Figure 1 shows an example of a functional block diagram of the clustering device of this embodiment. The clustering apparatus of the present embodiment may have a configuration in which some of the constituent elements (each part) of FIG. 1 are omitted.

The input unit 160 is for inputting a handwritten mathematical expression pattern (an example of a handwritten pattern) written on paper or the like, and its function is an optical reading device that reads the handwritten mathematical expression pattern as an image (black-and-white image or grayscale image). It can be realized by (scanner, camera, etc.).

The storage unit 170 stores programs and various data for operating the computer as each unit of the processing unit 100, and also functions as a work area of the processing unit 100, and the function can be realized by a hard disk, RAM, or the like.

The display unit 190 outputs an image generated by the processing unit 100, and its function can be realized by a display such as a touch panel, LCD, or CRT that also functions as an input unit 160.

The processing unit 100 (processor) performs processing such as classification (clustering) processing, learning processing, and display control based on data (image data) from the input unit 160, a program, and the like. The processing unit 100 performs various processes using the main storage unit in the storage unit 170 as a work area. The function of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), or a program. The processing unit 100 includes a scale image generation unit 110, a feature extraction unit 111, a feature vector generation unit 112, an integration unit 113, a classification unit 114, a display control unit 115, and a learning unit 116.

The scale image generation unit 110 reduces one handwritten mathematical expression pattern image to at least one step, and generates a plurality of scale images including the original image before reduction and at least one reduced image.

The feature extraction unit 111 gives each of the plurality of scale images to the learning model using the convolutional neural network, and extracts the symbol position feature relating to the appearance position of each symbol type for each of the plurality of scale images. Symbols are elements that make up mathematical formulas. Specifically, numbers, characters (alphabetic characters, Greek characters) and operators (arithmetic operators, logical operators, set operators, relational operations) that appear in mathematical formulas. Child), symbol (addition symbol, operation symbol, integration symbol,
Integral symbols, limit symbols (lim), special symbols (∞, etc.)), functions (log, sin, cos, tan, etc.), parentheses (large, medium, small).

The feature vector generation unit 112 performs a plurality of types of divisions for the symbol position feature, obtains the appearance probability of each symbol type for each division obtained by each of the plurality of types of divisions, and generates a feature vector. do. The plurality of types of divisions are divisions in which the values of n and / or m of the n × m divisions are different.

The integration unit 113 generates an integrated feature vector by taking the maximum value of the appearance probability of the same symbol at the same position of the feature vector generated from each of the plurality of scale images.

The classification unit 114 classifies the plurality of handwritten mathematical expression patterns into a plurality of groups based on the integrated feature vector generated from each of the plurality of handwritten mathematical expression pattern images.

The display control unit 115 controls the display unit 190 to display a plurality of handwritten mathematical expression patterns classified into a plurality of groups by the classification unit 114 for each group.

The learning unit 116 is obtained by giving each of a plurality of scale images (scale images for learning) to the learning model, acquiring a feature map for each of the plurality of scale images, and inputting the feature map into the classifier. A vector obtained by multiplying the symbol position feature and the feature map and averaging them is input to the classifier to generate a first appearance probability vector showing the appearance probability for each symbol type, and is generated from each of a plurality of scale images. Teacher data that generates the first integrated appearance probability vector by taking the maximum value of the appearance probability of the same symbol at the same location of the first appearance probability vector, and indicates the presence or absence of appearance for each type of the first integrated appearance probability vector and the symbol. The learning model is trained based on the above. Further, the learning unit 116 gives each of the plurality of scale images to the learning model, acquires a feature map for each of the plurality of scale images, and inputs a vector having the maximum value of the feature map to the classifier to symbolize. A second appearance probability vector indicating the appearance probability for each type of is generated, and the maximum value of the appearance probability of the same symbol is taken at the same place of the second appearance probability vector generated from each of the plurality of scale images and the second integration is performed. An appearance probability vector is generated, a third integrated appearance probability vector is generated by averaging (integrating) the first integrated appearance probability vector and the second integrated appearance probability vector, and based on the third integrated appearance probability vector and teacher data. The learning model may be trained.

2. FIG. 2 is a diagram showing a flow of processing for generating an integrated feature vector from an input image of a handwritten mathematical expression pattern (handwritten mathematical expression pattern image).

First, one handwritten mathematical expression pattern image (handwritten mathematical expression pattern image to be clustered) is reduced to at least one step to generate a plurality of scale images including the original image before reduction and at least one reduced image. In the example shown in FIG. 1, from one handwritten mathematical formula pattern image, a reduced image (1/4 image) in which the side length is reduced to 1/2 and a reduced image (1/16 image) in which the side length is reduced to 1/4. ) And three scale images (original image, 1/4 image, 1/16 image) are generated. A reduced image reduced to a smaller scale may be used, or a reduced image reduced to another scale may be additionally used.

Next, each of the plurality of scale images is given to the learning model, the deep convolutional neural network (Deep CNN) and the classifier (Classifier), and the appearance position of each symbol type (class) is given to each of the plurality of scale images. Extract the symbol position features (LC features) that are the features related to. FIG. 3 shows an example of the configuration of the deep convolutional neural network and the classifier. The deep convolutional neural network (Deep CNN) is composed of a convolutional layer CL and a pooling layer PL (maximum pooling layer), and outputs a feature map (CNN feature map). The feature map (feature map group) includes a plurality of feature maps which are two-dimensional arrays of features obtained by applying convolution or pooling to the input (scale image) of the two-dimensional array. The feature map is input to the classifier. Here, the feature maps output from the last two convolution layers CL are combined and input to the classifier. The classifier consists of a convolution layer CL and a dropout DO and outputs symbol position features.

The box shown as the symbol position feature (LC features) in FIG. 2 is a collective representation of a plurality of planes as shown in FIG. The symbol position feature is represented by a plurality of planes corresponding to a plurality of types (classes) of symbols, and each plane indicates the probability of occurrence of that class at each position on the input plane (scale image). Depth D (the number of planes lined up in the depth direction) indicates the number of classes (#classes), height H indicates the height of the input surface (number of pixels in the vertical direction), and width W indicates the width of the input surface (horizontal). The number of pixels in the direction) is shown.

Next, hierarchical spatial pooling (HSP: Hierarchical Spatial Pooling) is performed on a plurality of symbol position features extracted from each of the plurality of scale images. In hierarchical space pooling, multiple types of divisions are performed for each plane of the symbol position feature, and the appearance probability for each symbol type (class) is obtained for each division obtained by each of the multiple types of divisions. Generate a hierarchical spatial feature vector (HS features, corresponding to the feature vector of the present invention). FIG. 5 is a diagram schematically showing an example of hierarchical space pooling. In this example, three types of partitioning (1x1 partitioning, 3x5 partitioning and 5x7 partitioning) are applied to each plane of the symbol position feature, and three types of partitioning are applied. From each of the divisions, the maximum value of the appearance probability of each class is obtained for each division (maximum pooling is performed for each division), and a k-dimensional feature vector is generated for each division. k is the number of classes (#classes), and the k-dimensional feature vector is a vector whose elements are the appearance probabilities of each of the k types of symbols. Then, the k-dimensional feature vectors for the number of divided sections are connected, and the feature vectors for the number of types of the divided sections are further connected to generate the hierarchical space feature vector. A 1 × k = k-dimensional feature vector (probability of appearance for each class on the input surface) is obtained from the 1 × 1 partition, and a 3 × 5 × k = 15 k-dimensional feature vector is obtained from the 3 × 5 partition. (Probability of appearance for each class in each section when the input surface is divided into 3 × 5 sections) is obtained, and from the 5 × 7 section division, a feature vector of 5 × 7 × k = 35k dimensions (input surface is divided into 3 × 5 sections). The probability of appearance for each class in each section when divided into 5 × 7 sections) is obtained, and these are connected to obtain a hierarchical space feature vector of k + 15k + 35k = 51k dimensions. The feature vector obtained from the 1 × 1 partition does not represent the position information because each plane of the symbol position feature is not divided, but the feature vector obtained from the other partition expresses the position information. It should be noted that any of the above three types of division may be removed, or another type of division may be added.

Next, the maximum value of the appearance probability of the same symbol (same class) is taken at the same place of the hierarchical spatial feature vector generated from each of the plurality of scale images (maximum aggregation (MA) is performed), and the integrated feature is integrated. Generate a vector (multi-scale hierarchical spatial feature vector, MHS features). In the example shown in FIG. 2, the integrated feature vector is generated by integrating the three hierarchical spatial feature vectors generated from each of the original image, the 1/4 image, and the 1/16 image.

In the clustering of handwritten mathematical expression patterns, first, the similarity between the handwritten mathematical expression patterns (distance between the integrated feature vectors) is obtained based on the integrated feature vectors generated from each of the plurality of handwritten mathematical expression pattern images. As a function for obtaining the distance between integrated feature vectors, for example, a cosine distance function can be used. Then, the distance between the integrated feature vectors of the two handwritten mathematical expression pattern images is obtained for all combinations of the handwritten mathematical expression patterns to be clustered, and clustered into clusters for each similar handwritten mathematical expression pattern. A method such as the K-means ++ method can be used for clustering.

In mathematical formulas, symbols vary in size depending on their role. For example, subscripts (subscripts) and superscripts (superscripts) are smaller than baseline symbols. In the example shown in FIG. 3, “x” of different sizes appear in the handwritten mathematical expression pattern image. This makes it difficult to detect symbols in mathematical formulas. In the method of the present embodiment, each of the plurality of scale images is given to one deep convolutional neural network (Deep CNN), and the symbol position features (LC features) are extracted for each of the plurality of scale images to obtain the size. You can extract different symbol classes and positions. This is because a scale image of either scale is suitable for extracting the class and position of a symbol. That is, a small scale image is suitable for extracting the class and position of a symbol that is written large, and a large scale image (such as the original image) is suitable for extracting the class and position of a symbol that is written small. Suitable for.

In addition, the positional relationship of symbols is important in mathematical expression recognition. For example, recognition of parentheses and fractional rules requires not only the shape of the symbol, but also the context of the symbols around it. Therefore, in the method of the present embodiment, in order to obtain not only the symbol but also the information in the vicinity thereof, in the deep convolutional neural network, the feature maps output from the last two convolutional layers CL are combined and input to the classifier. ..

Further, in the method of the present embodiment, a plurality of types of divisions are performed on the symbol position feature, and the appearance probability of each symbol type is obtained for each division obtained by each of the plurality of types of divisions, and the hierarchical space feature is obtained. By adopting the process of generating vectors (HS features) (hierarchical spatial pooling), symbols can be detected regardless of the magnitude of the symbol misalignment. That is, even if the misalignment of the symbol is large, the symbol can be detected at a rough position by the feature vector obtained from the division with a small number of compartments, and if the misalignment of the symbol is small, the number of compartments The feature vector obtained from many compartments allows the symbol to be detected at a more precise position. In the hierarchical space pooling, in order to deal with the misalignment of symbols, a Gaussian function centered on the section may be superimposed on the appearance probability of each symbol in each section.

Further, if the size of the scale image is different, the size of the symbol position feature is also different, but in the method of the present embodiment, hierarchical spatial pooling is applied to a plurality of symbol position features extracted from each of the plurality of scale images. Therefore, it is possible to align the size of the features to the same size and integrate them. Then, by classifying a plurality of handwritten mathematical expression patterns into a plurality of groups based on the integrated feature vector (MHS features) obtained by integrating the hierarchical spatial feature vectors generated from each of the plurality of scale images, the handwritten mathematical expression patterns are similar. It is possible to cluster each item with high accuracy, reduce scoring errors and scoring variations of graders, and improve scoring efficiency.

Next, learning of a deep convolutional neural network (learning model) will be described. FIG. 6 is a diagram showing a flow of learning processing. At the time of learning, an appearance probability vector indicating the appearance probability for each type of symbol is extracted by global attention pooling (GAtP: Global Attentive Pooling) without using the position information of the symbol.

First, as in the case of clustering, one handwritten mathematical expression pattern image (handwritten mathematical expression pattern image for learning) is reduced to at least one step, and a plurality of scale images including the original image before reduction and at least one reduced image are included. To generate. Next, each of the plurality of scale images is given to the deep convolutional neural network (Deep CNN) to obtain a feature map (CNN feature map) output from the last two convolutional layers CL. The depth (D) of the box shown as the feature map indicates the number (#features) of the features (feature maps included in the feature map group) included in the feature map.

Next, global attention pooling (GAtP) is performed on the feature map. In this process, features are extracted from all occurrence positions of a symbol, but the appearance of notable classes is weighted. FIG. 7 is a diagram showing a flow of processing for global attention pooling. First, by entering a feature map (F) to the discriminator (Classifier), determine the symbol position feature _{(M i, i = 1..k)} . k is the number of classes (#classes). Next, for each symbol, the feature map (F) is multiplied (multiplied) by the symbol position feature to obtain k class attention feature maps (F × M _{1 to} F × M _k ). When the symbol position feature class c (c-th class) a (probability of class c at each position on the input surface) and M _c, by multiplying the symbol position feature (M _c) to the feature map (F) , Obtain a class attention feature map (F × _Mc ). The class attention feature map (F × _Mc ) is a feature map that weights the appearance of class c. Next, for each of the class attention feature maps (F × M _{1 to} F × M _k ), the values of the height H and the width W are averaged (averaged for each feature), and the class attention vector (Att _{F1) is taken.} ~ Att _Fk ) is obtained. Averaged class interest vector by multiplying the symbol position feature _{(M c)} to the feature map (F) _{(Att Fc)} can be obtained by the following equation (1).

ここで、ｘ，ｙは、特徴マップ（Ｆ）やシンボル位置特徴（Ｍ_ｃ）の高さＨと幅Ｗの次元を示す。

Here, x and y indicate the dimensions of the height H and the width W of the feature map (F) and the symbol position feature ( _Mc).

Next, the vectors (f (Att _F1 ) to f (Att _Fk )) obtained by inputting each of the class attention vectors (Att _{F1 to Att Fk) into the classifier are diagonalized.} , The first appearance probability vector (vector y having _{y 1 to y k} as each element) indicating the appearance probability for each class is obtained. The c-th element of the vector (f (Att _{Fc )), y c} (probability of appearance of class c), can be obtained by the following equation (2).

ここで、ｕ_ｃは、クラスｃのOne-hotベクトル（ｃ番目の要素だけ１で他の要素が０で
あるベクトル）であり、＜ａ,ｂ＞は、ベクトルａとベクトルｂの内積演算を示す。

Here, u _c is the One-hot vector class c (vector other elements only 1 c th element is 0), <a, b> is the inner product operation of the vector a and vector b show.

Next, as shown in FIG. 6, the maximum value of the appearance probability of the same symbol is taken at the same position of the first appearance probability vector (y) generated from each of the plurality of scale images (maximum aggregation (MA) is performed). The first integrated appearance probability vector (My) is generated.

Next, based on the first integrated appearance probability vector (My) and the teacher data (vector showing the appearance probability (presence / absence) of each class), a learning model (deep layer) is used using binary entropy loss. Train a convolutional neural network). Figure 6
In the example shown in, symbols "a", "b", "2", "+", and "fractional rule (frac)" appear in the handwritten mathematical expression pattern for learning, so these classes appear as teacher data. A vector is prepared in which the probability is "1" and the appearance probability of other classes is "0".

It should be noted that the global attention pooling (GAtP) may take a large amount of time to converge without prior learning, or may not converge in the worst case. In addition, there is a possibility that a certain probability will be given to a class that does not exist in the handwritten mathematical expression pattern image. Therefore, in order to solve this problem, the appearance probability (second appearance probability vector) for each class is obtained by Global Max Pooling (GMP), and the second appearance probability vector generated from each of a plurality of scale images is obtained. The second integrated appearance probability vector is generated by taking the maximum value of the appearance probability of the same symbol at the same place, and the second integrated appearance probability vector and the first integrated appearance probability vector (My) are integrated (averaged). A third integrated appearance probability vector may be generated, and the learning model may be trained based on the third integrated appearance probability vector and the teacher data. In the global maximum pooling, the d (depth, #features) -dimensional vector of the equation (3) in which the maximum values are taken for x and y for each feature of the feature map (F) is obtained.

このベクトルを識別器（Classifier）に入力して第２出現確率ベクトルを求める。この処理では、それぞれのクラスに対して、最もそれらしいものの位置からその特徴を抽出する。第１統合出現確率ベクトルと第２統合出現確率ベクトルの統合により、深層畳み込みニューラルネットワークは、大域注目プーリングと大域最大プーリングのそれぞれの損失関数の両方を低減する方向で学習を進める。学習の初期段階では、大域最大プーリングの利点を活用して学習が進み、それが進行した段階では、大域注目プーリングの機能が発揮される。

This vector is input to the classifier to obtain the second appearance probability vector. In this process, for each class, its features are extracted from the position of the most likely one. By integrating the first integrated appearance probability vector and the second integrated appearance probability vector, the deep convolutional neural network proceeds with learning in the direction of reducing both the loss functions of the global attention pooling and the global maximum pooling. In the initial stage of learning, learning progresses by taking advantage of the global maximum pooling, and in the advanced stage, the function of global attention pooling is demonstrated.

3. 3. Evaluation experiment An experiment was conducted to evaluate the clustering method (and learning method) of this embodiment. Purity shown in the following formula (4) was used as an evaluation index for clustering. Considering scoring for each cluster, the purity should be close to 1.

ここで、Ｋは、クラスタの数であり、Ｊは、実際のクラスの数であり、Ｈは、サンプルの数であり、Ｇ＝｛ｇ_１，ｇ_２，…，ｇ_Ｋ｝は、結果のクラスタであり、Ｃ＝｛ｃ_１，ｃ_２，…，ｃ_Ｊ｝は、実際のクラスである。

Where K is the number of clusters, J is the number of actual classes, H is the number of samples, and G = {g ₁ , g ₂ , ..., G _K } is the result. It is a cluster, and C = {c ₁ , c ₂ , ..., c _J } is the actual class.

In this experiment, CROHME 2016 dataset (H. Mouchere, C. Viard-Gaudin, R. Zanibbi, U. Garain, ICFHR2016 CROHME: Competition on Recognition of Online Handwritten Mathematical Expressions, in: 2016 15th Int . Conf. Front. Handwrit. Recognit., IEEE, 2016: pp. 607-612. Doi: 10.1109 / ICFHR.2016.0116.) Was used. 101 kinds of symbols appear in this. Also, this includes 8,
There are 834 training samples, 986 verification samples, and 1,147 test samples.
In this experiment, 620 samples of 36 kinds of mathematical formulas, which are a part of the learning sample, are used as the test sample without using the test sample. This is because the original test sample has only one sample per formula. On the other hand, the 620 samples selected from the training samples include 15 to 22 samples per formula. 8,214 samples were used for training, excluding these 620 samples from the original 8,834 training samples. The verification sample was used as it was. Hereinafter, this sample set will be referred to as S-CROHME.

In this experiment, the aspect ratio of all samples (mathematical images) is fixed and normalized so that the vertical direction is 128 pixels (usually, since the number of pixels is larger than this, it is reduced to 128 pixels, rarely. Smaller ones were enlarged to 128 pixels), and as a result, those larger than 800 pixels in the horizontal direction were linearly compressed so as to have 800 pixels in the horizontal direction. Those having less than 800 pixels in the horizontal direction were left as they were. Average pooling with a 2x2 mask was performed twice to generate a multi-resolution image (multi-step reduced image). As a result, mathematical images (three scale images) having 128 pixels, 64 pixels, and 32 pixels in the vertical direction were generated. note that. In mathematical formulas, multiple symbols appear, and in order to be able to read them, it is common for the vertical direction to be 128 pixels and the horizontal direction to be more than that. When the number of pixels is small (for example, 64 pixels), it can be dealt with by preparing a mathematical image having 64 pixels, 32 pixels, or 16 pixels in the vertical direction, or reducing the number of divisions.

Table 1 shows the specific configuration of the deep convolutional neural network used in this experiment. In Table 1, #maps is the number of feature maps, k is the number of kernels (filters), s is the stride magnitude, and p is the padding magnitude. ReLU is also an activation function.

階層的空間プーリング（ＨＳＰ）では、１×１の区画分割、３×５の区画分割、３×７の区画分割及び５×７の区画分割を行った。また、２つの数式画像の統合特徴ベクトル間
の距離を算出するために、コサイン距離関数を用いた。クラスタリング手法としては、K-means++法を採用した。

In Hierarchical Space Pooling (HSP), 1x1 partitioning, 3x5 partitioning, 3x7 partitioning and 5x7 partitioning were performed. In addition, a cosine distance function was used to calculate the distance between the integrated feature vectors of the two mathematical images. The K-means ++ method was adopted as the clustering method.

High purity can be easily obtained by increasing the number K of clusters. For example, if the number K of clusters is equal to the number H of samples, the purity is 1. However, this is the same as not clustering, and the scoring efficiency does not increase. Therefore, the number K of clusters must be about the number of types of mathematical expressions. However, in the actual scoring scene, the number of types cannot be known without accurate mathematical recognition. Here, for evaluation, the number of types of mathematical formulas known in advance was set to the number K of clusters. That is, S-CROHME was clustered into 36 clusters.

First, hierarchical spatial pooling when generating hierarchical spatial feature vectors (HS features) (
The effect of HSP) was evaluated. When no partitioning is performed (only 1x1 partitioning is performed), when a single partitioning (only one of 3x5, 3x7, 5x7) is performed, there are multiple types. Purity was determined by clustering S-CROHME test samples in each case of compartmentalization (hierarchical spatial pooling). The evaluation results are shown in Table 2.

区画分割を行わない場合の純度０．９２に対して、単一の区画分割を行う場合はいずれの区画分割でも０．９６以上の純度を達成した。また、複数種類の区画分割を行う場合、いずれの区画分割の組み合わせでも、単一の区画分割を行う場合より高い純度を達成した。これは、単一の区画分割を行うより、複数種類の区画分割を行う方が効果的であることを示している。特に、１×１の区画分割と３×５の区画分割の組み合わせで純度０．９９を達成した。

In contrast to the purity of 0.92 when no division was performed, a purity of 0.96 or higher was achieved in any of the divisions when a single division was performed. In addition, when a plurality of types of divisions were performed, higher purity was achieved in any combination of divisions than in the case of performing a single division. This indicates that it is more effective to perform multiple types of partitioning than to perform a single partitioning. In particular, a purity of 0.99 was achieved with a combination of 1x1 compartmentalization and 3x5 compartmentalization.

Next, the effect of the difference in global pooling was evaluated. At the time of learning, the purity was determined in each of the cases of performing global average pooling (GAP: Global Average Pooling), performing global maximum pooling (GMP), and integrating global attention pooling and global maximum pooling (GAtP + GMP). The evaluation results are shown in Table 3. The global average pooling is a method of obtaining the appearance probability vector by inputting the vector obtained by taking the average value for each feature of the feature map (F) into the classifier, and for each class, all the vectors are obtained. The feature is extracted from the appearance position.

区画分割を行う場合、ＧＡｔＰ＋ＧＭＰは、若干ではあるがＧＭＰ単独より良く、ＧＡＰよりは明らかに良い性能を示している。一方、区間分割を行わない（１×１の区間分割を行う）場合、ＧＡＰでは０．２８の純度しか得られない。

When partitioning, GAtP + GMP is slightly better than GMP alone and clearly better than GAP. On the other hand, when the section division is not performed (1 × 1 section division is performed), only 0.28 purity can be obtained by GAP.

The difference in global pooling is reflected not only in clustering but also in the average recognition rate of symbols. The average recognition rates of GAtP + GMP, GMP, and GAP were 0.94, 0.93, and 0.17, respectively. We were able to confirm the goodness of GAtP + GMP in terms of recognition performance.

FIG. 8 is a diagram in which a symbol position feature (appearance probability at each position on the image for each symbol type) extracted from the handwritten mathematical expression pattern image (scale image) is superimposed on a certain handwritten mathematical expression pattern image (scale image). .. The symbol position feature is shown with higher brightness as the probability of appearance of the symbol increases. The leftmost image in the first column of FIG. 8 is an image in which the symbol position features extracted from the original image are superimposed on the original image, and the image in the second column is a 1/4 image and a 1/4 image. It is an image in which the extracted symbol position features are superimposed, and the image in the third column is an image in which the symbol position features extracted from the 1/16 image are superimposed on the 1/16 image, and the image in the fourth column on the far right. The image of is a composite image of these three images (the maximum value at the same position is taken). In addition, from the first line to the eleventh line at the top, each symbol ("x", "Σ", "P", "n", "j", "i", "a", The appearance probabilities at each position of "0", "=", ")", and "(") are shown, and the image on the 12th line at the bottom is an image obtained by synthesizing these (each of all symbols). Appearance probability at the position). In FIG. 8, the reduced image (1/4 image, 1/16 image) and its symbol position feature are enlarged and displayed in the size of the original image.

From FIG. 8, it can be seen that symbols of different sizes are well processed in any of the three scale images. Small size symbols such as "x", "n", "j", "a" are detected in the original image and are of intermediate size such as "(", ")", "=", "P". The symbol is detected in the 1/4 image, and the large size symbol "Σ" is detected in the 1/16 image.

The position of the symbol is also estimated correctly. However, as shown in the images on the 10th and 11th lines, both the symbol "(" and the symbol ")" are detected at the same location. Since the symbol "(" and the symbol ")" appear as a pair in the mathematical formula, they are not distinguished into different positions in the semi-supervised learning in which the learning is performed only by the class information without using the position information. However, the adverse effect on clustering is minor.

The present invention is not limited to the above-described embodiment, and various modifications can be made. The present invention includes a configuration substantially the same as the configuration described in the embodiment (for example, a configuration having the same function, method and result, or a configuration having the same purpose and effect). The present invention also includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced. The present invention also includes a configuration that exhibits the same effects as the configuration described in the embodiment or a configuration that can achieve the same object. Further, the present invention includes a configuration in which a known technique is added to the configuration described in the embodiment.

In the above embodiment, the case of the offline method (image input method) in which the handwritten mathematical formula pattern is read as an image by an optical reading device has been described, but the coordinate data of the position (writing point) of the writing medium is read at regular time intervals by a tablet or the like. The present invention can also be applied to a detection online method (handwriting coordinate input method). In the case of the online method, the above embodiment can be applied as it is by connecting a time-series brush stroke sequence, giving it a thickness, and fleshing it to convert it into an image. Alternatively, the features of the one-dimensional data of the time series may be extracted, and the symbol position features (LC features) may be extracted as in the offline method. In the case of the online method, it is not necessary to reduce the handwritten mathematical expression pattern image and prepare multiple scale images as in the offline method, and the symbol position feature is used for each symbol using the LSTM (Long Short-Term Memory) neural network. Can be extracted. Then, by one-dimensional hierarchical space pooling, a plurality of types of divisions are performed on the symbol position features, the appearance probability of each symbol type is obtained for each division, and a feature vector is generated, and all types of divisions are generated. A hierarchical spatial feature vector is generated by concatenating the feature vectors from the division. Since a plurality of scale images are not prepared, it is not necessary to perform maximum aggregation (MA) of hierarchical space feature vectors.

Further, in the above embodiment, the case of classifying the handwritten mathematical expression pattern has been described, but the present invention is not limited to this. The present invention can be applied to the classification of structures (handwritten patterns such as structural formulas, chemical reaction formulas, and logical formulas of chemical substances) in which symbols are arranged in a two-dimensional space.

100 ... Processing unit, 110 ... Scale image generation unit, 111 ... Feature extraction unit, 112 ... Feature vector generation unit, 113 ... Integration unit, 114 ... Classification unit, 115 ... Display control unit, 116 ... Learning unit, 160 ... Input unit , 170 ... storage, 190 ... display

Claims (4)

A program for classifying multiple handwritten patterns that have been handwritten.
A scale image generator that reduces an image of one handwritten pattern to at least one step and generates a plurality of scale images including an original image before reduction and at least one reduced image.
A feature extraction unit that applies each of the plurality of scale images to a learning model using a convolutional neural network and extracts symbol position features related to the appearance position of each symbol type for each of the plurality of scale images.
A feature vector generation unit that performs a plurality of types of divisions on the symbol position feature, obtains the appearance probability of each symbol type for each division obtained by each of the plurality of types of divisions, and generates a feature vector. ,
An integration unit that generates an integrated feature vector by taking the maximum value of the appearance probability of the same symbol at the same location of the feature vector generated from each of the plurality of scale images.
A program characterized in that a computer functions as a classification unit that classifies a plurality of the handwritten patterns into a plurality of groups based on the integrated feature vector generated from each of the images of the plurality of handwritten patterns. In claim 1,
Each of the plurality of scale images is given to the training model, a feature map is acquired for each of the plurality of scale images, and the symbol position feature obtained by inputting the feature map into the classifier and the feature map are obtained. A vector obtained by multiplying and averaging is input to the classifier to generate a first appearance probability vector indicating the appearance probability for each symbol type, and the first appearance probability vector generated from each of the plurality of scale images is used. The first integrated appearance probability vector is generated by taking the maximum value of the appearance probability of the same symbol at the same location, and the learning is based on the first integrated appearance probability vector and teacher data indicating the presence or absence of appearance for each symbol type. A program characterized by further functioning a computer as a learning unit for learning a model. In claim 2,
The learning unit
Each of the plurality of scale images is given to the training model, a feature map is acquired for each of the plurality of scale images, and a vector having the maximum value of the feature map is input to the classifier for each symbol type. A second appearance probability vector indicating the appearance probability of is generated, and the maximum value of the appearance probability of the same symbol is taken at the same place of the second appearance probability vector generated from each of the plurality of scale images to obtain the second integrated appearance probability. A vector is generated, a third integrated appearance probability vector obtained by averaging the first integrated appearance probability vector and the second integrated appearance probability vector is generated, and the third integrated appearance probability vector and the teacher data are used as the basis. A program characterized by training a learning model. A clustering device that classifies multiple handwritten patterns that have been handwritten.
A scale image generator that reduces an image of one handwritten pattern to at least one step and generates a plurality of scale images including an original image before reduction and at least one reduced image.
A feature extraction unit that applies each of the plurality of scale images to a learning model using a convolutional neural network and extracts symbol position features related to the appearance position of each symbol type for each of the plurality of scale images.
A feature vector generation unit that performs a plurality of types of divisions on the symbol position feature, obtains the appearance probability of each symbol type for each division obtained by each of the plurality of types of divisions, and generates a feature vector. ,
An integration unit that generates an integrated feature vector by taking the maximum value of the appearance probability of the same symbol at the same location of the feature vector generated from each of the plurality of scale images.
A clustering apparatus including a classification unit that classifies a plurality of the handwritten patterns into a plurality of groups based on the integrated feature vector generated from each of the images of the plurality of handwritten patterns.

Images