JP3243700U - Syntax-driven formula identification system - Google Patents

Abstract

A syntax-driven mathematical expression identification system is provided. A syntax-driven formula identification system includes an online handwritten formula unit that stores strokes of online handwritten formulas and a specific program code in a mathematical programming language corresponding to the strokes, and an offline handwritten and printed version of the known program code. An offline handwritten and printed formula unit that stores formulas; a structured formula generation unit that collects a large number of program codes for formulas and generates a large number of structured formulas based on grammar; In the prediction stage, the mathematical formula identification neural network receives strokes of mathematical formulas that match the grammar, identifies them, and then outputs at least one program code corresponding to the mathematical formula. and a handwritten handwriting sample generation unit that outputs and displays possible identification results. [Selection diagram] Figure 2

Description

TECHNICAL FIELD The present invention relates to a mathematical expression identification system, and more particularly to a syntax-driven mathematical expression identification system.

The so-called online handwritten mathematical expression refers to a mathematical expression composed of mathematical symbols and related characters that are handwritten by a user when connecting to a device. The mathematical formula of the present invention includes a broad mathematical symbol, a mathematical formula, etc. Since the formula is entered while connected to the device, the device uses a stroke capture mechanism (e.g., stylus, handwriting pad, or touch pad) to track the stroke trajectory of the user in the process of writing the mathematical symbol or related character. Instantly convert to stroke coordinates and record the corresponding position coordinates of different mathematical symbols or related characters. The content is identified using the above-described method, that is, online recognition is performed.

So-called offline handwritten formulas or printed formulas are formulas obtained in a situation other than the above-mentioned online situation (i.e., where the input of stroke trajectories cannot be obtained immediately), and are, for example, saved in a two-dimensional image or printed on paper. This is a mathematical formula that does not include stroke order information. Identifying the content of such a mathematical formula is offline recognition.

Conventionally, when training a neural network for identifying mathematical expressions, it is necessary to use an extremely large number of mathematical expression samples in the training set, and the number of samples reaches 100,000 or more. The only way to collect these formulas is to generate them mechanically, however, the conventional formula generation method is very slow, and the number of conventional handwritten samples is quite small, making it difficult to collect. As a result, it took a long time, and it was not possible to modify or expand the classification range of a neural network that had been trained.

Therefore, the inventor of the present invention believed that the above-mentioned drawbacks could be improved, and as a result of intensive studies, he came up with the proposal of the present invention, which effectively improves the above-mentioned problems through rational design.

The present invention has been made in view of these circumstances, and its purpose is to provide a syntax-driven formula identification system.

According to one embodiment of the present invention, a syntax-driven formula identification system is provided. That is, one embodiment of the present invention includes the following configuration. Combine online handwriting, offline handwriting, and printed formulas, combine mathematical programming language rules to automatically synthesize a large number of samples, and train neural networks using an end-to-end learning method. It strengthens the learning effect, improves the discrimination ability, effectively reduces the overfitting problem of multi-layer neural network models, improves the generalization performance, and improves the accuracy of the test set. The neural network model according to the present invention can be simultaneously applied to the identification of online handwriting, offline handwriting, and printed mathematical expressions, and there is no need to develop multiple types of identification systems separately, which can significantly reduce costs.

1 is a schematic block diagram showing a combination of main parts of a syntax-driven formula identification system according to an embodiment of the present invention; FIG. FIG. 3 is a schematic diagram illustrating a training stage of a neural network for formula recognition according to an embodiment of the present invention; FIG. 6 is a schematic diagram illustrating a training stage of a neural network for formula recognition according to another embodiment of the present invention; FIG. 3 is a schematic diagram illustrating the application of the prediction stage of the formula identification neural network according to an embodiment of the present invention; 1 is a schematic configuration diagram showing the hardware of a syntax-driven formula identification system according to the present invention; FIG.

Embodiments of the present invention will be described in detail below. However, the present invention is not limited to this, and various modifications can be made within the scope described, and the present invention may also be applied to embodiments obtained by appropriately combining technical means disclosed in different embodiments. included in the technical scope of

The syntax-directed mathematical expression training and identification system 1 according to the present invention mainly includes a processor 2 and a memory 6 (see FIG. 5). The processor 2 is used to perform the operations required by the system 1. The memory 6 is used to store data or calculation programs of computer programs in the system 1, related operation results, and related data. All operation results and related data are stored in the memory 6.

1 to 5 illustrate a syntax-directed mathematical expression training and identification system according to the present invention, which mainly includes the following components.

<Online handwritten mathematical formula unit 10>
Used to receive Online Handwritten Mathematical Expressions, that is, receive strokes of mathematical symbols and associated characters in the Online Handwritten Mathematical Expressions, and calculate the coordinates of each stroke in the Online Handwritten Mathematical Expressions and the online handwritten mathematical expressions. A program code corresponding to the handwritten formula is obtained to form an online handwritten sample 12. Preferably, the coordinates of each stroke in the online handwriting sample 12 are saved in Ink Markup Language (InkML) format.

The program code corresponds to a specific program code in a mathematical programming language, and the mathematical programming language is LaTeX or MathML (Mathematical Markup Language), but in reality, the present invention can use any mathematical programming language. Therefore, the program code is converted into a corresponding mathematical formula after compilation of the mathematical programming language.

Taking LaTeX as an example of the mathematical programming language, for example, the formula "\1"/"\3" is obtained after compiling the program code "\\frac {1} {3}".

The program code according to the present invention is used for training a neural network. During training, the output value of the neural network is set as a program code to which the formula corresponds, and the stroke coordinates of the formula are used as input values of the neural network.

The online handwritten math formula unit 10 includes an online handwritten math formula sample library 15 that stores a large number of the online handwritten math samples 12 for use in training a neural network.

In this embodiment, the online handwritten formula unit 10 extracts a sub-formula from the online handwritten formula, that is, refers to a part of the online handwritten formula that constitutes a single formula. The strokes and program codes for these sub-mathematical expressions are acquired to generate corresponding online handwritten samples 12, and stored in the online handwritten mathematical expression sample library 15.

<Offline handwritten and printed mathematical formula unit 20>
Receiving offline handwritten and printed mathematical expressions of known program code and forming offline handwritten and printed mathematical expression samples 22 with the offline handwritten and printed mathematical expressions and their corresponding known program codes. It is also used to store in the offline handwritten and printed mathematical equation sample library 25. For example, the offline handwritten and printed formulas are two-dimensional images (eg, in PNG format), and the program code is stored in a mathematical markup language format. The mathematical markup language is MathML grammar or the like.

<Structured formula generation unit 30>
It is used to generate structured formulas. The structured formula generation unit 30 includes a corpus 34 and a mathematical grammar base 33. The structured formula generation unit 30 collects a large amount of program codes of formulas (for example, from the Internet) and stores them in the corpus 34 . The mathematical grammar base 33 stores a grammar of a specific mathematical programming language, for example, a stochastic context-free grammar, and the stochastic context-free grammar is a LaTeX or a Stochastic context-free grammar that describes a mathematical formula. This is a grammar that satisfies the MathML program code. The structured formula generation unit 30 combines the program codes of formulas in the corpus 34 into a number of structured formulas that conform to the format of the mathematical programming language based on the grammar of the mathematical grammar base 33. The structured formula is stored in a mathematical markup language (for example, MathML format).

<Handwritten handwriting sample generation unit 40>
It is connected to the online handwritten formula unit 10, the offline handwritten and printed formula unit 20, and the structured formula generation unit 30. The handwritten handwriting sample generating unit 40 converts each mathematical formula of the online handwritten mathematical formula unit 10, the offline handwritten and printed mathematical formula unit 20, and the structured mathematical formula generating unit 30 into a large number of mathematical handwritten handwriting samples 42, and converts each mathematical formula into a large number of mathematical handwritten handwriting samples 42, (The present invention generates hundreds of thousands of mathematical handwriting samples 42 during training). A method of generating the mathematical handwritten handwriting sample 42 will be described below.
(a) A corresponding mathematical handwritten handwriting sample 42 is formed based on the coordinates of each stroke in each online handwritten sample 12 and the corresponding program code of the online handwritten mathematical formula.
(b) Extract strokes according to the offline handwritten and printed formulas of each offline handwritten and printed formula sample 22, obtain the coordinates of each stroke, and the corresponding program code of the offline handwritten and printed formulas is a known one; Therefore, both form a corresponding mathematical handwritten handwriting sample 42.
(c) Synthesizing a corresponding mathematical handwriting sample 42 using each of the structured mathematical expressions of the structured mathematical expression generation unit 30, the method of which is as follows: (1) rendering the structured mathematical expression in a printed form; At the same time, the bounding box of each symbol in the typeface is recorded. (2) Decompose the structured mathematical formula in the printed form into some maximal parts having the same online handwritten mathematical formula samples 12, and for each part, create symbols in the printed form based on the corresponding online handwritten mathematical formula samples 12. Adjust the position of the corresponding bounding box. (3) After that, each symbol in the printed font is made to correspond to the coordinates of the corresponding stroke in the online handwritten sample 12, and a slight random affine transformation is performed, so that each symbol after the conversion corresponds to the corresponding stroke. A corresponding mathematical handwritten handwriting sample 42 is synthesized using the coordinates of and the program code to which the structured mathematical expression corresponds. (4) Randomly rotate or scale the mathematical handwritten handwriting samples 42 to generate more mathematical handwritten handwriting samples 42;

<Formula identification neural network 50>
It includes an input interface 51 and an output interface 55, the input interface 51 having a plurality of input terminals 52, and the output interface 55 having a plurality of output terminals 56. As shown in FIGS. 1 and 2, the input interface 51 is connected to the handwritten handwriting sample generation unit 40 and is used to receive the mathematical handwritten handwriting sample 42. The output interface 55 is used to output the program code of the mathematical formula to which the mathematical handwriting sample 42 corresponds.

Furthermore, in the training stage, tens of thousands or hundreds of thousands of mathematical handwritten handwriting samples 42 are employed for training. The stroke coordinates of the mathematical handwriting sample 42 are input to the plurality of input terminals 52 of the mathematical equation identification neural network 50, and the output result of the output interface 55 of the mathematical equation identification neural network 50 is corresponded to the mathematical handwriting sample 42. Set as the program code to be used. After training is repeated multiple times, the connection weight values in the mathematical formula identification neural network 50 are adjusted during the training process, so that the mathematical formula identification neural network 50 can identify the input mathematical handwritten handwriting sample 42. Become.

In the prediction stage, the stroke coordinates of the formula 100 that conform to the mathematical grammar are input to the input interface 51 of the formula identification neural network 50, the formula identification neural network 50 performs identification, and the output interface 55 selects at least one stroke coordinate. Output the program code. Each said program code corresponds to a mathematical formula.

The formula identification neural network 50 includes an encoder neural network 510 and a decoder neural network 520 (see FIG. 3). The input interface 51 is located in the encoder neural network 510 and the output interface 55 is located in the decoder neural network 520. The encoder neural network 510 is used to discover features from the input formula 100 and input them to the decoder neural network 520 to obtain the output of the formula identification neural network 50. The output includes the fragments and reliability of the originally input formula 100, and the fragments and reliability are re-inputted into the decoder neural network 520 for re-identification. After repeating this several times, the program codes and corresponding reliability of a plurality of possible formulas are obtained and stored in the result candidate set 81. The automatic regression method applies the predictive analysis method of LL(1) grammar. As shown in the application example shown in FIG. 4, the result candidate set 81 includes three program codes "\\frac { 1 } { 3 }", "-\\frac { 1 } { 3 }" and "\\frac { 1 }". The reliability of each program code is 50%, 30%, and 20%, respectively. This indicates that the input formula 100 is a formula to which these three program codes correspond, and "\\frac {1} {3}" has the highest reliability.

The present invention is not limited to the embodiments described above, and can be modified in various ways within the scope of the claims, and embodiments obtained by appropriately combining technical means devised in different embodiments. are also included in the technical scope of the present invention.

1 System 2 Processor 6 Memory 10 Online handwritten mathematical formula unit 12 Online handwritten sample 20 Offline handwritten and printed mathematical formula unit 22 Offline handwritten and printed mathematical formula sample 30 Structured mathematical formula generation unit 33 Mathematical grammar base 34 Corpus 40 Handwritten handwriting sample generation unit 42 Mathematical handwriting sample 50 Mathematical identification neural network 51 Input interface 52 Input end 55 Output interface 56 Output end 100 Mathematical formula 510 Encoder neural network 520 Decoder neural network 81 Result candidate set

Claims (11)

comprising a processor and a memory, the processor is used to perform operations necessary for the system, and the memory is for storing data or calculation programs of computer programs in the system, related operation results, and related data; A syntax-driven formula identification system used in
used for receiving online handwritten formulas, that is, receiving strokes of mathematical symbols and associated characters in the online handwritten formulas, and the coordinates of each stroke in the online handwritten formulas and the corresponding program of the online handwritten formulas; an online handwritten mathematical formula unit used to acquire a code and form an online handwritten sample, the program code corresponding to a specific program code in a mathematical programming language; an online handwritten formula unit including an online handwritten formula sample library that stores a large number of said online handwritten samples for training;
receiving offline handwritten and printed mathematical formulas of known program codes, forming offline handwritten and printed mathematical formula samples with said offline handwritten and printed mathematical formulas and their corresponding known program codes; and an offline handwritten and printed mathematical formula sample library. Offline handwritten and printed math units used to store
A structured formula generation unit used to generate structured formulas, the structured formula generation unit includes a corpus and a mathematical grammar base, and the structured formula generation unit generates a large amount of program code of formulas. The mathematical grammar base stores a grammar of a specific mathematical programming language, and the structured formula generation unit generates mathematical formulas in the corpus based on the grammar of the mathematical grammar base. a structured formula generation unit that combines the program code into a large number of structured formulas that conform to the format of the mathematical programming language;
a handwritten handwriting sample generation unit connected to the online handwritten math equation unit, the offline handwritten and printed math equation unit, and the structured math equation generation unit, wherein the handwritten handwriting sample generation unit is connected to the online handwritten math equation unit, the an offline handwritten and printed math equation unit; and a handwritten handwriting sample generation unit used to convert each equation of the structured math equation generation unit into a number of mathematical handwritten handwriting samples for training a neural network;
A mathematical expression recognition neural network including an input interface and an output interface, the input interface having a plurality of input ends, the output interface having a plurality of output ends, and the input interface connecting the handwriting sample generation unit to the handwriting sample generation unit. and in a training stage, the input interface is used to receive the mathematical handwriting sample, and the output interface is used to output a program code of the mathematical formula to which the mathematical handwritten handwriting sample corresponds, and in the prediction stage. In the input interface, the input interface is used to receive stroke coordinates of a mathematical formula that conforms to a mathematical grammar, and the mathematical formula identification neural network performs identification, and outputs at least one program code from the output interface to receive the identification result. a syntax-driven equation identification system, further comprising: an equation identification neural network for displaying, each of said program codes corresponding to a math equation. The generation method of the mathematical handwriting sample is as follows:
(a) forming a corresponding mathematical handwritten handwriting sample according to the coordinates of each stroke in each of the online handwritten samples and the corresponding program code of the online handwritten mathematical formula;
(b) extracting strokes by the offline handwritten and printed formulas of each of said offline handwritten and printed formula samples, and obtain the coordinates of each stroke, and the corresponding program code of said offline handwritten and printed formulas is known; Therefore, the two form a corresponding mathematical handwriting sample,
2. The syntax-driven formula identification system according to claim 1, wherein each of the structured formulas of the structured formula generation unit synthesizes a corresponding formula handwriting sample. The formula identification neural network includes an encoder neural network and a decoder neural network, the input interface is located in the encoder neural network, the output interface is located in the decoder neural network, and the encoder neural network recognizes the input formula from among the input formulas. It is used to discover features and input them into the decoder neural network to obtain the output of the formula identification neural network, the output including the fragments and reliability of the formula originally input, and the fragments and the reliability. is regressively input into the decoder neural network to perform identification again, and after repeating this several times, program codes and corresponding reliability of multiple possible formulas are obtained and stored in a result candidate set. 2. The syntax-driven formula identification system of claim 1. 4. The syntax-driven formula identification system according to claim 3, wherein the automatic regression method applies a predictive analysis method of LL(1) grammar. The online handwritten formula unit further extracts sub-formulas from the online handwritten formula, that is, refers to a part of the online handwritten formula that constitutes a single formula, and obtains strokes and program codes for these sub-formulas; The syntax-driven math equation identification system of claim 1, wherein a corresponding online handwritten math sample is generated and stored in the online handwritten math sample library. The syntax-driven formula identification system according to claim 1, wherein the mathematical programming language is LaTeX or MathML (Mathematical Markup Language). The syntax-driven formula identification system of claim 1, wherein the coordinates of each stroke in the online handwritten sample are stored in an Ink Markup Language (InkML) format. The syntax-driven formula of claim 1, wherein the offline handwritten and printed formula is stored in a two-dimensional image format, and the corresponding program code is stored in a mathematical markup language format. identification system. The syntax-driven formula identification system of claim 1, wherein the grammar of the mathematical programming language stored in the mathematical grammar base is a stochastic context-free grammar. 10. The syntax-driven formula identification system of claim 9, wherein the probabilistic context-free grammar is a grammar that satisfies a LaTeX or MathML program code that describes a formula. The syntax-driven formula identification system of claim 1, wherein the structured formula is stored in a mathematical markup language format.