DE202022104461U1 - Syntax guided recognition system for mathematical terms - Google Patents

Abstract

A syntax-directed recognition system for mathematical terms, the system of the present invention comprising a central processing unit and a main memory; wherein the main memory serves to store necessary software used in the system and the central processing unit serves to execute processes used in the system; the system of the present invention further comprising:
an on-line handwritten mathematical term unit for receiving on-line handwritten mathematical terms, including strokes of mathematical symbols and associated texts of the mathematical terms, for detecting coordinates of the strokes and program codes corresponding to the on-line handwritten mathematical term; wherein the coordinates of the strokes and the program codes are formed as on-line handwritten patterns, and the on-line handwritten mathematical terms unit includes an on-line database for storing a large number of on-line handwritten patterns for use in training the neural networks ;
an offline handwritten and printed mathematical terms unit for receiving the offline handwritten and printed mathematical terms with preset program codes, the offline handwritten and printed mathematical terms and associated program codes constituting templates for offline handwritten and printed mathematical terms , which are stored in an offline database;
a structured math term generator for generating structured math terms; the math term generator including a corpus and a math grammar database; the generator for structured mathematical expressions, which collects a large amount of program codes of mathematical terms, which are then stored in the corpus; the Mathematical Grammar Database stores grammars of certain mathematical programming languages, while the Structured Mathematical Terms Generator generates a large set of structured mathematical terms by using the program codes of the mathematical terms in the corpus based on the grammars in the Mathematical Grammar Database ; and
a handwriting pattern generator in communication with the online handwritten math term engine, the offline handwritten and printed math term engine, and the structured math term generator; the handwriting sample generator for generating a bulk of handwriting samples from mathematical terms for the online mathematical terms unit in the online handwritten mathematical terms unit, the offline handwritten and printed mathematical terms unit and the structured mathematical terms generator generate use in training the neural network; and
a neural network for recognizing mathematical terms, having an input interface and an output interface; the input interface including a plurality of input ports and the output interface including a plurality of output ports, the input interface communicating with the handwriting pattern generator for receiving the handwriting patterns for mathematical terms; wherein the output interface is for receiving the program codes corresponding to the mathematical term handwriting samples input to the input interface correspond to; and in the prediction stage, coordinates of strokes of a mathematical term for recognition are input to the input interface of the neural network for mathematical recognition; the output interface of the neural network outputs at least one program code which corresponds to the mathematical term entered.

Description

FIELD OF THE INVENTION

The invention relates to neural network recognition of mathematical terms, and more particularly to a syntax-directed recognition system for mathematical terms.

BACKGROUND OF THE INVENTION

Online handwritten math terms refer to math terms that are acquired by a user entering math terms by hand into an electronic device (e.g., touch pens, blackboards, or tablets, etc.). In this invention, the mathematical terms include mathematical symbols and mathematical formulas composed of mathematical symbols. Furthermore, the respective positional relationships between mathematical symbols are also detected. Detection based on the above data is called online detection.

The so-called offline handwritten mathematical term or printed mathematical term has the task of capturing mathematical terms which do not correspond to the above-mentioned online handwritten states. For example, two-dimensional figures or printed mathematical terms that do not contain information about the strokes and stroke sequences. The recognition of the mathematical term based on this data (offline) is called offline recognition.

Conventionally, when training a neural network to recognize mathematical terms, a large amount of mathematical terms is necessary in the training stage. The number of samples required can reach hundreds of thousands. Such a large amount of mathematical terms can only be generated mechanically, while conventional ways of generating mathematical terms take too much time. In addition, handwriting samples are very difficult to collect and, moreover, a trained neural network cannot be modified or even expanded. All of these are problems encountered in the current state of the art.

SUMMARY OF THE INVENTION

Accordingly, it is an object of this invention to provide a syntax-directed mathematical term recognition system that combines online handwritten mathematical terms, offline handwritten and printed mathematical terms, and structured mathematical terms to generate a vast set of mathematical terms that can be used in training of neural networks are applicable. An end-to-end learning path is applied to train the neural network and achieve a better learning effect, while also promoting cognitive abilities. In addition, the problem of overfitting in multi-layer neural networks is effectively reduced and generalization is increased. As a result, the accuracy of a test set is massively improved. The neural network mode of this invention is applicable to on-line handwritten math terms as well as off-line handwritten and printed math terms. Therefore, only a single detection system is used and no other system is required, thereby greatly reducing costs. More new mathematical terms are needed for recognition, but few new patterns are applied and grammars of the mathematical programming language need only minor adjustments to enhance the neural network's cognitive ability. In the invention, the mathematical language grammar is applied during training and thus the recognition results are adapted to the mathematical grammar. In addition, the invention is advantageous for customization.

character list

• Figure 1 is a structured block diagram showing the elements of the invention.
• Figure 12 is a schematic view of mathematical recognition training for the neural network according to the present invention.
• Figure 12 is another schematic view of mathematical recognition training for the neural network according to the present invention.
• Fig. 12 is a schematic view demonstrating the application of the invention, showing a mathematical recognition prediction stage for the neural network according to the present invention.
• shows the hardware structure of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT OF THE INVENTION

According to the until the system according to the invention is shown. The elements of the invention are described below. The system of the present invention includes a central processing unit 100; and a main memory 200. The main memory 200 is for storing software necessary in the present invention, and the central processing unit 100 is for performing the processes used in the present invention. The system of the present invention further includes the elements described below.

An on-line handwritten mathematical term unit 10 serves to receive on-line handwritten mathematical terms including strokes of mathematical symbols and related texts of mathematical terms, obtaining coordinates of the strokes and program codes corresponding to the on-line handwritten mathematical term (for certain computer languages, such as LaTeX or MathML - Mathematical Markup Language. However, all computer languages usable for mathematical terms are permissible in the invention). The program codes can be converted into mathematical terms after compilation or interpretation. The coordinates of the strokes and the program code are mapped as on-line handwritten patterns 12 . Preferably, coordinates of the strokes are stored by InkML (Ink Markup Language).

For example, in the mathematical programming language LaTeX, a program code of \\frac { 1 } { 3 } means nothing more than $\frac{1}{3}.$

In training neural networks, coordinates of the strokes of mathematical terms are used as inputs of the neural networks and the program codes serve as outputs of the neural networks.

The on-line handwritten mathematical terms unit 10 includes an on-line database 15 for storing larger amounts of on-line handwritten patterns 12 for training the neural networks.

In the unit for online handwritten mathematical terms 10, sub-mathematical terms are extracted from the original online handwritten mathematical terms to form them as independent mathematical terms, wherein a sub-mathematical term is a part of the original online handwritten mathematical term. The coordinates of the strokes and program codes of the sub-mathematical terms are also extracted to form on-line handwritten templates 12, which are then stored in the on-line database 15.

An off-line handwritten and printed mathematical terms unit 20 is for receiving the off-line handwritten and printed mathematical terms with preset program codes, the program codes of the off-line handwritten and printed mathematical expressions being known in advance. The handwritten and printed off-line math terms and associated program codes are formed as off-line handwritten and printed math expression templates 22, which are then stored in an off-line database 25. For example, the offline handwritten and printed mathematical term is a two-dimensional figure (e.g., formatted with PNG) whose program codes are stored in a markup language, such as the MathML language.

A structured math term generator 30 is used to generate structured math terms. The math term generator 30 includes a corpus 34 and a math grammar database 33 . To do this, the structured mathematical term generator 30 collects a large amount of mathematical term program codes (e.g. via networks) which are stored in the corpus 34 . The math grammar database 33 stores grammars of specific math programming languages, such as stochastic context-free grammars. The stochastic, context-free grammar describes grammars of program codes of mathematical terms for some specific programming languages like LaTeX or MathML. The structured math term generator 30 generates a large set of structured math terms using the program codes of the math terms in the corpus 34 based on the grammars in the math grammar database 33 . The structured mathematical terms can be stored using mathematical markup languages such as MathML.

1. (a) Die Koordinaten der Striche und die Programm-Codes der handgeschriebenen mathematischen Online-Termini in den handschriftlichen Online-Mustern 12 werden als entsprechende handschriftliche Muster mathematischer Termini 42 gebildet.
2. (b) Es fndet eine Einbeziehung der Striche aus den handgeschriebenen und gedruckten mathematischen Offline-Termini in die Probemenge für handgeschriebenen und gedruckten Offline-Termini 22 zur Erfassung der Koordinaten der Striche statt, wobei die Programm-Codes des handgeschriebenen und gedruckten mathematischen Offline-Terminus im Voraus bekannt sind, wie es auch schon weiter oben festegeigt wurde. Daher werden diese Koordinaten und Programm-Codes für die handgeschriebenen und gedruckten mathematischen Offline-Termini als entsprechende Handschriftmuster für mathematische Termini 42 gebildet.
3. (c) Erfassung von Handschriftmustern für mathematische Termini 42 für struktrierte mathematische Termini aus dem Generator für strukturierte mathematische Termini 30 auf folgende Weise: (1) die strukturierten mathematischen Termini werden in mathematische Termini in gedruckter Form umgewandelt (gerendert) und für jedes mathematische Symbol in dem mathematischen Ausdruck in gedruckter Form wird ein minimaler Rechteckrahmen erfasst, welcher gerade ein entsprechendes mathematisches Symbol enthält. (2) Die gedruckten mathematischen Termini sind in mehrere untergeordnete mathematische Termini unterteilt, welche identische mathematische Termini in den handschriftlichen mathematischen Mustern für mathematische Online-Termini 12 auffindbar machen. Wenn die untergeordneten mathematischen Termini keine entsprechenden mathematischen Online-Termini in den handschriftlichen mathematischen Mustern 12 finden, dann werden diese submathematischen Ausdrücke im folgenden Prozess nicht verwendet. Der minimale Rechtecksrahmen für jedes mathematische Symbol wird angepasst. (3) Ziel ist es, Koordinaten von Strichen in den Mustern handschriftlicher mathematischer Termini 12 zu finden, welche den Strichen jedes Symbols in den submathematischen Ausdrücken entsprechen. Dann wird eine affine Transformation an den oben gefundenen Koordinaten der Striche durchgeführt, um transformierte Koordinaten der Striche jedes Symbols und die Programm-Codes des strukturierten mathematischen Terminus zu erhalten, die als ein Handschriftmuster für mathematische Ausdrücke 42 kombiniert werden. (4) Das Handschriftmuster für mathematische Termini 42 wird zufällig gedreht oder in der Größe zufällig geändert, um mehr und mehr Handschriftproben 42 für mathematische Ausdrücke zu erzeugen.

A handwriting sample generator 40 is connected to the on-line handwritten math term engine 40, the off-line handwritten and printed math term engine 20 and the structured math term generator 20. The handwriting pattern generator 40 serves to generate a large set of handwriting patterns for mathematical terms 42 for the online mathematical terms unit 10, for the offline handwritten mathematical terms unit 20 and for both handwritten and printed mathematical terms and the generator 30 with the aim of Using structured mathematical terms to train the neural network. In the invention, the amount of handwriting samples 42 for mathematical expressions in the five-figure range. Ways of generating the mathematical expression handwriting patterns are described below.

1. (a) The coordinates of the strokes and the program codes of the on-line handwritten mathematical terms in the on-line handwritten patterns 12 are formed as corresponding on-line handwritten patterns 42.
2. (b) The strokes from the offline handwritten and printed mathematical terms are included in the sample set for offline handwritten and printed mathematical terms 22 for detecting the coordinates of the strokes, the program codes of the offline handwritten and printed mathematical terms are known in advance, as has already been established above. Therefore, these coordinates and program codes for the off-line handwritten and printed mathematical terms are formed as corresponding handwriting patterns for mathematical terms 42 .
3. (c) Acquisition of handwriting samples for mathematical terms 42 for structured mathematical terms from the structured mathematical terms generator 30 in the following way: (1) the structured mathematical terms are converted (rendered) into mathematical terms in printed form and for each mathematical symbol in the mathematical expression in printed form, a minimal rectangular frame is captured, which just contains a corresponding mathematical symbol. (2) The printed mathematical terms are divided into several subordinate mathematical terms, which make identical mathematical terms discoverable in the handwritten mathematical templates for online mathematical terms 12. If the sub-mathematical terms do not find corresponding online mathematical terms in the handwritten mathematical patterns 12, then those sub-mathematical expressions are not used in the following process. The minimum rectangle frame for each math symbol is adjusted. (3) The aim is to find coordinates of strokes in the patterns of handwritten mathematical terms 12 which correspond to the strokes of each symbol in the sub-mathematical expressions. Then, an affine transformation is performed on the stroke coordinates found above to obtain transformed stroke coordinates of each symbol and the program codes of the structured mathematical term, which are combined as a handwriting pattern for mathematical expressions 42 . (4) The handwriting sample for mathematical terms 42 is randomly rotated or resized to generate more and more handwriting samples 42 for mathematical expressions.

A neural network 50 for recognizing mathematical terms includes an input interface 51 and an output interface 55. The input interface 51 includes a plurality of input ports 52 and the output interface 55 includes a plurality of output ports 56. As in FIGS and As shown, the input interface 51 is connected to the handwriting pattern generator 40 for receiving the handwriting patterns of mathematical expressions 42 . The output interface 55 is for receiving the program codes corresponding to the handwriting patterns of mathematical terms input to the input interface 51 .

In the training phase, tens or even hundreds of thousands of samples of handwriting samples 42 are used for mathematical terms. The coordinates of the strokes in the handwriting patterns for mathematical terms are input to a plurality of input terminals 52 of the input interface 51 . The outputs of the output interface 55 are set as program codes corresponding to the handwriting patterns for mathematical terms. The weighting values in the connecting lines of the neural network 50 are adjusted by several training stages. The math term recognition neural network 50 is capable of identifying input handwriting samples 42 of math terms.

In the prediction stage, coordinates of strokes of a mathematical term 100 which conform to the mathematical grammars are input to the input interface 51 of the mathematical cognitive neural network 50 for recognition. The neural network output interface 55 outputs at least one program code corresponding to the mathematical expression 100 input.

As in As shown, the neural network 50 for mathematical recognition includes an encoding neural network 510, which includes the input interface 51 and a decoding neural network 520, which includes the output interface 55. The coding neural network 510 serves to find features of the input mathematical expression 510 . Then these features are fed into the decoding neural network 520 . the Output of the decoding neural network 520 comprises portions of program codes of the input mathematical expression 100, which are given with some confidence. These portions of program code and reliabilities are fed back for re-detection upon input of decoding neural network 520 to detect other portions of program code and reliabilities. These processes can be repeated over and over again until a desired reliability or preset time is reached. The resulting final program codes and reliabilities are stored in a candidate unit 81. For example, the type of autoregression is a predictive analysis method based on LL(1) grammar. Referring to an application is demonstrated in which the candidate set 81 contains the three program codes \\frac { 1 } { 3 }, - \\frac { 1 } { 3 }, and \\frac { 1 } { 2 } with a respective reliability of 50%, 30% and 20% included. This means that entering the mathematical term 100 \\frac { 1 } { 3 } is included with a probability of 50%.

Having described the present invention, it is obvious that the same can be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

In summary, the invention relates to a syntax-driven mathematical term recognition system combining online handwritten mathematical terms, handwritten and printed mathematical terms, and structured mathematical terms to generate a large set of mathematical terms applicable to neural network training. An end-to-end learning path is used to train the neural network to increase learning and cognitive ability. The problem of overfitting in multi-layer neural networks is effectively reduced to increase the accuracy of a test set. The neural network mode is applicable to online handwritten math terms and offline handwritten and printed math terms. Therefore, only one detection system is used and another system is not required.

Claims (11)

A syntax-directed recognition system for mathematical terms, the system of the present invention comprising a central processing unit and a main memory; wherein the main memory serves to store necessary software used in the system and the central processing unit serves to execute processes used in the system; the system of the present invention further comprising: an online handwritten mathematical terms unit for receiving online handwritten mathematical terms containing strokes of mathematical symbols and associated texts of the mathematical terms to capture coordinates of the strokes and program codes, which correspond to the online handwritten mathematical term; wherein the coordinates of the strokes and the program codes are formed as on-line handwritten patterns, and the on-line handwritten mathematical terms unit includes an on-line database for storing a large number of on-line handwritten patterns for use in training the neural networks ; an offline handwritten and printed mathematical terms unit for receiving the offline handwritten and printed mathematical terms with preset program codes, the offline handwritten and printed mathematical terms and associated program codes constituting templates for offline handwritten and printed mathematical terms , which are stored in an offline database; a structured math term generator for generating structured math terms; the math term generator including a corpus and a math grammar database; the Structured Mathematical Expression Generator, which collects a large amount of program code of mathematical terms, which are then stored in the corpus; the Mathematical Grammar Database stores grammars of certain mathematical programming languages, while the Structured Mathematical Terms Generator generates a large set of structured mathematical terms by using the program codes of the mathematical terms in the corpus based on the grammars in the Mathematical Grammar Database ; and a handwriting pattern generator in communication with the online handwritten math term engine, the offline handwritten and printed math term engine, and the structured math term generator; the handwriting pattern generator for generating a bulk of handwriting patterns from mathematical terms for the unit for online mathematical terms in the unit for online handwritten mathematical generate terms, the off-line handwritten and printed mathematical terms unit and the structured mathematical terms generator for use in training the neural network; and a neural network for recognizing mathematical terms, having an input interface and an output interface; the input interface including a plurality of input ports and the output interface including a plurality of output ports, the input interface communicating with the handwriting pattern generator for receiving the handwriting patterns for mathematical terms; the output interface for receiving the program codes corresponding to the mathematical term handwriting samples inputted to the input interface; and in the prediction stage, coordinates of strokes of a mathematical term for recognition are input to the input interface of the neural network for mathematical recognition; the output interface of the neural network outputs at least one program code which corresponds to the mathematical term entered. A system after claim 1 wherein the ways of generating a handwriting sample for mathematical terms are as follows: (a) the coordinates of the strokes and the program codes of the handwritten online mathematical terms in the handwritten online patterns are formed directly as corresponding handwritten patterns of mathematical terms; (b) obtaining the strokes of the off-line handwritten and printed mathematical terms in the off-line handwritten and printed mathematical expression templates to obtain the coordinates of individual strokes; these coordinates and the default program codes for the off-line handwritten and printed mathematical terms are formed as corresponding handwriting patterns for mathematical terms; (c) capturing handwriting samples of mathematical terms for the structured mathematical terms in the structured mathematical terms generator in the following way: (1) the structured mathematical terms are converted into mathematical terms in printed form or reproduced as such, for each mathematical symbol , which is formed in the printed mathematical term, a minimal rectangular frame is detected that just contains a corresponding mathematical symbol; (2) the printed math terms are divided into several subordinate math terms, which identical math terms may find in the handwritten math templates for online math terms; if the mathematical sub-expressions do not find corresponding mathematical terms in the handwritten examples of online mathematical terms, then these mathematical sub-expressions are not used in the following process; the minimum rectangular frame for each mathematical symbol is adjusted; (3) coordinates of strokes are found in the online handwritten mathematical patterns corresponding to the strokes of each symbol in the submathematical terms; then an affine transformation is performed on the coordinates of the strokes found above to obtain transformed coordinates of the strokes of each symbol and the program codes of the structured mathematical term, which are combined as a handwriting pattern of a mathematical term; (4) the handwriting pattern for mathematical terms is randomly rotated or resized to generate more and more handwriting patterns for mathematical terms. A system after claim 1 , wherein the neural network for mathematical recognition includes a neural coding network, which also includes the input interface; and a neural decoding network including the output interface; the coding neural network serves to find features of the input mathematical term; then these features are fed into the decoding neural network; the output of the decoding neural network contains parts of program codes of the input mathematical term, which are displayed with a certain statistical reliability; these parts of the program code and the reliabilities are automatically regressed back to the input of the decoding neural network for re-recognition to obtain other parts of the program code and their reliabilities; these processes are performed over and over again until a preset reliability or autoregression time is reached; the resulting final program codes and reliabilities are stored in a candidate unit. A system after claim 3 , where the autoregression approach is a predictive analysis method using LL(1) grammars. A system after claim 1 , wherein in the unit for online handwritten mathematical terms, sub-mathematical terms are extracted from the original online handwritten mathematical terms to form them as independent mathematical terms, wherein a sub-mathematical term is a part of the original of handwritten online math terms; the coordinates of the strokes and program codes of the sub-mathematical terms are also extracted to form on-line handwritten patterns, which are then stored in the on-line database. A system after claim 1 , where the mathematical programming language is LaTeX or MathML (Mathematical Markup Language). A system after claim 1 , where coordinates of the strokes of the online handwritten pattern are stored by InkML (Ink Markup Language). A system after claim 1 , where the offline handwritten and printed mathematical term represents a two-dimensional figure and the program codes are stored by markup language. A system after claim 1 , wherein the mathematical grammar database stores syntax rules of specific mathematical programming languages that map stochastic context-free grammar. A system after claim 9 , whereby the stochastic context-free grammar describes grammars of program codes of mathematical terms for program languages such as LaTeX or MathML. A system after claim 1 , where the structured mathematical terms are stored using mathematical markup languages.

Images