CN106980856B - Formula identification method and system and symbolic reasoning calculation method and system - Google Patents
Formula identification method and system and symbolic reasoning calculation method and system Download PDFInfo
- Publication number
- CN106980856B CN106980856B CN201610028002.0A CN201610028002A CN106980856B CN 106980856 B CN106980856 B CN 106980856B CN 201610028002 A CN201610028002 A CN 201610028002A CN 106980856 B CN106980856 B CN 106980856B
- Authority
- CN
- China
- Prior art keywords
- character
- formula
- result
- characters
- data information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Character Discrimination (AREA)
Abstract
The formula identification method comprises the following steps: acquiring data information of a formula; carrying out character segmentation on the data information to obtain a single character; identifying the single character to obtain a character identification result of the single character; identifying the position relation among the single characters to obtain the position relation among the single characters; and carrying out formula analysis on the character recognition result by combining the position relation to obtain a formula recognition result. According to the method, the data information of the formula is acquired through the human-computer interface, and the formula identification result is acquired through character segmentation, character identification, character position relation identification and formula analysis, so that the learning cost of the user for formula input according to the specification is greatly reduced, and the accuracy of mathematical formula identification is improved.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a formula identification method and system and a symbolic reasoning calculation method and system.
Background
With the rapid development of artificial intelligence and online education in recent years, human-computer-friendly computer-aided education becomes an urgent field to be explored. Mathematics is receiving increasing attention as an important discipline ranging from Kindergarten (usually 5-6 years old) to the twelve-year (Grade 12, usually 17-18 years old) (kingergen through twelfeth Grade, K12 or K-12). Typically, mathematical knowledge engines require a user to input data in a prescribed way in a format similar to a "programming language," and the engine provides the corresponding results by analyzing the user's input. Since such a data input format is different from a habit in daily life, it takes a lot of time for a user to learn and adapt, resulting in high learning cost.
In the related art, Mathematica is a piece of scientific computing software that well combines high-level connections between numeric and symbolic computing engines, graphics systems, programming languages, text systems, and other applications. Many functions are in the world leading position in the corresponding field, and the functions are also one of the more widely used mathematical software at present. While Mathematica is a powerful symbolic computing engine, it is not specifically optimized for education at stage K12. Furthermore, the Mathematica must write complex code using its prescribed programming language, and thus, when using Mathematica, a user needs to grasp an entirely new programming language in advance. This is clearly very difficult for elementary and middle school students in stage K12.
The MyScript Calculator, which is introduced by MyScript corporation in the united states, is a mathematical Calculator supporting handwriting input, which analyzes input data of a user using a conventional shallow neural network and context-based grammar recognition, and can perform basic operations such as addition, subtraction, multiplication, division, parentheses, and exponentiation. Although MyScript Calculator can support handwriting input, it can only recognize numbers and fixed several mathematical constants. In addition, only a few very simple numerical calculations such as addition, subtraction, multiplication, division, etc. can be made in terms of calculation, and it is impossible to perform a slightly complicated mathematical calculation such as sign calculation.
Another product MathPad of MyScript is simply a handwriting recognition tool, with no computing power at all. In the K12 education field, simple numerical calculation only occupies a small part of the whole mathematics education system, and more scenes are that symbolic calculation and reasoning are needed.
Disclosure of Invention
In view of the above, the present invention provides a formula recognition method and system and a symbolic reasoning calculation method and system, which can obtain data information of a formula through a handwritten mathematical formula or a file input interface, and obtain a formula recognition result through character segmentation, character recognition and formula analysis, thereby greatly reducing learning cost and improving accuracy of mathematical formula recognition.
In a first aspect, a formula identification method provided in an embodiment of the present invention includes:
acquiring data information of a formula;
carrying out character segmentation on the data information to obtain a single character;
identifying the single character to obtain a character identification result of the single character;
identifying the position relation among the single characters to obtain the position relation among the single characters;
and carrying out formula analysis on the character recognition result by combining the position relation to obtain a formula recognition result.
In a second aspect, the symbolic reasoning calculation method provided in the embodiment of the present invention includes obtaining a formula recognition result by the formula recognition method, and further includes:
analyzing and calculating the formula identification result through a symbol calculation engine to obtain an analysis calculation result of the formula;
rendering the formula identification result and the analysis calculation result;
and outputting the rendered formula identification result and the analysis calculation result.
In a third aspect, an embodiment of the present invention further provides a formula identification system, where the formula identification system includes:
the data information acquisition unit is used for acquiring data information of the formula;
the character segmentation unit is used for carrying out character segmentation on the data information to obtain a single character;
the character recognition unit is used for recognizing the single character to obtain a character recognition result of the single character;
the position relation identification unit is used for identifying the position relation among the single characters to obtain the position relation among the single characters; and
and the formula analysis unit is used for carrying out formula analysis on the character recognition result by combining the position relation to obtain a formula recognition result.
In a fourth aspect, an embodiment of the present invention further provides a symbolic reasoning calculation system, including a formula recognition result obtaining unit, configured to obtain a formula recognition result through the formula recognition system, and further including:
the analysis calculation unit is used for analyzing and calculating the formula identification result through a symbol calculation engine to obtain an analysis calculation result of the formula;
the rendering unit is used for rendering the recognition result and the analysis calculation result of the formula; and
and the output unit is used for outputting the rendered formula identification result and the analysis calculation result.
The technical scheme provided by the invention has the following beneficial effects:
the method comprises the steps that an intelligent terminal such as a mobile terminal obtains data information similar to a mathematical formula through an input interface, character segmentation is carried out on the data information to obtain single characters, character recognition is carried out on the single characters to obtain character recognition results, the position relation among the single characters is recognized to obtain the position relation among the single characters, and formula structure analysis is carried out on the character recognition results according to the position relation to obtain formula recognition results. The formula recognition method can acquire the data information of the formula through the handwritten mathematical formula input interface, and acquire the formula recognition result through character segmentation, character recognition, character position relation recognition and formula structure analysis, thereby greatly reducing the learning cost of the user for carrying out formula input according to the specification and improving the accuracy of the mathematical formula recognition.
Similarly, the formula identification result is obtained by the formula identification method, the formula identification result is analyzed and calculated to obtain the analysis and calculation result of the formula, the formula identification result and the analysis and calculation result are rendered, and the rendered formula identification result and the rendered analysis and calculation result are output, so that the learning cost of formula input by a user according to the specification is greatly reduced, and the accuracy of mathematical formula identification is improved.
Drawings
FIG. 1 is a schematic flow chart of a formula identification method according to an embodiment of the present invention;
fig. 2A is a schematic flowchart of a method for character segmentation of data information according to a second embodiment of the present invention;
FIG. 2B is a diagram illustrating pre-segmentation of characters provided by an embodiment of the present invention;
FIG. 3A is a flowchart illustrating a method for recognizing a single character according to a second embodiment of the present invention;
FIG. 3B is a schematic diagram of a network structure of a long-short memory neural network model used in an embodiment of the present invention;
FIG. 3C is a schematic diagram of a single long-short memory neural network model used in an embodiment of the present invention;
FIG. 3D is a flowchart illustrating another method for recognizing a single character according to a second embodiment of the present invention;
FIG. 3E is a schematic diagram of the structure of a deep convolutional neural network model used in an embodiment of the present invention;
fig. 4A is a flowchart illustrating a method for identifying a position relationship between single characters according to a second embodiment of the present invention;
FIG. 4B is a diagram illustrating extraction of location features between single characters according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for formula resolution according to a second embodiment of the present invention;
fig. 6A is a schematic flow chart of a symbolic reasoning calculation method according to a third embodiment of the present invention;
FIGS. 6B and 6C are exemplary diagrams of software execution results provided by embodiments of the present invention;
FIG. 7 is a schematic diagram of a formula recognition system according to a fourth embodiment of the present invention;
FIG. 8 is a block diagram of a character segmentation unit according to a fifth embodiment of the present invention;
fig. 9A is a schematic diagram of an architecture of a character recognition unit according to a fifth embodiment of the present invention;
FIG. 9B is a schematic diagram of another structure of the character recognition unit according to the fifth embodiment of the present invention;
fig. 10 is a schematic structural diagram of a location relationship identification unit according to a fifth embodiment of the present invention;
fig. 11 is a schematic structural diagram of a formula parsing unit according to the fifth embodiment of the present invention;
fig. 12 is a schematic structural diagram of a symbolic reasoning computing system according to a sixth embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it should be understood that the specific embodiments described herein are only for explaining the present invention and are not limiting the present invention. In addition, for convenience of description, only a part, not all of the contents related to the present invention are shown in the drawings.
Example one
Referring to fig. 1, the scheme of this embodiment may be applied to a scenario where formula identification is required, and may be executed by an intelligent terminal similar to a mobile phone, and specifically may be implemented by a software program that is configured at the intelligent terminal such as a mobile phone or deployed at a cloud and is called by the intelligent terminal, where the formula identification method includes the following steps:
s110: and acquiring data information of the formula.
Illustratively, data information of a mathematical formula input by a user can be acquired through a human-computer interface, and the user can input the data information in a handwriting mode, or can input the data information in an uploading mode of a file with a format of MS Word, Excel, pdf, png, jpg and the like, which is editable or non-editable and in which the formula is recorded, or can input the data information in other common input modes.
S120: and carrying out character segmentation on the data information to obtain a single character.
For example, the data information input by the user is character-divided by a character division technology, and the data information can be divided into a series of single characters. In the prior art, there are many specific character segmentation methods, for example, character segmentation methods based on connected component, projection, cluster analysis, template matching, or using strtok function segmentation. The present embodiment does not limit the specific implementation of character segmentation, but the second embodiment of the present invention provides an alternative implementation.
S130: and identifying the single character to obtain a character identification result of the single character.
For example, the character recognition is performed on the obtained segmented characters by a character recognition technique, and the character recognition result of the single character is obtained. In the prior art, there are many specific character recognition methods, such as structural pattern recognition, statistical pattern recognition, and recognition methods based on artificial neural networks. The embodiment of character recognition is not limited in any way, but the second embodiment of the present invention provides an alternative implementation.
It should be noted that the character segmentation in step S120 is a prerequisite for the character recognition in step S130, and the character recognition in step S130 provides reference information for the character segmentation in step S120.
S140: and identifying the position relationship among the single characters to obtain the position relationship among the single characters.
For example, the position relationship may be: left-right relationship, up-down relationship, upper superscript relationship, lower superscript relationship, and root relationship. The second embodiment of the present invention provides an alternative implementation manner for a specific identification method for identifying a positional relationship.
S150: and carrying out formula analysis on the character recognition result by combining the position relation to obtain a formula recognition result.
Illustratively, according to the position relationship between the character recognition result and the single character, the data information of the formula input by the user is analyzed, the type corresponding to the formula is determined, and the recognition result of the formula is obtained. For example, the second embodiment of the present invention provides an alternative implementation for a specific analysis method of formula analysis.
In summary, in this embodiment, an intelligent terminal such as a mobile terminal obtains data information of a formula input by a user through a human-computer interface, performs character segmentation on the data information to obtain single characters, performs character recognition on the single characters to obtain character recognition results thereof, performs position relationship recognition on the single characters to obtain position relationship recognition results of the single characters, and obtains recognition results of the formula after analyzing the character recognition results. According to the formula identification method, data information of a formula is acquired through a handwritten mathematical formula input interface, and the identification result of the formula is acquired through character segmentation, character identification, character position relation identification and formula analysis, so that the learning cost of a user for carrying out formula input according to the specification is greatly reduced, and the accuracy of mathematical formula identification is improved.
Example two
On the basis of the first embodiment of the present invention, this embodiment further provides optional implementation manners of step S120, step S130, step S140, and step S150 in the technical solution of the first embodiment.
Referring to the first embodiment of the present invention, as shown in fig. 2A, step S120, namely, performing character segmentation on the data information to obtain a single character, may include:
s121: and performing character pre-segmentation on the data information to obtain an initial stroke.
Illustratively, the data information of the formula entered by the user is divided into a series of initial strokes by a character pre-segmentation. For example, in FIG. 2B, pre-dividing the data information of "-1 < X" formula in the figure, such as "1", "<", "in X \ and" in X/"can correspond to obtaining initial strokes 1, 2, 3 and 4, respectively.
S122: and combining the initial strokes to obtain a combined stroke.
Illustratively, in the above-mentioned series of initial strokes, strokes that are likely to be combined together are found, and the strokes that can be combined together are combined to obtain a combined stroke.
It should be noted that, due to different pre-segmentation methods of the character, there may exist a plurality of different combinations of the initial strokes, which results in the same set of initial strokes and may also exist a plurality of combinations of the initial strokes. For example, in FIG. 2B, there are two combination schemes for initial strokes 1 and 2 as follows: the first is to divide the initial stroke 1 and the initial stroke 2, and not to combine them, the initial stroke combination result is "1 <; another combination is to combine initial stroke 1 and initial stroke 2 together, resulting in a combined stroke "K". Similarly, in FIG. 2B, there is also the possibility that the initial strokes 3 and 4 are combined into the English letter "X" or the mathematical operation symbol "X", respectively.
S123: and respectively identifying initial strokes and/or combined strokes by using a single character identification model obtained by training in advance, and taking the high probability as a segmentation result to obtain the single character.
Since the combination between the initial strokes may also exist in different combinations, the same initial set of strokes may also exist in different combinations. To solve this problem, for example, a single-character recognition model obtained by training in advance may be used to respectively recognize the character formed by the initial stroke and the character formed by the combined stroke, and a character formed by a group of strokes with a high probability may be taken as a character segmentation result obtained after character segmentation. The single character recognition model in the embodiment is a model obtained through big data learning, training and analysis in advance, and compared with the model in the prior art, the single character recognition model after training has better robustness of an output result.
For example, it can be said that the recognition probability of each scheme in the character segmentation process is the recognition result probability corresponding to the scheme, and the recognition probability of each scheme in the combination process is the recognition result corresponding to the scheme, and the final result is the result
For another example, there are many possible ways of dividing and combining strokes, and the single character recognition model recognizes all possible ways to obtain recognition results of various possible ways, and the recognition results and the division results work together to determine the final division results and recognition results of the single character. It can be seen that this step results in both the single character segmentation and recognition results. For example, in strokes 1 and 2 in FIG. 2B, the determination of the set of strokes may be as follows: the strokes are divided and combined, and the following two schemes are adopted: the first scheme is that strokes 1 and 2 are two separate characters; the second scheme is that stroke 1 and stroke 2 are combined into one character. As an example, the decision process for the two schemes may be as follows: for the first scheme, a single-character recognition model is used to respectively recognize two single characters of stroke 1 and stroke 2, two recognition results, namely A1 ("1") and A2 ("<"), are obtained, and a corresponding probability P1 is calculated. Similarly, for the second scheme, the character combined by stroke 1 and stroke 2 is recognized by using a single-character recognition model to obtain a recognition result B1 ("K"), and a corresponding probability P2 is calculated. Comparing the sizes of P1 and P2, if P1> P2, the first scheme corresponds to that '1' and '<' are two separated characters, namely a segmentation result and a recognition result; on the contrary, if P1 < P2, the combination of "1" and "<" corresponding to the second scheme into a character "K" is the segmentation result and the recognition result.
In summary, the present disclosure further provides an optional implementation manner of step S120 on the basis of the first embodiment, and a series of initial strokes in the data information can be obtained through the character pre-segmentation of step S121. Through the stroke combination of step S122, a combined stroke after the original stroke combination can be obtained. In step S123, the original strokes and the combined strokes are recognized respectively by using the single character recognition model obtained through training in advance, and the result with high probability is selected as the character segmentation result.
Referring to the first embodiment of the present invention, as shown in fig. 3A, 3B, 3C, 3D and 3E, step S130, namely, identifying a single character, and obtaining a character identification result of the single character may include the following two cases in parallel:
first, when the data information is input by online handwriting, as shown in fig. 3A, step S130 may include:
s131 and 131 a: and extracting the characteristics of the single character.
For example, in the present embodiment, the following features may be extracted from the single character: coordinate point information (x, y), coordinate point first derivative information (x ', y'), coordinate point second derivative information (x ', y') and coordinate point curvature information k, although other more suitable features such as coordinate point third derivative information (x ', y') may be extracted in other embodiments.
S132 a: inputting the characteristics into a long and short memory neural network model obtained by training in advance to obtain a character recognition result of a single character.
Long Short Term Memory (LSTM) is a special recurrent neural network that can learn Long-Term dependencies, remembering that information is the inherent behavior of Long Short Term memory neural networks for a Long period of time, rather than the result of an effort to learn.
For example, as shown in fig. 3B and 3C, the extracted features are input into a long-short memory neural network model obtained through training, so as to obtain character recognition results corresponding to all single characters, wherein each character recognition result is obtained from the result of the output layer. The long and short memory neural network model in this embodiment is a neural network model obtained through big data learning and training in advance, and the output of the neural network model has better robustness compared with the neural network model in the prior art.
When the data information is input through a file, as shown in fig. 3D, step S130 may include:
s131 and 131 d: and preprocessing the single character to obtain a preprocessed character.
For example, preprocessing such as image scaling, image deblurring, and image gray scale stretching may be performed on data information input through a file to obtain preprocessed characters.
S132 d: inputting the preprocessed characters into a deep convolutional neural network model obtained by training in advance to obtain a character recognition result of the single character.
Illustratively, the obtained preprocessed character is input into a deep convolutional neural network model shown in fig. 3E obtained by training in advance, and the character recognition result of the single character can be obtained after recognition.
The deep convolutional neural network model processes an input picture through a convolutional block for multiple times, performs full connection processing twice, and finally outputs the processed picture through a softmax function, wherein the process of convolution processing each time comprises the following steps: convolution (Convolution), Batch Normalization (Batch Normalization), linear correction (ReLU), and Pooling (Pooling). The deep convolutional neural network model in this embodiment is a neural network model obtained through big data learning and training in advance, and its output has a higher accuracy rate compared with the neural network model in the prior art.
In summary, the present disclosure provides an optional implementation manner of step S130 based on the first embodiment, when the data information is input by online handwriting, the single character feature can be extracted in step S131a, and the character recognition result of the single character can be obtained through a long-short memory (LSTM) neural network model in step S132 a; when the data information is inputted through the file, the preprocessed character may be obtained by preprocessing the single character in step S131d, and the character recognition result of the single character may be obtained by inputting the preprocessed character to the deep convolutional neural network model in step S132 d.
Referring to the first embodiment of the present invention, as shown in fig. 4A, the step S140 of identifying the position relationship between the single characters to obtain the position relationship between the single characters may include:
s141: the single characters are grouped according to a mutual visibility principle.
For example, the mutually visible definitions may be: a straight line connecting two single characters exists, and the line does not pass through the third character, so that the two single characters are mutually visible. According to the mutual visibility principle, two single characters which accord with the mutual visibility principle are divided into a group.
S142: for two single characters classified into the same group, the position feature between the two single characters is extracted.
Illustratively, for two single characters grouped in the same group, 9 features can be extracted as shown in FIG. 4B, where B and C represent two "mutually visible" characters, dx1Denotes the distance, dx, between the left end of B and the left end of C2Denotes the distance between the right end of B and the right end of C, dx denotes the distance between the right end of B and the left end of C, dy1Denotes the distance, dy, between the upper end of B and the upper end of C2Denotes a distance between a lower end of B and a lower end of C, dy denotes a distance between a lower end of B and an upper end of C, D denotes a distance between an intermediate position of a right end of B and an intermediate position of a right end of C, H denotes a distance between an upper end and a lower end of a region occupied by B and C, dhc denotes a distance between an intermediate position of a lower end of B and an intermediate position of a lower end of C, and H (H, C) ([ H, D, dhc, dx)1,dx2,dy,dy1,dy2]Representing the extracted 9 feature sets.
S143: and inputting the position characteristics into a character position relation classifier obtained by training in advance to obtain the position relation between the two single characters.
Illustratively, the extracted position relationship between the single characters is input into a character position relationship classifier obtained by training in advance, and the position relationship between the two single characters is obtained after analysis. The character position relation classifier in this embodiment is obtained by big data learning and training in advance, and the output of the classifier has better robustness compared with the classifier in the related art.
In summary, the present disclosure further provides an optional implementation manner of step S140 in the technical solution of the first embodiment on the basis of the first embodiment. In step S141, the single characters are classified into the same group visible to each other, in step S1412, the position characteristics between two single characters in the same group are extracted, and in step S1413, the position characteristics are analyzed by the character position classifier obtained through training in advance, so that the position relationship between two single characters can be obtained
Referring to the first embodiment of the present invention, as shown in fig. 5, step S150, performing formula analysis on the character recognition result according to the position relationship, and obtaining a formula recognition result may include:
s151: the stroke order, the maximum stroke number and the position relation of the single character are input into a statistical context-free grammar model obtained through training in advance.
Illustratively, the stroke order O of a single character and the maximum number of strokes L that may be present in all single charactersmaxInputting a statistical context-free grammar model obtained by training in advance, wherein L is in the embodimentmaxThe value is 2 and may be other positive integer values in other embodiments. The statistical context-independent grammar model in the embodiment is a grammar model obtained by big data learning and training in advance, and compared with the grammar model in the related technology, the output of the grammar model has better robustness.
S152: and selecting the analytic tree with the maximum output probability result to obtain a formula identification result.
Illustratively, after the statistical context-free grammar model is analyzed and identified, the parse tree with the largest probability result in the output options is selected as the final identification result of the formula, and the specific process of output can be as follows:
where N represents the number of strokes of the character, γ (B, B)B,lB) Syntax, γ (C, B), indicating that character B satisfiesC,lC) A syntax representing the satisfaction of the character C,
the characters B and C can be combined into a under the condition of grammar rule r, and the grammar of the combined result conforms to the grammar system G of the system.
In summary, the present embodiment further provides an optional implementation manner of step S150 on the basis of the first embodiment. And (S151) inputting the stroke sequence, the maximum stroke number and the position relation of the character into a grammar model independent of the statistical context, and S1422 selecting the analytic tree with the maximum probability in the output to obtain the final formula recognition result.
The following is an embodiment of a symbolic reasoning computation method provided by the implementation of the present invention. In this embodiment, the formula identification method is used to obtain the identification result of the formula, analyze and calculate the identification result to obtain the analysis and calculation result of the formula, and finally render and output the identification result and the calculation result of the formula. Therefore, reference may be made to the above embodiments of the formula identification method for details that are not explicitly described in this embodiment.
EXAMPLE III
On the basis of the foregoing embodiment, referring to fig. 6A and 6B, the scheme of this embodiment may be applied to a scenario that requires formula calculation, and may be executed by a smart terminal similar to a mobile phone, and may specifically be implemented by a software program configured in the smart terminal such as a mobile phone or in a cloud, where the symbolic reasoning calculation method includes the following steps:
s310: and obtaining a formula identification result through a formula identification method.
Exemplary, Intelligent terminals, etcThe recognition result of the formula can be obtained by any formula recognition method in the foregoing embodiments one to three. As shown in FIG. 6B, the formula data input by the user is shown in the lower right corner of the figure, and the recognition result is
S320: and analyzing and calculating the formula identification result through a symbol calculation engine to obtain an analysis calculation result of the formula.
The symbol calculation engine may be a commercial symbol calculation engine such as Mathematica, a commercial symbol calculation engine that is developed or purchased twice on the basis of a commercial symbol calculation engine such as Mathematica, or an open-source symbol calculation engine such as Sympy (mathematical symbol calculation library of python). It should be further noted that, in this embodiment, the symbol calculation engine is not limited in any way.
Illustratively, the formula recognition result obtained above may be analyzed and calculated by the symbolic calculation engine, so as to obtain a final analysis calculation result of the formula. As shown in fig. 6B, the formula is identified according to the formula, and the calculation result of the formula is obtained by analyzing and calculating the formula is x-y.
S330: rendering formula recognition results and analyzing calculation results.
Illustratively, after returning the formula recognition result and the analysis calculation result, it is rendered into a form of a common mathematical formula that is convenient for a user to read.
S340: and outputting a rendered formula identification result and an analysis calculation result.
Illustratively, the rendered formula recognition result and the mathematical formula form of the analysis calculation result are output and displayed on a display screen of an intelligent terminal such as a mobile phone. As shown in fig. 6B, the rendering result is output on a software interface displayed on a display screen, and the rendered formula recognition result and the analysis calculation result are respectively as shown in the upper left corner of the figure
And x-y.
This embodiment further takes a typical solution equation set in mathematical operation as an example, as shown in fig. 6C, and illustrates an implementation interface of the technical solution of the present invention under the condition of a software program.
When the user inputs the mathematical equation set in the lower right corner of FIG. 6C in a handwriting manner
Including the common alphanumerical characters x, y and the parenthesis, subtraction and root operators. As can be seen from the figure: an intelligent terminal such as a mobile phone collects input data of a user and uploads the input data to a cloud server; the cloud server receives the data and identifies the data; after the recognition result is obtained, calling a symbolic calculation engine to perform analysis calculation on the recognized mathematical formula to obtain an analysis calculation result; then the cloud server sends the recognition result and the analysis and calculation result to the intelligent terminal; and rendering and displaying after the intelligent terminal receives the identification result and the analysis and calculation result. I.e. a system of mathematical equations
Rendering is performed, and the rendered input result is displayed in the upper left corner of fig. 6C. Rendering the received calculation result, and displaying the rendered calculation result in the middle of the software interface, namely in the middle of fig. 6C, so as to facilitate the user to check.
In summary, in this embodiment, a formula recognition result is obtained by any one of the formula recognition methods in the first to third embodiments, the formula recognition result is analyzed and calculated by the symbolic calculation engine to obtain an analysis calculation result of the formula, and the formula recognition result and the analysis calculation result are rendered and output. The symbolic reasoning calculation method greatly reduces the learning cost of the formula input by the user according to the specification and improves the accuracy of the mathematical formula identification.
The following is an embodiment of the formula identification system provided in the embodiment of the present invention, which belongs to the same concept as the embodiment of the formula identification method described above, and reference may be made to the embodiment of the formula identification method described above for details that are not described in detail in the embodiment of the formula identification system.
Example four
Referring to fig. 7, the formula identification system 400 of the present embodiment corresponds to the method of the first embodiment, and the system 400 includes: a data information acquisition unit 410, a character segmentation unit 420, a character recognition unit 430, a positional relationship recognition unit 440, and a formula analysis unit 450. Wherein:
a data information obtaining unit 410, configured to obtain data information of the formula.
The character segmentation unit 420 is configured to perform character segmentation on the data information to obtain a single character.
The character recognition unit 430 is configured to recognize a single character and obtain a character recognition result of the single character.
The position relationship identifying unit 440 is configured to identify a position relationship between the single characters to obtain a position relationship between the single characters. And
and the formula analyzing unit 450 is configured to perform formula analysis on the character recognition result according to the position relationship to obtain a formula recognition result.
In summary, according to the symbolic reasoning calculation system of the embodiment, the learning cost of the user for formula input according to the specification is greatly reduced, and the accuracy of mathematical formula identification is improved.
EXAMPLE five
On the basis of the fourth embodiment of the present invention, this embodiment further provides alternative implementations of the character segmentation unit 420, the character recognition unit 430, the position relationship recognition unit 440, and the formula analysis unit 450.
Referring to the fourth embodiment, as shown in fig. 8, the character segmentation unit 420 may include:
and a character pre-segmentation subunit 421, configured to perform character pre-segmentation on the data information to obtain an initial stroke.
And a stroke combination subunit 422, configured to combine the initial strokes to obtain a combined stroke. And
a single character obtaining subunit 423, configured to separately recognize the initial stroke and/or the combined stroke by using a single character recognition model obtained through training in advance, and obtain a single character by using the segmentation result with a high probability.
Referring to the fourth embodiment, as shown in fig. 9A and 9B, the character recognition unit 430 may include two cases in parallel as follows:
when the data information is input by online handwriting, as shown in fig. 9A, the character recognition unit 430 may include:
and a feature extraction subunit 431a, configured to extract features of the single character. And
the feature input subunit 432a is configured to input features into a long and short memory neural network model obtained through training in advance, so as to obtain a character recognition result of a single character.
When the data information is inputted through a file, as shown in fig. 9B, the character recognition unit 430 may include:
and a single character preprocessing subunit 431b, configured to perform preprocessing on the single character to obtain a preprocessed character. And
and the preprocessed character input subunit 432b is configured to input the preprocessed characters into a deep convolutional neural network model obtained through training in advance, so as to obtain a character recognition result of a single character.
Referring to the fourth embodiment, as shown in fig. 10, the position relation obtaining subunit 440 may include:
a single-character grouping subunit 441, configured to group the single characters according to a mutual visibility principle.
The position feature extracting subunit 442 is configured to, for two single characters classified into the same group, extract a position feature between the two single characters. And
a position relation obtaining subunit 443, configured to input the position feature into a character position relation classifier obtained through training in advance, and obtain a position relation between the two single characters.
Referring to the fourth embodiment, as shown in fig. 11, the formula parsing unit 450 may include:
and an input subunit 451, configured to input the stroke order of the single character, the maximum stroke number, and the position relationship to a statistical context-free grammar model obtained through training in advance. And
and the recognition result obtaining subunit 452 is configured to select the parse tree with the largest output probability result, and obtain the recognition result of the formula.
In summary, according to the technical solution of this embodiment, the data information may be pre-divided by the character pre-dividing subunit 421 to obtain an initial stroke, the initial strokes may be combined by the stroke combining subunit 422 to obtain a combined stroke, the single-character obtaining subunit 423 may be used to respectively identify the initial stroke and/or the combined stroke by using a single-character identification model obtained through training in advance, and the initial stroke and/or the combined stroke is analyzed to obtain a single-character result by taking the high probability as a division and identification result. When the data information is input by online handwriting, the characteristics of the single character can be extracted through the characteristic extraction subunit 431a, and the character recognition result of the single character can be obtained through the characteristic input subunit 432a by analyzing a long and short memory neural network model obtained through training in advance; when the data information is inputted through a file, the single character can be preprocessed through the single character preprocessing subunit 431b to obtain a preprocessed character, and the character recognition result of the single character can be obtained through a deep convolutional neural network model obtained through training through the preprocessed character input subunit 432 b. The single-character grouping subunit 441 can group the single characters, the position feature extraction subunit 442 can divide the two single characters into the same group, extract the position features, the position relationship obtaining subunit 443 can obtain the position relationship between the two single characters by inputting the position features into the character position relationship classifier obtained by training in advance. The stroke order, the maximum stroke number and the position relation of the single character are input to a statistical context-free grammar model obtained through training in advance through the input subunit 451, the subunit 452 is obtained through the recognition result, the parse tree with the maximum output probability result is selected, and the recognition result of the formula can be obtained.
The following is an embodiment of the symbolic reasoning computing system provided in the embodiments of the present invention, which belongs to the same concept as the above embodiments of the symbolic reasoning computing method, and reference may be made to the above embodiments of the symbolic reasoning computing method for details that are not described in detail in the embodiments of the symbolic reasoning computing system.
EXAMPLE six
Referring to fig. 12, the symbolic inference calculation system 600 of the present embodiment corresponds to the method of the third embodiment, and includes a formula recognition result obtaining unit 610, an analysis calculation unit 620, a rendering unit 630, and an output unit 640. Wherein:
a formula recognition result obtaining unit 610, configured to obtain a formula recognition result through the formula recognition system.
The analysis unit 620 is configured to perform analysis and calculation on the formula identification result through the symbol calculation engine to obtain an analysis and calculation result of the formula.
And a rendering unit 630, configured to render the formula recognition result and the analysis calculation result. And
an output unit 840, configured to output the rendered formula identification result and the analysis calculation result.
In conclusion, according to the symbolic reasoning calculation system in the technical scheme of the embodiment, the learning cost of formula input by a user according to the specification is greatly reduced, and the accuracy of mathematical formula identification is improved.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes in the detailed description of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A formula identification method, comprising:
acquiring data information of a formula;
carrying out character segmentation on the data information to obtain a single character;
identifying the single character to obtain a character identification result of the single character;
identifying the position relation among the single characters to obtain the position relation among the single characters;
carrying out formula analysis on the character recognition result by combining the position relation to obtain a formula recognition result;
when the data information is input by online handwriting, the identifying the single character to obtain the character identifying result of the single character comprises the following steps:
extracting the characteristics of the single character; the features at least include: coordinate point information, first derivative information of the coordinate points, second derivative information of the coordinate points and curvature information of the coordinate points;
inputting the characteristics into a long and short memory neural network model obtained by training in advance to obtain a character recognition result of the single character; or
When the data information is input through a file, the single character is identified to obtain the character identification result of the single character, and the method comprises the following steps:
preprocessing the single character to obtain a preprocessed character;
inputting the preprocessed characters into a deep convolutional neural network model obtained by training in advance to obtain a character recognition result of the single character;
wherein, the identifying the position relationship among the single characters to obtain the position relationship among the single characters comprises:
grouping the single characters according to a mutual visibility principle; the mutual visibility principle includes: a straight line connecting line exists between the two single characters, and the straight line connecting line does not pass through a third character;
for two single characters which are divided into the same group, extracting the position characteristics between the two single characters;
and inputting the position characteristics into a character position relation classifier obtained by training in advance to obtain the position relation between the two single characters.
2. The method of claim 1, wherein said character segmenting said data information to obtain single characters comprises:
performing character pre-segmentation on the data information to obtain an initial stroke;
combining the initial strokes to obtain combined strokes;
and respectively identifying the initial strokes and/or the combined strokes by using a single character identification model obtained by training in advance, and taking the high probability as a segmentation result to obtain the single character.
3. The method of claim 1, wherein the performing formula analysis on the character recognition result in combination with the position relationship to obtain a formula recognition result comprises:
inputting the stroke sequence, the maximum stroke number and the position relation of the single character into a statistical context-free grammar model obtained in advance through training;
and selecting the analytic tree with the maximum output probability result to obtain a formula identification result.
4. A symbolic reasoning calculation method comprising obtaining a formula recognition result by the formula recognition method according to any one of claims 1 to 3, further comprising:
analyzing and calculating the formula identification result through a symbol calculation engine to obtain an analysis calculation result of the formula;
rendering the formula identification result and the analysis calculation result;
and outputting the rendered formula identification result and the analysis calculation result.
5. A formula identification system comprising:
the data information acquisition unit is used for acquiring data information of the formula;
the character segmentation unit is used for carrying out character segmentation on the data information to obtain a single character;
the character recognition unit is used for recognizing the single character to obtain a character recognition result of the single character;
the position relation identification unit is used for identifying the position relation among the single characters to obtain the position relation among the single characters; and
the formula analysis unit is used for carrying out formula analysis on the character recognition result by combining the position relation to obtain a formula recognition result;
wherein, when the data information is input by online handwriting, the character recognition unit includes:
the characteristic extraction subunit is used for extracting the characteristics of the single character; the features at least include: coordinate point information, first derivative information of the coordinate points, second derivative information of the coordinate points and curvature information of the coordinate points; and
the characteristic input subunit is used for inputting the characteristics into a long and short memory neural network model obtained by training in advance to obtain a character recognition result of the single character; or
When the data information is input through a file, the character recognition unit includes:
the single character preprocessing subunit is used for preprocessing the single character to obtain a preprocessed character; and
the preprocessing character input subunit is used for inputting the preprocessing characters into a deep convolutional neural network model obtained by training in advance to obtain a character recognition result of the single character;
wherein the positional relationship identifying unit includes:
a single character grouping subunit, configured to group the single characters according to a mutual visibility principle; the mutual visibility principle includes: a straight line connecting line exists between the two single characters, and the straight line connecting line does not pass through a third character;
the position feature extraction subunit is used for extracting the position features between the two single characters which are divided into the same group; and
and the position relation obtaining subunit is used for inputting the position characteristics into a character position relation classifier obtained in advance through training to obtain the position relation between the two single characters.
6. The system of claim 5, wherein the character segmentation unit comprises:
the character pre-segmentation subunit is used for performing character pre-segmentation on the data information to obtain an initial stroke;
the stroke combination subunit is used for combining the initial strokes to obtain combined strokes; and
and the single character obtaining subunit is used for respectively identifying the initial strokes and/or the combined strokes by using a single character identification model obtained by training in advance, and taking the high probability as a segmentation result to obtain the single character.
7. The system of claim 5, wherein the formula parsing unit comprises:
the input subunit is used for inputting the stroke sequence, the maximum stroke number and the position relation of the single character into a statistical context-free grammar model obtained in advance through training; and
and the formula identification result obtaining subunit is used for selecting the analytic tree with the maximum output probability result to obtain a formula identification result.
8. A symbolic reasoning calculation system including a formula recognition result obtaining unit for obtaining a formula recognition result by the formula recognition system according to any one of claims 5 to 7, further comprising:
the analysis calculation unit is used for analyzing and calculating the formula identification result through a symbol calculation engine to obtain an analysis calculation result of the formula;
the rendering unit is used for rendering the formula identification result and the analysis calculation result; and
and the output unit is used for outputting the rendered formula identification result and the analysis calculation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610028002.0A CN106980856B (en) | 2016-01-15 | 2016-01-15 | Formula identification method and system and symbolic reasoning calculation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610028002.0A CN106980856B (en) | 2016-01-15 | 2016-01-15 | Formula identification method and system and symbolic reasoning calculation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106980856A CN106980856A (en) | 2017-07-25 |
| CN106980856B true CN106980856B (en) | 2020-11-27 |
Family
ID=59339912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610028002.0A Active CN106980856B (en) | 2016-01-15 | 2016-01-15 | Formula identification method and system and symbolic reasoning calculation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106980856B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729865A (en) * | 2017-10-31 | 2018-02-23 | 中国科学技术大学 | A kind of handwritten form mathematical formulae identified off-line method and system |
| CN109977737A (en) * | 2017-12-28 | 2019-07-05 | 新岸线(北京)科技集团有限公司 | A kind of character recognition Robust Method based on Recognition with Recurrent Neural Network |
| CN108334839B (en) * | 2018-01-31 | 2021-09-14 | 青岛清原精准农业科技有限公司 | Chemical information identification method based on deep learning image identification technology |
| CN108805224B (en) * | 2018-05-28 | 2021-10-01 | 中国人民解放军国防科技大学 | Multi-symbol hand-drawn sketch recognition method and device with sustainable learning ability |
| CN109063561B (en) * | 2018-06-28 | 2022-03-08 | 广州视源电子科技股份有限公司 | Formula identification calculation method and device |
| CN109102079B (en) * | 2018-08-16 | 2022-01-11 | 深圳市德瑞信息技术有限公司 | Intrusion detection algorithm based on value derivative GRU |
| CN110858307B (en) * | 2018-08-24 | 2022-09-13 | 国信优易数据股份有限公司 | Character recognition model training method and device and character recognition method and device |
| CN109614944B (en) * | 2018-12-17 | 2021-05-04 | 科大讯飞股份有限公司 | Mathematical formula identification method, device, equipment and readable storage medium |
| CN109858627B (en) * | 2018-12-24 | 2021-08-20 | 上海仁静信息技术有限公司 | Inference model training method and device, electronic equipment and storage medium |
| CN111400491A (en) * | 2018-12-27 | 2020-07-10 | 北大方正集团有限公司 | Formula body positioning method, device and equipment and computer readable storage medium |
| CN110263631B (en) * | 2019-05-10 | 2022-03-15 | 南京大学 | Handwritten chemical formula recognition and balancing method |
| CN110738027B (en) * | 2019-10-15 | 2023-07-14 | 深圳逻辑汇科技有限公司 | Computer-implemented method, apparatus, and storage medium for generating spreadsheet formulas |
| CN111340020B (en) * | 2019-12-12 | 2024-04-19 | 科大讯飞股份有限公司 | Formula identification method, device, equipment and storage medium |
| CN112712075B (en) * | 2020-12-30 | 2023-12-01 | 科大讯飞股份有限公司 | Arithmetic detection method, electronic equipment and storage device |
| CN113610073A (en) * | 2021-06-29 | 2021-11-05 | 北京搜狗科技发展有限公司 | Method and device for identifying formula in picture and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101149790A (en) * | 2007-11-14 | 2008-03-26 | 哈尔滨工程大学 | Chinese printing style formula identification method |
| CN101388068A (en) * | 2007-09-12 | 2009-03-18 | 汉王科技股份有限公司 | Mathematical formula identifying and coding method |
| CN103425974A (en) * | 2012-05-15 | 2013-12-04 | 富士施乐株式会社 | Appratus and method for processing images |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013030850A2 (en) * | 2011-08-26 | 2013-03-07 | Council Of Scientific & Industrial Research | Chemical structure recognition tool |
-
2016
- 2016-01-15 CN CN201610028002.0A patent/CN106980856B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101388068A (en) * | 2007-09-12 | 2009-03-18 | 汉王科技股份有限公司 | Mathematical formula identifying and coding method |
| CN101149790A (en) * | 2007-11-14 | 2008-03-26 | 哈尔滨工程大学 | Chinese printing style formula identification method |
| CN103425974A (en) * | 2012-05-15 | 2013-12-04 | 富士施乐株式会社 | Appratus and method for processing images |
Non-Patent Citations (1)
| Title |
|---|
| 二维文法的研究及其在联机手写数学公式识别的应用;吴炳玮;《中国优秀硕士学位论文全文数据库 信息科技辑》;20150115;第6,29,42-44,49,58页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106980856A (en) | 2017-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106980856B (en) | Formula identification method and system and symbolic reasoning calculation method and system | |
| US10817741B2 (en) | Word segmentation system, method and device | |
| CN110032998B (en) | Method, system, device and storage medium for detecting characters of natural scene picture | |
| CN111160352A (en) | Workpiece metal surface character recognition method and system based on image segmentation | |
| CN109685065B (en) | Layout analysis method and system for automatically classifying test paper contents | |
| CN115063875B (en) | Model training method, image processing method and device and electronic equipment | |
| CN110942004A (en) | Handwriting recognition method and device based on neural network model and electronic equipment | |
| CN113657098B (en) | Text error correction method, device, equipment and storage medium | |
| CN113239818B (en) | Table cross-modal information extraction method based on segmentation and graph convolution neural network | |
| CN116543404A (en) | Table semantic information extraction method, system, equipment and medium based on cell coordinate optimization | |
| CN114092938B (en) | Image recognition processing method and device, electronic equipment and storage medium | |
| CN105117740A (en) | Font identification method and device | |
| CN112115921A (en) | True and false identification method and device and electronic equipment | |
| CN111414913B (en) | Character recognition method, recognition device and electronic equipment | |
| CN114863429A (en) | Text error correction method and training method based on RPA and AI and related equipment thereof | |
| CN113223011B (en) | Small sample image segmentation method based on guide network and full-connection conditional random field | |
| CN110147785B (en) | Image recognition method, related device and equipment | |
| CN113673528A (en) | Text processing method and device, electronic equipment and readable storage medium | |
| CN110263631B (en) | Handwritten chemical formula recognition and balancing method | |
| CN112200216A (en) | Chinese character recognition method, device, computer equipment and storage medium | |
| CN113837157B (en) | Topic type identification method, system and storage medium | |
| CN110555462A (en) | non-fixed multi-character verification code identification method based on convolutional neural network | |
| CN113723367B (en) | Answer determining method, question judging method and device and electronic equipment | |
| CN115984886A (en) | Table information extraction method, device, equipment and storage medium | |
| CN115578739A (en) | Training method and device for realizing IA classification model by combining RPA and AI |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20180716 Address after: Room 3835, Building 3, 2528 Zhennan Road, Putuo District, Shanghai, 2003 Applicant after: Shanghai Ren Jing Information Technology Co., Ltd. Address before: Room 2002, Building 5, 200 Guoding East Road, Yangpu District, Shanghai, 2003 Applicant before: Shanghai Qian Wan answer Cloud Computing Technology Co., Ltd. |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20181011 Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: BEIJING ZIJIE TIAODONG NETWORK TECHNOLOGY CO., LTD. Address before: Room 3835, Building 3, 2528 Zhennan Road, Putuo District, Shanghai, 2003 Applicant before: Shanghai Ren Jing Information Technology Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |