JP2007072718A - Handwritten mathematical expression recognizing device and recognizing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize a handwritten mathematical expression by adopting a stochastic approach. <P>SOLUTION: This handwritten mathematical expression recognizing device comprises: an input means for reading the mathematical expression; a means for calculating the stroke likelihood of a stroke candidate constituting the mathematical expression; a means for expressing the inputted mathematical expression by the use of a plurality of mathematical expression candidates constituted of the plurality of stroke candidates; a means for calculating a structure likelihood concerning position relation between mathematical expression components which constitute the inputted handwritten mathematical expression by the use of stochastic context-free grammar; a means for calculating the likelihood of each mathematical expression candidate by the use of the calculated stroke likelihood and the structure likelihood; and a means for determining the mathematical expression based on the likelihood of each calculated mathematical expression candidate. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

The present invention relates to a handwritten mathematical expression recognition apparatus and method.

Currently, mathematical expressions are input to computers mainly by text descriptions in dedicated languages such as TeX and matlab, and mathematical expression editors operated with a mouse. However, these input methods require training and are not intuitive. If an intuitive mathematical expression can be input using a pen tablet or the like, there is a wide range of applications in educational and research fields, such as mathematical expression input to papers, mathematical expression search, and numerical software input interfaces.

There are many existing studies on online handwritten mathematical expression recognition (Non-Patent Document 1). In these studies, mathematical expression recognition is performed in two stages: symbol recognition and structure recognition. First, symbol recognition is performed by dividing the input stroke series into symbols. Then, the recognized symbols are combined according to a rule using their positional relationship to form a mathematical formula.

However, mathematical formulas are always written according to a strict grammar, but handwritten mathematical formulas are not always interpreted in a single way because of their fluctuations, as shown in equation (1). Number 1 is an example showing the ambiguity formula recognition, or 2x of the or 2 ^x thing is not always obvious even for humans.

In the existing method using the rule base, the stroke recognition and the structure recognition are treated as completely separated problems. However, the structure recognition and the stroke recognition cannot be separated as shown in Equation 2. Equation 2 is an example in which a change in structure affects stroke recognition. The center part may be parentheses “) (” in the left expression, but it is natural to interpret it as “x” in the right expression. is there.
KFChan, DYYeung: “Mathematical Expression Recognition: A Survey” 'Int. J.DocumentAnal., Vol.3, no.1, pp.3.15,2000.

The present invention aims to recognize handwritten mathematical expressions by adopting a probabilistic approach to the conventional rule-based approach. Another object of the present invention is to recognize handwritten mathematical expressions based on the position that stroke recognition and structure recognition are closely related.

The present invention considers that, in handwritten mathematical expression recognition, probabilistic recognition of finding the maximum likelihood among various mathematical expression candidates is essential, and the product of stroke likelihood and structure likelihood is maximized. It is characterized in that it is formulated as a problem for obtaining mathematical expression candidates to be converted. In obtaining a mathematical expression candidate that maximizes the product of the stroke likelihood and the structure likelihood, the stroke likelihood can be calculated by a model of an existing method, and the structure likelihood can be calculated by a probability context free grammar. Further, in the calculation of the structure likelihood according to the present invention, the application probability of the formula generation rule is determined by the positional relationship between the formula parts. In one preferable aspect, a likelihood distribution of an area (hidden writing area) in which a mathematical expression part is to be written is used as a position feature amount for evaluating the positional relationship between the mathematical expression parts. Alternatively, the structure likelihood may be calculated using the circumscribed rectangle of each observed stroke as the position feature amount. The likelihood of a mathematical formula candidate can be efficiently calculated by CYK-algorithm. The handwriting recognition according to the present invention is configured as a handwritten mathematical expression recognition device, a handwritten mathematical expression recognition method, a computer program for causing a computer to execute a handwritten mathematical expression, or a computer-readable medium recording the program. The specific configuration of the present invention will be described below.

The handwritten mathematical expression recognition apparatus according to the present invention includes an input means for reading a handwritten mathematical expression, a means for calculating stroke likelihood of stroke candidates constituting the mathematical expression, and a plurality of input mathematical expressions that are composed of a plurality of stroke candidates. Means for expressing with mathematical expression candidates, means for calculating the structural likelihood related to the positional relationship between mathematical parts constituting the input handwritten mathematical expression using the probability context free grammar, and the calculated stroke likelihood and structural likelihood And means for calculating the likelihood of each mathematical expression candidate and means for determining the mathematical expression based on the calculated likelihood of each mathematical expression candidate. In one aspect, the means for determining the mathematical expression is a mathematical expression candidate having the maximum likelihood. In another aspect, the means for determining a mathematical expression includes a means for extracting a plurality of mathematical expression candidates having a high likelihood, and a means for selecting one mathematical expression candidate from the plurality of extracted mathematical expression candidates.

In one preferred aspect, the structural likelihood calculating means is configured to write the stroke based on the stroke generation rule candidate for each observed circumscribed rectangle of the stroke and the generation rule candidate for the stroke. The structure likelihood is calculated by means of a likelihood context free grammar in which the probability of application is determined by the positional relationship between mathematical parts including the stroke, using the calculated likelihood distribution as the position feature quantity of each stroke. Means for calculating the degree. In one preferred aspect, the circumscribed rectangle is evaluated only for the stroke among the mathematical parts, so the hidden writing area likelihood distribution for the stroke is the actual circumscribed rectangle and the stroke generation rule (terminal symbol generation rule). Candidate (= candidate of what the stroke is) is determined by two (P (b | a, F)), and other formula parts are determined by the rules for generating the formula parts (non-terminal symbol generation rules) (P (a _l , a _r | a _p , R)).

In one preferable aspect, the likelihood distribution of the area in which the stroke mathematical component is to be written is set for each parameter of the circumscribed rectangle, the vertical center, vertical size, horizontal start point, horizontal end point, and each stroke generation rule. Calculated from the random variable. The apparatus includes a context-free grammar composed of a plurality of formula generation rules and means for storing the application probability of each formula generation rule determined by the positional relationship between the formula parts.

The handwritten mathematical expression recognition method according to the present invention includes an input step of reading a handwritten mathematical expression, a step of calculating a stroke likelihood of a stroke candidate constituting the mathematical expression, and a plurality of input mathematical expressions composed of a plurality of stroke candidates. A step of expressing with mathematical expression candidates, a step of calculating a structural likelihood related to a positional relationship between mathematical parts constituting the input handwritten mathematical expression using a probability context free grammar, and a calculated stroke likelihood and structural likelihood. Are used to calculate the likelihood of each mathematical expression candidate, and the step is to determine an input handwritten mathematical expression based on the calculated likelihood of each mathematical expression candidate. About the more concrete structure of the handwritten mathematical formula recognition method, it describes in the description which concerns on the said handwriting recognition apparatus, and the below-mentioned embodiment.

The handwritten mathematical expression recognition apparatus according to the present invention includes, as specific hardware, a computer including an input unit, an output unit, a storage unit, a calculation unit, and a display unit. Therefore, in the handwritten mathematical expression recognition according to the present invention, in order to recognize the handwritten mathematical expression, the computer inputs input means for reading the handwritten mathematical expression, means for calculating the stroke likelihood of the stroke candidates constituting the mathematical expression, and the inputted mathematical expression. Means for expressing a plurality of mathematical expression candidates composed of a plurality of stroke candidates, and means for calculating a structural likelihood related to the positional relationship between mathematical parts constituting the inputted handwritten mathematical expression using a probability context free grammar And a means for calculating the likelihood of each formula candidate using the calculated stroke likelihood and structure likelihood, and a means for determining a formula based on the calculated likelihood of each formula candidate Or a computer-readable medium storing the computer program. For a more specific configuration of the computer program for executing handwritten mathematical expression recognition, the description related to the handwriting recognition device can be used.

In the present invention, a stroke is basically handled as a minimum unit, but a symbol can be handled as a minimum unit. Strokes and symbols are terminal formula parts in a stochastic context free grammar. Therefore, other technical means adopted by the present invention are input means for reading handwritten mathematical formulas and terminal formula component likelihoods of terminal formula component candidates constituting the mathematical formulas. Calculating means, means for expressing the input mathematical expression by a plurality of mathematical expression candidates composed of a plurality of terminal mathematical expression component candidates, and a mathematical expression including a terminal mathematical expression component and a non-terminal mathematical expression component constituting the input handwritten mathematical expression Means for calculating the structural likelihood related to the positional relationship between the parts using the probability context free grammar; means for calculating the likelihood of each mathematical expression candidate using the calculated terminal mathematical expression part likelihood and the structural likelihood; And means for determining an input handwritten mathematical expression based on the calculated likelihood of each mathematical expression candidate. This technical means is also configured as a handwritten mathematical expression recognition device, a handwritten mathematical expression recognition method, a computer program for causing a computer to execute a handwritten mathematical expression, or a computer-readable medium recording the program. In one aspect, the minimum unit for dividing the handwritten mathematical expression input is a stroke, and the terminal mathematical expression component is a stroke. In another aspect, the smallest unit for dividing the handwritten mathematical expression input is a symbol, and the terminal mathematical expression component is a symbol.

Here, terms necessary for understanding the present invention will be described.
Mathematical parts: Stroke mathematical parts and other mathematical parts (characters, terms, factors, etc., which are defined as high-order mathematical parts in this specification). Everything related to non-terminal symbol generation rules is a mathematical component. In the terminal symbol generation rule, the left side is a stroke mathematical component, and the right side is an actual stroke (this is not a mathematical component).
Parent formula component: For each generation rule, the formula component (one) on the left side is called a parent formula component.
Child mathematical component: For each fire terminal symbol production rule, the mathematical component on the right side (single or plural. If the production rule is expressed in Chomsky standard form, there are always two).
Stroke: "Stroke" has the meaning of "input stroke / actual stroke" indicating from the pen down to the pen up of the actual input (sample point sequence), and the meaning of "stroke formula part" as a recognition candidate of the input stroke . The input stroke is an actual sample point sequence, and the stroke formula part is symbolic information. The input stroke is generated from the stroke formula part by the terminal symbol generation rule.
Stroke generation rule: A terminal symbol generation rule in which an input stroke is generated from a stroke formula part.
Stroke generation rule candidates: For each input stroke, there may be a plurality of stroke recognition candidates. That is, the terminal symbol generation rule for which the input stroke is generated cannot be uniquely specified, and a plurality of terminal symbol generation rule candidates must be considered simultaneously. In other words, it is considered that a certain input stroke may have been generated from a terminal symbol generation rule of minus → input stroke, or may be generated from a terminal symbol generation rule of positive horizontal bar → input stroke. The likelihood to which each rule is applied is performed by stroke likelihood calculation.
Stroke sequence: An expression when the input is viewed as an input stroke sequence.
Symbol: A part of a mathematical component. It is a so-called “character”. It is the smallest unit that has meaning alone.
Terminal formula component: Refers to the parent formula component of the terminal symbol generation grammar. An input stroke is generated from the terminal mathematical component. In this method, it is a stroke formula part. It is also possible to recognize mathematical expressions that do not necessarily have a stroke as the minimum unit. If the mathematical expression can be divided into discrete symbols by some preprocessing, the same recognition can be performed with the symbol as a minimum unit. In that case, the termination formula part is a symbol.
Non-terminal mathematical part: the same as the upper mathematical part. A mathematical expression component other than a stroke mathematical expression component (a mathematical expression component other than a symbol mathematical expression component when the symbol is the minimum unit). There is no generation rule that directly generates an input stroke.
Likelihood distribution of hidden writing area: There is a hidden writing area representing a portion occupied by the mathematical expression part in the mathematical expression part, and this "area where the mathematical expression part is to be written" is called a hidden writing area. For example, if the mathematical components are in a “horizontal” positional relationship, the hidden writing area of each mathematical component is considered to be connected horizontally. An input stroke is generated from the stroke mathematical component (terminal mathematical component), but the position of the generated input stroke (the circumscribed rectangle) is determined probabilistically depending on the hidden writing area of the stroke mathematical component. Therefore, when a certain input stroke is observed and an input stroke generation rule candidate is obtained, the likelihood distribution of the hidden writing area of the stroke mathematical expression part that has generated the input stroke can be obtained. The likelihood distribution of the hidden writing area can be calculated as the probability distribution of the hidden writing area when the circumscribed rectangle is obtained.

The recognition of the handwritten mathematical expression of the present invention is particularly advantageous in the structure recognition as compared with the conventional recognition method. It is also possible to correct stroke recognition errors in the structure recognition process, and as a result, more accurate handwritten mathematical expressions can be recognized.

[A] Basic Configuration of Handwritten Formula Recognition The present invention relates to probabilistic online handwritten formula recognition using probabilistic context free grammar, and recognizes formulas from handwritten formula information input in time series. In the present invention, the recognition of the mathematical expression is formulated as a problem for obtaining a mathematical expression candidate that maximizes the product of the stroke likelihood and the structure likelihood. As shown in FIG. 1, the handwritten mathematical expression recognition method includes likelihood calculation for each stroke and structural likelihood calculation based on the positional relationship of mathematical expression parts (strokes and symbols).

The stroke likelihood is a probability that the handwritten stroke is actually generated from the stroke model, and the stroke likelihood can be calculated by an existing model-based character recognition method. Referring to FIG. 3, the stroke likelihood indicates how much probability “is the stroke”, and indicates the likelihood of the stroke “a” for three handwritten symbols. Further, the stroke likelihood will be described with reference to FIG. 2. According to the time series, the likelihood that the first stroke is the root symbol (candidate) is 0.2, and the likelihood that v (candidate) is 0. 2, and the likelihood of being a fractional line (candidate) is 0.1. The likelihood that the second stroke is a (candidate) is 0.2, and the likelihood that the second stroke is d (candidate) is 0.1. The likelihood that the third stroke is 2 (candidate) is 0.2, and the likelihood that the third stroke is z (candidate) is 0.1. The likelihood that the fourth stroke is a minus sign (candidate) is 0.2, and the likelihood that the fourth stroke is a horizontal bar-(candidate) is 0.1. The likelihood that the fifth stroke is 1 (candidate) is 0.2, and the likelihood that the vertical stroke of the plus sign | (candidate) is 0.1. The likelihood that the sixth stroke is 6 (candidate) is 0.2, and the likelihood that it is the first stroke (candidate) of G is 0.1.

The structure likelihood is a probability that the positional relationship between the strokes is actually generated from the mathematical expression candidate. Based on FIG. 3, the structure likelihood is the probability of “the structure”, and for the three patterns of symbols a and b, the two symbols are “connected side by side”. Likelihood ". The structural likelihood can be calculated by modeling with a stochastic context-free grammar (SCFG) described later.

In the present invention, a mathematical expression candidate that maximizes stroke likelihood × structure likelihood is obtained from various possible mathematical expression candidates, and the mathematical expression candidate with the maximum likelihood is recognized as a handwritten mathematical expression. As shown in FIG. 3, when there are two strokes that intersect at right angles as shown in FIG. 2, the likelihood that the fourth stroke is a minus sign (candidate) is 0 when viewed from the stroke likelihood alone. .2, the likelihood that the horizontal bar of the plus sign-(candidate) is 0.1, the likelihood that the fifth stroke is 1 (candidate) is 0.2, and the likelihood of the plus sign vertical The likelihood of being a bar | (candidate) is 0.1. However, since the structure likelihood of -1 (candidate) is low and the structure likelihood of + (candidate) is high, the candidate as + is dominant in the product of the stroke likelihood and the symbol structure likelihood. .

[B] Modeling of mathematical structure using probabilistic context-free grammar Context-Free Grammar (CFG) is a grammar written in the form of A → B, AB → C, ABC → D, etc. Suitable for generating and analyzing recursive languages. For example, various mathematical expressions (for example, ab ^abc ) composed of three characters a, b, and c can be analyzed by repeatedly applying several simple mathematical expression generation rules as shown in FIG.

The stochastic context free grammar SCFG is a grammar in which the production rules in CFG have application probabilities. The application probability is determined by the positional relationship between the mathematical formula parts. For example, in FIG. 5, the application probability of the third generation rule from the top is determined by “how the position of the expression seems to be upper right with respect to the character”. In this way, the structural likelihood of a handwritten mathematical expression can be modeled by the SCFG whose application probability is determined by the positional relationship between mathematical parts. The structural likelihood by SCFG can be calculated efficiently by CYK-algorithm.

[C] Formula Grammar The formula grammar is a terminal symbol generation rule for generating an actual stroke from a stroke (end) formula part, and a non-terminal that generates one or more other formula parts from a non-stroke (non-end) formula part. Consists of symbol generation rules.
[C-1] The non-terminal symbol generation rule mathematical grammar suffices to prepare a context-free grammar as shown in FIG. Each formula generation rule constituting the context-free grammar has a probability of being applied depending on the positional relationship between the formula parts. Among the context-free grammars shown in FIG. 6 that are equivalently converted to the Chomsky standard form, the ones related to formula generation in FIG. 12 are the five shown in FIG.

[C-2] Terminal Symbol Generation Rule The terminal symbol generation rule F is a rule for generating an actual handwritten stroke from the stroke equation part εST. The terminal symbol generation rules related to FIG. 12 are the five shown in FIG.

[D] Likelihood distribution of hidden writing area as position feature quantity Evaluation of the positional relationship between mathematical formula parts is not a simple problem. Conventionally, a circumscribed rectangle generally used as a positional relationship feature amount cannot handle various symbol shapes in a unified manner as shown in FIG. 9, and needs to be classified for each character category. In addition to the alphabet, there are many symbols with special shapes such as accents, dots, and commas in the mathematical formula, so the number of cases increases, resulting in complicated SCFG rules. Figure 9 is an example illustrating the limits of the circumscribed rectangle, the positional relationship between the symbols of qx and d ^x are different, (quoted from non-patent document 1) enclosing rectangle are equal.

In the present invention, in order to avoid this problem, the likelihood distribution of the hidden writing area is used as a feature quantity that uniformly handles various mathematical parts (symbols and strokes). The hidden writing area indicates where the mathematical parts (symbols and strokes) actually written are “where they are going to be written”, and the likelihood distribution of this “area where they are going to be written” is calculated. First, a formula generation model as a background of the hidden writing area will be described. As shown in FIG. 17, the formula “abc ^d ” is assumed to be written as follows, assuming a human formula generation model.
(1) First, determine the center and vertical size of the entire formula.
(2) Determine the start and end points of the first character “a”. The area where “a” is to be written is determined.
(3) “a” is written in that area. The place actually written has a stochastic fluctuation.
(4) Next, determine the end point of “b”. The area where “b” is to be written is determined.
(5) “b” is written in that area. “B” tends to be written above the area.
(6) Similarly, “c” is written.
(7) The center and vertical size of “d” are stochastically determined from the vertical center and vertical size of “c”.
(8) The end point of “d” is determined, and “d” is written in that area.
That is, we are assuming that we are writing mathematical formulas along a hidden grid that is consistent in the horizontal direction. The structure likelihood is considered to be “probability generated from this generation model”.

Assuming the mathematical expression generation model as described above, the “rectangular area where the mathematical expression part is to be written” in the model is set as a hidden writing area. In this model, it is assumed that each stroke is written with a probabilistic fluctuation in the hidden writing area of each stroke formula part. In other words, the position and size at which each stroke is actually written (the two can be represented by a circumscribed rectangle) are stochastically determined from the hidden writing area of each stroke formula component. Moreover, the hidden writing areas of the respective mathematical formula parts have a certain positional relationship. For example, when the formula part ABC is generated according to the rule of A → B + C (for example, A: 2 (x + 1), B: 2, C: (x + 1), etc.), A, B and C Each hidden writing area has a certain positional relationship ("horizontal").

FIG. 11 is a conceptual diagram of a hidden writing area. The formula abc is considered to be written with stochastic fluctuations in the rectangular area as shown in the figure above. At this time, b tends to be written above the rectangular area as compared to a and c. On the other hand, when there is a character actually written as shown in the figure below, conversely, the “hidden writing region” that is the “rectangular region to be written” of the character has a likelihood distribution. Using the likelihood distribution of this “hidden writing area book” as the position feature quantity, the positions of all the mathematical components (symbols) are handled uniformly. It is considered that alphabets, dots, commas, and accents can all be handled with the same feature amount by the likelihood distribution of the hidden writing area.

The likelihood distribution of the hidden writing area will be further described in detail. When a stroke is written, the likelihood distribution of the area to be written is obtained by the bounding rectangle of the written stroke and the generation rule F of that stroke (what stroke was written for). Is the conditional probability distribution of the hidden writing area when A certain hidden writing area can be expressed by four parameters a = (a ^c , a ^s , a ^b , a ^e ). Here, a ^c : vertical center, a ^s : vertical size, a ^b : horizontal start point, a ^e : horizontal end point. When representing the probability distribution P (a), it is necessary to represent the probability distribution in the four-dimensional space of the four-dimensional vector a = (a ^c , a ^s , a ^b , a ^e ). Assuming a normal distribution of diagonal covariance in 4D space (i.e., a ^c , a ^s , a ^b , and a ^e are each independently a normal distribution), the parameter representing P (a) is The mean vector and the covariance matrix μa = (μa ^c , μa ^s , μa ^b , μa ^e ), σa = diag (σa ^c , σa ^s , σa ^b , σa ^e ). The circumscribed rectangle b = (b ^c , b ^s , b ^b , b ^e ) of each actual stroke is stochastically derived from the hidden writing region a = (a ^c , a ^s , a ^b , a ^e ) P (b | We think that it can be obtained according to a, F). Conversely, when the circumscribed rectangle b of each stroke is obtained, the probability distribution of the hidden writing area a can be obtained by viewing P (b | a, F) as a function of a. This function is prepared in advance for each stroke generation rule. It is learned in advance what kind of hidden writing area likelihood distribution is generated from each stroke. Even if the circumscribed rectangle of the input stroke is the same, the likelihood distribution of the hidden writing area changes depending on the recognition result (generation rule) of the stroke. For example, even if the circumscribed rectangles are the same, the likelihood of the hidden writing area is different between the case where the stroke generation rule candidate is “a → (actual stroke)” and the case where “d → (actual stroke)”. Distribution is different. Parameters of the bounding rectangle b of each actual stroke (b ^c , b ^s , b ^b , b ^e , where b ^c : vertical center b ^s : vertical size, b ^b : horizontal start point, b ^e : horizontal end point The parameter of the likelihood distribution of the hidden writing area a (random variable) is a ^c = b ^c + Pc, a ^s =
b ^s + Ps, a ^b = b ^b + Pb, a ^e = b ^e + Pe, and learn Pc, Ps, Pb, Pe for each stroke, Based on this, a function for calculating the likelihood distribution of the region in which the mathematical formula part is to be written is stored.

[E] Structural likelihood calculation using the hidden writing area of the mathematical part [E-1] A set of various defined stroke mathematical parts,
And a set of all mathematical parts including stroke mathematical parts,
And In addition, a set of positional relationships between mathematical parts
And

A set of non-terminal symbol generation rules among the formula generation rules is set as a set R. Information of each non-terminal symbol generation rule is R (εR) = (R ^p εMP, R ¹ εMP, R ^r εMP, R ^RP εRP). Here, R ^p , R ^l , R ^r , and R ^RP indicate the positional relationship that the parent formula part name, the child formula part name (left), the child formula part name (right), and the child formula part satisfy in order.

Let F be a set of terminal symbol generation rules. Information that each terminal symbol generation rule has is F (εF) = (F ^p εST). An example of a list of rules is shown in FIG.

For each stroke of the handwritten formulas observed, the i-th stroke shape data (sample coordinate series information) t _i, expressed as the circumscribed rectangle b _i, handwritten formulas observed,
It expresses. Here, N is the total number of observation strokes.

Arbitrary formula candidate H is an input formula,
In this case, it is represented by a tree structure as shown in FIG. This tree structure is a nested form of formula generation rules, for example
Can be described as follows.

[E-2] "Writing area" of mathematical parts
It is assumed that there is a rectangular hidden writing area behind each mathematical expression part as shown in FIG. A writing region of the parent math part _R ^{i p} in Equation productions _{R i,}
It expresses. Here, a _i ^c , a _i ^s , a _i ^b , and a _i ^e are the coordinates and lengths shown in FIG.

Next, for each positional relationship, the probability that the child formula part writing area is a _l , a _r when the parent formula part writing area is _ap ,
Determine.

[E-2-1] Positional relationship: The condition that the lateral positional relationship is horizontal is described by the following equation.
That is,
(Δ (x) is the Dirac Delta function)

[E-2-2] Positional relationship: upper right The condition that the positional relationship is upper right is described by the following equation.
Here, q _{upper right} ¹ and q _{upper right} ² are respectively given normal distributions.
Follow. Therefore,
It becomes.

[E-2-3] Other positional relations Similarly for the lower right and other positional relations,
Or normal distribution given in advance
Introducing a random variable q according to
Write like this.

[E-3] Terminal Symbol Generation Rule Applicability Probability Stroke output t, b represents the probability P (t, b | a, F) generated by the stroke writing area and terminal symbol generation rules a, F as follows: Define as follows.
The first term on the right side is the probability (stroke likelihood) that the stroke shape t is the stroke F ^p and can be calculated by an existing character recognition method. The second term on the right side is the probability distribution of the actual stroke circumscribing rectangle b when the stroke is generated in the writing area a by the terminal symbol generation rule F. This probability is expressed as ab normal distribution
Defined by

[E-4] Likelihood Calculation The likelihood calculation method for obtaining the observation (t, b) from the formula candidate H is shown in the example of FIG. If the writing area of each mathematical component is a _i ,
It becomes. The character at the lower right of the integration symbol indicates which character the integration is for. Π ¹¹ _{i = 1} P (t _i | R ^MP ) is a stroke likelihood and can be calculated by an existing character recognition method. In addition, the remaining part is an equation of structure likelihood, which is independent of a _i ^c , a _i ^s , a _i ^b , and a _i ^e in the writing probability distribution P (a _i ) of each character part. Approximation to handle
It is possible to perform sequential calculation as follows.

(1) Introduction
Calculate the part.
(a) P (b ₂ | a ₂ , R ₂ ) and P (b ₃ | a ₃ , R ₃ ) are normal distributions of a ₂ and a ₃ when b ₂ and b ₃ are given, respectively. is there.
(b) If P (a ₂ , a ₃ | a ₇ , R ₇ ) is given by Equation 1, that is,
If it is described,
It becomes. Here, since the product of the normal distribution function is a normal distribution function, and the convolution of the normal distribution function and the delta function is also a normal distribution function, P (a ₂ ) and P (a ₃ ) are normal distributions. Thus, P (a ₂ , a ₃ | a ₇ , R ₇ ) is a normal distribution function of a ₇ .
(c) If P (a ₂ , a ₃ | a ₇ , R ₇ ) is given by Equation 2, that is,
Similarly, when
Where P (a ₂ , a ₃ | a ₇ , R ₇ ) is a normal distribution of a ₇ because P (a ₂ ), P (a ₃ ), and P (q) are normally distributed. It becomes a function.
(d)
Is a normal distribution function P of _{a 7 (a 7).}

(2) Therefore, the one outer integral is
It becomes. Here, since both P (a ₇ ) and P (b ₄ | a ₄ , R ₄ ) are normal distributions, the integral value is the normal distribution P (a ₈ ) as before.

(3) Similarly, by performing the integral calculation outward, the structural likelihood becomes ∫P (a ₁₁ ) da ₁₁ , and the structural likelihood is obtained by this integral calculation.

[E-5] Calculation of likelihood by CYK-algorithm Calculation of the likelihood of a mathematical expression candidate can be efficiently analyzed by CYK-algorithm. 15 and 16 show examples of likelihood calculation. FIG. 15 shows a case where the time series of 4 strokes is “2” → “a” → “2” → “route symbol”. For example, the likelihood of “2a” is 0.1 (likelihood of 2) × 0.2 (likelihood of a) × 0.05 (horizontal link likelihood) = 0.001. The likelihood of “2 ^a ” is 0.1 (likelihood of 2) × 0.2 (likelihood of a) × 0.025 (upper right connection likelihood) = 0.001. For the entire four strokes, the likelihood of √2a ² (candidate) is 0.0000001, and the likelihood of √2a2 (candidate) is 0.00000005. FIG. 16 shows a case where the time series of the four strokes is “(“ → ”5 portion“ → ”5 portion“ → “)”. For the entire four strokes, the likelihood of (5) (candidate) is 0.0000001, and the likelihood of 151 (candidate) is 0.000000005.

[F] Operation confirmation experiment [F-1] Confirmation of effectiveness of likelihood distribution in hidden writing area
8 formulas (30 to 60 strokes) that seem to have moderate complexity in the IEEE Transaction, strokes were fully recognized using 80 handwritten formulas by the same author 10 times each as evaluation data The structure recognition experiment was conducted under the conditions. The parameters that determine the likelihood distribution of the hidden writing area of each stroke were determined heuristically, and the mathematical expression grammar allowed fractions, square roots and accents in addition to those in FIG. As a result, the structure of 63 of 80 formulas was completely recognized. In addition, errors are only local errors such as subscript mistakes, and it is considered that accuracy can be further improved by parameter tuning. From this experiment, it was confirmed that the likelihood distribution of the hidden writing area is effective as a feature quantity for the structure likelihood calculation.

[F-2] Operation Confirmation of Entire System The operation of the entire system was confirmed by manually setting the experimental stroke likelihood. The eight formulas used in [F-1] are each evaluated as eight handwritten formulas once each for evaluation data, and the stroke likelihood first-rank candidates for all strokes are all alphabetic in order to check the correction of the stroke recognition error due to the structure recognition. ”A” and 2nd place were correct strokes, and the mathematical formula was recognized. As a result, many correct strokes were finally included in the recognition results. This experiment confirmed that stroke recognition errors can be corrected in structure recognition. In the above experiment, the stroke likelihood is given manually, but a hidden Markov model HMM may be used for stroke likelihood calculation.

The present invention can be used as online handwritten mathematical expression recognition. Specifically, it can be used for education, paper creation, mathematical expression search interface, numerical calculation software interface, and the like.

It is a figure explaining the whole structure of handwritten numerical formula recognition. It is a figure explaining the stroke likelihood of a stroke candidate. It is a figure explaining the relationship between stroke likelihood and structure likelihood. It is a figure explaining probabilistic context free grammar. It is a figure which shows the example of the probability context free grammar which produces | generates a numerical formula. It is a figure which shows the example of the probability context free grammar which produces | generates a numerical formula. FIG. 13 is a diagram showing the rules related to formula generation in FIG. 12, in which the grammar shown in FIG. 6 is equivalently changed to the Chomsky standard form. It is a figure which shows the terminal symbol production | generation rule relevant to FIG. It is an example which shows the limit of a circumscribed rectangle. It is a figure explaining the evaluation of the positional relationship between mathematical formula parts. It is a conceptual diagram of a hidden writing area. It is a figure which shows a response | compatibility of a handwritten numerical formula and the tree structure of the correct numerical formula candidate with respect to it. It is a figure which shows a formula component and a hidden writing area. It is a figure which shows the parameter of a hidden writing area. It is a figure which shows the example of likelihood calculation using CYK-algorithm. It is a figure which shows the example of likelihood calculation using CYK-algorithm. It is a figure explaining a human mathematical formula generation model.

Claims (18)

Input means for reading handwritten mathematical formulas;
Means for calculating a stroke likelihood of stroke candidates constituting the mathematical formula;
Means for expressing an input mathematical expression with a plurality of mathematical expression candidates composed of a plurality of stroke candidates;
Means for calculating the structural likelihood related to the positional relationship between the mathematical parts constituting the input handwritten mathematical expression using a probability context free grammar;
Means for calculating the likelihood of each mathematical formula candidate using the calculated stroke likelihood and structure likelihood;
Means for determining an input handwritten mathematical formula based on the calculated likelihood of each mathematical formula candidate;
An apparatus for recognizing handwritten mathematical formulas.   2. The handwritten mathematical expression recognition apparatus according to claim 1, wherein the mathematical expression determination means uses the mathematical expression candidate having the maximum likelihood as the mathematical expression. The means to determine the formula is
Means for extracting a plurality of mathematical formula candidates having a high likelihood;
Means for selecting one mathematical expression candidate from the plurality of mathematical expression candidates extracted;
The recognition apparatus of the handwritten numerical formula of Claim 1 consisting of. The structure likelihood calculating means includes
A means for calculating a likelihood distribution of an area in which the stroke is to be written based on the stroke generation rule candidate for each of the observed circumscribed rectangles of the strokes and the generation rule candidates for the strokes;
Means for calculating the structure likelihood by a probability context free grammar in which the application probability is determined by the positional relationship between the mathematical components including the stroke, using the calculated likelihood distribution as the position feature amount of each stroke;
The recognition apparatus of the handwritten numerical formula in any one of Claims 1 thru | or 3 which has these. The structural likelihood calculating means is, for each mathematical component,
If the mathematical part is a stroke mathematical part, based on the observed circumscribed rectangle of the mathematical part and the generation rule of the mathematical part,
In the case where the mathematical part is a mathematical part other than a stroke, based on the region where each mathematical part constituting the mathematical part is to be written and the generation rule of the mathematical part,
5. The handwritten mathematical expression recognition apparatus according to claim 1, wherein a likelihood distribution of an area in which the mathematical formula part is to be written is calculated. The likelihood distribution of the area in which the stroke formula part is to be written is calculated from the parameters of the vertical center, vertical size, horizontal start point, and horizontal end point of the circumscribed rectangle and the random variables set for each stroke generation rule. To be
The recognition apparatus of the handwritten numerical formula in any one of Claim 4, 5.   7. The apparatus according to claim 1, further comprising means for storing a context-free grammar composed of a plurality of formula generation rules and an application probability of each formula generation rule determined by a positional relationship between formula components. A handwritten mathematical expression recognition device. An input step for reading a handwritten mathematical formula;
Calculating stroke likelihood of stroke candidates constituting the mathematical formula;
Expressing the input mathematical expression with a plurality of mathematical expression candidates composed of a plurality of stroke candidates;
Calculating a structural likelihood related to the positional relationship between mathematical parts constituting the input handwritten mathematical expression using a probability context free grammar;
Calculating the likelihood of each formula candidate using the calculated stroke likelihood and structure likelihood;
Determining an input handwritten mathematical formula based on the calculated likelihood of each mathematical formula candidate;
The recognition method of the handwritten mathematical formula which consists of.   The method for recognizing a handwritten mathematical expression according to claim 8, wherein the step of determining the mathematical expression uses a mathematical expression candidate having the maximum likelihood as a mathematical expression. The step of determining the formula is:
Extracting a plurality of mathematical formula candidates having a high likelihood;
Selecting one mathematical expression candidate from the extracted plurality of mathematical expression candidates;
The recognition method of the handwritten numerical formula of Claim 8 consisting of. The structural likelihood calculating step includes:
For each circumscribed rectangle of each observed stroke, and for each candidate for the generation rule of the stroke, calculating a likelihood distribution of an area in which the stroke is to be written based on the stroke generation rule candidate;
The structure likelihood is calculated by a probability context free grammar in which the application probability is determined by the positional relationship between mathematical parts including the stroke, using the calculated likelihood distribution as the position feature amount of each stroke;
The recognition method of the handwritten numerical formula in any one of Claims 8 thru | or 10 which has. In the structural likelihood calculation step, for each mathematical component,
If the mathematical part is a stroke mathematical part, based on the observed circumscribed rectangle of the mathematical part and the generation rule of the mathematical part,
In the case where the mathematical part is a mathematical part other than a stroke, based on the region where each mathematical part constituting the mathematical part is to be written and the generation rule of the mathematical part,
The method for recognizing a handwritten mathematical expression according to any one of claims 8 to 11, wherein a likelihood distribution of an area in which the mathematical formula part is to be written is calculated. The likelihood distribution of the area in which the stroke formula part is to be written is calculated from the parameters of the vertical center, vertical size, horizontal start point, and horizontal end point of the circumscribed rectangle and the random variables set for each stroke generation rule. To be
The method for recognizing a handwritten mathematical formula according to claim 11.   A computer program for causing a computer to function as any one of claims 1 to 7 for recognizing a handwritten mathematical expression.   A computer-readable recording medium having recorded thereon a computer program for causing a computer to function as any one of claims 1 to 7 for recognizing a handwritten mathematical expression. Input means for reading handwritten mathematical formulas;
Means for calculating a terminal formula component likelihood of a terminal formula component candidate constituting the formula;
Means for expressing an input mathematical expression with a plurality of mathematical expression candidates composed of a plurality of terminal mathematical expression component candidates;
Means for calculating a structural likelihood related to the positional relationship between the mathematical expression parts including the terminal mathematical expression part and the non-terminal mathematical part constituting the input handwritten mathematical expression using a probability context free grammar;
Means for calculating the likelihood of each formula candidate using the calculated terminal formula component likelihood and structure likelihood;
Means for determining an input handwritten mathematical formula based on the calculated likelihood of each mathematical formula candidate;
An apparatus for recognizing handwritten mathematical formulas.   The handwritten mathematical expression recognition apparatus according to claim 16, wherein the terminal mathematical expression component is a stroke.   The handwritten mathematical expression recognition apparatus according to claim 16, wherein the terminal mathematical component is a symbol.