CN107209862B - Identification device and information storage medium - Google Patents

Identification device and information storage medium Download PDF

Info

Publication number
CN107209862B
CN107209862B CN201680006635.XA CN201680006635A CN107209862B CN 107209862 B CN107209862 B CN 107209862B CN 201680006635 A CN201680006635 A CN 201680006635A CN 107209862 B CN107209862 B CN 107209862B
Authority
CN
China
Prior art keywords
stroke
character
gesture
numerical expression
character string
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201680006635.XA
Other languages
Chinese (zh)
Other versions
CN107209862A (en
Inventor
中川正树
潘明庆
黎德英
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tokyo University of Agriculture and Technology NUC
Original Assignee
Tokyo University of Agriculture and Technology NUC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tokyo University of Agriculture and Technology NUC filed Critical Tokyo University of Agriculture and Technology NUC
Publication of CN107209862A publication Critical patent/CN107209862A/en
Application granted granted Critical
Publication of CN107209862B publication Critical patent/CN107209862B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/18—Extraction of features or characteristics of the image
    • G06V30/1801—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Abstract

Provided is a recognition device capable of recognizing a numerical expression including a character string. The recognition device includes: a registration unit that determines whether or not an input stroke or a stroke string matches a specific gesture character for distinguishing between a character string and a numerical expression, and registers information on a range specified by the specific gesture character when the input stroke or the stroke string matches the specific gesture character; a distinguishing section that sets, as a stroke corresponding to the character string, a stroke included in a range specified by the determination gesture character among the inputted strokes, and sets, as a stroke corresponding to the numerical expression, a stroke not included in the range specified by the determination gesture character; and a recognition unit that recognizes the character string from a stroke set as a stroke corresponding to the character string by using a character string recognition engine, and recognizes the numerical expression from a stroke set as a stroke corresponding to the numerical expression by using a numerical expression recognition engine.

Description

Identification device and information storage medium
Technical Field
The invention relates to a program, an information storage medium, and a recognition device.
Background
Conventionally, as a method for inputting numerical expressions, a method using a specialized language such as LaTeX, a method using a numerical input editor, or the like has been used, but the former requires mastering specialized words, and the latter requires repetition of menu selection and symbol selection, which are inconvenient for the general public to use. In contrast, a method of recognizing a numerical expression handwritten on a tablet or the like by a computer has been studied for a long time. Further, the recognition rate has been improved by many years of research, and further, the possibility of penetration into the general public has been increased due to rapid spread of tablet PCs in recent years.
Documents of the prior art
Patent document
Patent document 1: japanese laid-open patent publication No. 2009-80615
Disclosure of Invention
Problems to be solved by the invention
The numerical expression is composed of a relation between a symbol (symbol) and positions and sizes of four operators, brackets, score symbols, root symbols, integral symbols, symbols representing power, and the like. In the handwritten numerical recognition system, a symbol is generally assumed to be an alphanumeric symbol, but actually, a character string other than the alphanumeric symbol is often written as a symbol, and in the conventional handwritten numerical recognition system, there is a problem that a character string included in a numerical expression cannot be recognized. Such numeric expressions containing strings are frequently found in textbooks and are also frequently written on a blackboard.
The present invention has been made in view of the above problems, and an object thereof is to provide a program, an information storage medium, and a recognition device capable of recognizing a numerical expression including a character string.
Means for solving the problems
(1) The present invention relates to a program for recognizing a numerical expression including a character string from a stroke string input by handwriting, the program causing a computer to function as a registration unit that determines whether or not an input stroke or a stroke string matches a specific gesture for distinguishing a character string from a numerical expression, and when it is determined that the input stroke or the stroke string matches the specific gesture, registers information relating to a range specified by the specific gesture, the discrimination unit setting, as a stroke corresponding to one of the character string and the numerical expression, a stroke included in the range specified by the specific gesture among the input strokes, and setting, as a stroke corresponding to the other of the character string and the numerical expression, a stroke not included in the range specified by the specific gesture, and a recognition unit that recognizes a character string from a stroke set as the one corresponding to the character string by a character string recognition engine, a numerical expression is recognized by a numerical recognition engine from strokes set as strokes corresponding to the numerical expression. The present invention also relates to a computer-readable information storage medium storing a program for causing a computer to function as each of the above-described units. The present invention also relates to an identification device including the above-described components.
According to the present invention, it is determined whether or not an input stroke matches a specific gesture character, a stroke included in a range specified by the specific gesture character is set as a stroke corresponding to one of a character string and a numerical expression, a stroke not included in the range specified by the specific gesture character is set as a stroke corresponding to the other of the character string and the numerical expression, the character string is recognized by a character string recognition engine from the stroke set as the stroke corresponding to the character string, and the numerical expression is recognized by a numerical recognition engine from the stroke set as the stroke corresponding to the numerical expression. Further, a method of automatically separating (distinguishing) an expression from a character string without using a specific gesture character may be considered. However, it is difficult to automatically distinguish between a numeric expression and a character string, and erroneous division cannot be avoided, and therefore, in the present invention, a determination gesture for distinguishing between a character string and a numeric expression is introduced.
(2) In the program, the information storage medium, and the identification device according to the present invention, the following may be used: the distinguishing unit sets, when the input stroke or the stroke string is determined to match the specific gesture character, a stroke included in a range specified by the specific gesture character as a stroke corresponding to one of a character string and a numerical expression, determines, when the input stroke is determined not to match the specific gesture character, whether the input stroke is included in the range specified by the specific gesture character, and sets, when the input stroke is included in the range specified by the specific gesture character, the input stroke as a stroke corresponding to one of the character string and the numerical expression.
According to the present invention, even when a specific gesture is written after a character string (or numerical expression) is written, or when a character string (or numerical expression) is written after a specific gesture is written, the character string and the numerical expression can be distinguished from each other, and convenience for the user can be improved.
(3) In the program, the information storage medium, and the identification device according to the present invention, the following may be used: the distinguishing unit sets, as a stroke corresponding to a character string, a stroke included in a range specified by the specific gesture character, among inputted strokes, and sets, as a stroke corresponding to a numerical expression, a stroke not included in a range specified by the specific gesture character, and the recognizing unit recognizes the numerical expression using a numerical recognition engine, with strokes included in a range specified by 1 specific gesture character being regarded as 1 mark.
According to the present invention, a stroke included in a range specified by 1 fixed gesture character (a stroke set as a stroke corresponding to a character string) is regarded as 1 mark and a numerical expression is recognized by a numerical recognition engine, whereby a numerical expression including a character string can be recognized by using a numerical recognition engine that cannot recognize a character string.
(4) In addition, the program and the information storage medium according to the present invention may be: the computer is caused to function as a display control unit that controls a display unit to display an input stroke string, and the display control unit is caused to perform control to display a stroke or a stroke string determined to match the specific gesture character as an image representing a shape corresponding to the specific gesture character.
In addition, the identification device according to the present invention may be: the display control unit performs control to display the stroke or the stroke string determined to match the specific gesture character as an image representing a shape corresponding to the specific gesture character.
According to the present invention, by displaying the stroke determined to match the fixed gesture character as the image indicating the shape corresponding to the fixed gesture character, the user can be made aware that the fixed gesture character has been correctly determined, and the convenience of the user can be improved.
(5) In the program, the information storage medium, and the identification device according to the present invention, the following may be used: the display control unit performs control to display strokes included in the range specified by the specification gesture character in a color different from strokes not included in the range specified by the specification gesture character.
According to the present invention, by displaying the strokes included in the range specified by the fixed gesture character in a different color from the strokes not included in the range specified by the fixed gesture character, the user can be made aware that the character string and the numerical expression have been correctly distinguished by the fixed gesture character, and the convenience of the user can be improved.
(6) In the program, the information storage medium, and the identification device according to the present invention, the following may be used: the login unit determines that the input stroke or the stroke string matches the specific gesture character when the input stroke or the stroke string is substantially rectangular.
Drawings
Fig. 1 is an example of a functional block diagram of the identification device according to the present embodiment.
Fig. 2A is a diagram showing an example of writing in which a numerical expression, a character string, and a fixed gesture are mixed.
Fig. 2B is a diagram showing an example of the determination gesture character table.
Fig. 3A is a diagram for explaining the order in which the determination gesture characters are written.
Fig. 3B is a diagram for explaining the order in which the gesture characters are determined by writing.
Fig. 3C is a diagram for explaining the order in which the determination gesture characters are written.
Fig. 3D is a diagram for explaining the order in which the gesture characters are determined by writing.
Fig. 4A is a diagram showing an example of writing of a rectangular fixed gesture character.
Fig. 4B is a diagram showing an example of writing of a rectangular fixed gesture character.
Fig. 5 is a flowchart showing an example of processing in the sequential method.
Fig. 6A is a diagram showing a specific display example in the successive mode.
Fig. 6B is a diagram showing a specific display example in the successive mode.
Fig. 6C is a diagram showing a specific display example in the sequential manner.
Fig. 6D is a diagram showing a specific display example in the sequential manner.
Fig. 6E is a diagram showing a specific display example in the successive mode.
Fig. 6F is a diagram showing a specific display example in the sequential mode.
Fig. 7 is a flowchart showing an example of the processing in the batch system.
Fig. 8A is a diagram showing a specific display example in the batch mode.
Fig. 8B is a diagram showing a specific display example in the batch mode.
Fig. 8C is a diagram showing a specific display example in the batch mode.
Detailed Description
The present embodiment will be described below. The embodiments described below are not intended to unduly limit the scope of the present invention set forth in the claims. The configurations described in the present embodiment are not necessarily all essential components of the present invention.
1. Form a
Fig. 1 shows an example of a functional block diagram of the identification device according to the present embodiment. Note that the recognition device of the present embodiment may be configured such that a part of the components (each part) in fig. 1 is omitted.
The character input unit 160 is used for allowing a user to input handwritten characters through a writing medium (pen, fingertip, or the like), and its function can be realized by a writing surface such as a tablet or a touch panel. The character input unit 160 detects coordinate data indicating a position of the writing medium from the time the writing medium contacts the writing surface to the time the writing medium is separated from the writing surface at regular time intervals, and outputs a detected coordinate data string (coordinate point sequence) to the processing unit 100 as data of strokes (strokes). Further, a vector from the end point of a stroke to the start point of the next stroke is referred to as an off-stroke (pen-motion vector), and a sequence in which the stroke and the pen-motion are continuous is referred to as a stroke string.
The storage unit 170 stores programs and various data for causing a computer to function as each unit of the processing unit 100, and functions as a work area of the processing unit 100, and the functions thereof can be realized by a hard disk, a RAM, or the like.
The display unit 190 is used to output the image generated by the processing unit 100, and its function can be realized by a display such as a touch panel, an LCD, or a CRT, which also functions as the character input unit 160.
The processing unit 100 (processor) performs processing such as recognition processing and display control based on the coordinate data, program, and the like from the character input unit 160. The processing unit 100 performs various processes with the main storage unit in the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.) and ASIC (gate array, etc.) or programs. The processing unit 100 includes a registration unit 110, a discrimination unit 112, an identification unit 114, and a display control unit 120.
The registration unit 110 determines whether or not the input stroke or stroke string matches a specific gesture character for distinguishing between a character string and a numerical expression, and if it is determined that the input stroke or stroke string matches the specific gesture character, performs a process of registering information on a range (region) specified by the specific gesture character. The registered information is stored in the storage unit 170. Here, the login section 110 may determine that the input stroke or the stroke string matches the above-described specific gesture character when the stroke or the stroke string is formed substantially rectangular.
The differentiating unit 112 sets, as a stroke corresponding to one of the character string and the numerical expression, a stroke included in the range specified by the specific gesture character among the inputted strokes, and sets, as a stroke corresponding to the other of the character string and the numerical expression, a stroke not included in the range specified by the specific gesture character. That is, the differentiating unit 112 may set, as the stroke corresponding to the character string, the stroke included in the range specified by the above-described determination gesture character among the inputted strokes, and may set, as the stroke corresponding to the numerical expression, the stroke not included in the range specified by the above-described determination gesture character, or may reverse the setting.
The recognition unit 114 performs the following processing: the character string is recognized by the character string recognition engine from the strokes set by the differentiating section 112 as the strokes corresponding to the character string, and the numerical expression is recognized by the numerical expression recognition engine from the strokes set by the differentiating section 112 as the strokes corresponding to the numerical expression. Here, when the distinguishing unit 112 sets the strokes included in the range specified by the fixed gesture character as the strokes corresponding to the character string, the recognition unit 114 may recognize the numerical expression using the numerical expression recognition engine, with 1 of the fixed gesture characters and the strokes included in the range specified by the fixed gesture character being regarded as 1 mark. For example, in the example shown in fig. 2A, the character strings "area", "height", and "width" may be recognized by the mathematical expression recognition engine as 1 mark such as "x", "y", and "z", respectively.
The display control unit 120 performs control to display the input stroke string on the display unit 190. Here, the display control unit 120 may perform control to display the stroke or the stroke string determined to match the specific gesture character as an image (for example, an image representing a rectangle) representing a shape corresponding to the specific gesture character. The display control unit 120 may perform control to display strokes included in the range specified by the specification gesture in a color different from strokes not included in the range specified by the specification gesture. The display control unit 120 performs control to display the numerical expression (recognition result) including the character string recognized by the recognition unit 114 on the display unit 190.
2. Method of the present embodiment
Next, the method of the present embodiment will be described with reference to the drawings.
In the case of causing a computer to recognize a numerical expression including a character string to be handwritten, it is desirable that the numerical expression and the character string be automatically separated (distinguished) from each other and recognized from each other from an input stroke string (handwritten pattern). However, it is inherently difficult to automatically distinguish between the numerical expression and the character string, and if the numerical expression and the character string are to be distinguished separately without complete distinction, the recognition rate may be significantly reduced.
Therefore, in the method of the present embodiment, a gesture character for distinguishing a character string from a numerical expression is introduced. This gesture character is referred to as a determination gesture character. Here, the determination of the gesture symbol means the following operation: when a character to be recognized or a symbol other than a numerical expression is written, a stroke included in a range specified by the gesture character is specified, and processing corresponding to the stroke is performed.
In the method of the present embodiment, a rectangle surrounding a character string is used as the determination gesture. That is, when a stroke (or a stroke string) of the handwriting input is formed in a substantially rectangular shape, it is determined that the stroke matches the specific gesture character, a character string is recognized from strokes included in a range specified by the specific gesture character, and a numerical expression is recognized from strokes not included in the range specified by the specific gesture character. The rectangle is used as the shape for determining the gesture symbol because: in general, no rectangle appears in numerical expressions, and in addition, it is natural for humans to write a rectangle in order to specify a range. In some fields of mathematics, a square or a rectangle may be used as a marker, but in this case, a marker having another shape may be used as a gesture-determining symbol instead of the rectangle. That is, the gesture character may be determined as long as the gesture character has a unique shape (is easily distinguished from other marks and symbols) and the range indicated by the gesture character is clear. For example, the shape of the gesture character may be a shape in which one or two diagonal lines of a rectangle are written from right to left (this is because if writing from left to right, it cannot be distinguished from "X"), or a shape in which two diagonal lines of a rectangle are written continuously (a shape in which "α" is inverted). Note that, although the fixed gesture character is used to distinguish the character strings included in the numerical expressions, the fixed gesture character (for example, a rectangle surrounding the numerical expressions) may be used to distinguish the numerical expressions included in the character strings.
Fig. 2A is a diagram showing an example of writing in which a numerical expression, a character string, and a fixed gesture are mixed. The registration unit 110 monitors whether or not a stroke is a fixed gesture character every time a stroke is input, and determines that the stroke is a fixed gesture characterNext, the stroke order (the order of strokes corresponding to the determined gesture character) and the coordinates (the coordinates of 4 corners of the rectangle or the coordinates of 2 opposite corners of the rectangle) of the determined gesture character are registered in the determined gesture character table. Fig. 2B is a diagram showing a determination gesture symbol table corresponding to the writing example of fig. 2A. For example, in the determination gesture character table shown in FIG. 2B, the determination gesture character BG is referred to1The coordinates of the upper left corner and the lower right corner (an example of information on the range specified by the fixed gesture character) of the character string "area" written after the character string "area" drawn 4 are registered, and the fixed gesture character BG is registered2Register with coordinates that it is written after the 1 st "x" drawing 13 and register with its upper left and lower right corners, with respect to the determination gesture BG3The coordinates of the upper left corner and the lower right corner thereof, which are written after the 2 nd "x" are registered. Referring to the specific gesture character table shown in fig. 2B, the differentiating section 112 classifies strokes surrounded by any specific gesture character into a part of a character string (strokes corresponding to the character string), and differentiates strokes other than the part of the character string into a part of a numerical expression (strokes corresponding to the numerical expression).
In the example shown in fig. 2A, the case where the definite gesture character is written in 1 stroke is shown, but it is also permissible to write the definite gesture character in multiple strokes. In this case, for convenience of processing, the maximum number of strokes (for example, 4 strokes) of the writing specification gesture symbol is defined, and it is conditioned that the stroke corresponding to the numerical expression or the character is not written from the start of writing the writing specification gesture symbol to the end of writing. If the conditions are of such a degree, the user is not greatly restricted. In addition, in the case where the specific gesture character is allowed to be written in multiple strokes, both the stroke order of the first stroke and the stroke order of the last stroke constituting the specific gesture character are registered in the specific gesture character table. Note that "order after adjustment" in the determination gesture character table of fig. 2B is described later.
In the method of the present embodiment, the process of distinguishing a character string from a numerical expression is executed regardless of the timing at which the specific gesture is written. This is because, if the order in which the gesture characters are written is designated, the user is greatly restricted, which makes the use inconvenient. As shown in fig. 3, regarding the order in which the gesture characters are determined by writing, 4 cases are considered. Fig. 3A shows a case where the specific gesture BG is initially written, and then a character string is written in the specific gesture BG, fig. 3B shows a case where a character string is initially written, and then a specific gesture BG surrounding the character string is written, fig. 3C shows a case where a character string is initially written, and then other strokes are written, and then a specific gesture BG surrounding the character string is written, and fig. 3D shows a case where a character string is additionally written in the specific gesture BG surrounding the character string that has been written.
Fig. 4 shows an example of writing of a rectangular fixed gesture character. When writing the fixed gesture BG, the user starts writing from the upper left corner of the rectangle, continues writing clockwise (see fig. 4A) or counterclockwise (see fig. 4B), and returns to the upper left corner. When at least a part of the stroke points (e.g., 1/3 or more) constituting the input stroke is included in the specific gesture character, the differentiating section 112 determines that the stroke is surrounded by the specific gesture character (the stroke is included in the range specified by the specific gesture character). This is to take into account that certain gestures written by the user sometimes intersect with other strokes (see FIG. 3).
In the method of the present embodiment, in order to recognize a specific gesture character with high accuracy, an online recognition method using time series information of a stroke and an offline recognition method using a feature of the stroke as an image are used in combination to recognize the specific gesture character. The recognition objects for recognizing the specific gesture symbol include numerical expressions (symbols such as english characters, operators such as four arithmetic operators, score symbols, root numbers, and integral symbols, and parentheses) and specific gesture symbols (bounding rectangles).
In the online recognition method, an MRF (Markov Random Field) model is used. Firstly, normalizing an input pattern to a standard size, and extracting characteristic points such as end points, angles and the like by using a Ramer-Douglas-Puck (Ramer-Douglas-Peucker) algorithm. Then, elastic matching is performed with the feature points of the MRF model. A threshold is used to evaluate the score and if the score is less than the threshold, the score is discarded.
In the off-line recognition method, MQDF (Modified Quadratic Discriminant Function) is used. By connecting the stroke point strings in time series order, the strokes can be easily transformed into images. Although time series information is lost when a stroke is converted into an image, by converting the stroke into an image, the influence of stroke order difference, rewriting, or the like can be eliminated. For this image, gradient features are extracted by a Sobel (Sobel) filter and projected in 8 directions. Let the original image be 64 × 64 pixels, which are divided into 8 × 8 partitions of the same size. To which a gaussian filter of 18 x 18 pixels is applied to blur it. Thus, the directional feature near the center contributes more and the peripheral contribution is less, and some contribution is obtained from the adjacent partitions, thereby resisting the position shift. As a result, 8-direction features are extracted for each 8 × 8 partition, and 512-dimensional features are obtained in total. Further, the dimensions are compressed by discriminant analysis of Fisher (Fisher) to obtain 256-dimensional features. Identification was performed using MQDF. Further, a threshold value is set, and if the distance to the rectangle of the input pattern is greater than the threshold value even when the distance is shortest, the threshold value is discarded.
And finally, integrating results of the online identification method and the offline identification method. Here, the online recognition method obtains a logarithmic value of the probability value (although it is a negative value, the larger the probability value is), and the offline recognition method obtains a distance (the smaller the probability value is, the better the distance is), so that it is necessary to make their dimensions uniform. Therefore, the score of the online recognition is multiplied by-1 to be a positive value (therefore, the smaller the score, the better), and the weighted sum is taken to calculate the total score. Composite scorecombinationThe calculation is performed by the following equation.
[ number 1]
scorecombination＝(-1)×w1×scoreonline+w2×scoreoffline (1)
Herein, scoreonline、scoreofflineThe scores are the score of the online identification method and the score of the offline identification method, respectively. In addition, w1、w2Are the integrated weights that are optimized by using the training pattern. In more detail, generally, scoreonlineScore of about-10 to 0offlineTake a value of more than 0 and less than 1024. Namely, -scoreonlineIs scoreofflineAbout one-hundredth of (a), and therefore, in reality, 0 < w is satisfied1＜0.1、w2＝1-w1The optimum value is obtained under the condition of (1). The registration unit 110 identifies score among objectscombinationThe smallest recognized object is recognized as the determination gesture.
3. Treatment of
Next, a flow of processing of the recognition device of the present embodiment will be described. The identification device has: an interactive device capable of displaying time-series stroke points in real time and performing feedback for recognition for each stroke, such as a tablet PC; and a non-interactive device in which a time-series stroke point is recorded by a tablet or an electronic pen having a memory, and data is read and processed by a PC or the like in a batch manner.
3-1. treatment in a sequential manner
In the sequential method employed in the interactive apparatus, a user designates a language (for example, english or japanese) for recognizing a character string, starts a system (program), and starts writing a numerical expression or a character string on an input unit such as a touch panel. The system identifies the determined gesture characters and distinguishes character strings and numerical expressions when writing strokes each time, and identifies the character strings and the numerical sequences and displays identification results after writing is finished. Fig. 5 is a flowchart showing an example of processing in the sequential method.
First, the processing unit 100 acquires a stroke input to the character input unit 160 (step S10). At this time, the display control unit 120 performs control to display the acquired stroke on the display unit 190. Next, the registration unit 110 determines whether or not the latest stroke traced from the acquired stroke to the maximum number of strokes (for example, 4 strokes) of the fixed gesture character matches the fixed gesture character (constitutes the fixed gesture character) (step S12). For example, the registration unit 110 calculates the total score for each of the latest 1 stroke, the latest 2-stroke string, the latest 3-stroke string, and the latest 4-stroke (maximum number of strokes)combinationDetermining that the gesture symbol gives the smallest score among the recognized objectscombinationAnd then recognized as a determination gesture character. However, score is not calculated for strokes discarded by either of the online recognition method and the offline recognition methodcombinationIn the case where each stroke is discarded, it is determined that the gesture is not consistent with the determination gesture.
When the determined gesture is matched (yes in step S12), the registration unit 110 registers the information for specifying the gesture (the stroke order and the coordinates of the 2 opposite corners) in the determined gesture table based on the information of the stroke determined to be matched with the determined gesture (step S14). Here, the registration unit 110 cancels the setting when it is determined that the stroke matching the determination gesture character is set as the stroke corresponding to the numerical expression before the stroke matching the determination gesture character. Further, the display control unit 120 performs control to display the image of the stroke determined to match the determination gesture character as an image of a rectangle (shape corresponding to the determination gesture character) representing a color (for example, red) different from the other strokes. This enables the user to be fed back with a determination that the gesture has been correctly recognized.
Next, the discrimination unit 112 refers to the determination gesture symbol table, and sets, as a stroke corresponding to the character string, a stroke included in the range specified by the registered determination gesture symbol (step S16). At this time, the display control unit 120 performs control to change the color of the stroke (for example, from black to blue) included in the range specified by the determination gesture. This enables the user to be fed back a case where the character strings have been correctly distinguished by the specific gesture. Further, in the case where the stroke included in the range specified by the determination gesture character does not exist, the process of step S16 is skipped.
If the determined gesture does not match the determined gesture (no in step S12), the discrimination unit 112 refers to the determined gesture list and determines whether or not the acquired stroke is included in the range specified by any of the determined gestures (step S18). When the acquired stroke is included in the range specified by any one of the fixed gestures (yes in step S18), the differentiating unit 112 sets the acquired stroke as a stroke corresponding to the character string (step S20), and the display control unit 120 performs control to change the color of the acquired stroke (for example, from black to blue). On the other hand, if the acquired stroke is not included in the range specified by the fixed gesture (no in step S18), the discrimination unit 112 sets the acquired stroke as a stroke corresponding to the numerical expression (step S22).
Next, the processing unit 100 determines whether or not a stroke is input (step S24), and if a stroke is input (yes in step S24), the process proceeds to step S10, and the process from step S10 onward is repeated until writing by the user is completed.
If it is determined that no stroke has been input (no in step S24), the recognition unit 114 recognizes the character string from the strokes set in steps S16 and S20 as the strokes corresponding to the character string by the character string recognition engine, and recognizes the numerical expression from the strokes set in step S22 as the strokes corresponding to the numerical expression by the numerical expression recognition engine (step S26). Then, the display control unit 120 performs control to display the recognition result (the numerical expression including the character string) on the display unit 190.
Here, the numerical expression recognition engine does not understand the strokes corresponding to the identified gesture characters nor the strokes corresponding to the character strings, and cannot analyze the structure of the numerical expression if there is no information on the character strings surrounded by the identified gesture characters (that is, only information on the strokes corresponding to the numerical expression). Therefore, in the method of the present embodiment, 1 specific gesture character and a character string surrounded by the specific gesture character are regarded as 1 mark (1 symbol in the numerical expression), and the numerical expression is recognized from coordinate information of the specific gesture character regarded as the mark and information of a stroke corresponding to the numerical expression. For this purpose, the stroke order of the gesture character is adjusted. Specifically, the first stroke constituting the fixed gesture character and the earlier stroke order of the first stroke corresponding to the character string surrounded by the fixed gesture character are registered in the fixed gesture character table as the adjusted stroke order of the fixed gesture character. In the example shown in FIG. 2, BG is the determined gesture symbol1、 BG2、BG3The adjusted stroke order of (2) is registered with "1", "9" and "12", respectively. In this case, the recognition unit 114 (mathematical recognition)Engine) can determine the gesture symbol BG by referring to the determined gesture symbol table1And the character string 'area' contained in the gesture recognition device is regarded as 1 mark written in the 1 st drawing, and the gesture character BG is determined2And the character string 'height' contained in the gesture recognition device is regarded as 1 mark written in the 9 th drawing, and the gesture character BG is determined3And the character string "width" included in the character string are regarded as 1 mark written on the 12 th drawing, and the structure of the numerical expression is analyzed based on the relationship between the mark and the position and size of the stroke corresponding to the numerical expression.
In the sequential system, the user may have written mistakes, and therefore, it is preferable to have an Undo (Undo) function. In this case, when the user inputs the cancel command and the latest cancelled stroke is the fixed gesture, the fixed gesture and the registration information thereof are deleted, and the stroke included in the range specified by the fixed gesture (the stroke set as the stroke corresponding to the character string) is reset as the stroke corresponding to the numerical expression. On the other hand, in the case where the latest stroke that is cancelled is a stroke corresponding to a character string or a numerical expression, the stroke is simply deleted.
Fig. 6 is a diagram showing a specific display example in the sequential mode. In the example shown in fig. 6, the user first writes the character string "value" and the numerical expression, and writes the determination gesture BG enclosing the character string "value1(FIG. 6A). Then, the registration unit 110 recognizes the fixed gesture BG1The display control unit 120 specifies the gesture symbol BG1Substitution with red symbol BG1Image BI of equally sized rectangles1(determination of gesture character image) of the determined gesture character BG1The color of the surrounding character string "value" changes to blue (fig. 6B). Next, the user writes a new definite gesture BG2(FIG. 6C). Then, the registration unit 110 recognizes the fixed gesture BG2The display control unit 120 specifies the gesture symbol BG2Transposing to determine gesture symbol image BI2(FIG. 6D). Then, when the user is determining the gesture symbol image BI2When the character string "result" is written, the display control unit 120 displays the character string "result" in blue every time a stroke is acquired (fig. 6E).Finally, when the user performs an operation to instruct execution of recognition, the recognition unit 114 recognizes a character string and a numerical expression, and the display control unit 120 displays the recognition result (fig. 6F). Furthermore, here, the gesture symbol image BI is to be determined1、BI2Removed to display the recognition result.
3-2. batch mode treatment
In the batch mode employed by non-interactive devices, the user starts the system and writes numerical expressions and character strings to an interface such as a tablet. After writing, data of the stroke string is read into a PC or the like (recognition device), and at this time, a language for recognizing the character string is specified. In the batch mode, the 3 processes of determining the recognition of the gesture character, the distinction of the character string and the number expression, the recognition of the character string and the number sequence, and the display of the recognition result are performed after the data of all the stroke strings are transferred to the PC or the like. In this case, when the input strokes are sequentially processed, the same processing as the sequential method can be performed. In the batch method, since it is not necessary to identify a gesture character and distinguish a character string from a numerical expression in real time, the following method may be adopted: after all the determined gestures are recognized, the strokes outside the determined gestures are distinguished into character strings and numerical expressions, and finally the character strings and the numerical expressions are recognized. Fig. 7 is a flowchart showing an example of processing in the latter method in the batch system.
First, the processing unit 100 acquires a stroke string input to the character input unit 160 (step S30). Next, the processing unit 100 sets the variable i to 1 (step S32). Next, the registration unit 110 determines whether or not the latest stroke traced from the ith stroke (i-th drawing) to the maximum drawing number (for example, 4 drawings) of the determination gesture symbol matches the determination gesture symbol (step S34).
When the determined gesture is matched (yes in step S34), the registration unit 110 registers the information for specifying the gesture (the stroke order and the coordinates of the 2 opposite corners) in the determined gesture table based on the information of the stroke determined to be matched with the determined gesture (step S36). In the case of non-coincidence with the determined gesture character (no at step S34), the i-th stroke is set as a non-gesture character stroke (a stroke not coincident with the determined gesture character) (step S38).
Next, the processing unit 100 increments the value of the variable i by 1 (step S40), and determines whether or not the value of the variable i is equal to or less than n (n is the total number of strokes included in the acquired stroke string) (step S42). If the value of the variable i is equal to or less than n (yes in step S42), the process proceeds to step S34, and the process from step S34 is repeated until the value of the variable i exceeds n.
When the value of the variable i exceeds n (no in step S42), the processing unit 100 sets the variable i to 1 (step S44). Next, the discrimination unit 112 refers to the determination gesture symbol table, and determines whether or not the i-th non-gesture symbol stroke set in step S38 is included in the range specified by any one of the determination gesture symbols (step S46). When the i-th non-gesture character stroke is included in the range specified by any one of the determination gestures (yes in step S46), the discrimination section 112 sets the i-th non-gesture character stroke as a stroke corresponding to the character string (step S48). On the other hand, if the i-th non-gesture character stroke is not included in the range specified by the determination gesture character (no in step S46), the differentiating section 112 sets the i-th non-gesture character stroke as a stroke corresponding to the numerical expression (step S50).
Next, the processing unit 100 increments the value of the variable i by 1 (step S52), and determines whether or not the value of the variable i is equal to or less than m (m is the total number of non-gesture character strokes) (step S54). If the value of the variable i is equal to or less than m (yes in step S54), the process proceeds to step S46, and the process from step S46 is repeated until the value of the variable i exceeds m.
When the value of the variable i exceeds m (no in step S54), the recognition unit 114 recognizes the character string from the stroke set in step S48 as the stroke corresponding to the character string using the character string recognition engine, and recognizes the numerical expression from the fixed gesture table (coordinate information of the fixed gesture regarded as a marker) and the stroke set in step S50 as the stroke corresponding to the numerical expression using the numerical expression recognition engine (step S56). Then, the display control unit 120 performs control to display the recognition result (the numerical expression including the character string) on the display unit 190.
Fig. 8 is a diagram showing a specific display example in the batch mode. In batchesIn the method, all strokes written by the user and taken into the recognition device are displayed (fig. 8A), and when the user performs an operation to instruct the execution of the recognition, the recognition of the specific gesture character, the distinction of the character string and the numerical expression, the recognition of the character string and the numerical sequence are executed, and the recognition result is displayed (fig. 8C). Further, after the recognition of the gesture character and the discrimination of the character string and the numerical expression are performed, and before the recognition of the character string and the numerical sequence is performed, the recognition result of the gesture character and the discrimination of the numerical expression and the character string may be displayed as shown in fig. 8B. In the example shown in FIG. 8B, the gesture character BG will be determined1、 BG2、BG3Respective substitution into definite gesture symbol images BI1、BI2、BI3Gesture symbol BG to be determined1、BG2、BG3The color of the surrounding character strings "area", "height", and "width" is changed to blue.
According to the present embodiment, a specific gesture character for distinguishing a character string from a numerical expression is introduced, strokes surrounded by the specific gesture character are distinguished as character strings, strokes not surrounded by the specific gesture character are distinguished as numerical expressions, and recognition is performed by a character string recognition engine and a numerical expression recognition engine, respectively, whereby character strings included in the numerical expression can be clearly distinguished from the numerical expressions and recognized, respectively. In recognizing the numerical expression, the fixed gesture character and the stroke surrounded by the fixed gesture character are regarded as 1 mark, and thereby the structural analysis of the numerical expression including the character string can be performed using the numerical recognition engine.
Further, according to the present embodiment, in the sequential mode, when the fixed gesture character is written, the process of dividing the stroke surrounded by the fixed gesture character into character strings is performed (step S16 in fig. 5), and when the written stroke is surrounded by the fixed gesture character, the process of dividing the stroke into character strings is performed (step S20 in fig. 5), whereby the character strings and the numerical expressions can be distinguished regardless of the timing at which the fixed gesture character is written, and convenience for the user can be improved. That is, according to the present embodiment, the process of distinguishing a character string from a numerical expression can be performed regardless of whether the character string is written after the character string is specified (fig. 3A), the character string is written after the character string is written (fig. 3B and 3C), or the character string is added to the specified character string (fig. 3D).
The present invention is not limited to the above-described embodiments, and various modifications can be made. The present invention includes substantially the same configurations (for example, configurations having the same functions, methods, and results or configurations having the same objects and effects) as those described in the embodiments. The present invention includes a configuration in which the immaterial portions of the configurations described in the embodiments are replaced. The present invention includes a configuration that achieves the same operational effects or the same objects as those of the configuration described in the embodiment. The present invention includes a configuration in which a known technique is added to the configuration described in the embodiment.
Description of the reference numerals
100 processing part, 110 registration part, 112 distinguishing part, 114 identification part, 120 display control part, 160 character input part, 170 storage part and 190 display part.

Claims (7)

1. A recognition apparatus for recognizing a numerical expression including a character string from a stroke string input by handwriting, comprising:
a registration unit that sequentially determines whether or not a latest stroke traced back from a stroke of the handwriting input to a maximum number of strokes for distinguishing a character string from a numeric confirmation gesture symbol coincides with the confirmation gesture symbol, and registers information on a range specified by the confirmation gesture symbol when it is determined that the latest stroke coincides with the confirmation gesture symbol;
a distinguishing unit that sets, as a stroke corresponding to one of a character string and a numerical expression, a stroke included in a range specified by the specific gesture character, among inputted strokes, and sets, as a stroke corresponding to the other of the character string and the numerical expression, a stroke not included in the range specified by the specific gesture character; and
and a recognition unit that recognizes the character string from a stroke set as a stroke corresponding to the character string by using a character string recognition engine, and recognizes the numerical expression from a stroke set as a stroke corresponding to the numerical expression by using a numerical expression recognition engine.
2. Identification device according to claim 1,
the distinguishing unit sets, when the input stroke or the stroke string is determined to match the specific gesture character, a stroke included in a range specified by the specific gesture character as a stroke corresponding to one of a character string and a numerical expression, determines, when the input stroke is determined not to match the specific gesture character, whether the input stroke is included in the range specified by the specific gesture character, and sets, when the input stroke is included in the range specified by the specific gesture character, the input stroke as a stroke corresponding to one of the character string and the numerical expression.
3. Identification device according to claim 1 or 2,
the distinguishing unit sets, as a stroke corresponding to the character string, a stroke included in a range specified by the specific gesture character, among the inputted strokes, sets a stroke not included in the range specified by the specific gesture character as a stroke corresponding to the numerical expression,
the recognition unit recognizes a stroke included in a range specified by 1 of the determination gestures as 1 marker, and recognizes a numerical expression using a numerical expression recognition engine.
4. Identification device according to claim 1 or 2,
further comprises a display control section for controlling the display section to display the inputted stroke string,
the display control unit performs control to display the stroke or the stroke string determined to match the specific gesture character as an image representing a shape corresponding to the specific gesture character.
5. Identification device according to claim 4,
the display control unit performs control to display strokes included in the range specified by the specification gesture character in a color different from strokes not included in the range specified by the specification gesture character.
6. Identification device according to claim 1 or 2,
the login unit determines that the stroke or the stroke string matches the specific gesture character when the input stroke or the stroke string forms a rectangle.
7. A computer-readable information storage medium, characterized in that,
a program for causing a computer to function as each of the above-described units of the recognition device according to any one of claims 1 to 6 is stored.
CN201680006635.XA 2015-01-21 2016-01-19 Identification device and information storage medium Expired - Fee Related CN107209862B (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2015009207 2015-01-21
JP2015-009207 2015-01-21
PCT/JP2016/051457 WO2016117564A1 (en) 2015-01-21 2016-01-19 Program, information storage medium, and recognition device

Publications (2)

Publication Number Publication Date
CN107209862A CN107209862A (en) 2017-09-26
CN107209862B true CN107209862B (en) 2021-03-09

Family

ID=56417105

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201680006635.XA Expired - Fee Related CN107209862B (en) 2015-01-21 2016-01-19 Identification device and information storage medium

Country Status (4)

Country Link
JP (1) JP6694638B2 (en)
KR (1) KR101909485B1 (en)
CN (1) CN107209862B (en)
WO (1) WO2016117564A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7104965B2 (en) * 2018-04-23 2022-07-22 国立大学法人東京農工大学 Program and clustering device
CN111079622A (en) * 2019-12-10 2020-04-28 黄淮学院 Method for miniaturizing handwritten text recognizer under unified recognition framework
CN111539383B (en) * 2020-05-22 2023-05-05 浙江蓝鸽科技有限公司 Formula knowledge point identification method and device

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3150468B2 (en) * 1993-01-07 2001-03-26 株式会社東芝 Online character recognition device
JPH09319828A (en) * 1996-05-30 1997-12-12 Nec Corp On-line character recognition device
JPH11203412A (en) * 1998-01-16 1999-07-30 Toshiba Corp Method and device for processing document image and computer readable record medium storing document image processing program
JP3827453B2 (en) * 1998-09-07 2006-09-27 積水樹脂株式会社 Sound absorbing structure
US20060001667A1 (en) * 2004-07-02 2006-01-05 Brown University Mathematical sketching
JP5098504B2 (en) * 2007-08-09 2012-12-12 富士通株式会社 Character recognition program, character recognition device, and character recognition method
CN101388068B (en) * 2007-09-12 2011-06-22 汉王科技股份有限公司 Mathematical formula identifying and coding method
JP2009080615A (en) 2007-09-26 2009-04-16 Univ Of Tokyo Online recognition method and device for handwritten mathematical expression
CN101685497B (en) * 2008-09-28 2011-10-12 汉王科技股份有限公司 Method and device for processing hand-written information
JP2011053986A (en) * 2009-09-02 2011-03-17 Sharp Corp Computing device, computing program, and recording medium recording the program
CN103679164A (en) * 2012-09-21 2014-03-26 阿里巴巴集团控股有限公司 A method and a system for identifying and processing a mark based on a mobile terminal
JP2014127188A (en) * 2012-12-27 2014-07-07 Toshiba Corp Shaping device and method
CN107066999A (en) * 2013-05-22 2017-08-18 华为终端有限公司 A kind of character recognition method and user terminal
CN104063063B (en) * 2014-06-05 2018-10-09 北京搜狗科技发展有限公司 The mathematical computations input method and input method system of input method
CN104268118A (en) * 2014-09-23 2015-01-07 赵方 Mathematical formula calculator including touch screen and method for identifying mathematical formulas

Also Published As

Publication number Publication date
KR101909485B1 (en) 2018-10-18
JP6694638B2 (en) 2020-05-20
KR20170100541A (en) 2017-09-04
WO2016117564A1 (en) 2016-07-28
CN107209862A (en) 2017-09-26
JPWO2016117564A1 (en) 2017-10-26

Similar Documents

Publication Publication Date Title
CN110232311B (en) Method and device for segmenting hand image and computer equipment
US10964057B2 (en) Information processing apparatus, method for controlling information processing apparatus, and storage medium
US7633512B2 (en) Information processing apparatus, information processing method and program
EP1768058A2 (en) Information processing apparatus and control method therefor
KR20180064371A (en) System and method for recognizing multiple object inputs
CN111626297A (en) Character writing quality evaluation method and device, electronic equipment and recording medium
US20140184610A1 (en) Shaping device and shaping method
CN108701215B (en) System and method for identifying multi-object structures
KR101559502B1 (en) Method and recording medium for contactless input interface with real-time hand pose recognition
CN107209862B (en) Identification device and information storage medium
CN114365075A (en) Method for selecting a graphical object and corresponding device
JP2008108177A (en) Handwritten character input system for displaying point such as start point, endpoint, break point and control point as featured point of stroke
GB2462903A (en) Single Stroke Character Recognition
US7295707B2 (en) Method for aligning gesture features of image
KR20220038477A (en) Extract line of text
CN111259888A (en) Image-based information comparison method and device and computer-readable storage medium
CN114511853B (en) Character image writing track recovery effect discrimination method
CN115019396A (en) Learning state monitoring method, device, equipment and medium
CN114241486A (en) Method for improving accuracy rate of identifying student information of test paper
JP3977473B2 (en) Handwritten character recognition method and handwritten character recognition apparatus
JPH0442316A (en) Electronic computer
CN115705753A (en) Gesture recognition method
JP2022041468A (en) Identification method, information processing system, method, and program
JP2023153238A (en) Digital ink generation apparatus, method, and program
KR20150137887A (en) Korean Alphabet Learning Apparatus and method using Motion Recognition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20210309