JP2014182685A - Handwritten information extraction device, handwritten information extraction method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract and output a portion intended by a user with the handwritten information of the user as an input.SOLUTION: A handwritten information extraction device for extracting and outputting a portion intended by a user with handwritten information of the user as an input includes: coordinate information detection means; time information detection means; information storage means; and information extraction means. The coordinate information detection means detects coordinates of the handwritten information of the user. The time information detection means detects the handwritten time information in which the handwritten information of the user is handwritten. The information storage means stores both of detection results by the coordinate information detection means and the time information detection means. The information extraction means extracts the portion intended by the user based on the handwritten information stored by the information storage means.

Description

  The present invention relates to a handwritten information extracting apparatus, a handwritten information extracting method, and a program that take user's handwritten information as input and extract and output a portion intended by the user.

  2. Description of the Related Art Conventionally, as one input method, a handwritten character recognition technique has been used in which a handwritten character is written with a pen or the like, and the written character is recognized and input. For example, in Patent Literature 1, handwritten information input in the input area is sequentially recognized and characters are input by handwriting by inputting the recognized characters into the input field.

  As another form of character recognition technology, a technology for recognizing text from image data of a document including text and making it into an electronic document is used. For example, Patent Document 2 realizes generation of a structured document tagged with XML or HTML from various types of printed documents.

JP 2013-025370 A JP 2005-043990 A

  However, in the handwriting input device as in Patent Document 1, all handwritten information is handled as input, and thus the user originally inputs to the input field, for example, a midway calculation formula of a calculation problem or a memo for organizing ideas. Handwritten input that is not supposed to be handled is also treated as input to the input field. In Patent Document 2, it is possible to extract only text from an image, but it is not possible to determine which part of the extracted text is the text that the user wants to extract, and it is originally input It is not supposed to be a method of entering in the field.

  Therefore, the present invention has been made in view of the above-described problems, and provides a handwritten information extraction device, a handwritten information extraction method, and a program that extract user-written information and extract a portion intended by the user. The purpose is to do.

  The present invention proposes the following matters in order to solve the above problems.

  (1) The present invention is a handwriting information extraction device that takes user's handwritten information as input, extracts and outputs a part intended by the user, and includes coordinate information detection means for detecting coordinates of the user's handwritten information; Time information detecting means for detecting time information in which the user's handwritten information is handwritten, handwritten information storage means for storing both the detected coordinates of the user's handwritten information and time information in which the user's handwritten information is handwritten, and the handwriting Proposed is a handwritten information extracting apparatus comprising: handwritten information extracting means for extracting a portion intended by a user based on handwritten information stored in an information storage means.

  (2) The present invention relates to the handwritten information extracting device of (1), comprising an imaging means, wherein the coordinate information detecting means detects coordinates of the user's handwritten information, and the time information detecting means is the user's handwritten information. Has proposed a handwritten information extracting device characterized by detecting time information handwritten.

  (3) The present invention relates to the handwritten information extraction apparatus according to (1) or (2), wherein the user's handwritten information is an addition formula, a subtraction formula, a multiplication formula, a division formula, a differential formula, an integral formula, a chemical reaction formula In this case, a handwritten information extracting device is proposed in which the handwritten information extracting means detects an equal sign and extracts the handwritten information described last after the equal sign.

  (4) In the handwritten information extraction device according to (1) or (2), the present invention provides the handwritten information extraction means when the user's handwritten information is an arithmetic operation (addition, subtraction, multiplication, division). Is the handwritten information described above the handwritten information described last or the symbol indicating the division operation (hanging symbol) below the line segment below the mathematical symbol indicating addition, subtraction, and multiplication Has proposed a handwritten information extraction device characterized by the extraction of.

  (5) The present invention is a handwritten information extraction method in a handwritten information extraction apparatus that takes user's handwritten information as input and extracts and outputs a portion intended by the user, and detects the coordinates of the user's handwritten information. A second step of detecting time information in which the user's handwritten information is handwritten, and a third step of storing both the detected coordinates of the user's handwritten information and the time information in which the user's handwritten information is handwritten. The present invention proposes a handwritten information extraction method comprising: a step; and a fourth step of extracting a portion intended by the user based on the stored handwritten information.

  (6) The present invention is a handwriting information extraction method in a handwriting information extraction device that takes user's handwriting information as input and extracts and outputs a part intended by the user, and detects the coordinates of the user's captured handwriting information. The first step, the second step of detecting time information in which the handwritten information of the user that has been imaged is detected, and the detected coordinates of the user's handwritten information and the time information in which the user's handwritten information is handwritten are stored together. And a fourth step of extracting a portion intended by the user based on the stored handwritten information, and proposes a handwritten information extracting method.

  (7) The present invention is a program for causing a computer to execute a handwriting information extraction method in a handwriting information extraction apparatus that extracts and outputs a portion intended by the user, using the user's handwriting information as input, and the user's handwriting 1st step which detects the coordinate of information, 2nd step which detects the time information which handwritten handwritten information of this user, and the time information which handwritten the coordinate of said detected user's handwritten information and user's handwritten information A program for causing a computer to execute a third step of storing both and a fourth step of extracting a portion intended by the user based on the stored handwritten information is proposed.

  (8) The present invention is a program for causing a computer to execute a handwriting information extraction method in a handwriting information extraction apparatus that takes a user's handwriting information as an input and extracts and outputs a portion intended by the user. The first step of detecting the coordinates of the handwritten information of the user, the second step of detecting the time information of handwriting the handwritten information of the captured user, the coordinates of the detected handwritten information of the user and the handwritten information of the user Proposed is a program for causing a computer to execute a third step of storing together handwritten time information and a fourth step of extracting a portion intended by the user based on the stored handwritten information. ing.

  According to the present invention, it is possible to extract a part intended by the user from the user's handwritten information, and it is necessary to extract the answer, idea, etc., finally derived, or input it to another input field. There is an effect that the labor of display and input can be reduced.

It is a figure which shows the structure of the handwritten information extraction apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of an answer extraction by the equal sign detection which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of an answer extraction by the handwriting detection which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of an answer extraction by the handwriting detection which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of answer extraction by the symbol which shows the answer which concerns on the 1st Embodiment of this invention. It is a figure which shows the answer extraction example of the choice problem which concerns on the 1st Embodiment of this invention. It is a figure which shows the process of the handwritten information extraction apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the structure of the handwritten information extraction apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the process of the handwritten information extraction apparatus which concerns on the 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Note that the constituent elements in the present embodiment can be appropriately replaced with existing constituent elements and the like, and various variations including combinations with other existing constituent elements are possible. Therefore, the description of the present embodiment does not limit the contents of the invention described in the claims.

<First Embodiment>
The handwritten information extraction apparatus according to the present embodiment will be described with reference to FIGS.

<Configuration of handwritten information extraction device>
As shown in FIG. 1, the handwritten information extraction device according to the present embodiment includes a handwritten information input unit 110, a coordinate information detection unit 120, a time information detection unit 130, a storage unit 140, and a handwriting information extraction unit 150. It is composed of

  The handwritten information input unit 110 is, for example, a touch panel of a smartphone or a tablet terminal. The coordinate information detection unit 120 detects the coordinates of the user's handwritten information. The time information detection unit 130 detects time information obtained by handwriting the user's handwritten information.

  The memory | storage part 140 memorize | stores both the coordinate of the detected user's handwritten information, and the time information which handwritten the user's handwritten information. Note that the storage unit is preferably formed of a rewritable element such as a RAM (Random Access Memory). The handwritten information extraction unit 150 extracts a part intended by the user based on the handwritten information stored in the storage unit.

<Processing of handwritten information extraction device>
Processing of the handwritten information extraction device according to the present embodiment will be described with reference to FIG.

  First, the coordinate of a user's handwritten information is detected (step S110), and the time information which handwritten the user's handwritten information is detected (step S120). Next, the coordinates of the detected handwritten information of the user and the time information of handwriting the user's handwritten information are stored together (step S130), and the part intended by the user is extracted based on the stored handwritten information (step S130). S140).

  As described above, according to the present embodiment, handwritten information of a part intended by the user can be extracted from handwritten information from a symbol representing a part intended by the user or a feature of a handwriting input interval.

<Example>
The present embodiment will be described with reference to FIGS.

  The present embodiment illustrates the case where the problem of the problem collection reproduced on the electronic device is answered. Here, the electronic device includes a display unit that can be touch-inputted with a pen such as a tablet or a finger and can display the input content, a storage unit that stores input information and problem data, a processing unit that processes a program, and the like. However, another device having an equivalent function may be used.

  The coordinates and time on the electronic device display unit when handwriting is performed on the electronic device are stored as handwritten information.

  It is handwritten input that is input without leaving time, and includes a period in which input is not performed for a certain period of time before or after, or handwritten input that is input in the vicinity, and a certain amount of handwritten input is around A group including a space that is not performed is detected as a group of handwritten input. Here, a group of handwritten input that is input last is extracted as an answer and input to the answer column.

Next, recognition processing is performed on the handwritten information to detect characters / symbols.
Specifically, a vector in the direction in which a character / symbol is input is detected from the coordinates of the detected character / symbol, and the direction of the vector is 90 ° rotated in the positive x direction and counterclockwise from the positive x direction. Is the positive y direction, and the positive x direction and the 180 ° reverse direction of the positive y direction are the negative x direction and negative y direction, respectively.

  When an equal sign (=) (equal sign 1) is detected from the handwritten information, a group of handwritten information located in the positive x direction is detected as the answer 1 from the coordinates of the equal sign 1 and input to the answer column. . When another equal sign (equal sign 2) is detected in the negative y direction from the detected coordinates of equal sign 1, a set of handwritten input located in the positive x direction of equal sign 2 is detected as answer 2. The answer 2 is overwritten in the answer field where the answer 1 is entered. Here, the equal sign may be other than “=”, “≈”, “≠”, “→” or the like, or may be another symbol read from a table defined as an equal sign (FIG. 2).

  Alternatively, the characteristic of writing calculation is detected, the portion corresponding to the answer of writing calculation is detected as an answer, and the answer is input to the answer column. Here, the characteristic of writing is that the numerical values are arranged in the vertical direction, and a set of four arithmetic operation symbols (+,-, x) and a horizontal straight line at the bottom. Or, it is a set of symbols that indicate division of division (sag droop symbol) and numerical values above and below it. The part that corresponds to the answer of writing is the group of numerical values located at the bottom of the horizontal line at the bottom or the group of numerical values located above the symbol indicating the divisional calculation (the drooping symbol) (Fig. 3, Fig. 3) 4).

  Alternatively, a group of handwritten input indicated by a symbol indicating an answer, such as circled or underlined, is detected as an answer and input to the answer column. Here, the symbol indicating the answer may be another symbol read from the answer symbol table (FIG. 5).

  Alternatively, a group of handwritten inputs located in the vicinity of a user operation such as click, tap, flick, etc. is detected as an answer and input to the answer field. Here, the user operation may be another operation read from the user operation table.

  Alternatively, when a symbol indicating an answer such as a circle is input on the option question displayed on the electronic device, the option indicated by the symbol is input to the answer column. Alternatively, when a cancellation symbol such as “×” or a cancellation symbol is input on the option question, an option in which the cancellation symbol is not input is input to the answer column. Here, the symbol indicating the answer and the cancel symbol may be another symbol read from the answer symbol and cancel symbol table (FIG. 6).

  Here, when a plurality of answers are extracted by the above-described method, the answer extracted by the user operation as the first priority, the equal sign as the second priority, the answer extracted by the symbol indicating the answer, the third priority The priority is assigned in the order of the answer extracted by detecting the writing and the answer extracted by the last input as the fourth priority, and the extraction result having a high priority is input to the answer column. This priority may be changed by the system and the user.

  Here, the extracted answer may be converted into text data using a handwritten character recognition technique and input to the answer column. Further, although an example of the electronic problem collection is described, another device having an equivalent function may be used. For example, questionnaire software and document creation software.

  As described above, according to the present embodiment, a symbol representing a part intended by the user, for example, a mathematical symbol such as equal (=), a symbol of calculation, or a characteristic symbol such as an underline or a circle Then, handwritten information intended by the user can be extracted from features of the handwriting input interval, for example, features that are separated in terms of time, such as no input for a certain period of time.

  In addition to the above, characteristic operations such as clicks and flicks may be detected, and a collection of handwritten information near the coordinates where the operations were performed may be extracted as a part intended by the user, as described above. By extracting the part intended by the user, it is possible to automatically extract and display only the part intended by the user from handwritten information including a part not intended by the user, such as a memo or intermediate calculation formula. .

<Second Embodiment>
The handwritten information extraction apparatus according to the present embodiment will be described with reference to FIGS.

<Configuration of handwritten information extraction device>
As shown in FIG. 8, the handwritten information extraction apparatus according to the present embodiment includes a coordinate information detection unit 120, a time information detection unit 130, a storage unit 140, a handwriting information extraction unit 150, and an imaging unit 160. Has been.

  The coordinate information detection unit 120 detects the coordinates of the user's handwritten information. The time information detection unit 130 detects time information obtained by handwriting the user's handwritten information.

  The memory | storage part 140 memorize | stores both the coordinate of the detected user's handwritten information, and the time information which handwritten the user's handwritten information. Note that the storage unit is preferably formed of a rewritable element such as a RAM (Random Access Memory). The handwritten information extraction unit 150 extracts a part intended by the user based on the handwritten information stored in the storage unit. The imaging unit 160 captures, for example, a user's handwritten information by imaging a blackboard, an electronic blackboard, a white board, paper, and the like.

  Specifically, a camera for optically shooting the blackboard is installed, and the blackboard is shot. The user performs handwriting on the photographed blackboard with chalk, a pen, a marker, or the like. Coordinates handwritten with chalk or the like are detected from difference information between frames before and after the photographed blackboard moving image.

Alternatively, infrared lasers may be installed at both ends of the upper side of the blackboard to monitor the blackboard surface.
The user performs handwriting on the blackboard with a stylus pen or marker, and the reflecting plate attached to the stylus pen or marker reflects the laser beam to detect handwritten coordinates.

  Alternatively, the position of the finger or stylus pen may be detected using a camera instead of the infrared laser. Specifically, feature detection is performed from the captured images, and the coordinates on the image of the finger or stylus pen shown in the two images are specified. It is possible to calculate the three-dimensional position of the finger or stylus pen from the coordinates where the camera is installed and the coordinates on the image.

  Alternatively, handwritten coordinates such as chalk, a finger, and a stylus pen may be detected by installing infrared sensors on the upper, lower, left, and right sides, running infrared rays vertically and horizontally, and detecting blocked positions. The detected handwritten coordinates and the time are stored as handwritten information.

<Processing of handwritten information extraction device>
The process of the handwritten information extraction device according to this embodiment will be described with reference to FIG.

  First, the user's handwritten information is imaged and captured, the coordinates of the user's handwritten information are detected (step S110), and time information in which the user's handwritten information is handwritten is detected (step S120). Next, the coordinates of the detected handwritten information of the user and the time information of handwriting the user's handwritten information are stored together (step S130), and the part intended by the user is extracted based on the stored handwritten information (step S130). S140).

  As described above, according to the present embodiment, the user intends based on the symbol indicating the portion intended by the user from the handwritten information written on the blackboard, the electronic blackboard, the white board, the paper, or the feature of the handwriting input interval. The handwritten information of the part to be performed can be extracted.

  The processing of the handwritten information extracting apparatus is recorded on a recording medium readable by the computer system, and the program recorded on the recording medium is read into the handwritten information extracting apparatus and executed, thereby executing the handwritten information extracting apparatus of the present invention. Can be realized. The computer system here includes an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW (World Wide Web) system is used. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiments, and includes designs and the like that do not depart from the gist of the present invention.

100; Handwritten information extraction device 110; Handwritten information input unit 120; Coordinate information detection unit 130; Time information detection unit 140; Storage unit 150; Handwritten information extraction unit 160;

Claims (8)

A handwritten information extraction device that takes user's handwritten information as input and extracts and outputs a portion intended by the user,
Coordinate information detecting means for detecting coordinates of user's handwritten information;
Time information detecting means for detecting time information in which the user's handwritten information is handwritten;
Handwritten information storage means for storing both the coordinates of the detected user handwritten information and the time information of handwriting the user's handwritten information;
Handwritten information extracting means for extracting a part intended by the user based on the handwritten information stored in the handwritten information storage means;
An apparatus for extracting handwritten information, comprising: Comprising imaging means,
2. The handwritten information extraction according to claim 1, wherein the coordinate information detecting means detects coordinates of handwritten information of the user, and the time information detecting means detects time information obtained by handwriting the user's handwritten information. apparatus.   When the user's handwritten information is an addition formula, a subtraction formula, a multiplication formula, a division formula, a differential formula, an integral formula, a chemical reaction formula, the handwritten information extraction means detects the equal sign, and the equal sign The handwritten information extracting apparatus according to claim 1, wherein the handwritten information described last is extracted after.   When the user's handwritten information is an arithmetic operation (addition, subtraction, multiplication, division), the handwritten information extraction means is at the bottom of the line segment below the mathematical symbol indicating addition, subtraction, and multiplication. The handwritten information extraction device according to claim 1 or 2, wherein the handwritten information described on the handwritten information described in (2) or a symbol (a drooping symbol) indicating divisional writing is extracted. . It is a handwritten information extraction method in a handwritten information extraction device that takes user's handwritten information as input and extracts and outputs the part intended by the user,
A first step of detecting coordinates of user's handwritten information;
A second step of detecting time information in which the user's handwritten information is handwritten;
A third step of storing both the detected coordinates of the handwritten information of the user and the time information of handwriting the user's handwritten information;
A fourth step of extracting a portion intended by the user based on the stored handwritten information;
A handwritten information extraction method characterized by comprising: It is a handwritten information extraction method in a handwritten information extraction device that takes user's handwritten information as input and extracts and outputs the part intended by the user,
A first step of detecting the coordinates of the handwritten information of the imaged user;
A second step of detecting time information obtained by handwriting the handwritten information of the imaged user;
A third step of storing both the detected coordinates of the handwritten information of the user and the time information of handwriting the user's handwritten information;
A fourth step of extracting a portion intended by the user based on the stored handwritten information;
A handwritten information extraction method characterized by comprising: A program for causing a computer to execute a handwriting information extraction method in a handwriting information extraction device that takes user's handwriting information as input and extracts and outputs a portion intended by the user,
A first step of detecting coordinates of user's handwritten information;
A second step of detecting time information in which the user's handwritten information is handwritten;
A third step of storing both the detected coordinates of the handwritten information of the user and the time information of handwriting the user's handwritten information;
A fourth step of extracting a portion intended by the user based on the stored handwritten information;
A program that causes a computer to execute. A program for causing a computer to execute a handwriting information extraction method in a handwriting information extraction device that takes user's handwriting information as input and extracts and outputs a portion intended by the user,
A first step of detecting the coordinates of the handwritten information of the imaged user;
A second step of detecting time information obtained by handwriting the handwritten information of the imaged user;
A third step of storing both the detected coordinates of the handwritten information of the user and the time information of handwriting the user's handwritten information;
A fourth step of extracting a portion intended by the user based on the stored handwritten information;
A program that causes a computer to execute.