JP5717831B2 - Electronic device and handwritten document processing method - Google Patents

Description

  Embodiments of the present invention relate to processing of handwritten documents.

  In recent years, various electronic devices such as tablets, PDAs, and smartphones have been developed. Many electronic devices of this type are equipped with a touch screen display to facilitate an input operation by a user.

  The user can instruct the electronic device to execute a function associated with the menu or object by touching the menu or object displayed on the touch screen display with a finger or the like.

  Some of such electronic devices have a function for a user to handwrite characters, figures, and the like on a touch screen display. A handwritten document (handwritten page) including such handwritten characters and figures is stored and viewed as necessary.

  By the way, an optical character recognition (OCR) technique for generating image data of a page by scanning a character handwritten on a paper page such as a notebook and recognizing the handwritten character using the image data. Is being used. With this technology, handwritten characters can be converted into character codes.

JP 2003-99713 A

  In OCR, for example, handwritten characters are recognized in order from the upper left of the scanned handwritten document, and the character code of the characters is output. Therefore, for example, character codes arranged in the order of output are displayed on the screen as a recognition result.

  However, the characters on the handwritten document may be handwritten at a position intended by the user so as to be visually recognized as a group such as a paragraph, a bullet, or a headline. Therefore, it may be expected that not only the character itself on the handwritten document but also the arrangement of the character is recognized.

  An object of one embodiment of the present invention is to provide an electronic device and a handwritten document processing method capable of converting a handwritten document including a handwritten character into a shaped document including a character code.

According to the embodiment, the electronic device includes an acquisition unit and a display control unit. The acquisition unit can acquire character codes of a plurality of handwritten characters corresponding to the plurality of lines using data of a handwritten document including a plurality of handwritten characters corresponding to the plurality of lines. When the first condition is satisfied , the display control means uses at least one first character code using a plurality of first character codes corresponding to the first row and a plurality of second character codes corresponding to the second row. Can be displayed at a position corresponding to the second line, or at least one character corresponding to the second character code can be displayed at a position corresponding to the first line. When the first condition is not satisfied , a plurality of characters corresponding to the plurality of first character codes are associated with the first row using the plurality of first character codes and the plurality of second character codes. Second formatted document data including a plurality of characters corresponding to the plurality of second character codes at a position corresponding to the second row can be displayed. Whether or not the first condition is satisfied depends on whether (1) the character code corresponding to the first character in the first row matches the character code corresponding to the first character in the second row. Or (2) whether or not the character code corresponding to the first character of the first line and the character code corresponding to the first character of the second line correspond to a symbol used in the bullets, or (3) The relationship between the horizontal position of the first row and the horizontal position of the second row in the case of horizontal writing, or (4) the vertical position of the first row in the case of vertical writing. (5) A plurality of characters corresponding to the plurality of first character codes and a plurality of characters corresponding to the plurality of second character codes correspond to a source code. Or (6) a plurality of characters corresponding to the plurality of first character codes. A plurality of characters corresponding to the character and the plurality of second character codes are determined by using at least one of whether corresponding to formula.

FIG. 2 is a perspective view illustrating an appearance of the electronic apparatus according to the embodiment. 6 is an exemplary view showing an example of a handwritten document processed by the electronic apparatus of the embodiment. The figure for demonstrating the time series information corresponding to the handwritten document of FIG. 2 preserve | saved at a storage medium by the electronic device of the embodiment. 2 is an exemplary block diagram showing the system configuration of the electronic apparatus of the embodiment. FIG. 2 is an exemplary block diagram showing the functional configuration of a digital notebook application program executed by the electronic apparatus of the embodiment. FIG. The figure which shows the example by which a handwritten document is character-recognized. FIG. 7 is an exemplary view illustrating an example in which the handwritten document of FIG. 6 is converted into a shaped document including a character code by the electronic device of the embodiment. The figure for demonstrating the line recognized from the handwritten document of FIG. The figure for demonstrating the character recognized from the handwritten document of FIG. The figure for demonstrating the group recognized from the handwritten document of FIG. The figure which shows the example by which the handwritten document containing a table | surface is character-recognized. The figure for demonstrating the group recognized from the handwritten document of FIG. FIG. 12 is an exemplary view illustrating an example in which the handwritten document of FIG. 11 is converted into a shaped document including a character code by the electronic device of the embodiment. 6 is an exemplary flowchart illustrating an example of a procedure of handwriting input processing executed by the electronic apparatus of the embodiment. 6 is an exemplary flowchart illustrating an example of a procedure of handwritten document conversion processing executed by the electronic apparatus of the embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 is a perspective view illustrating an external appearance of an electronic apparatus according to an embodiment. This electronic device is, for example, a pen-based portable electronic device that can be handwritten with a pen or a finger. This electronic device can be realized as a tablet computer, a notebook personal computer, a smartphone, a PDA, or the like. Below, the case where this electronic device is implement | achieved as the tablet computer 10 is assumed. The tablet computer 10 is a portable electronic device also called a tablet or a slate computer, and includes a main body 11 and a touch screen display 17 as shown in FIG. The touch screen display 17 is attached to be superposed on the upper surface of the main body 11.

  The main body 11 has a thin box-shaped housing. The touch screen display 17 incorporates a flat panel display and a sensor configured to detect a contact position of a pen or a finger on the screen of the flat panel display. The flat panel display may be, for example, a liquid crystal display (LCD). As the sensor, for example, a capacitive touch panel, an electromagnetic induction digitizer, or the like can be used. In the following, it is assumed that two types of sensors, a digitizer and a touch panel, are incorporated in the touch screen display 17.

  Each of the digitizer and the touch panel is provided so as to cover the screen of the flat panel display. The touch screen display 17 can detect not only a touch operation on a screen using a finger but also a touch operation on a screen using the pen 100. The pen 100 may be an electromagnetic induction pen, for example.

  The user can perform a handwriting input operation on the touch screen display 17 using an external object (the pen 100 or a finger). During the handwriting input operation, the trajectory of the movement of the external object (the pen 100 or the finger) on the screen, that is, the stroke trajectory (handwriting) handwritten by the handwriting input operation is drawn in real time. Displayed on the screen. The trajectory of the movement of the external object while the external object is in contact with the screen corresponds to one stroke. A set of many strokes corresponding to handwritten characters or figures, that is, a set of many trajectories (handwriting) constitutes a handwritten document.

  In the present embodiment, the handwritten document is stored in the storage medium as handwritten document data including time series information indicating the coordinate sequence of the trajectory of each stroke and the order relationship between the strokes, not image data. The details of this time series information will be described later with reference to FIG. 3. This time series information generally means a set of time series stroke data respectively corresponding to a plurality of strokes. Each stroke data may be any data as long as it can express a single stroke that can be input by handwriting. For example, a coordinate data series corresponding to each point on the locus of this stroke (time Series coordinates). The order of arrangement of the stroke data corresponds to the order in which the strokes are handwritten, that is, the stroke order.

  The tablet computer 10 reads arbitrary existing handwritten document data from the storage medium, and the handwritten document corresponding to the handwritten document data, that is, the handwritten document in which the trajectory corresponding to each of the plurality of strokes indicated by the time series information is drawn. Can be displayed on the screen.

  Next, the relationship between strokes (characters, marks, figures, tables, etc.) handwritten by the user and time-series information will be described with reference to FIGS. FIG. 2 shows an example of a handwritten document (handwritten character string) handwritten on the touch screen display 17 using the pen 100 or the like.

  In a handwritten document, there are many cases where another character or graphic is handwritten on the character or graphic once handwritten. In FIG. 2, the handwritten character string “ABC” is handwritten in the order of “A”, “B”, and “C”, and then the handwritten arrow is handwritten in the immediate vicinity of the handwritten character “A”. The case is envisaged.

  The handwritten character “A” is represented by two strokes (“∧” -shaped trajectory, “−”-shaped trajectory) handwritten using the pen 100 or the like, that is, two trajectories. The trajectory of the first “∧” -shaped pen 100 handwritten is sampled in real time, for example, at equal time intervals, thereby obtaining the time-series coordinates SD11, SD12,... SD1n of the “∧” -shaped stroke. Similarly, the trajectory of the “−”-shaped pen 100 to be handwritten next is also sampled, thereby obtaining the time-series coordinates SD21, SD21,... SD2n of the “−”-shaped stroke.

  The handwritten character “B” is expressed by two strokes handwritten using the pen 100 or the like, that is, two trajectories. The handwritten character “C” is represented by one stroke handwritten by using the pen 100 or the like, that is, one locus. The handwritten “arrow” is expressed by two strokes handwritten using the pen 100 or the like, that is, two trajectories.

  FIG. 3 shows time-series information 200 corresponding to the handwritten document of FIG. The time series information includes a plurality of stroke data SD1, SD2,. In the time series information 200, the stroke data SD1, SD2,..., SD7 are arranged in time series in the order of handwriting, that is, the order in which a plurality of strokes are handwritten.

  In the time series information 200, the first two stroke data SD1 and SD2 indicate two strokes of the handwritten character “A”, respectively. The third and fourth stroke data SD3 and SD4 indicate two strokes constituting the handwritten character “B”, respectively. The fifth stroke data SD5 indicates one stroke constituting the handwritten character “C”. The sixth and seventh stroke data SD6 and SD7 indicate two strokes constituting the handwritten arrow, respectively.

  Each stroke data includes a coordinate data series (time series coordinates) corresponding to one stroke, that is, a plurality of coordinates corresponding to a plurality of points on the trajectory of one stroke. In each stroke data, a plurality of coordinates are arranged in time series in the order in which the strokes are written. For example, for the handwritten character “A”, the stroke data SD1 is a coordinate data series (time series coordinates) corresponding to each point on the locus of the stroke of the “∧” shape of the handwritten character “A”, that is, n coordinates. Data SD11, SD12,... SD1n are included. The stroke data SD2 includes coordinate data series corresponding to each point on the trajectory of the stroke of the “−” shape of the handwritten character “A”, that is, n pieces of coordinate data SD21, SD22,. Note that the number of coordinate data may be different for each stroke data.

  Each coordinate data indicates an X coordinate and a Y coordinate corresponding to a certain point in the corresponding locus. For example, the coordinate data SD11 indicates the X coordinate (X11) and the Y coordinate (Y11) of the start point of the “∧” -shaped stroke. SD1n indicates the X coordinate (X1n) and Y coordinate (Y1n) of the end point of the “∧” -shaped stroke.

  Further, each coordinate data may include time stamp information T corresponding to the time when a point corresponding to the coordinate is handwritten. The handwritten time may be either absolute time (for example, year / month / day / hour / minute / second) or relative time based on a certain time. For example, the absolute time (for example, year / month / day / hour / minute / second) when the stroke is started is added to each stroke data as time stamp information, and each coordinate data in the stroke data indicates a difference from the absolute time. The relative time may be added as time stamp information T.

As described above, by using the time series information in which the time stamp information T is added to each coordinate data, the temporal relationship between the strokes can be expressed more accurately.
Information (Z) indicating writing pressure may be added to each coordinate data.

  Furthermore, in the present embodiment, as described above, the handwritten document is stored not as an image or a character recognition result but as time-series information 200 composed of a set of time-series stroke data, so it does not depend on a language. Can handle handwritten characters and figures. Therefore, the structure of the time-series information 200 according to the present embodiment can be used in common in various countries around the world with different languages.

FIG. 4 is a diagram illustrating a system configuration of the tablet computer 10.
As shown in FIG. 4, the tablet computer 10 includes a CPU 101, a system controller 102, a main memory 103, a graphics controller 104, a BIOS-ROM 105, a nonvolatile memory 106, a wireless communication device 107, an embedded controller (EC) 108, and the like. .

  The CPU 101 is a processor that controls the operation of various modules in the tablet computer 10. The CPU 101 executes various software loaded into the main memory 103 from the nonvolatile memory 106 that is a storage device. These software include an operating system (OS) 201 and various application programs. The application program includes a digital notebook application program 202. The digital notebook application program 202 has a function of creating and displaying the above-mentioned handwritten document, a function of converting the handwritten document into a formatted document including a character code, and the like.

  The CPU 101 also executes a basic input / output system (BIOS) stored in the BIOS-ROM 105. The BIOS is a program for hardware control.

  The system controller 102 is a device that connects the local bus of the CPU 101 and various components. The system controller 102 also includes a memory controller that controls access to the main memory 103. The system controller 102 also has a function of executing communication with the graphics controller 104 via a PCI Express standard serial bus or the like.

  The graphics controller 104 is a display controller that controls the LCD 17 </ b> A used as a display monitor of the tablet computer 10. A display signal generated by the graphics controller 104 is sent to the LCD 17A. The LCD 17A displays a screen image based on the display signal. A touch panel 17B and a digitizer 17C are disposed on the LCD 17A. The touch panel 17B is a capacitance-type pointing device for inputting on the screen of the LCD 17A. The touch position on the screen where the finger is touched and the movement of the touch position are detected by the touch panel 17B. The digitizer 17C is an electromagnetic induction type pointing device for inputting on the screen of the LCD 17A. The digitizer 17C detects the contact position on the screen where the pen 100 is touched, the movement of the contact position, and the like.

  The wireless communication device 107 is a device configured to perform wireless communication such as wireless LAN or 3G mobile communication. The EC 108 is a one-chip microcomputer including an embedded controller for power management. The EC 108 has a function of turning on or off the tablet computer 10 in accordance with the operation of the power button by the user.

  Next, the functional configuration of the digital notebook application program 202 will be described with reference to FIG. The digital notebook application program 202 creates, displays, and edits a handwritten document by using stroke data input by a handwriting input operation using the touch screen display 17. Also, the digital notebook application program 202 formats the handwritten document, that is, converts the handwritten characters in the handwritten document into character codes, and converts the formatted document data configured based on the character size and arrangement on the handwritten document. Can be generated.

  The digital notebook application program 202 includes, for example, a trajectory display processing unit 301, a time-series information generation unit 302, a line recognition unit 303, a character recognition unit 304, a character group recognition unit 305, a formatted document generation unit 306, and a page storage processing unit 307. A page acquisition processing unit 308, a document display processing unit 309, and the like.

  The touch screen display 17 is configured to detect the occurrence of events such as “touch”, “move (slide)”, and “release”. “Touch” is an event indicating that an external object has touched the screen. “Move (slide)” is an event indicating that the contact position has been moved while an external object is in contact with the screen. “Release” is an event indicating that an external object has been released from the screen.

  The trajectory display processing unit 301 and the time-series information generation unit 302 receive a “touch” or “move (slide)” event generated by the touch screen display 17 and thereby detect a handwriting input operation. The “touch” event includes the coordinates of the contact position. The “movement (slide)” event includes the coordinates of the contact position of the movement destination. Therefore, the trajectory display processing unit 301 and the time-series information generation unit 302 can receive a coordinate sequence corresponding to the trajectory of the movement of the contact position from the touch screen display 17.

  The trajectory display processing unit 301 receives a coordinate string from the touch screen display 17, and based on the coordinate string, the trajectory of each stroke handwritten by a handwriting input operation using the pen 100 or the like is displayed on the LCD 17A in the touch screen display 17. On the screen. The trajectory display processing unit 301 draws the trajectory of the pen 100 while the pen 100 is in contact with the screen, that is, the trajectory of each stroke, on the screen of the LCD 17A.

  The time-series information generation unit 302 receives the above-described coordinate sequence output from the touch screen display 17, and based on this coordinate sequence, generates time-series information (stroke data) having the structure described in detail in FIG. Generate. In this case, time series information, that is, coordinates and time stamp information corresponding to each point of the stroke may be temporarily stored in the work memory 401.

  The page storage processing unit 307 stores the generated time series information (time series information temporarily stored in the work memory 401) in the storage medium 402 as handwritten document data. The storage medium 402 is, for example, a storage device in the tablet computer 10.

The page acquisition processing unit 308 reads arbitrary handwritten document data already stored from the storage medium 402. The read handwritten document data is sent to the document display processing unit 309. The document display processing unit 309 analyzes the handwritten document data, and displays the trajectory of each stroke indicated by the time series information as a handwritten document (handwritten page) on the screen based on the analysis result.
With the above-described configuration, the user can create and browse a handwritten document including handwritten characters.

  The created handwritten document can be converted into a character code by character recognition. FIG. 6 shows an example in which a handwritten document (handwritten page) 61 is recognized. Character recognition converts the character in the handwritten document 61 into a character code, and the character code is output as a character recognition result 62. In the character recognition result 62, the recognized characters are arranged for each line on the handwritten document 61. For example, a single character string “summary” on the handwritten document 61 is also displayed as a single character string (character code string) “summary” on the character recognition result 62. Further, for example, a sentence “Please meet today” on the handwritten document 61 is handwritten over a plurality of lines on the handwritten document 61. On the character recognition result 62, this one sentence is broken for each line on the handwritten document 61 and displayed in a state where it is interrupted even though it is a single sentence.

  In this way, the recognized characters (character codes) are simply arranged by simply recognizing the characters of the handwritten document 61, and information on the position and size of characters in the handwritten document 61 and the structure such as paragraphs are lost. I will be broken. Therefore, for example, there is a possibility that one sentence is displayed in an interrupted state and is difficult to use as a document. Also, such a document is likely to be difficult for a user to read.

  Therefore, in the present embodiment, as shown in FIG. 7, a character code corresponding to a handwritten character included in the handwritten document 61 is recognized, and a document in which the arrangement of the recognized character code is formatted (hereinafter, also referred to as a formatted page). 63) is generated. That is, in the present embodiment, a document in which the handwritten document 61 is clarified is generated.

  In the formatted document (formatted page) 63, the character string “2012 03/28” is arranged at a position corresponding to the position on the handwritten document 61. In the formatted document 63, the indentation (indent) at the head of the two itemized items in the handwritten document 61 is maintained. Further, in the formatted document 63, the corresponding characters are arranged so that one sentence “Please contact us today ... please” is not interrupted. Further, in the formatted document 63, characters (character codes) are displayed in a size (font size) corresponding to the size on the handwritten document 61.

  As described above, in the present embodiment, the formatted document 63 is generated without losing information on the position and size of the character on the handwritten document 61 and the configuration of the group to which the character belongs. This group is a group to which a group of characters to be treated as one unit such as paragraphs, bullets, headings, tables, and mathematical formulas belongs.

  Below, the process which converts a handwritten document into a formatted document using the handwritten document data containing the time series information produced | generated by the time series information generation part 302 is demonstrated.

  First, the line recognition unit 303 recognizes a plurality of lines included in handwritten document data. For example, the line recognition unit 303 recognizes a plurality of lines each including one or more handwritten characters on the handwritten document 61 using handwritten document data. More specifically, the line recognition unit 303 recognizes a line based on a change in the coordinates of the handwritten stroke by using a plurality of stroke data corresponding to the plurality of strokes handwritten on the handwritten document 61. To do.

  In the example shown in FIG. 8, eight lines 661 to 668 on the handwritten document 61 are recognized. When characters are horizontally written in the handwritten document 61 as in the handwritten document 61, it is assumed that the user handwrites characters belonging to one line from left to right in the handwritten page. . Therefore, the line recognition unit 303 moves the contact position of the object (finger or pen 100) on the touch screen display 17 from the end of one line to the beginning of the next line from time series information (a plurality of stroke data). Detect the coordinates that indicate

More specifically, the line recognizing unit 303 uses the coordinate data series of handwritten time order, and the X coordinate (horizontal coordinate) changes greatly from right to left in the handwritten document 61. Two strokes handwritten in succession are detected. In the example shown in FIG. 8, the line recognition unit 303 uses the last coordinate data SD4n of the Nth stroke and the first coordinate data SD51 of the (N + 1) th stroke subsequent to the Nth stroke, When the absolute value of the difference between the X coordinate of the data SD4n and the X coordinate of the coordinate data SD51 is greater than or equal to the threshold value, one or more strokes up to the Nth stroke and one or more strokes from the (N + 1) th stroke Detect that they belong to different rows. That is, a row 665 up to the Nth stroke SD4n and a row 666 from the (N + 1) th stroke are detected.
Similarly, the line recognition unit 303 recognizes lines 661 to 668 in the handwritten document 61 by detecting line breaks.

  Next, the character recognition unit 304 recognizes character codes corresponding to a plurality of handwritten characters included in the recognized lines. That is, the character recognition unit 304 converts the handwritten characters into character codes by recognizing the handwritten characters included in each of the plurality of lines. The character recognition unit 304 recognizes a character corresponding to one or more strokes among a plurality of strokes in each line.

  As shown in FIG. 9, the plurality of strokes handwritten in each of the plurality of lines 661 to 668 on the handwritten document 61 are associated with one or more corresponding strokes for each recognized character. That is, as a result of character recognition by the character recognition unit 304, a plurality of strokes are divided into blocks 651, 652, 653, 654,. For example, a plurality of strokes corresponding to “2012 03/28” handwritten in the row 661 are “2”, “0”, “1”, “2”, “0”, “3”, “2”, And a block for each character corresponding to “8”.

  The character recognition unit 304 uses, for example, one or more stroke data corresponding to one or more strokes (strokes to be processed) among a plurality of strokes, and the first feature amount indicating the shape of the one or more strokes Is calculated. Then, the character recognition unit 304 detects characters having a feature amount similar to the calculated first feature amount using character dictionary data stored in advance in the storage medium 402. In the character dictionary data, for example, a plurality of characters and a plurality of feature amounts corresponding to the plurality of characters are defined. Therefore, the character recognizing unit 304 recognizes a character having the second feature amount whose similarity to the calculated first feature amount is equal to or greater than a threshold value from a plurality of characters defined in the character dictionary data. Thus, handwritten characters in the block to be processed are converted into character codes.

  Note that the character recognition unit 304 may detect a plurality of character candidates having a feature amount whose similarity to the first feature amount is greater than or equal to a threshold value from a plurality of characters defined in the character dictionary data. In that case, the character recognition unit 304 is recognized from, for example, language dictionary data indicating the co-occurrence probabilities of words and characters, and strokes in the vicinity of the processing target stroke (for example, the left and right strokes of the processing target stroke). Based on the characters, the likely characters (characters with high likelihood) are narrowed down from the detected plurality of character candidates with respect to the stroke to be processed. As a result, the character recognition unit 304 recognizes a character corresponding to the stroke to be processed.

  The character recognition unit 304 further calculates a font size for displaying the character (character code) based on the recognized character size. The character recognizing unit 304, for example, based on the size of a plurality of handwritten characters included in each of the plurality of rows recognized by the line recognizing unit 303 (for example, the average of the sizes of the plurality of handwritten characters) The font size for displaying a plurality of character codes is calculated. The character recognizing unit 304 determines that the size of the plurality of handwritten characters included in each of the plurality of lines is within the first range (that is, the size variation is within a certain range). ), The average of the sizes of a plurality of handwritten characters may be set to a font size for displaying a plurality of character codes. For example, the character recognizing unit 304 determines that the size of the handwritten character “charge” 655 and the size of the handwritten character “te” 656 shown in FIG. 9 are within the first range, and these characters 655 and 656. One font size associated with is calculated.

  More specifically, the character recognition unit 304 sets the first font size based on the sizes of the plurality of first handwritten characters included in the first line among the plurality of lines recognized by the line recognition unit 303. calculate. In addition, the character recognition unit 304 calculates a second font size based on the sizes of the plurality of second handwritten characters included in the second line of the plurality of lines. Through such processing, the character recognition unit 304 determines the font size for displaying the character code. Note that the character recognition unit 304 may determine the font size for each group, which will be described later, instead of for each row.

  Next, the character group recognition unit 305 and the formatted document generation unit 306 recognize the group indicating the configuration in the handwritten document 61, and the character codes (characters recognized by the character recognition unit 304) arranged based on the recognized group. Coded document data including code) is generated. The character group recognition unit 305 and the formatted document generation unit 306, for example, have the first condition among the first line among the plurality of lines recognized by the line recognition unit 303 and the second line following the first line. When satisfying, using a plurality of first character codes corresponding to the first line and a plurality of second character codes corresponding to the second line, characters corresponding to at least one first character code are set in the second line. It is possible to generate the first formatted document data including the character corresponding to the first character or the character corresponding to the at least one second character code at the position corresponding to the first line. Therefore, for example, when the first line and the second line satisfy the first condition, the character group recognition unit 305 and the formatted document generation unit 306 have a line break between the plurality of first character codes and the plurality of second character codes. First formatted document data is generated without inserting a code. Further, when the first line and the second line do not satisfy the first condition, the character group recognition unit 305 and the formatted document generation unit 306 use a plurality of first character codes and a plurality of second character codes, Second formatted document data including a plurality of characters corresponding to the plurality of first character codes at a position corresponding to the first line and a plurality of characters corresponding to the plurality of second character codes at a position corresponding to the second line. Can be generated. Therefore, for example, when the first line and the second line do not satisfy the first condition, the character group recognizing unit 305 and the formatted document generating unit 306 are arranged between the plurality of first character codes and the plurality of second character codes. A line feed code is inserted to generate second formatted document data.

  More specifically, the character group recognizing unit 305 includes a plurality of lines recognized by the line recognizing unit 303 and a plurality of characters (character codes) recognized by the character recognizing unit 304 in the handwritten document 61. Recognize the group that shows the configuration This group is a group to which a group of characters to be treated as one unit such as paragraphs, bullets, headings, tables, and mathematical formulas belongs.

  For example, the character group recognizing unit 305 determines that the first line and the second line when the first line of the recognized lines and the second line subsequent to the first line satisfy the first condition. Line is included in a group of paragraphs, and when the first line and the second line do not satisfy the first condition, the first line and the second line are included in one paragraph. Recognize that it is not in a group (a separate paragraph group).

  This first condition is, for example, that the horizontal position of the first line and the horizontal position of the second line are aligned in the handwritten document 61. The character group recognition unit 305, for example, in the handwritten document 61, the horizontal position (X coordinate) of the first line (for example, the first character of the first line) and the second line (for example, the first line of the second line). If the difference between the horizontal position (X coordinate) of the first character and the second character is less than the threshold value, it is recognized that the first line and the second line are included in one paragraph group. When the difference between the horizontal position (X coordinate) of the first row and the horizontal position (X coordinate) of the second row is equal to or greater than the threshold value, the character group recognition unit 305 Recognize that the line and the second line are not included in one paragraph group.

  The character group recognition unit 305 and the formatted document generation unit 306 can also convert a handwritten document including itemized items into a formatted document. The character group recognition unit 305 and the formatted document generation unit 306, for example, in the first character code of a plurality of first character codes corresponding to the first line in the handwritten document and the second line following the first line. When the first character code of the corresponding plurality of second character codes is a specific character code (third character code), using the plurality of first character codes and the plurality of second character codes, Second formatted document data including a plurality of characters corresponding to the plurality of first character codes at a position corresponding to the first line and a plurality of characters corresponding to the plurality of second character codes at a position corresponding to the second line. Is generated. In addition, the character group recognition unit 305 and the formatted document generation unit 306 are configured such that the first character code of the plurality of first character codes and the first character code of the plurality of second character codes are the third character code. If not, whether a character corresponding to at least one of the plurality of first character codes is included in a position corresponding to the second line using the plurality of first character codes and the plurality of second character codes Or third formatted document data including a character corresponding to at least one of the plurality of second character codes at a position corresponding to the first line.

  More specifically, the character group recognizing unit 305 further recognizes that the recognized paragraph group is a group including a plurality of lines constituting the itemized list. For example, in the handwritten document 61, the character group recognition unit 305 includes a character code of the first character of the first line of the plurality of lines, and a character code of the first character of the second line following the first line. Is a specific character code (third character code), it is recognized that the first line and the second line are included in one bulleted group. Also, the character group recognition unit 305 determines whether the first line and the second line are different when the character code of the first character in the first line and the character code of the first character in the second line are not specific character codes. Recognize that it is not included in a single bulleted group. This specific character is, for example, a character code corresponding to a symbol or character that is specified to be used in a bulleted list, such as “•”, “□”, and “◯”.

FIG. 10 shows groups 671 to 674 recognized in the handwritten document 61.
The character group recognition unit 305 displays the X coordinate of the first character “2” of the character string “2012 03/28” corresponding to the line 661 shown in FIG. 8 and the first character of the character string “summary” corresponding to the line 662. Since the difference between the character “MA” and the X coordinate (absolute value of the difference) is equal to or greater than the threshold value, line 661 and line 662 are recognized as separate paragraph groups 671 and 672. Also, the character group recognition unit 305 prepares the character string “Today's ...” corresponding to the row 665 shown in FIG. 8, the character string “decided ...” corresponding to the row 666, and the character string “corresponding to the row 667. .. ”And the difference between the X coordinates (absolute value of the difference) between the first characters of the character string“ sai ”corresponding to the line 668 is less than the threshold value, so that the lines 665, 666, 667, 668 are This is recognized as one paragraph group 674. Thereby, the character group recognition unit 305 can recognize the paragraph group 674 corresponding to the paragraph in the handwritten document 61.

  Furthermore, the character group recognizing unit 305 has the character code “□” of the first character of the line 663 and the character code “□” of the first character of the line 664 shown in FIG. These rows 663 and 664 are recognized as one itemized group 673. Thereby, the character group recognition unit 305 can recognize the group 673 corresponding to the itemized list in the handwritten document 61.

  Based on the group recognized by the character group recognition unit 305, the formatted document generation unit 306 generates formatted document data in which a plurality of characters (character codes) recognized by the character recognition unit 304 are arranged. The formatted document generation unit 306 generates formatted document data in which the recognized character codes are arranged on the formatted document 63 based on the positions of the recognized groups 671 to 674 on the handwritten document 61. The formatted document generation unit 306 recognizes the formatted document data by inserting a line feed code between a plurality of lines in a bulleted group without inserting a line feed code between a plurality of lines in the paragraph group. Insert a line feed code between specified groups.

  The formatted document generation unit 306 associates the font size calculated by the character recognition unit 304 with the character code included in the formatted document data. For example, the formatted document generation unit 306 associates the first font size with the plurality of first character codes corresponding to the first line among the plurality of recognized lines in the generated formatted document data, and sets the second line to the second line. The second font size is associated with the corresponding plurality of second character codes. Thereby, in the formatted document 63, the corresponding character code can be displayed with the font size based on the size of the character on the handwritten document 61.

  As illustrated in FIG. 7, the formatted document generation unit 306 arranges the recognized character codes on the formatted document 63 based on the positions of the recognized groups 671 to 674 on the handwritten document 61. The formatted document generation unit 306 arranges a character code string “2012 03/28” at a position on the formatted document 63 corresponding to the position of the paragraph group 671 on the handwritten document 61. The formatted document generation unit 306 arranges a character code string “Summary” at a position on the formatted document 63 corresponding to the position of the paragraph group 672 on the handwritten document 61. In addition, the formatted document generation unit 306 adds a character code string corresponding to the itemized list including the two items indicated by “□” at the position on the formatted document 63 corresponding to the position of the group 673 on the handwritten document 61. It is arranged. At that time, the formatted document generation unit 306 inserts a line feed code between two lines included in the bulleted group 673 so that the positions of the first character codes of the two lines are aligned (that is, two items). The same number of spaces (blank character code) is inserted at the beginning of each line (so that the indent at the beginning of the line is maintained).

  Further, the formatted document generation unit 306 arranges a character code string “Please meet today” in the position on the formatted document 63 corresponding to the position of the paragraph group 674 on the handwritten document 61. The formatted document generation unit 306 inserts a line feed code between the four character code strings because four character code strings corresponding to the four lines of character strings on the handwritten document 61 are included in one paragraph group 674. Not done. As a result, in the formatted document 63, the corresponding character code can be arranged so that one sentence “Please meet with us today ...” is not interrupted.

  The page storage processing unit 307 stores the generated formatted document data in the storage medium 402.

  The page acquisition processing unit 308 reads arbitrary formatted document data already stored from the storage medium 402. The read formatted document data is sent to the document display processing unit 309. The document display processing unit 309 analyzes the formatted document data, and displays on the screen a formatted document (formatted page) in which characters indicated by the character codes are arranged with an associated font size based on the analysis result. .

  Next, a case where the handwritten document includes a handwritten table will be described with reference to FIGS. 11, 12, and 13.

  FIG. 11 shows an example in which a handwritten document 71 including a handwritten table 711 is character-recognized. In this handwritten table 711, vertical lines and horizontal lines that clearly indicate the table are not handwritten, but a plurality of items are arranged in the vertical direction and the horizontal direction to display a table of 4 rows × 4 columns. ing.

  With only character recognition, the characters in the handwritten document 71 are converted into character codes without considering the groups shown in Table 711, and the character codes are output as the character recognition result 72. In the character recognition result 72, the recognized characters are arranged for each line on the handwritten document 71.

  For example, on the handwritten document 71, the character strings “June”, “July”, and “August” are respectively handwritten in three columns in the table 711. However, on the character recognition result 72, the columns in the table 711 are not considered, and the character strings in the three columns are displayed as continuous character strings “June, July, August”.

  Similar to the example shown in FIG. 6, the recognized characters (character codes) are simply arranged by simply recognizing the handwritten document 71. For example, the items in the table 711 are displayed in a linked state. Will be. That is, information regarding the configuration in Table 711 is lost.

  Therefore, the character group recognition unit 305 recognizes the group in the table 711 included in the handwritten document 71. In the following description, it is assumed that the lines and characters in the handwritten document 71 have been recognized by the line recognition unit 303 and the character recognition unit 304.

  The character group recognition unit 305 detects a space between a plurality of handwritten character strings in each of a plurality of lines recognized by the line recognition unit 303. For example, in the line including “June July August” in the handwritten document 71, the character group recognition unit 305, the space between “June” and “July”, “July” and “8 Detects a space between "month". Then, the character group recognition unit 305 detects whether or not the detected blank is at the same horizontal position across a plurality of lines. For example, the character group recognition unit 305 determines the horizontal position of the space (first space) between “June” and “July” and the space between “4” and “10” ( When the horizontal position of (second blank) is within the predetermined range (first range), it is recognized that “June” and “4” are in the first column. , "July" and "10" are recognized in the second column. More specifically, the character group recognizing unit 305 determines that the absolute value of the difference between the leftmost X coordinate of the first blank and the leftmost X coordinate of the second blank is within a threshold value, and the right edge of the first blank If the absolute value of the difference between the X-coordinate of X and the X-coordinate of the right end of the second blank is within the threshold, it is recognized that “June” and “4” are in the first column. , "July" and "10" are recognized in the second column.

  Accordingly, as shown in FIG. 12, in the table 711 in the handwritten document 71, the position of the blank 73S in each line is within a predetermined range over a plurality of lines, so the columns 73A, 73B, 73C and 73D are recognized. Thus, the character group recognition unit 305 can recognize the table group 73 in the handwritten document 71.

  The formatted document generation unit 306 is a format in which the character code recognized from the handwritten characters in the table 711 is arranged at the position on the formatted document (formatted page) 74 corresponding to the position of the group 73 of the table on the handwritten document 71. Generate document data. Based on the positions of the columns 73A, 73B, 73C, and 73D in the table group 73, the formatted document generation unit 306 assigns the character codes corresponding to the characters (character strings) in the table 711 to the same horizontal direction. (That is, align the left ends of a plurality of items in each column).

  As shown in FIG. 13, in the formatted document 74, items in the handwritten table 711 are arranged in consideration of columns. For example, the heads of the items “June”, “4 pieces”, “6 pieces”, and “11 pieces” that belong to the same column 73B in the table 711 are aligned. As a result, the formatted document 74 can be generated from the handwritten document 71 without losing information on the structure of the handwritten table 711.

  In the present embodiment, a handwritten document including a mathematical expression and a program source code may be converted into a formatted document. In the formula and program source code, if the line structure on the handwritten document is changed, the contents (interpretation) of the formula and the program may change. Therefore, it is desirable to maintain lines on the handwritten document in the mathematical formulas and program source code.

  For example, when a plurality of first character codes corresponding to the first line among a plurality of lines included in a handwritten document include a character code of a predetermined mathematical symbol, the character group recognition unit 305 first Are recognized as a group of mathematical expressions. The formatted document generation unit 306 generates formatted document data in which a plurality of character codes corresponding to a group of mathematical expressions are arranged at a position on the formatted document corresponding to the position of the group of mathematical expressions on the handwritten document. Therefore, for example, when the plurality of first character codes include a character code of a predetermined mathematical symbol, the formatted document generation unit 306 positions a plurality of characters corresponding to the plurality of first character codes in the first line. And the formatted document data (second formatted document data) including the plurality of characters corresponding to the plurality of second character codes corresponding to the second line following the first line at the position corresponding to the second line. . For example, in the formatted document data, the formatted document generation unit 306 includes a plurality of first character codes corresponding to the first line and a plurality of second character codes corresponding to the second line following the first line. Insert a line feed code between them.

In addition, the character group recognition unit 305 uses, for example, description specification data indicating specifications related to the description of a predetermined programming language, and a plurality of character codes corresponding to the first line among the plurality of lines included in the handwritten document are stored. Determine whether the source code of the program. In this description specification data, for example, a symbol (character code of the symbol) used for describing the source code in the predetermined programming language or a character string (for example, a name used as a class, method, data type, function, etc.) Character codes corresponding to) are specified. The description specification data is stored in advance in the storage medium 402, for example. When the plurality of character codes are program source codes, the character group recognition unit 305 recognizes the first line as one source code group. The formatted document generation unit 306 generates formatted document data in which a plurality of character codes corresponding to the source code group are arranged at positions on the formatted document corresponding to the positions of the source code group on the handwritten document. . Therefore, for example, when the plurality of first character codes are the source code of the program, the formatted document generation unit 306 includes a plurality of characters corresponding to the plurality of first character codes at positions corresponding to the first line, Formatted document data (second formatted document data) including a plurality of characters corresponding to a plurality of second character codes corresponding to the second row following the row at positions corresponding to the second row is generated. In this formatted document data, the formatted document generation unit 306 includes, for example, a plurality of first character codes corresponding to the first line and a plurality of second character codes corresponding to the second line following the first line. Insert a line feed code between them.
As described above, in the handwritten document including the mathematical formula and the program source code, the formatted document data in which the lines of the mathematical formula and the program source code on the handwritten document can be generated.

  In the above description, the processing in a horizontally written document in which a plurality of characters are handwritten in the horizontal direction has been described. However, the above processing is also applied to a vertically written document in which a plurality of characters are handwritten in the vertical direction. Can do. In that case, the line recognition unit 303 recognizes a vertically written line from the handwritten document. The character recognition unit 304 converts the handwritten character included in the recognized vertical writing line into a character code. Then, the character group recognition unit 305 and the formatted document generation unit 306 recognize the group indicating the configuration of the vertically written handwritten document using the recognized vertical writing line and the character code, and based on the recognized group. Formatted document data including vertically arranged character codes (vertically written document data) is generated.

  Further, in the above description, an example in which handwritten document data including time-series information (stroke data) is converted into formatted document data has been described. However, scanning characters printed on paper pages or handwritten characters is scanned. Thus, the image data of the page may be generated, and the image data may be converted into formatted document data. The line recognition unit 303 recognizes a plurality of lines on the image using the image data. The character recognition unit 304 uses the image data to convert handwritten characters included in the recognized lines into character codes. The character group recognition unit 305 and the formatted document generation unit 306 recognize a group indicating the configuration in the image (handwritten document), and generate formatted document data including a character code arranged based on the recognized group. .

  Next, an example of a procedure of handwriting input processing executed by the digital notebook application 202 will be described with reference to FIG.

  First, the trajectory display processing unit 301 displays a trajectory (stroke) of the movement of the pen 100 or the like by a handwriting input operation on the display 17A (block B11). In addition, the time-series information generation unit 302 generates the above-described time-series information (a plurality of stroke data arranged in time-series order) based on the coordinate sequence corresponding to the locus by the handwriting input operation (block B12). The time series information generation unit 302 may temporarily store the time series information in the work memory 401. The page storage processing unit 307 may store the time series information generated by the time series information generation unit 302 (time series information temporarily stored in the work memory 401) as handwritten document data in a storage medium.

  FIG. 15 shows an example of a procedure of handwritten document conversion processing executed by the digital notebook application 202.

  First, the line recognition unit 303 recognizes a plurality of lines from a plurality of strokes using the generated time-series information (handwritten document data) (block B21). The character recognition unit 303 recognizes a plurality of characters from a plurality of strokes (block B22). For example, the character recognition unit 303 converts each handwritten character into a character code. By this character recognition, a plurality of strokes are divided into blocks for each character.

  Next, the group recognition unit 305 recognizes a group of characters such as paragraphs, bullets, headings, tables, and mathematical expressions based on the recognized lines and characters (block B23). Then, the formatted document generation unit 306 generates a formatted page in which characters (character codes) are arranged based on the recognized group (block B24).

  As described above, according to the present embodiment, a handwritten document including handwritten characters can be converted into a formatted document including character codes. The line recognition unit 303 recognizes a plurality of lines each including a plurality of handwritten characters on the handwritten document 61 using the handwritten document data. The character recognition unit 304 converts a plurality of handwritten characters included in each of the recognized plurality of lines into a plurality of character codes. Then, the character group recognition unit 305 and the formatted document generation unit 306 recognize a group indicating the configuration in the handwritten document 61, and generate formatted document data including character codes arranged based on the recognized group.

  As a result, the structure (letter size, column position, etc.) that shows handwritten conversion (clear text) from a handwritten document containing characters handwritten with a device such as a digital notebook to a document file (formatted document data). This can be realized without any loss and without any work by the user.

  Note that all the processing procedures of the present embodiment described in the flowcharts of FIGS. 14 and 15 can be executed by software. For this reason, the same effect as this embodiment can be easily realized only by installing and executing this program on a normal computer through a computer-readable storage medium storing the program for executing this processing procedure. .

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  17A ... LCD, 202 ... Digital notebook application, 301 ... Trajectory display processing unit, 302 ... Time series information generation unit, 303 ... Line recognition unit, 304 ... Character recognition unit, 305 ... Character group recognition unit, 306 ... Formatted document generation 307 ... Page storage processing unit, 308 ... Page acquisition processing unit, 309 ... Document display processing unit, 401 ... Work memory, 402 ... Storage medium

Claims (24)

An acquisition means capable of acquiring character codes of a plurality of handwritten characters corresponding to the plurality of lines using data of a handwritten document including a plurality of handwritten characters corresponding to the plurality of lines;
When the first condition is satisfied, a character corresponding to at least one first character code is obtained by using a plurality of first character codes corresponding to the first row and a plurality of second character codes corresponding to the second row. The first formatted document data including the character corresponding to the second line or including the character corresponding to at least one second character code in the position corresponding to the first line can be displayed.
When the first condition is not satisfied, a plurality of characters corresponding to the plurality of first character codes are associated with the first row using the plurality of first character codes and the plurality of second character codes. Display control means capable of displaying second formatted document data including a plurality of characters corresponding to the plurality of second character codes in a position corresponding to the second row,
Whether or not the first condition is satisfied is as follows: (1) The character code corresponding to the first character of the first line is different from the character code corresponding to the first character of the second line. Or (2) whether or not the character code corresponding to the first character of the first line and the character code corresponding to the first character of the second line correspond to a symbol used in the bullets, or (3) The relationship between the horizontal position of the first row and the horizontal position of the second row in the case of horizontal writing, or (4) the vertical position of the first row in the case of vertical writing. (5) A plurality of characters corresponding to the plurality of first character codes and a plurality of characters corresponding to the plurality of second character codes correspond to a source code. Or (6) a plurality of characters corresponding to the plurality of first character codes. An electronic device defined by using at least one of whether or not a plurality of characters and a plurality of characters corresponding to the plurality of second character codes correspond to mathematical expressions. The display control means includes
If the it is starve below threshold in the first row of the horizontal position and horizontal position of the second row, between the second character codes said plurality of first character codes of the plurality Without inserting a line feed code
When the difference between the horizontal position of the first row and the horizontal position of the second row is equal to or greater than the threshold value, the interval between the plurality of first character codes and the plurality of second character codes. The electronic device according to claim 1, wherein a line feed code is inserted in The display control means includes
When the first character code of the plurality of first character codes and the first character code of the plurality of second character codes are a third character code, the plurality of first character codes and the Insert a line feed code between multiple second character codes,
When the first character code of the plurality of first character codes and the first character code of the plurality of second character codes are not the third character code, the plurality of first character codes and the The electronic device according to claim 1, wherein no line feed code is inserted between the plurality of second character codes. The display control means further includes a plurality of the plurality of display blanks when the horizontal position of the first blank in the first row and the horizontal position of the second blank in the second row are within a first range. The character code of the character that follows the first space in the first character code and the character code of the character that follows the second space in the plurality of second character codes are the same horizontal The electronic device according to claim 1, wherein the second formatted document data arranged at the position in the direction can be displayed. The acquisition means further determines a first font size based on the sizes of the plurality of first handwritten characters in the first row, and a second font based on the sizes of the plurality of second handwritten characters in the second row. Determine the size,
Before Symbol the first font size associated with the plurality of first character codes, the plurality of second the character code 請 Motomeko 1 wherein the second font size that is associated with the electronic device.   The display control means further includes, when the plurality of first character codes include a character code of a predetermined mathematical symbol, a plurality of characters corresponding to the plurality of first character codes corresponding to the first row. 2. The electronic device according to claim 1, wherein the second formatted document data including a plurality of characters corresponding to the plurality of second character codes at a position corresponding to the second row can be displayed.   The display control means further determines whether or not the plurality of first character codes are program source codes using data indicating specifications relating to a description of a predetermined program language, and the plurality of first characters When the code is a source code of a program, the plurality of characters corresponding to the plurality of first character codes are included at positions corresponding to the first line, and the plurality of characters corresponding to the plurality of second character codes are The electronic device according to claim 1, wherein the second formatted document data included in the position corresponding to the second row can be displayed. A touch screen display;
The electronic device according to claim 1, wherein the data of the handwritten document includes a plurality of stroke data corresponding to a plurality of strokes based on a handwriting input operation using the touch screen display. Using data of a handwritten document including a plurality of handwritten characters corresponding to a plurality of lines, obtaining character codes of a plurality of handwritten characters corresponding to the plurality of lines,
When the first condition is satisfied, a character corresponding to at least one first character code is obtained by using a plurality of first character codes corresponding to the first row and a plurality of second character codes corresponding to the second row. Displaying the first formatted document data including at a position corresponding to the second line, or including a character corresponding to at least one second character code at a position corresponding to the first line;
When the first condition is not satisfied, a plurality of characters corresponding to the plurality of first character codes are associated with the first row using the plurality of first character codes and the plurality of second character codes. Displaying second formatted document data including a plurality of characters corresponding to the plurality of second character codes in a position corresponding to the second line,
Whether or not the first condition is satisfied depends on whether (1) the character code corresponding to the first character in the first row matches the character code corresponding to the first character in the second row. Or (2) whether or not the character code corresponding to the first character of the first line and the character code corresponding to the first character of the second line correspond to a symbol used in the bullets, or (3) The relationship between the horizontal position of the first row and the horizontal position of the second row in the case of horizontal writing, or (4) the vertical position of the first row in the case of vertical writing. (5) A plurality of characters corresponding to the plurality of first character codes and a plurality of characters corresponding to the plurality of second character codes correspond to a source code. Or (6) a plurality of characters corresponding to the plurality of first character codes. The method in which a plurality of characters corresponding to the second character code of the character and the plurality is determined using at least one of whether corresponding to formula. The first formed document data be displayed, when the a first row horizontal direction than a position said starve threshold and horizontal position of the second row of the said plurality of first character codes Do not insert a line feed code between multiple second character codes,
The second formatted document data is displayed when the horizontal position of the first row and the horizontal position of the second row are equal to or greater than the threshold value, and the plurality of first character codes The method according to claim 9, wherein a line feed code is inserted between the plurality of second character codes. The display of the second formatted document data means that a first character code of the plurality of first character codes and a first character code of the plurality of second character codes are a third character code. If there is, a line feed code is inserted between the plurality of first character codes and the plurality of second character codes,
Displaying the first formatted document data means that a first character code of the plurality of first character codes and a first character code of the plurality of second character codes are the third character code. The line feed code is not inserted between the plurality of first character codes and the plurality of second character codes if not. Displaying the second formatted document data further means that the horizontal position of the first blank in the first row and the horizontal position of the second blank in the second row are within the first range. The character code of the character that follows the first space in the plurality of first character codes, and the character code of the character that follows the second space in the plurality of second character codes. 10. The method according to claim 9, wherein the second formatted document data arranged at the same horizontal position is displayed. The obtaining further determines a first font size based on the sizes of the plurality of first handwritten characters in the first row, and second based on the sizes of the plurality of second handwritten characters in the second row. Determine the font size,
Before Symbol the plurality of first character code associated the first font size, wherein the plurality of second character codes 請 Motomeko 9 The method according the second font size is that associated with it.   The display of the second formatted document data further includes a plurality of characters corresponding to the plurality of first character codes when the plurality of first character codes include a character code of a predetermined mathematical symbol. The method according to claim 9, wherein second formatted document data including a plurality of characters corresponding to the plurality of second character codes at a position corresponding to the second line is displayed at a position corresponding to one line.   Displaying the second formatted document data further determines whether or not the plurality of first character codes are program source codes using data indicating specifications relating to a description of a predetermined programming language; When the plurality of first character codes are program source codes, a plurality of characters corresponding to the plurality of first character codes are included at positions corresponding to the first line, and correspond to the plurality of second character codes. The method according to claim 9, wherein second formatted document data including a plurality of characters to be displayed at a position corresponding to the second line is displayed.   The method according to claim 9, wherein the data of the handwritten document includes a plurality of stroke data corresponding to a plurality of strokes based on a handwriting input operation using a touch screen display. A program executed by a computer, wherein the program is
A procedure for acquiring character codes of a plurality of handwritten characters corresponding to the plurality of lines using data of a handwritten document including a plurality of handwritten characters corresponding to the plurality of lines;
When the first condition is satisfied, a character corresponding to at least one first character code is obtained by using a plurality of first character codes corresponding to the first row and a plurality of second character codes corresponding to the second row. A step of displaying first formatted document data including at a position corresponding to the second line, or including a character corresponding to at least one second character code at a position corresponding to the first line;
When the first condition is not satisfied, a plurality of characters corresponding to the plurality of first character codes are associated with the first row using the plurality of first character codes and the plurality of second character codes. Causing the computer to execute a procedure for displaying second formatted document data including a plurality of characters corresponding to the plurality of second character codes in a position corresponding to the second row,
Whether or not the first condition is satisfied depends on whether (1) the character code corresponding to the first character in the first row matches the character code corresponding to the first character in the second row. Or (2) whether or not the character code corresponding to the first character of the first line and the character code corresponding to the first character of the second line correspond to a symbol used in the bullets, or (3) The relationship between the horizontal position of the first row and the horizontal position of the second row in the case of horizontal writing, or (4) the vertical position of the first row in the case of vertical writing. (5) A plurality of characters corresponding to the plurality of first character codes and a plurality of characters corresponding to the plurality of second character codes correspond to a source code. Or (6) a plurality of characters corresponding to the plurality of first character codes. Characters and the program in which a plurality of characters corresponding to the plurality of second character codes are determined by using at least one of whether corresponding to formula. The procedure for first displaying the formed document data, if the it is starve below threshold in the first row of the horizontal position and the position of the second row in the horizontal direction, the plurality of first character codes And no line feed code between the plurality of second character codes,
The procedure for displaying the second formatted document data is such that the difference between the horizontal position of the first row and the horizontal position of the second row is equal to or greater than the threshold value. The program according to claim 17, wherein a line feed code is inserted between the code and the plurality of second character codes. The procedure for displaying the second formatted document data is such that the first character code of the plurality of first character codes and the first character code of the plurality of second character codes are a third character code. If there is, a line feed code is inserted between the plurality of first character codes and the plurality of second character codes,
The procedure for displaying the first formatted document data is such that the first character code of the plurality of first character codes and the first character code of the plurality of second character codes are the third character code. If not, a program according to claim 17, wherein no line feed code is inserted between the plurality of first character codes and the plurality of second character codes. In the procedure of displaying the second formatted document data, the horizontal position of the first blank in the first row and the horizontal position of the second blank in the second row are within the first range. The character code of the character that follows the first space in the plurality of first character codes, and the character code of the character that follows the second space in the plurality of second character codes. 18. The program according to claim 17, wherein the second formatted document data arranged at the same horizontal position is displayed. The obtaining step further includes determining a first font size based on the sizes of the plurality of first handwritten characters in the first row, and a second based on the sizes of the plurality of second handwritten characters in the second row. Determine the font size,
Before Symbol the plurality of first character code associated the first font size, wherein the plurality of second character codes said second font size is that請 Motomeko 17 wherein the program associated to.   In the procedure of displaying the second formatted document data, when the plurality of first character codes include a character code of a predetermined mathematical symbol, a plurality of characters corresponding to the plurality of first character codes are displayed. 18. The program according to claim 17, wherein the second formatted document data including a plurality of characters corresponding to the plurality of second character codes at a position corresponding to the second line is displayed at a position corresponding to one line.   The procedure of displaying the second formatted document data further determines whether or not the plurality of first character codes are source codes of the program, using data indicating specifications relating to a description of a predetermined programming language, When the plurality of first character codes are program source codes, a plurality of characters corresponding to the plurality of first character codes are included at positions corresponding to the first line, and correspond to the plurality of second character codes. The program according to claim 17, wherein second formatted document data including a plurality of characters to be displayed at a position corresponding to the second line is displayed. The computer further comprises a touch screen display,
The program according to claim 17, wherein the handwritten document data includes a plurality of stroke data corresponding to a plurality of strokes based on a handwriting input operation using the touch screen display.

Images