JP7452155B2 - Handwriting input device, handwriting input method, program - Google Patents

Description

The present invention relates to a handwriting input device, a handwriting input method, and a program.

In a typical computer-controlled whiteboard device or an application that allows handwriting input (hereinafter referred to as a handwriting input device), the input means is limited to a pen or a finger. For this reason, an operation menu is provided so that the user can switch between pen functions, such as changing the color of text, and editing functions, such as erasing text. Normally, the pen function menu allows selection of color, thickness, etc., and the edit function menu allows selection of deletion, movement, scaling, rotation, cutting, copying, pasting, etc. (see, for example, Patent Document 1).

Patent Document 1 discloses a handwriting input device in which a menu of color settings, transparency settings, thickness settings, line type settings, stamp settings, and operation settings is displayed by pressing a pen button.

However, the conventional technology has a problem in that it is difficult to input handwriting with different settings depending on the pen. For example, multiple users may use a handwriting input device with a pen, but each user cannot handwrite with different settings depending on the pen.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a handwriting input device that allows handwriting input with different settings depending on the pen.

In view of the above-mentioned problems, the present invention provides a handwriting input device that displays handwritten stroke data based on the position of an input means in contact with a touch panel, wherein control data storage means for storing control data of the input means in association with each other; and display control means for displaying an operation guide including text data on a display section at a position corresponding to a position of the stroke data, the input means display control means for displaying information based on the stroke data on the display unit by reflecting the control data associated with the identification information of the input means received from the input means; The control data storage means associates the angle information determined based on the angle between the straight line and the predetermined direction with the identification information of the input means received from the input means when the stroke data of the straight line was handwritten. and handwriting recognition control means for storing the angle information when a straight line is handwritten using the operating guide, and inputting the angle information when a straight line is handwritten using a method other than the operating guide. It is characterized by recognizing straight lines as characters .

It is possible to provide a handwriting input device that allows handwriting input with different settings depending on the pen.

It is a figure which shows the comparative example of the operation menu displayed by a handwriting input device. It is a figure showing an example of use of a handwriting input device. It is an example of a figure explaining the outline of processing of a handwriting input device. It is a figure which shows an example of the perspective view of a pen. 1 is a diagram showing an example of an overall configuration diagram of a handwriting input device. 1 is an example of a hardware configuration diagram of a handwriting input device. It is a figure explaining the function which a handwriting input device and a pen have. FIG. 3 is a block diagram showing functions related to user authentication that the handwriting input device has. It is a figure which shows an example of defined control data. It is a figure which shows an example of the dictionary data of a handwriting recognition dictionary part. It is a figure which shows an example of the dictionary data of a character string conversion dictionary part. It is a figure which shows an example of the dictionary data of a predictive conversion dictionary part. FIG. 3 is a diagram illustrating an example of operation command definition data and system definition data held by an operation command definition unit. FIG. 7 is a diagram illustrating an example of operation command definition data when there is a selected object selected by a handwritten object. FIG. 3 is a diagram illustrating an example of handwritten input storage data stored by a handwritten input storage unit. FIG. 3 is a diagram illustrating pen ID control data stored by a pen ID control data storage unit. FIG. 7 is a diagram illustrating an example of an operation guide and selectable candidates displayed by the operation guide. FIG. 7 is a diagram illustrating the relationship between the position of an operation guide and the position of a handwritten object rectangular area display. FIG. 7 is a diagram showing an operation guide displayed above a handwritten object rectangular area display. FIG. 3 is an example of a diagram illustrating an example of specifying a selected object; FIG. FIG. 7 is a diagram illustrating a display example of operation command candidates based on operation command definition data when there is a handwritten object. FIG. 7 is a diagram illustrating a display example of operation command candidates based on operation command definition data when there is a handwritten object. It is an example of a figure explaining the input method of 90 degrees angle information. It is an example of a figure explaining the input method of the angle information of 45 degrees. 7 is an example of a diagram illustrating another method of inputting angle information. FIG. FIG. 7 is an example sequence diagram (part 1) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 7 is an example sequence diagram (part 2) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 7 is an example sequence diagram illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (Part 3); FIG. 12 is an example sequence diagram (part 4) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 6 is an example sequence diagram (part 5) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates; FIG. 6 is an example sequence diagram (part 6) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 7 is an example sequence diagram illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 7); FIG. 7 is a diagram illustrating a display example in which a plurality of pieces of text data are displayed with their orientations aligned. It is a figure which shows the other example of a structure of a handwriting input device. It is a figure which shows the other example of a structure of a handwriting input device. It is a figure which shows the other example of a structure of a handwriting input device. It is a figure which shows the other example of a structure of a handwriting input device. It is an example of a system configuration diagram of a handwriting input system (Example 2). 1 is an example of a hardware configuration diagram of an information processing system. 1 is an example of a functional block diagram showing functions of the handwriting input system in a block form. FIG. 7 is an example sequence diagram (part 1) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 7 is an example sequence diagram (Part 2) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 7 is an example sequence diagram illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (Part 3); FIG. 12 is an example sequence diagram (part 4) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 6 is an example sequence diagram (part 5) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates; FIG. 6 is an example sequence diagram (part 6) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates. FIG. 7 is an example sequence diagram illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 7); FIG. 8 is an example sequence diagram (part 8) illustrating a process in which the handwriting input device displays character string candidates and operation command candidates;

DETAILED DESCRIPTION OF THE INVENTION Hereinafter, as an example of a mode for carrying out the present invention, a handwriting input device and a handwriting input method performed by the handwriting input device will be described with reference to the drawings.

<Comparative example of handwriting input device>
For convenience of explanation of the handwriting input device in this embodiment, first, a comparative example of the operation procedure of the handwriting input device will be briefly described.

FIG. 1 shows a comparative example of an operation menu displayed by the handwriting input device 2. The entire operation panel 101 displays a pen function menu button 1021 for displaying the pen function menu 102, an edit function menu button 1031 for displaying the edit function menu 103, an input/output function menu button 1041 for displaying the input/output function menu 104, etc. ing. The pen function menu 102 allows the user to select the color, thickness, and operation mode of the pen. The pen function menu 102 has a handwriting input mode button 1022 and a fairhand input mode button 1023. Further, in the editing function menu 103, you can select deletion, selection, cutting, copying, and pasting of objects, and in the input/output function menu 104, you can execute template loading, file loading, saving to file, printing, etc. .

When the handwriting input mode button 1022 is pressed, the information handwritten by the user (pen coordinates) is input as is, and when the fair input mode button 1023 is pressed, the information (hereinafter simply referred to as "letters, numbers, symbols, alphabets, etc.") handwritten by the user is input. , characters, etc.) are recognized, and the text data that has been converted (written) into characters, etc. is displayed.

As an example of the operation procedure of the conventional handwriting input device 2, when a user is handwriting a straight line or a curved line (hereinafter referred to as a stroke), in order to erase multiple strokes and start writing again with a pen, The following operating steps are required. Note that unless otherwise specified, pressing down with a pen means pressing down with the tip of a pen.
(A1) Press the edit function menu button 1031 with a pen to display the edit function menu 103.
(A2) When the user presses the "delete" button 1032 in the editing function menu 103, the handwriting input device 2 automatically deletes the editing function menu 103.
(A3) If the user writes with a pen across multiple strokes to be erased, the strokes will be erased.
(A4) Press the pen function menu button 1021 with the pen.

This step erases the strokes to be erased and allows the user to continue writing strokes with the pen. Just erasing a stroke requires a four-step operating procedure. Even though there are differences in the number of steps and screen transitions, the handwriting input devices 2 often have the same operation system. Further, this type of operation system is common to all computer products, and is based on an operation system in which the user calls up necessary functions. If the user does not know how to call up a function, he or she cannot operate it at all, so it is common for the handwriting input device 2 to provide a method for calling up a function using a tutorial function or a help function.

Similarly, when input/output such as files or printing is required, the user opens the input/output function menu 104. For example, to load a template, the following steps are required:
(B1) When the user presses the input/output function menu button 1041 with a pen, the input/output function menu 104 opens.
(B2) When the user presses the "template selection" button 1042 on the input/output function menu 104 with a pen, the input/output function menu closes and a template selection window opens.
(B3) Press the left and right buttons with a pen to scroll until the desired template is displayed.
(B4) When a desired template is pressed with a pen, that template is loaded and the template selection window is closed.

Similarly, in order to convert characters, etc. handwritten by a user into text data, the following operation steps are required.
(C1) The user presses the pen function menu button 1021 with a pen.
(C2) The user presses the fair copy input mode button 1023 with a pen.

When the fair copy input mode button 1023 is pressed, the handwritten "abcde" is converted to text data "abcde" as shown in FIG. 1(b). Since the text data is displayed in font, it will be in a clean-cut state.

As described above, the conventional handwriting input device 2 requires a step-by-step operation procedure by the user. In other words, since the handwriting input device 2 is based on an operation system that guides the user to the next step in an explanatory manner, it is difficult for the user to operate the system intuitively.

<Example of usage of handwriting input device>
The handwriting input device 2 is not only used while being held perpendicularly to the floor, but may also be used when placed horizontally on the floor.

FIG. 2 is a diagram showing an example of how the handwriting input device 2 is used. In FIG. 2A, the handwriting input device 2 is installed on a desk with the display facing upward (this type of installation is called flat placement). Users sitting on chairs facing each other are writing by hand around the handwriting input device 2 (facing each other in FIG. 2A).

In this case, a normal user handwrites characters and the like in the vertical direction to himself (FIG. 2(b)). However, in the fair handwriting input mode, correct handwriting recognition is possible only when characters are handwritten by a user at a certain fixed location (in a fixed direction when viewed from the handwriting input device 2). Assuming that the user at the fixed location shown in FIG. 2(b) is Mr. A, the handwritten characters etc. of Mr. B and Mr. C will not be recognized correctly.

The reason for this is that current handwriting recognition engines can only correctly recognize handwritten text when it is oriented in a certain direction. Correct recognition means that recognition is possible with a certain practical recognition rate. Further, the certain fixed direction is, for example, the vertical direction when the handwriting input device 2 is held perpendicular to the floor or the like (the vertical direction in the state shown in FIG. 5(a), which will be described later).

When the handwriting input device 2 is used in a state where it is held perpendicular to the floor or the like (mounted on a wall, held on a stand, etc.), inconveniences are unlikely to occur because the user does not handwrite upside down. However, in the usage example shown in FIG. 2, the characters of Mr. B and Mr. C are rotated 180 degrees (upside down) when viewed from the handwriting recognition engine, so that the handwriting cannot be recognized correctly.

Depending on the computer or OS (Operating System), the entire screen can be rotated in 90 degree increments using a predetermined operation (for example, Ctrl+Alt+arrow keys). In other words, when the user wants to recognize handwritten characters written by the user, the user only has to rotate the entire screen. However, with this method, for example, it is not possible to accurately recognize handwritten characters written by multiple users facing each other.

<Outline of processing of handwriting input device in this embodiment>
Therefore, the handwriting input device 2 of this embodiment allows handwriting input with different settings depending on the user.

FIG. 3 is an example of a diagram illustrating an outline of processing of the handwriting input device 2. There are four users (A to D) around the handwriting input device 2 (four sides). For convenience of explanation,
When handwriting letters etc. from Mr. A's position, set the angle of the letters to 0 degrees,
When handwriting letters etc. from Mr. B's position, the angle of the letters is 90 degrees,
When handwriting characters etc. from Mr. C's position, the angle of the characters is 180 degrees,
When writing letters etc. by hand from Mr. D's position, the angle of the letters is 270 degrees,
shall be.

Each user holds a pen 2500 and writes by hand with the pen 2500. This pen 2500 stores identification information (hereinafter referred to as pen ID). Using this, the handwriting input device 2 associates the pen ID with the angle information at which the user handwrites characters and the like. That is, when starting to use the handwriting input device 2, the user uses the pen 2500 to input angle information (90 degrees, 180 degrees, 270 degrees) when handwriting characters, etc. (0 degrees). (No input required). The details will be described later, but with a simple operation, you can input the angle information when writing characters, etc., and you can also associate the pen ID and the angle information.

In FIG. 3, each user is handwriting characters such as "abc". The handwriting input device 2 identifies angle information using the pen ID of the pen 2500 that handwritten "abc", rotates the handwritten "abc" clockwise by the angle information, and then performs character recognition. Rotation is done internally, and displayed characters etc. are not rotated. Internally means that the handwriting input device processes the data. However, the operation guide described later is displayed rotated counterclockwise. As an example, FIG. 3 shows "abc", which is obtained by rotating the characters "abc" handwritten by Mr. C by 180 degrees (the characters are not displayed because they are only internally rotated). Since the characters or the like are oriented vertically by the rotation (the angle becomes 0 degrees), the handwriting input device 2 can correctly recognize the characters.

In this way, in this embodiment, it is possible to input handwriting with different settings (here, angle information) depending on the pen (or user), so users around the handwriting input device 2 placed horizontally can input characters. Even if the characters are handwritten, the characters can be recognized correctly. In addition to angle information, font thickness, color, texture, etc. can be set for each user.

<About terms>
The input means may be any means that allows handwriting on a touch panel. Examples include a pen, a human finger or hand, and a rod-shaped member. Additionally, line of sight input may be possible.

Stroke data is a freely handwritten line. It has a continuous set of points and may be interpolated as appropriate.

The operation command is an instruction prepared for operating the handwriting input device 2 and instructing execution of a specific process. In this embodiment, editing, modification, input/output, and pen state operation commands are taken as examples, but all commands for operating the handwriting input device 2, such as screen reversal, page switching, and operation mode settings, are used as examples. is the target.

The control data of the input means is information regarding how to process the stroke data input by the input means. Examples include color, thickness, pattern, and angle. In this embodiment, this will be explained as pen ID control data. These are reflected in the stroke data.

The information based on stroke data may be information generated based on stroke data. For example, there are convertible candidates after handwriting recognition, operation commands, etc.

<Example of pen appearance>
FIG. 4 shows an example of a perspective view of the pen 2500. FIG. 4 shows an example of a multifunctional pen 2500. A pen 2500 that has a built-in power supply and can send commands to the handwriting input device 2 is called an active pen (a pen that does not have a built-in power supply is called a passive pen). The pen 2500 in Figure 4 has one physical switch on the pen tip, one on the pen tip, and two on the side of the pen; the tip is for writing, the tip is for erasing, and the side of the pen is for assigning user functions. It is. The pen 2500 of this embodiment has a nonvolatile memory and stores a pen ID that does not overlap with other pens.

Note that if the pen is equipped with a switch, it is also possible to reduce the number of operating procedures for the handwriting input device 2 by the user. A pen with a switch mainly refers to an active pen, but it includes not only an active pen but also a passive pen using an electromagnetic induction method, since even a passive pen without a built-in power supply can generate power using only an LC circuit. Pens with optical, infrared, and capacitive switches other than electromagnetic induction are active pens.

The user can assign one of the pen side switches to the pen function menu 102 and the other to the edit function menu 103. This is convenient in that the user can display the pen function menu 102 or the edit function menu 103 by pressing the side button of the pen 2500 held in hand, but the user must The complication of pressing the side button to open the pen function menu 102 or the editing function menu 103 remains the same.

On the other hand, if you use the switch on the end of the pen for erasing, steps (A1), (A2), and (A4) explained in Fig. The operation procedure required for erasing can be reduced to one step.

It is assumed that the hardware configuration of the pen 2500 is similar to a general control system including a communication function and a microcomputer. Coordinate input methods for the pen 2500 include an electromagnetic induction method, an active capacitive coupling method, and the like. Further, the pen 2500 may have functions such as pen pressure detection, tilt detection, and a hover function (displaying a cursor before touching the pen).

<Overall configuration of the device>
The overall configuration of the handwriting input device 2 according to this embodiment will be described using FIG. 5. FIG. 5 is a diagram showing the overall configuration of the handwriting input device 2. As shown in FIG. FIG. 5A shows, as an example of the handwriting input device 2, the handwriting input device 2 used as a horizontally elongated electronic blackboard hung on a wall.

As shown in FIG. 5A, a display 220 as an example of a display device is installed above the handwriting input device 2. User U can handwrite (also referred to as input or drawing) characters and the like on display 220 using pen 2500.

FIG. 5(b) shows a handwriting input device 2 used as a vertically elongated electronic blackboard hung on a wall.

FIG. 5C shows the handwriting input device 2 placed flat on a desk 230. Since the handwriting input device 2 has a thickness of about 1 cm, there is no need to adjust the height of the desk even if it is placed flat on an ordinary desk. It can also be moved easily.

<Device hardware configuration>
Next, the hardware configuration of the handwriting input device 2 will be explained using FIG. 6. The handwriting input device 2 has the configuration of an information processing device or a computer as shown in the figure. FIG. 6 is an example of a hardware configuration diagram of the handwriting input device 2. As shown in FIG. As shown in FIG. 6, the handwriting input device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204. We are prepared.

Among these, the CPU 201 controls the operation of the handwriting input device 2 as a whole. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). RAM 203 is used as a work area for CPU 201. The SSD 204 stores various data such as programs for the handwriting input device 2.

The handwriting input device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I/O F223, a power control circuit 224, an AC adapter 225, and a battery 226.

The display controller 213 controls and manages screen display in order to output the output image to the display 220 or the like. The touch sensor 216 detects that the pen 2500, the user's hand, or the like (the pen or the user's hand serves as an input means) contacts the display 220. The touch sensor 216 also receives the pen ID.

Touch sensor controller 215 controls processing of touch sensor 216. The touch sensor 216 inputs and detects coordinates. For example, in the case of an optical system, two light receiving/emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays in parallel to the display 220. This is a method of receiving light that is reflected by a reflecting member provided around the light receiving element 220 and returns along the same optical path as the light emitted by the light receiving element. The touch sensor 216 outputs the position information of the infrared rays emitted by the two light receiving and emitting devices blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position that is the contact position of the object. Furthermore, the touch sensor controller 215 has a communication unit 215a and can communicate with the pen 2500 wirelessly. For example, when communicating using a standard such as Bluetooth (registered trademark), a commercially available pen can be used. By registering one or more pens 2500 in the communication unit 215a in advance, the user can communicate with the handwriting input device 2 without performing connection settings for communicating the pen 2500 with the handwriting input device 2.

The power switch 227 is a switch for switching the power of the handwriting input device 2 on and off. The tilt sensor 217 is a sensor that detects the tilt angle of the handwriting input device 2. It is mainly used to detect whether the handwriting input device 2 is used in the installation state shown in FIG. 5(a), FIG. 5(b), or FIG. 5(c), and depending on the installation state. You can automatically change the thickness of characters, etc.

The serial interface 218 is a communication interface with an external device such as a USB. Used for inputting information from outside. The speaker 219 is used for audio output, and the microphone 221 is used for audio input. The wireless communication device 222 communicates with a terminal carried by a user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), etc., but the communication standard does not matter. The wireless communication device 222 forms an access point, and when the user sets the obtained SSID (Service Set Identifier) and password on the terminal carried by the user, the user can connect to the access point.

Note that the wireless communication device 222 is preferably provided with two access points.
a. Access point → Internet
b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the internal network but can use the Internet. Access point b is for internal users, who can use the internal network and the Internet.

The infrared I/F 223 detects the handwriting input device 2 placed next to it. Utilizing the straightness of infrared rays, only the handwriting input device 2 placed next to it can be detected. Preferably, one infrared I/F 223 is provided on each side, and it is possible to detect in which direction of the handwriting input device 2 another handwriting input device 2 is placed. As a result, the screen expands, and it is possible to display previously handwritten information (handwritten information on another page with the width of one display 220 as one page) etc. on the adjacent handwriting input device 2.

A power supply control circuit 224 controls an AC adapter 225 and a battery 226, which are the power sources of the handwriting input device 2. The AC adapter 225 converts alternating current shared by commercial power sources into direct current.

If the display 220 is a so-called electronic paper, it consumes little or no power to maintain image display, so it can also be driven by the battery 226. This allows the handwriting input device 2 to be used for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

Furthermore, the handwriting input device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG. 6.

Note that the touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies a touch position by detecting a change in capacitance, or a touch panel that identifies a touch position by a voltage change between two opposing resistive films. Various detection means may be used, such as a resistive film type touch panel or an electromagnetic induction type touch panel that detects electromagnetic induction generated when a touch object contacts the display section to specify the contact position. The touch sensor 216 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the pen tip. In this case, you can use your fingertips or a pen-shaped stick to perform touch operations. Note that the pen 2500 does not need to be an elongated pen type.

<About device functions>
Next, the functions of handwriting input device 2 and pen 2500 will be described using FIG. 7. FIG. 7A is an example of a functional block diagram showing functions of the handwriting input device 2 in a block form. The handwriting input device 2 includes a handwriting input section 21, a display section 22, a handwriting input display control section 23, a candidate display timer control section 24, a handwriting input storage section 25, a handwriting recognition control section 26, a handwriting recognition dictionary section 27, and a character string conversion section. It includes a control section 28, a character string conversion dictionary section 29, a predictive conversion control section 30, a predictive conversion dictionary section 31, an operation command recognition control section 32, an operation command definition section 33, and a pen ID control data storage section 36. . Each function that the handwriting input device 2 has is a function that is realized when any of the components shown in FIG. or means.

The handwriting input unit 21 is realized by a touch sensor 216 and the like, and receives handwritten input from the user and receives a pen ID. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) with a pen ID, and sends it to the handwriting input display control unit 23. Pen coordinate data is periodically transmitted as discrete values, and coordinates between the discrete values are calculated by interpolation.

The display unit 22 is realized by a display 220 or the like, and displays handwritten objects and operation menus. The display unit 22 converts the drawing data d3 written into the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits the data to the display 220. The display unit 22 displays information based on stroke data according to the user's position.

The handwriting input display control unit 23 performs overall control regarding handwriting input and display. The handwriting input display control section 23 processes the pen operation data d2 from the handwriting input section 21 and transmits it to the display section 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be explained later with reference to FIGS. 26 to 32.

The candidate display timer control unit 24 is a timer for display control of selectable candidates. A timer is started or stopped to generate a timing to start displaying selectable candidates and a timing to erase the display. The selectable candidates are handwritten recognition character string/language character string candidates, conversion character string candidates, character string/predictive conversion candidates, and operation command candidates that are selectably displayed in the operation guide described later. The candidate display timer control unit 24 receives a timer start request d4 (or a timer stop request) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

The handwritten input storage unit 25 has a storage function for storing user data (handwritten objects/character string objects). The handwritten input storage unit 25 receives the user data d6-1 from the handwritten input display control unit 23 and stores it in the handwritten input storage unit 25. The handwritten input storage section 25 receives the acquisition request d6-2 from the handwritten input display control section 23, and transmits the user data d7 stored in the handwritten input storage section 25. The handwritten input storage section 25 transmits the position information d36 of the confirmed object to the operation command recognition control section 32.

The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike general OCR (Optical Character Reader), characters (not only Japanese but also multiple languages such as English), numbers, symbols (%, $, &, etc.), figures (lines, etc.), , circles, triangles, etc.). Various algorithms have been devised for the recognition method, but in this embodiment, the details will be omitted because known techniques can be used.

The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, executes handwriting recognition, and holds handwritten recognition character string candidates. Further, the handwriting recognition control unit 26 uses the handwriting recognition dictionary unit 27 to hold language character string candidates converted from the handwritten recognition character string candidates d12. Separately, when receiving the acquisition request d8-2 from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the held handwritten recognition character string candidates and language character string candidates d9 to the handwriting input display control unit 23. .

The handwriting recognition dictionary section 27 is dictionary data for language conversion for handwriting recognition. The handwriting recognition dictionary section 27 receives the handwritten recognition character string candidate d12 from the handwriting recognition control section 26, converts it into a linguistically likely linguistic character string candidate d13, and sends it to the handwriting recognition control section 26. For example, in the case of Japanese, hiragana is converted to kanji or katakana.

The character string conversion control unit 28 controls the conversion of converted character string candidates into character strings. The converted character string is a character string that is likely to be generated including a handwritten recognition character string or a language character string. The character string conversion control unit 28 receives the handwritten recognition character string and the language character string candidate d11 from the handwriting recognition control unit 26, converts it into a converted character string candidate using the character string conversion dictionary unit 29, and holds it. Separately, when an acquisition request d14 is received from the handwritten input display control unit 23, the held conversion character string candidate d15 is transmitted to the handwritten input display control unit 23.

The character string conversion dictionary section 29 is dictionary data for character string conversion. The character string conversion dictionary unit 29 receives the handwritten recognition character string and the language character string candidate d17 from the character string conversion control unit 28, and transmits the conversion character string candidate d18 to the character string conversion control unit 28.

The predictive conversion control unit 30 receives the handwritten recognition character string and the language character string candidate d10 from the handwriting recognition control unit 26, and receives the converted character string candidate d16 from the character string conversion control unit 28. The predictive conversion control unit 30 uses the predictive conversion dictionary unit 31 to convert each of the handwritten recognition character string, language character string candidate d10, and conversion character string candidate d16 into predicted character string candidates. The predicted converted character string is a character string that is likely to be generated including a handwritten recognition character string, a language character string, or a converted character string. Separately, when the acquisition request d19 is received from the handwritten input display control unit 23, the predicted character string candidate d20 is transmitted to the handwritten input display control unit 23.

The predictive conversion dictionary section 31 is dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives the handwritten recognition character string, language character string candidates, and conversion character string candidates d21 from the predictive conversion control unit 30, and transmits the predicted character string candidates d22 to the predictive conversion control unit 30.

The operation command recognition control unit 32 receives the handwritten recognition character string and the language character string candidate d30 from the handwriting recognition control unit 26, and receives the converted character string candidate d28 from the character string conversion control unit 28. The operation command recognition control unit 32 receives the predicted character string candidate d29 from the predictive conversion control unit 30. Then, an operation command conversion request d26 is sent to the operation command definition section 33 for each, and an operation command candidate d27 is received from the operation command definition section 33. The operation command recognition control unit 32 holds operation command candidates d27.

If the operation command conversion request d26 partially matches (matches in part or in whole) the operation command conversion request d26, the operation command definition unit 33 transmits the operation command candidate d27 to the operation command recognition control unit 32.

Further, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23, and transmits a position information acquisition request d23 of a confirmed object that has been input in the past to the handwriting input storage unit 25. The operation command recognition control unit 32 holds the confirmed object specified by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 specifies the position of the pen operation data d24-1 and a selected object that satisfies predetermined criteria. Separately, when the acquisition request d24-2 is received from the handwriting input display control unit 23, the selected object d25 identified as the held operation command candidate is transmitted to the handwriting input display control unit 23.

The pen ID control data storage section 36 holds pen ID control data (it can also be called a storage means). Before the handwriting input display control section 23 transmits the display data to the display section 22, the pen ID control data storage section 36 transmits the pen ID control data d41 to the handwriting input display control section 23. The handwriting input display control unit 23 draws display data under operating conditions stored in association with the pen ID. Furthermore, before the handwriting recognition control unit 26 executes handwriting recognition, the pen ID control data storage unit 36 transmits the angle information d44 of the pen ID control data to the handwriting recognition control unit 26, and the handwriting recognition control unit 26 Handwriting recognition is executed by rotating the stroke using angle information stored in association with the ID.

Further, after the handwriting recognition control unit 26 recognizes a straight line for setting angle information when the user handwrites characters, etc., the handwriting recognition control unit 26 converts the angle information d43 of the pen ID control data into the pen ID control data. The angle information d43 is transmitted to the storage unit 36 and stored in association with the pen ID. Further, after the handwriting input display control unit 23 executes the operation command for setting the angle information, the handwriting input display control unit 23 transmits the pen ID control data d42 to the pen ID control data storage unit 36, and outputs the result of the execution of the operation command. (angle information set by the user) is associated with the pen ID and saved. Thereafter, the stroke of that pen ID is rotated by the set angle information, and then handwriting recognition is executed.

FIG. 7(b) is a functional block diagram showing the functions of the pen 2500 in block form. The pen 2500 has a pen event transmitter 41. The pen event transmitting unit 41 attaches a pen ID to event data of pen up, pen down, and pen coordinates and transmits the data to the handwriting input device 2.

<About user authentication>
In this embodiment, since control is performed using the result of user authentication, it is preferable that the handwriting input device 2 has a function of user authentication. Therefore, functions related to user authentication will be explained using FIG. 8.

FIG. 8 is a block diagram showing functions related to user authentication that the handwriting input device 2 has. Although FIG. 8 shows only the handwriting input display control section 23 as a function related to the user authentication section 34 (user authentication means), each of the functions shown in FIG. 7 may be used for user authentication results.

The authentication information acquisition unit 35 acquires authentication information d31 from the user U. The authentication information d31 may be an IC card number, a user ID and password, or biometric information such as a fingerprint. The user authentication section 34 acquires authentication information d32 from the authentication information acquisition section 35 and searches the user information DB 37 for authentication information d33. If a user matching the search exists, user information d34 is acquired from the user information DB 36. The user information may be any information that represents the user's attributes, such as a user name, password, user's computer name, department, authority, etc.

Since the user authentication section 34 transmits the user information d35 to the handwritten input display control section 23, the handwritten input display control section 23 can execute the operation command using the user information. Operation commands using user information will be explained with reference to FIG.

Note that the authentication function is not only provided by the handwriting input device 2 but may also be provided by an external authentication server or the like. In this case, the handwriting input device 2 transmits the authentication information to the authentication server and obtains the authentication result and user information.

<About predefined control data>
Next, defined control data used by the handwriting input device 2 for various processes will be described using FIG. 9. FIG. 9 shows an example of predefined control data. In the example of FIG. 9, control data is shown for each control item.

The selectable candidate display timer 401 defines the time until displaying selectable candidates (an example of a first time). This is because selectable candidates are not displayed during handwriting. In FIG. 9, it means that selectable candidates are displayed if pen down does not occur within TimerValue=500 [ms] after pen up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when starting the selectable candidate display timer in step S18-2 of FIG. 28, which will be described later.

The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted (an example of a second time). This is to delete the selectable candidates when the user does not select the selectable candidates. In FIG. 9, it means that if a selectable candidate is not selected within TimerValue=5000 [ms] after the selectable candidate is displayed, the selectable candidate display data is deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used at step S54 in FIG. 30 when starting the selectable candidate display deletion timer.

A handwritten object neighborhood rectangular area 403 defines a rectangular area that is considered to be near the handwritten object. In the example of FIG. 9, the handwritten object vicinity rectangular area 403 is a rectangle obtained by enlarging the rectangular area of the handwritten object by 50% (Horizontal) of the estimated character size in the horizontal direction and by 80% (Vertical) of the estimated character size in the vertical direction. It becomes an area. In the example of FIG. 9, the estimated character size is a percentage (specified as a percentage), but if the unit is set to "mm" or the like, it is also possible to set the length to a fixed length. The handwritten object vicinity rectangular area 403 is held by the handwritten input storage unit 25. The estimated character size 405 is used in step S10 of FIG. 27 to determine the overlapping state of the handwritten object vicinity rectangular area and the stroke rectangular area.

Estimated writing direction/character size determination condition 404 defines constants for determining the writing direction and the measurement direction of character size. In the example in Figure 9, the difference between the time when the first stroke was added to the handwritten object rectangular area and the time when the last stroke was added is MinTime = 1000 [ms] or more, and the horizontal distance (width) of the handwritten object rectangular area If the difference between and the vertical distance (height) is MinDiff = 10 [mm] or more, and the horizontal distance is longer than the vertical distance, it means that the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. do. If the horizontal distance is shorter than the vertical distance, it means that the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If the above conditions are not met, the estimated writing direction is determined to be "horizontal writing" (DefaultDir="Horizontal"), and the estimated character size is determined to be the longer distance between the horizontal distance and the vertical distance. The estimated writing direction/character size determination condition 404 is held in the handwritten input storage unit 25. The estimated writing direction/character size determination condition 404 is used in obtaining the estimated writing direction in step S50 of FIG. 30 and in obtaining the character string object font in step S72 of FIG.

Estimated character size 405 defines data for estimating the size of characters, etc. In the example of FIG. 9, the estimated character size determined by the estimated writing direction/character size determination condition 404 is the estimated character size 405 of a small character 405a (hereinafter referred to as the minimum font size) and a large character 405c (hereinafter referred to as the minimum font size). maximum font size). If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the character size is determined to be medium character 405b. The estimated character size 405 is held in the handwritten input storage unit 25. The estimated character size 405 is used in obtaining the character string object font in step S72 of FIG.

Specifically, the handwritten input storage unit 25 compares the estimated character size determined by the estimated writing direction/character size determination condition 404 with the FontSize of the estimated character size 405, and uses the font with the closest size. For example, if the estimated font size is 25 [mm] (FontSize for small fonts) or less, "Small font" is used, and if the estimated font size is over 25 mm and 50 mm (FontSize for medium fonts) or less, "Medium font" is used. If the font size exceeds 100mm (FontSize for large fonts), it is determined as "large font". "Small font" 405a is a 25mm Mincho font (FontStyle="Mincho" FontSize="25mm"), "Medium font" 405b is a 50mm Mincho font (FontStyle="Mincho" FontSize= "50mm"), "Large font" 405c uses a Gothic 100mm font (FontStyle="Gothic" FontSize="100mm"). If you want to increase the number of font sizes or styles, you can increase the number of estimated font sizes 405.

The straddle line determination condition 406 defines data used to determine whether multiple objects have been selected. The handwritten object is a single stroke, and in the example shown in Figure 9, the length of the long side of the handwritten object is 100 [mm] or more (MinLenLongSide="100mm"), and the length of the short side is 50 [mm] or less. (MaxLenShortSide="50mm"), and if there is an object with an overlapping ratio of 80% or more (MinOverLapRate="80%") in the long side direction and short side direction with the handwritten object, multiple objects will be selected. (selected object). The straddle line determination condition 406 is held by the operation command recognition control unit 32. The straddle line determination condition 406 is used in the straddle line determination for determining the selected object in step S41 of FIG.

The enclosing line determination condition 407 defines data used to determine whether an object is an enclosing line. In the example of FIG. 9, the operation command recognition control unit 32 determines a confirmed object whose overlap rate in the long side direction and short side direction of the handwritten object is 100% or more (MinOverLapRate="100%") as the selected object. The enclosing line determination condition 407 is held by the operation command recognition control unit 32. The enclosing line determination condition 407 is used in enclosing line determination for determining the selected object in step S41 of FIG.

Note that either the straddling line determination condition 406 or the enclosing line determination condition 407 may be determined with priority. For example, if the straddle line judgment condition 406 is made loose (to make it easier to select the straddle line), and the enclosing line judgment condition 407 is made strict (the value is such that only the enclosing line can be selected), the operation command is recognized. It is preferable that the control unit 32 prioritizes the enclosing line determination condition 407 for determination.

<Example of dictionary data>
The dictionary data will be explained using FIGS. 10 to 12. FIG. 10 is an example of dictionary data of the handwriting recognition dictionary section 27. FIG. 11 is an example of dictionary data of the character string conversion dictionary section 29. FIG. 12 is an example of dictionary data of the predictive conversion dictionary section 31. Note that these dictionary data are used in steps S24 to S33 in FIG. 29, respectively.

In this embodiment, the conversion result using the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 10 is used as a language string candidate, the conversion result using the dictionary data of the character string conversion dictionary unit 29 in FIG. The conversion result based on the dictionary data of the conversion dictionary unit 31 is called a predicted character string candidate. "Before conversion" of each dictionary data represents the character string to be searched for in the dictionary data, "after conversion" represents the character string after conversion corresponding to the character string to be searched, and "probability" represents the probability of the user's selection. The probability is calculated from past user selections of each string. Therefore, the probability may be calculated for each user. Although various algorithms have been devised as methods for calculating probability, it is sufficient to use an appropriate method to calculate the probability, and the details will be omitted. The present embodiment is characterized in that character string candidates are displayed in descending order of selection probability from the estimated writing direction.

In the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 10, the handwritten "gi" is "gi" with a probability of 0.55, "waza" with a probability of 0.4, and the handwritten "gishi" is "technician" with a probability of 0.5. Indicates that the probability is 0.45 to convert to "Engineer". The same applies to other "before conversion" character strings. In FIG. 10, the character string "before conversion" is handwritten in hiragana, but anything other than hiragana may be registered in "before conversion."

The dictionary data of the character string conversion dictionary section 29 in FIG. 11 shows that the character string "Ki" is converted to "minutes" with a probability of 0.95, and the character string "waza" is converted to "skill test" with a probability of 0.85. The same applies to other "before conversion" character strings.

In the dictionary data of the predictive conversion dictionary unit 31 in FIG. 12, the character string "minutes" is converted to "destination of minutes" with a probability of 0.65, and the character string "skills test" is converted to "approval of skills test" with a probability of 0.75. Indicates that the In the example of FIG. 12, the character strings before conversion are all kanji, but characters other than kanji may be registered.

Note that the dictionary data is language-independent, and any character strings may be registered before and after conversion.

<Operation command definition data held by the operation command definition section>
Next, the operation command definition data used by the operation command recognition control section 32 will be explained using FIGS. 13 and 14. FIG. 13 shows an example of operation command definition data and system definition data held by the operation command definition unit 33.

FIG. 13(a) shows an example of operation command definition data. The operation command definition data in FIG. 13A is an example of operation command definition data when there is no selected object selected by the handwritten object, and all operation commands for operating the handwriting input device 2 are targeted. The operation command in FIG. 13A includes an operation command name (Name), a character string (String) that partially matches the character string candidate, and an operation command string (Command) to be executed. “%~%” in the operation command character string are variables, and are associated with system definition data as shown in FIG. 13(b). That is, "%~%" is replaced with the system definition data shown in FIG. 13(b).

First, in the operation command definition data 701, the operation command name is "Read Minutes Template", the character string that partially matches the character string candidate is "Minutes" or "Template", and the operation command string to be executed is "ReadFile https ://%username%:%password%@server.com/template/minutes.pdf". In this example, the operation command string to be executed includes system definition data "%~%", indicating that "%username%" and "%password%" are replaced with system definition data 704 and 705, respectively. Therefore, the final operation command string to be executed will be "ReadFile https://taro.tokkyo:x2PDHTyS@server.com/template/minutes.pdf" and "https://taro.tokkyo:x2PDHTyS@ Indicates that the file "server.com/template/minutes.pdf" is to be read (ReadFile).

In the operation command definition data 702, the operation command name is "save in minutes folder", the character string that partially matches the character string candidate is "minutes" or "save", and the operation command string to be executed is "WriteFile https: //%username%:%password%@server.com/minutes/%machinename%_%yyyy-mm-dd%.pdf". Similar to the operation command definition data 701, "%username%", "%password%", and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates replacement with the current date. For example, if the current date is September 26, 2018, this indicates that it will be replaced with "2018-09-26". The final operation command to be executed is "WriteFile https://taro.tokkyo:x2PDHTyS@server.com/minutes/%My-Machine_2018-09-26.pdf", and the minutes are written as "https://taro.tokkyo". :x2PDHTyS@server.com/minutes/%My-Machine_2018-09-26.pdf" (WriteFile).

In the operation command definition data 703, the operation command name is "print", the character string that partially matches the character string candidate is "print" or "print", and the operation command string to be executed is "PrintFile https://%username%". :%password%@server.com/print/%machinename%－"%yyyy－mm－dd%.pdf". If you replace the operation command string in the same way as the operation command definition data 702, the final operation command to be executed will be "PrintFile https://taro.tokkyo:x2PDHTyS@server.com/print/%My-Machine_2018-09-26 .pdf", indicating that the file "https://taro.tokkyo:x2PDHTyS@server.com/print/%My-Machine_2018-09-26.pdf" is to be printed (PrintFile). That is, the file is sent to the server. The user communicates the printer with the server, specifies a file, and the printer prints the file's contents on paper.

In this way, since the operation command definition data 701 to 703 can be identified from the character string candidates, the user can display the operation command by handwriting it. Additionally, if user authentication is successful, "%username%", "%password%", etc. in the operation command definition data are replaced with user information, making it possible to input and output files in association with the user.

If the user is not authenticated (including the case of authentication failure if the user is able to use the handwriting input device 2 even though the authentication fails), the handwriting input device 2 uses the preset “ Replace with %username%”, “%password%”, etc. Therefore, it is possible to input and output files in association with the handwriting input device 2 without user authentication.

Operation command definition data 709, 710, and 711 are operation commands for changing the pen state. The pen state may also be referred to as the type of pen. The operation command names of the operation command definition data 709, 710, and 711 are "thin pen," "thick pen," and "marker," and the character strings that partially match the character string candidates are "thin," "pen," "thick," and "pen." ”, “marker” or “pen”, and the operation command character string is “ChangePen fine”, “ChangePen bold”, or “Changepen marking”. When this operation command is executed, the pen state is stored in the pen ID control data storage section 36, so the user can handwrite strokes in the set pen state.

The operation command definition data 712 is an operation command for aligning the direction of text data in a certain direction. The operation command name of the operation command definition data 712 is "Align text direction," the character string that partially matches the character string candidate is "Text," "Direction," or "Direction," and the operation command string is "AlignTextDirection." Text data written by a user from a direction other than the top and bottom has different orientations, making it difficult to read everything from one direction. When the user executes the operation command definition data 712, the handwriting input device 2 aligns the recognized handwritten character strings in the same direction (for example, vertically). Aligning in this case means rotating the text data by the angle information.

Next, based on FIG. 14, operation command definition data when there is a handwritten object, that is, editing-related and modification-related operation command definition data will be described. FIG. 14 shows an example of operation command definition data when there is a selected object selected by a handwritten object. The operation command definition data in FIG. 14 includes an operation command name (Name), a group name (Group) of operation command candidates, and an operation command character string to be executed (Command).

The operation command definition data 707 defines editing-related operation commands (Group="Edit"), and is an example of the definition data of editing-related operation command names "erase," "move," "rotate," and "select." In other words, these operation commands are displayed for the selected object, and the user can select the desired operation command.

The operation command definition data 708 defines the operation commands of the decoration type (Group="Decorate"), and the definition data examples of the operation command names of the decoration type "thick", "thin", "large", "small", and "underline" are shown below. be. These operation commands are displayed for the selected object, and the user can select the desired operation command. In addition, color operation commands may be displayed.

Therefore, since the operation command definition data 707 and 708 are specified when the user selects the selected object with the handwritten object, the operation command can be displayed by handwriting the user.

<Handwritten input saved data saved by the handwritten input storage section>
Next, handwritten input storage data will be explained using FIG. 15. FIG. 15 shows an example of handwritten input storage data stored by the handwritten input storage section 25. One line in FIG. 15 represents one stroke. One piece of handwritten input storage data has the following items: DataId, Type, PenId, Color, Width, Pattern, Angle, StartPoint, StartTime, EndPoint, EndTime, Point, and Pressure.

DataId is stroke identification information. Type is the type of stroke. Types include stroke, group, and text. The type of handwritten input storage data 801 and 802 is Stroke, and the type of handwritten input storage data 803 is Group. Group means to group other strokes, and for handwritten input saved data of type Group, specify the strokes to be grouped by DataId. PenId, Color, Width, Pattern, and Angle are transcribed pen ID control data described below. StartPoint is the coordinates of the start point of the stroke, and StartTime is the time of the start point of the stroke. EndPoint is the coordinate of the end point of the stroke, and EndTime is the time of the end point of the stroke. Point is a coordinate string from the start point to the end point, and Pressure is the pen pressure from the start point to the end point. As shown in Angle, handwritten input saved data 804 and 805 are rotated clockwise by 180 degrees and 270 degrees, respectively, and then handwritten recognition is performed.

<Pen ID control data stored by the pen ID control data storage unit>
Next, pen ID control data will be explained using FIG. 16. FIG. 16 is a diagram illustrating pen ID control data stored by the pen ID control data storage section 36. One line in FIG. 16(a) indicates pen ID control data for one pen. Further, FIG. 16(b) is a diagram illustrating angle information when the user writes by hand on the handwriting input device 2. The angle information can be said to be the angle in the direction in which the user is present, the angle in the direction in which the pen is used, or the angle related to the rotation of characters handwritten by the user. The angle information for each user is 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees counterclockwise, with the predetermined direction (for example, the vertical direction) of the handwriting input device 2 being 0 degrees (reference). degrees.

The user's angle information is the position of the user relative to the handwriting input device 2 when the handwriting input device 2 is placed flat. In other words, the user's angle information can be said to be information related to position. It is possible to specify in which direction the user is located when viewed from the handwriting input device 2. In addition to the angle information, the direction seen from the handwriting input device 2 is likened to a clock: 0 degrees: 6 o'clock direction, 45 degrees: between 4 o'clock direction and 5 o'clock direction, 90 degrees: 3 o'clock direction, 135 degrees: 1 o'clock direction 180 degrees: 12 o'clock direction, 225 degrees: between 10 o'clock direction and 11 o'clock direction, 270 degrees: between 9 o'clock direction, 315 degrees: between 7 o'clock direction and 8 o'clock direction. You may.

Note that the angle information is not automatically determined based on the user's position, but each user inputs (specifies) the angle information. The resolution of angle information that can be specified (45 degrees in FIG. 16) is just one example, and may be smaller, such as 5 degrees to 30 degrees. However, it is believed that users can read characters that are rotated at around 45 degrees.

The pen ID control data includes PenId, Color, Width, Pattern, and Angle. PenId is identification information stored inside the pen. Color is the stroke color set for this pen (can be changed arbitrarily by the user). Width is the stroke thickness set for this pen (can be changed arbitrarily by the user). Pattern is the stroke line type set for this pen (can be changed arbitrarily by the user). Angle is the stroke angle information set for this pen (can be changed arbitrarily by the user). In the example of FIG. 16A, the angle information of each pen is 0 degrees, 90 degrees, 180 degrees, and 270 degrees counterclockwise.

The pen ID control data 901 is control data with a pen ID of 1, the color is black, the thickness is 1 pixel (1 px), the pattern is solid, and the angle information is 0 degrees. Similarly, in the pen ID control data 902, the pen ID is 2, the color is black, the thickness is 1 pixel, the pattern is solid, and the angle information is 90 degrees. In the pen ID control data 903, the pen ID is 3, the color is black, the thickness is 10 pixels, the pattern is solid, and the angle information is 180 degrees. The pen ID control data 904 has a black color, a thickness of 10 pixels, a pattern of halftone dots, and an angle of 270 degrees.

This data includes step S5 in FIG. 26 (obtain pen ID control data), step S20 in FIG. 28 (save angle information of pen ID control data), step S21 (obtain angle information of pen ID control data), and It is used in step S51 of FIG. 32 (obtaining pen ID control data) and step S78 of FIG. 32 (saving angle information of pen ID control data).

<Display example of selectable candidates>
FIG. 17 is an example of the operation guide 500 and selectable candidates 530 displayed by the operation guide. The operation guide 500 is displayed by the user handwriting the handwritten object 504 (by the timeout of the selectable candidate display timer). The operation guide 500 includes an operation header 520, operation command candidates 510, handwritten recognition character string candidates 506, conversion character string candidates 507, character string/predictive conversion candidates 508, and handwritten object rectangular area display 503. . The selectable candidates 530 are an operation command candidate 510, a handwritten recognition character string candidate 506, a conversion character string candidate 507, and a character string/predictive conversion candidate 508. In this example, there is no language conversion string, but it may be displayed. Furthermore, the selectable candidates 530 excluding the operation command candidate 510 are referred to as character string candidates 539.

Operation header 520 has buttons 501, 509, 502, 505. A button 501 accepts an operation for switching between predictive conversion and kana conversion. In the example of FIG. 17, when the user presses the button 509 that is displayed as "prediction", the handwriting input section 21 accepts it, notifies the handwriting input display control section 23 to that effect, and the display section 22 displays "Kana". The display changes to button 509. After the change, the character string candidates 539 are arranged in descending order of probability of "kana conversion".

A button 502 operates the page for displaying candidates. In the example of FIG. 17, there are three candidate display pages, and the first page is currently being displayed. Button 505 accepts deletion of operation guide 500. When the user presses the button 505, the handwritten input unit 21 accepts the button, notifies the handwritten input display control unit 23 of this, and the display unit 22 erases the display other than the handwritten object. Button 509 accepts batch display deletion. When the user presses the button 509, the handwritten input unit 21 accepts this and notifies the handwritten input display control unit 23 of this fact. The display unit 22 erases all the displays shown in FIG. 17, including the handwritten object. Users can start over by handwriting.

A handwritten object 504 is the character "gi" handwritten by the user. A handwritten object rectangular area display 503 surrounding the handwritten object 504 is displayed. The display procedure will be explained using sequence diagrams in FIGS. 26 to 32. In the example of FIG. 17, a handwritten object rectangular area display 503 is displayed in a dotted line frame.

The handwritten recognition character string candidates 506, the conversion character string candidates 507, and the character string/predictive conversion candidates 508 are arranged in descending order of probability. “gi” in the handwritten recognition character string candidate 506 is a recognition result candidate. In this example, "gi" is correctly recognized.

A converted character string candidate 507 is a converted character string candidate converted from a language character string candidate. In this example, "skill test" is an abbreviation for "technical mass production trial." The character string/predicted conversion candidate 508 is a predicted character string candidate converted from a language character string candidate or a converted character string candidate. In this example, "Approval of skill test" and "Delivery address for minutes" are displayed.

The operation command candidate 510 is an operation command candidate selected based on the operation command definition data 701 to 703 and 709 to 712 in FIG. 13(a). In the example of FIG. 17, the bullet ">" 511 indicates that it is a candidate for an operation command. In FIG. 17, there is no selection object selected by the handwritten object 504 "gi", and "minutes", which is a character string candidate for "gi", partially matches the operation command definition data 701, 702 in FIG. 13(a). Therefore, it is displayed as an operation command candidate 510.

When the user selects "Load minutes template", the operation command defined in the operation command definition data 701 is executed, and when the user selects "Save in minutes folder", the operation defined in the operation command definition data 702 is executed. The command is executed. In this way, operation command candidates are displayed when operation command definition data that includes a converted character string are found, so they are not always displayed.

As shown in FIG. 17, since the character string candidate and the operation command candidate are displayed simultaneously (both), the user can arbitrarily select either the character string candidate he or she intended to input or the operation command.

<Relationship between the position of the operation guide and the position of the handwritten object rectangular area display>
The display unit 22 displays an operation guide 500 including text data at a position corresponding to the position of the stroke data. The display unit 22 displays an operation guide 500 including text data at a position on the screen based on the position of the stroke data. In this way, the position of the operation guide 500 is determined by the position of the stroke data.

FIG. 18 is a diagram illustrating the relationship between the position of the operation guide and the position of the handwritten object rectangular area display. First, the width A and height H1 of the operation guide 500 are constant. The right end of handwritten object rectangular area display 503 and the right end of operation guide 500 match.

The width B of the handwritten object rectangular area display 503 is determined by the length of the handwritten object 504 handwritten by the user. In FIG. 18A, the width B of the handwritten object rectangular area display 503 is one character, and A>B, so the coordinates (x ₀ , y ₀ ) of the upper left corner P of the operation guide 500 are calculated as follows. be done. Note that the coordinates of the upper left corner Q of the handwritten object rectangular area display 503 are (x ₁ , y ₁ ). The height of the handwritten object rectangular area display 503 is assumed to be _H2 .
x ₀ = x ₁ - (A-B)
y ₀ = y ₁ + H ₂

On the other hand, as shown in FIG. 18(b), when the width B of the handwritten object rectangular area display is larger than A, the coordinates (x ₀ , y ₀ ) of the upper left corner P of the operation guide 500 are calculated as follows. .
x ₀ = x ₁ + (B-A)
y0 = _y1 + _H2

Note that although the operation guide 500 is displayed below the handwritten object rectangular area display 503 in FIG. 18 and the like, the operation guide 500 may be displayed above the handwritten object rectangular area display 503.

FIG. 19 shows an operation guide 500 displayed above the handwritten object rectangular area display 503. The method for calculating x ₁ is the same as in FIG. 18, but the method for calculating y ₀ is different.
y ₀ = y ₁ - H ₁
Note that the operation guide 500 may be displayed on the right or left side of the handwritten object rectangular area display 503. Further, if there is no display space for the operation guide 500 because the user has handwritten on the edge of the display, the operation guide 500 is displayed on the side where there is display space.

<Example of specifying selected object>
The handwriting input device 2 of this embodiment allows the user to designate a selected object by selecting the confirmed object by handwriting. The selected object is subject to editing or modification.

FIG. 20 is an example of a diagram illustrating an example of specifying a selected object. In FIG. 20, the handwritten object 11 is shown with a solid black line, the handwritten object rectangular area 12 is shown with gray shading, the confirmed object 13 is shown with a black line, and the selected object rectangular area 14 is shown with a dotted line. In addition, when distinguishing between each, a lowercase alphabet is added to the code. Furthermore, as a determination condition (whether or not there is a predetermined relationship) for determining a confirmed object as a selected object, the straddle line determination condition 406 or enclosing line determination condition 407 of the defined control data shown in FIG. 9 is used. ing.

FIG. 20(a) is an example in which the user specifies two horizontally written confirmed objects 13a and 13b with a straddling line (handwritten object 11a). In this example, the length H1 of the short side and the length W1 of the long side of the handwritten object rectangular area 12a satisfy the straddle line determination condition 406, and the overlap rate with the confirmed objects 13a and 13b is the straddle line determination condition. Since the condition 406 is satisfied, both the confirmed objects 13a and 13b of "minutes" and "giji" are designated as selected objects.

FIG. 20(b) is an example in which a horizontally written confirmed object 13c is designated with a surrounding line (handwritten object 11b). In this example, only the confirmed object 13c "Minutes" whose overlap rate between the confirmed object 13c and the handwritten object rectangular area 12c satisfies the enclosing line determination condition 407 is designated as the selected object.

FIG. 20(c) is an example in which a plurality of vertically written confirmed objects 13d and 13e are designated by a straddling line (handwritten object 11c). In this example, as in FIG. 20(a), the short side length H1 and the long side length W1 of the handwritten object rectangular area 12d satisfy the straddle line determination condition 406, and the "minutes" Since the overlap rate with each of the two confirmed objects 13d and 13e called "Giji" satisfies the straddle line determination condition 406, both the confirmed objects 13d and 13e of "Minutes" and "Giji" are selected as selected objects. specified.

FIG. 20(d) is an example in which a vertically written confirmed object 13f is designated with a surrounding line (handwritten object 11d). In this example, like FIG. 20(b), only the confirmed object 13f "minutes" is designated as the selected object.

<Example of displaying operation command candidates>
FIG. 21 shows a display example of operation command candidates based on the operation command definition data when the handwritten object shown in FIG. 14 is present. FIG. 21(a) shows candidates for editing-related operation commands, and FIG. 21(b) shows candidates for modification-related operation commands. Further, FIG. 21(a) shows an example in which a selected object is designated by the handwritten object 11a of FIG. 20(a).

As shown in FIGS. 21(a) and 21(b), the operation command candidate displayed following the bullet ">" 511 is the main menu 550. The main menu 550 displays the name of the last executed operation command or the name of the first operation command in the operation command definition data. The first bullet “》” 511a on the first line is a candidate for an editing operation command, and the first bullet “》” 511b on the second line is a candidate for a modification operation command.

“>” 512 at the end of the line indicates that there is a submenu (an example of a submenu button). ">" 512a in the first line displays a submenu (last selected) for editing operation command candidates. ">" 512b on the second line displays the remaining submenus regarding candidates for modification-type operation commands. When the user presses ">" 512, a submenu 560 is displayed on the right side. All operation commands defined in the operation command definition data are displayed in the submenu 560. In the display example of FIG. 21(a), a submenu 560 corresponding to ">" 512a on the first line from when the main menu is displayed is also displayed. It may be displayed by pressing ">" 512a on the first line.

When the user presses any operation command name with a pen, the handwriting input display control unit 23 executes the command of the operation command definition data associated with the operation command name on the selected object. That is, when "Delete" 521 is selected, "Delete" is selected, when "Move" 522 is selected, "Move" is selected, when "Rotate" 523 is selected, "Rotate" is selected, and when "Select" 524 is selected, " Execute "Select" respectively.

For example, when the user presses "Erase" 521 with a pen, "Minutes" and "Giji" can be deleted, and when the user presses "Move" 522, "Rotate" 523, and "Select" 524, the bounding box (the circumscribed rectangle of the selected object) can be deleted. ) are displayed, "Move" 522 and "Rotate" 523 allow you to move or rotate by dragging the pen, and "Select" 524 allows you to perform other bounding box operations.

Character string candidates other than operation command candidates, such as “1” 541, “1,” 542, “~” 543, “→” 544, and “⇒” 545, are the recognition results of the straddle line (handwritten object 11a), If the user intends to input a string rather than an operation command, a string candidate can be selected.

In FIG. 21(b), by pressing ">" 512b on the second line, the submenu of FIG. 21(b) is displayed. The display example in FIG. 21(b) also displays a main menu 550 and a submenu 560, similar to FIG. 21(a). Based on the operation command definition data in FIG. 14, when "thick" 531 is selected, "Thick" is selected, when "thin" 532 is selected, "Thin" is selected, and when "large" 533 is selected, "Large" is selected. When "small" 534 is selected, the handwriting input display control unit 23 executes "Small" and when the underline 535 is selected, "Underline" is executed for the selected object.

In addition, how thick should it be made when "Thick" 531 is selected, how thin should it be made when "Thin" 532 is selected, how large should it be made when "Large" 533 is selected, and "Small" Fixed values are separately defined for how much to reduce when 534 is selected, the line type when underline 535 is selected, etc. Alternatively, it is even better if a selection menu is opened separately when the submenu shown in FIG. 21(b) is selected so that the user can make adjustments.

When the user presses ``thick'' 531 with a pen, the handwritten input display control unit 23 thickens the lines that constitute the confirmed objects 13a and 13b, ``minutes'' and ``giji''. When you press "Thin" 532 with a pen, the handwriting input display control unit 23 can make the lines that make up the "minutes" and "giji" thinner, and when you press "Large" 533 with a pen, the handwriting input display control unit 23 can make the lines that make up the "minutes" and "giji" thinner. By pressing "Small" 534 with a pen, the handwritten input display control section 23 can be made smaller, and by pressing "Underline" 535 with a pen, an underline can be added to the handwritten input display control section 23.

FIG. 22 shows a display example of operation command candidates based on the operation command definition data when the handwritten object shown in FIG. 14 is present. The difference from FIGS. 21(a) and 21(b) is that FIGS. 22(a) and 22(b) show an example in which the selected object is designated by the handwritten object 11b (encircled line) in FIG. 20(b). As can be seen by comparing FIGS. 21 and 22, there is no difference in the operation command candidates displayed depending on whether the handwritten object is a line or an enclosing line. The purpose of the handwriting input display control section 23 is to display operation command candidates on the display section 22 when a selected object is designated. However, the handwritten object may be recognized and the operation command candidates to be displayed may be changed depending on the handwritten object. In this case, the recognized handwritten objects (one, circle, etc.) are associated with operation command definition data as shown in FIG.

In FIG. 22, character string candidates other than operation command candidates, such as "○" 551, "∞" 552, "0" 553, "00" 554, and "RO" 555, are the recognition results of the enclosing line (handwritten object 11b). Yes, and if the user intends to input a string rather than an operation command, the user can select a string candidate.

<Example of inputting angle information>
Next, a method for inputting angle information will be explained using FIG. 23. FIG. 23 is an example of a diagram illustrating a method of inputting angle information. In FIG. 23, a case will be described in which a user located at the 3 o'clock direction of the handwriting input device 2 inputs angle information. Since characters etc. handwritten from the 3 o'clock direction are correctly recognized when rotated 90 degrees clockwise, it is preferable to input 90 degree angle information.

In FIG. 23(a), when the angle information of the pen ID control data is 0 degrees (initial value), the user who is present in the 3 o'clock direction of the handwriting input device 2 has handwritten "gi", so the operation guide 500 is Indicates the displayed state. Since the handwriting input device 2 recognizes the character "gi" handwritten from the 3 o'clock direction with the angle information being 0 degrees, selectable candidates 530 that are different from the expected are displayed.

When inputting angle information, the user handwrites a straight line in the operation guide 500 (an example of a predetermined range) from top to bottom as seen from the user. FIG. 23(b) shows an example of this straight line 521. The angle α formed by the straight line 521 in the counterclockwise direction and the 6 o'clock direction where the angle information is 0 degrees is the angle information. That is, the angle α formed by the straight line 522 drawn from the starting point S in the 6 o'clock direction and the straight line 521 input by the user in a counterclockwise direction is the angle information. Simply put, the direction of the end point of the straight line 521 is the angle information. Therefore, the angle information input by the user in FIG. 23(b) is 90 degrees.

Note that to detect a straight line, for example, there is a method of converting the coordinates from the starting point S to the ending point E into a straight line using the least squares method, and comparing the obtained correlation coefficient with a threshold value to determine whether it is a straight line. .

Immediately after the user starts writing the straight line 521 (immediately after touching the starting point S of the straight line 521 with the pen 2500), the handwriting input device 2 erases the operation guide 500. Immediately after writing the straight line 521 (immediately after the pen 2500 is released from the end point E of the straight line 521), the handwriting input device 2 can write the lines at 45 degrees, 90 degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, and 315 degrees. , 360 degrees, the angle α formed above searches for the closest value and determines it as the angle information. The formed angle α itself may be used as angle information. The determined angle information is set in Angle of the pen ID control data. Since the pen event transmitting unit 41 of the pen 2500 transmits the pen ID to the handwriting input device 2 when the pen tip is pressed for handwriting, etc., the handwriting input device 2 can associate the angle information with the pen ID control data. Can be done.

Note that only the operation guide 500 allows the user to input angle information by hand-drawing a straight line. As a result, when the user draws a straight line by hand with a device other than the operation guide 500, it is recognized as "1" or "1", and when the user draws a straight line by hand with the operation guide 500, angle information can be input. That is, the handwriting recognition control unit 26 detects a straight line from a predetermined range and converts stroke data handwritten outside the predetermined range into text data.

FIG. 23(c) shows the operation guide 500 immediately after the operation in FIG. 23(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object (stroke data) is internally rotated clockwise by 90 degrees for handwriting recognition, and the operation guide 500 only rotates by 90 degrees. It is displayed rotated counterclockwise.

FIG. 24 is an example of a diagram illustrating a method of inputting 45 degree angle information. In FIG. 24, a case will be described in which a user who is present in the direction between 4 o'clock and 5 o'clock on the handwriting input device 2 inputs angle information. Since characters etc. handwritten from the middle between the 4 o'clock direction and the 5 o'clock direction are correctly recognized when rotated clockwise by 45 degrees, it is preferable that angle information of 45 degrees is input.

In FIG. 24(a), when the angle information is 0 degrees (initial value), the user located in the middle direction between the 4 o'clock direction and the 5 o'clock direction of the handwriting input device 2 handwritten "gi", so the operation guide 500 and selectable candidates are displayed. The handwriting input device 2 recognizes the character "gi" handwritten from a direction halfway between the 4 o'clock direction and the 5 o'clock direction, with the angle information being 0 degrees, so selectable candidates that are different from expectations are displayed.

When inputting angle information, the user handwrites a straight line in the operation guide 500 from top to bottom as viewed from the user. FIG. 24(b) shows an example of this straight line 521. The angle α formed by the straight line 521 in the counterclockwise direction and the 6 o'clock direction where the angle information is 0 degrees is the angle information. Therefore, the angle information input by the user in FIG. 24(b) is 45 degrees. This angle information (45 degrees) is set in Angle of the pen ID control data.

FIG. 24(c) shows the operation guide 500 immediately after the operation of FIG. 24(b). Since 45 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object is internally rotated clockwise by 45 degrees for handwriting recognition, and the operation guide 500 is rotated counterclockwise by 45 degrees. displayed rotated.

FIG. 25 is an example of a diagram illustrating another method of inputting angle information. In FIG. 25, it is assumed that the user exists in the 3 o'clock direction of the handwriting input device 2. In FIG. 25, the operation guide 500 is displayed because the user who is present in the 3 o'clock direction of the handwriting input device 2 has handwritten "gi" when the angle information is 0 degrees (initial value). Further, the operation guide 500 in FIG. 25 has a rotation operation button 511 in the operation header 520. The icon of the rotation operation button 511 is circular, making it easy for the user to recognize the rotation operation button 511 from any direction.

Each time the user presses the rotation operation button 511 with the pen 2500, the rotation operation button 511 is a button that adds the angle information of the pen ID control data by 90 degrees and changes it to the remainder of dividing by 360 degrees. This remainder becomes angle information. Note that the angle added by pressing the rotation operation button 511 once may be 45 degrees. Further, it is preferable to display the current angle information each time the button is pressed (for example, display the angle information as a pop-up).

<Operating procedure>
The operation of the handwriting input device 2 will be explained using the above configuration and FIGS. 26 to 32. 26 to 32 are sequence diagrams illustrating an example of a process in which the handwriting input device 2 displays character string candidates and operation command candidates. The process in FIG. 26 starts when the handwriting input device 2 starts up (when the application starts up). Note that in FIGS. 26 to 32, the functions in FIG. 7 are indicated by symbols for space reasons.

S1: First, the handwritten input display control unit 23 transmits a handwritten object start signal to the handwritten input storage unit 25. The handwritten input storage unit 25 secures a handwritten object area (memory area for storing handwritten objects). The handwritten object area may be secured after the user contacts the handwriting input section 21 with a pen.

S2: Next, the user touches the handwriting input section 21 with a pen. The handwriting input section 21 detects pen down and transmits the detected pen down to the handwriting input display control section 23 .

S3: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 secures a stroke area.

S4: When the user moves the pen while keeping it in contact with the handwriting input section 21, the handwriting input section 21 transmits pen coordinates to the handwriting input display control section 23.

S5: The handwriting input display control unit 23 specifies the pen ID received from the pen 2500 at the same time as inputting the coordinates, and acquires the current pen ID control data stored in the pen ID control data storage unit 36. Since the pen ID is transmitted when coordinates are input, the stroke and pen ID are associated with each other. The pen ID control data storage section 36 transmits pen ID control data (color, thickness, pattern, angle information) to the handwriting input display control section 23 . Note that the angle information is still at the initial value of zero.

S6: The handwriting input display control unit 23 transmits pen coordinate complementary display data (stroke data obtained by interpolating discrete pen coordinates) to the display unit 22. The display unit 22 interpolates pen coordinates using the pen coordinate complementary display data and displays a line.

S7: The handwritten input display control section 23 transmits the pen coordinates and the reception time to the handwritten input storage section 25. The handwritten input storage unit 25 adds pen coordinates to the stroke. While the user is moving the pen, the handwriting input section 21 periodically repeats sending the pen coordinates to the handwriting input display control section 23, so the processes of steps S4 to S7 are repeated until the pen is raised.

S4: When the user releases the pen from the handwriting input section 21, the handwriting input section 21 transmits pen up to the handwriting input display control section 23.

S9: The handwriting input display control unit 23 transmits the end of the stroke to the handwriting input storage unit 25, and the handwriting input storage unit 25 determines the pen coordinates of the stroke. Once the pen coordinates of a stroke are determined, it is no longer possible to add pen coordinates to the stroke.

S10: Next, the handwritten input display control unit 23 transmits to the handwritten input storage unit 25, based on the handwritten object nearby rectangular area 403, the overlapping state acquisition of the handwritten object nearby rectangular area and the stroke rectangular area. The handwritten input storage section 25 calculates the overlap situation and transmits the overlap situation to the handwriting input display control section 23 .

Subsequent steps S11 to S17 are executed when the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap.

S11: If the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap, one handwritten object is determined, and the handwritten input display control unit 23 sends a held data clear command to the handwritten recognition control unit 26.

S12-14: The handwriting recognition control unit 26 sends the held data clear to the character string conversion control unit 28, predictive conversion control unit 30, and operation command recognition control unit 32, respectively. Clears data related to character string candidates and operation command candidates that have been held by the handwriting recognition control unit 26, character string conversion control unit 28, predictive conversion control unit 30, and operation command recognition control unit 32. Note that at the time of clearing, the last handwritten stroke has not been added to the handwritten object.

S15: The handwritten input display control unit 23 transmits the end of the handwritten object to the handwritten input storage unit 25. The handwritten input storage unit 25 fixes the handwritten object. Confirmation of a handwritten object means that one handwritten object is completed (no more strokes are added).

S16: The handwritten input display control unit 23 transmits the handwritten object start to the handwritten input storage unit 25. In preparation for starting handwriting (pen down) of the next handwritten object, the handwritten input storage unit 25 secures a new handwritten object area.

S17: Next, the handwritten input display control unit 23 transmits a stroke addition request to the handwritten input storage unit 25 regarding the stroke completed in step S9. When steps S11 to S17 are executed, the stroke to be added is the first stroke of the handwritten object, and the handwritten input storage unit 25 adds stroke data to the handwritten object that is being started. If steps S11 to S17 are not performed, the added strokes are added to the handwritten object that is already being handwritten.

S18: Next, the handwriting input display control unit 23 transmits the stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds stroke data to a stroke data holding area (an area where stroke data is temporarily stored) where character string candidates are stored.

S19: The handwriting recognition control unit 26 executes gesture handwriting recognition on the stroke data holding area. Gesture handwriting recognition refers to recognizing angular information from straight lines. Note that since gesture handwriting recognition is performed inside the operation guide 500, the handwriting recognition control unit 26 detects straight lines inside the operation guide 500. The position information of the operation guide 500 is transmitted to the handwriting recognition control unit 26 in step S57, which will be described later.

S20: When a straight line in the operation guide 500 is detected, the angle α between the straight line 522 drawn from the starting point of the straight line in the 6 o'clock direction and the straight line 521 input by the user in a counterclockwise direction is determined in units of 45 degrees. do. Then, the handwriting recognition control unit 26 stores the determined angle information in the pen ID control data storage unit 36 in association with the pen ID included in the stroke data of the straight line 521. Note that step S20 is executed when a straight line is detected within the operation guide 500.

S21: Next, the handwriting recognition control unit 26 specifies the pen ID received from the handwriting input unit 21 and acquires the angle information of the pen ID control data from the pen ID control data storage unit 36.

S22: The handwriting recognition control unit 26 rotates clockwise using the angle information obtained from the stroke data in the stroke data holding area.

S23: As a result, the stroke data becomes 0 degrees with respect to the vertical direction, so the handwriting recognition control unit 26 executes handwriting recognition on the stroke data.

S24: The handwriting recognition control unit 26 transmits the handwritten recognition character string candidate, which is the result of this execution, to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary section 27 sends linguistically likely linguistic character string candidates to the handwriting recognition control section 26 .

S25: The handwriting recognition control unit 26 transmits the handwritten recognition character string candidates and the received language character string candidates to the character string conversion control unit 28.

S26: The character string conversion control unit 28 transmits the handwritten recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 sends the converted character string candidates to the character string conversion control unit 28 .

S27: The character string conversion control unit 28 transmits the received conversion character string candidates to the predictive conversion control unit 30.

S28: The predictive conversion control unit 30 transmits the received conversion character string candidates to the predictive conversion dictionary unit 31. The predictive conversion dictionary section 31 transmits the predicted character string candidates to the predictive conversion control section 30.

S29: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

S30: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 sends the operation command candidates to the operation command recognition control unit 32. Thereby, the operation command recognition control unit 32 can obtain operation command candidates corresponding to the operation command definition data having a character string (String) that matches the predicted character string candidate.

Thereafter, similar processing is performed up to the transmission of operation command candidates explained in steps S31 to S38.
S31: The character string conversion control unit 28 transmits the received conversion character string candidate to the operation command recognition control unit 32.

S32: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 sends the operation command candidates to the operation command recognition control unit 32. Thereby, the operation command recognition control unit 32 can obtain operation command candidates corresponding to the operation command definition data having a character string (String) that matches the converted character string candidate.

S33: The handwriting recognition control unit 26 transmits the handwritten recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

S34: The predictive conversion control unit 30 transmits the handwritten recognition character string candidates and the received language character string candidates to the predictive conversion dictionary unit 31. The predictive conversion dictionary section 31 transmits the predicted character string candidates to the predictive conversion control section 30.

S35: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

S36: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 sends the operation command candidates to the operation command recognition control unit 32. Thereby, the operation command recognition control unit 32 can obtain operation command candidates corresponding to the operation command definition data having a character string (String) that matches the predicted character string candidate.

S37: The handwriting recognition control unit 26 transmits the handwritten recognition character string candidates and the received language character string candidates to the operation command recognition control unit 32.

S38: The operation command recognition control unit 32 transmits the handwritten recognition character string candidates and the received language character string candidates to the operation command definition unit 33. The operation command definition unit 33 sends the operation command candidates to the operation command recognition control unit 32. Thereby, the operation command recognition control unit 32 can obtain operation command candidates corresponding to the operation command definition data having a string that matches the language character string candidate.

S39: Next, the handwriting recognition control unit 26 transmits stroke addition to the operation command recognition control unit 32.

S40: The operation command recognition control unit 32 transmits the acquisition of position information of the confirmed object to the handwritten input storage unit 25. The handwritten input storage section 25 transmits the position information of the confirmed object to the operation command recognition control section 32.

S41: In order to determine the selected object, the operation command recognition control unit 32 uses the stroke position information received from the handwriting recognition control unit 26 due to stroke addition in step S39 and the position information of the confirmed object received from the handwriting input storage unit 25. It is determined whether or not there is a predetermined relationship based on the straddle line determination condition 406 and the enclosing line determination condition 407, and a confirmed object that can be determined to be selected is saved as a selected object. Furthermore, in this case, since the selected object is specified, candidates for input/output operation commands are obtained from the operation command definition unit 33.

In addition, the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each include a handwritten recognition character string candidate, a language character string candidate, a converted character string candidate, and a predicted character string. Data related to candidates, operation command candidates, and selected objects are held so that they can be obtained in subsequent steps S46 to S49.

S18-2: Immediately after transmitting the stroke addition to the handwriting recognition control unit 26 in step S18, the handwriting input display control unit 23 transmits a selectable candidate display timer start command to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

Subsequent steps S42 to S44 are executed if the pen down occurs before a certain period of time has elapsed (before the timer times out).

S42: If the user touches the handwriting input unit 21 with a pen before the timer times out, the handwriting input unit 21 transmits pen down (the same event as in step S2) to the handwriting input display control unit 23.

S43: The handwritten input display control unit 23 transmits a stroke start (same as step S3) to the handwritten input storage unit 25. The subsequent sequence is the same as step S3 and subsequent steps.

S44: Furthermore, the handwritten input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer. This is because a timer is no longer necessary since pen-down has been detected.

Steps S45 to S79 are executed if pen-down does not occur before a certain period of time has elapsed (before the timer times out). Therefore, the operation guide 500 shown in FIG. 17 is displayed.

S45: If the user does not touch the handwriting input unit 21 with the pen while the selectable candidate display timer is starting, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

S46: The handwritten input display control unit 23 transmits the handwritten recognition character string/language character string candidate acquisition to the handwritten recognition control unit 26. The handwriting recognition control unit 26 transmits the currently held handwritten recognition character string/language character string candidates to the handwriting input display control unit 23 .

S47: The handwritten input display control unit 23 transmits the conversion character string candidate acquisition to the character string conversion control unit 28. The character string conversion control unit 28 transmits the currently held converted character string candidates to the handwriting input display control unit 23.

S48: The handwritten input display control unit 23 transmits the prediction character string candidate acquisition to the prediction conversion control unit 30. The predictive conversion control unit 30 transmits the currently held predicted character string candidates to the handwriting input display control unit 23.

S49: The handwritten input display control unit 23 transmits the operation command candidate acquisition to the operation command recognition control unit 32. The operation command recognition control section 32 transmits the currently held operation command candidates and selected objects to the handwriting input display control section 23 .

S50: Further, the handwriting input display control unit 23 transmits the estimated writing direction acquisition to the handwriting input storage unit 25. The handwritten input storage section 25 determines the stroke addition time, horizontal distance, and vertical distance of the handwritten object rectangular area, and transmits the estimated writing direction to the handwritten input display control section 23 .

S51: Next, the handwriting input display control section 23 specifies the pen ID received from the handwriting input section 21 and acquires the angle information of the current pen ID control data from the pen ID control data storage section 36. This is to rotate the selectable candidates and the operation guide 500.

S52: The handwriting input display control unit 23 selects these handwritten recognition character string candidates (“gi” in FIG. 17), language character string candidates (not displayed in FIG. 17, but for example “Ki”), and conversion character string candidates (“gi” in FIG. 17). ``Minutes'', ``Skills test''), predicted character string candidates (``Settle skill test'', ``Send minutes'' in Figure 17), operation command candidates (``Load minutes template'', ``Send minutes'' in Figure 17), (save in the minutes folder), each selection probability, and estimated writing direction to create selectable candidate display data as shown in FIG. The selectable candidate display data (operation guide 500) is rotated counterclockwise, and the rotated selectable candidate display data (operation guide 500) is transmitted to the display unit 22 to be displayed.

S53: The handwriting input display control unit 23 also rotates the handwritten object and the selected object's rectangular area display data (rectangular frame) (handwritten object rectangular area display 503 in FIG. 17) counterclockwise using the angle information acquired in step S51. The image is rotated and transmitted to the display unit 22 for display.

S54: The handwritten input display control unit 23 transmits a selectable candidate display deletion timer start command to the candidate display timer control unit 24 in order to delete the selectable candidate display data after a certain period of time from the display. The candidate display timer control unit 24 starts this timer.

Steps S55 to S60 are performed when the user erases the selectable candidate display displayed on the display unit 22 or when a change in the handwritten object occurs (that is, when a stroke of the handwritten object is added, (deleted, moved, transformed, or split), or if no candidate is selected by the timeout.

Furthermore, steps S55, S56, and S57 are executed when the candidate display is deleted or when a change in the handwritten object occurs.

S55: The handwriting input unit 21 transmits the occurrence of deletion of the selectable candidate display or change of the handwritten object to the handwriting input display control unit 23.

S56: The handwritten input display control unit 23 transmits a selectable candidate deletion timer stop command. The candidate display timer control unit 24 stops the timer. This is because the handwritten object is operated within a certain period of time, so the timer becomes unnecessary.

S57: The handwriting input display control unit 23 stores the position information of the operation guide 500 in the handwriting recognition control unit 26 so that it can be used in the gesture determination of gesture handwriting recognition in step S19. The position information is, for example, the coordinates of the upper left corner and the lower right corner, or equivalent coordinates. Thereby, the handwriting recognition control unit 26 can determine whether the straight line used for inputting angle information is within the operation guide 500.

S59: The handwritten input display control unit 23 sends the selectable candidate display data deletion command to the display unit 22 to erase the display.

S60: The handwritten input display control unit 23 sends a request to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22, thereby erasing the display. Therefore, if the display of the operation command candidate is deleted under a condition other than that the operation command candidate is selected, the handwritten object remains displayed.

S58: On the other hand, if the selectable candidate display does not disappear or the handwritten object changes while the selectable candidate deletion timer is starting (if the user does not perform a pen operation), the candidate display timer control unit 24 times out. is transmitted to the handwritten input display control section 23.

Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S59 and S60. This is because the selectable candidate display data and the rectangular area display data of the handwritten object and the selected object may be deleted after a certain period of time has elapsed.

If the user selects a selectable candidate while the selectable candidate deletion timer is starting, steps S61 to S79 are executed.

S61: If the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the selection of character string candidates or operation command candidates to the handwriting input display control unit 23.

S62: The handwritten input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

S63: Next, the handwriting input display control section 23 transmits a held data clear command to the handwriting recognition control section 26.

S64: The handwriting recognition control unit 26 transmits a held data clear command to the character string conversion control unit 28.

S65: The handwriting recognition control unit 26 transmits the held data clear to the predictive conversion control unit 30.

S66: The handwriting recognition control unit 26 transmits a held data clear command to the operation command recognition control unit 32.

The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data related to the character string candidates and operation command candidates that have been held so far.

S67: Next, the handwritten input display control unit 23 sends a command to delete selectable candidate display data to the display unit 22 to erase the display.

S68: The handwritten input display control unit 23 erases the display by transmitting a message to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22.

S69: The handwritten input display control unit 23 erases the display by transmitting to the display unit 22 the message for erasing the handwritten object display data and erasing the pen coordinate complementary display data transmitted in step S5. This is because a handwritten object or the like is no longer necessary because a character string candidate or an operation command candidate has been selected.

S70: The handwritten input display control unit 23 transmits handwritten object deletion to the handwritten input storage unit 25.

If a character string candidate is selected, steps S71 to S73 are executed.

S71: If a character string candidate is selected, the handwritten input display control unit 23 transmits the addition of a character string object to the handwritten input storage unit 25.

S72: Furthermore, the handwritten input display control unit 23 transmits the character string object font acquisition to the handwritten input storage unit 25. The handwritten input storage section 25 selects a defined font from the estimated character size of the handwritten object and transmits it to the handwritten input display control section 23 .

S73: Next, the handwritten input display control unit 23 uses the predefined font received from the handwritten input storage unit 25 to cause the display unit 22 to transmit character string object display data to be displayed at the same position as the handwritten object. Display.

If an operation command candidate is selected, steps S74 to S78 are executed. Further, if there is a selected object, steps S74 to S76 are executed.

S74: If a candidate for the operation command for the selected object is selected (if the selected object exists), the handwritten input display control unit 23 erases the display by sending a message to erase the selected object display data to the display unit 22. . This is to temporarily erase the original selected object.

S75: Next, the handwritten input display control unit 23 transmits an instruction to execute an operation command for the selected object to the handwritten input storage unit 25. The handwritten input storage section 25 transmits the display data (edited or modified display data) of the new selected object to the handwritten input display control section 23 .

S76: Next, the handwritten input display control unit 23 transmits selected object display data to the display unit 22 to redisplay the selected object after the operation command has been executed.

If there is no selected object (if an input/output operation command is selected), steps S77 and S78 are executed.

S77: When an input/output type operation command is selected, the handwritten input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user. Note that when the user authentication unit 34 successfully authenticates the user, the handwriting input display control unit 23 sets information regarding the user who has been successfully authenticated to % to % of the operation command and executes the operation command.

S78: As shown in FIG. 25, when the user presses the rotation operation button 511 of the operation header 520, the handwriting input display control unit 23 receives angle information depending on the number of presses of the rotation operation button 511. The handwriting input display control unit 23 stores the received angle information in the pen ID control data storage unit 36 in association with the pen ID received from the pen 2500 when the rotation operation button 511 is pressed.

S79: For the next handwritten object, the handwritten input display control unit 23 transmits a handwritten object start signal to the handwritten input storage unit 25. The handwritten input storage unit 25 secures a handwritten object area. After this, the processes of steps S2 to S79 are repeated.

<Displaying multiple text data in alignment>
When the user selects a language character string candidate, a conversion character string candidate, or a character string/predictive conversion candidate from the selectable candidate display data as shown in FIG. Display column candidates or character string/predictive conversion candidates as text data.

FIG. 33(a) shows an example of text data obtained by converting stroke data handwritten by the user at the 6 o'clock, 3 o'clock, 12 o'clock, and 9 o'clock positions. In FIG. 33(a), text data "abc" is displayed to users at respective positions. If a user wants to read such text data, it will be easier to read if the orientation of the text data is aligned. Therefore, in this embodiment, operation command definition data 712 for aligning text directions is prepared in the operation command definition section 33. When this operation command is executed, the handwritten input display control unit 23 converts the character string object stored in the handwritten input storage unit 25 to the angle (Angle) of the handwritten input stored data associated with the character string object. Rotate each stroke clockwise using .

FIG. 33(b) shows each text data rotated clockwise. The angle of each text data is 0 degrees with respect to the vertical direction. Therefore, Mr. A can read the text data without moving his location.

<Summary>
As explained above, the handwriting input device 2 of this embodiment allows each user to input handwriting with different settings, so users around the handwriting input device 2 placed horizontally can handwrite characters, etc. However, the characters can be recognized correctly. In other words, even if multiple users input or operate the pen 2500 at the same time, each user's input or operation will be independently reflected in the operating conditions of the pen ID. I can concentrate. Since each user's position (angle information) can be set independently, even if each user writes from different angles at the same time, handwriting can be recognized correctly and operation guides can be displayed. Further, since the angle information can be set by handwriting a straight line in the operation guide, the operation can be performed intuitively.

Furthermore, the handwriting input device 2 of this embodiment does not require the operation of selecting an operation menu and selecting an operation from a list of buttons, and can input operation commands in the same way as when handwriting characters or the like. Since the operation command and the selectable candidates 530 are displayed simultaneously on the operation guide, the user can use the handwriting input device 2 without distinguishing between inputting characters and the like and selecting the operation command. Candidates for arbitrary operation commands can be displayed by the user handwriting a handwritten object or by enclosing a confirmed object with an enclosing line. Therefore, any function (editing function, input/output function, pen function, etc.) can be called from the handwritten state. This eliminates the need for a step-by-step operation to call up a desired function by pressing a menu button, and reduces the number of operating procedures from the user's handwriting to calling up a desired function.

<<Another example 1 of the configuration of the handwriting input device>>
Although the handwriting input device of this embodiment is described as having a large touch panel, the handwriting input device is not limited to having a touch panel.

FIG. 34 is a diagram showing another example of the configuration of the handwriting input device. In FIG. 34, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to a handwriting input device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user writes directly with a marker. Note that the whiteboard may be a blackboard, as long as it is a flat surface large enough to project an image.

The projector 411 has an ultra-short focus optical system and can project an image with little distortion onto the whiteboard 413 from a distance of about 10 cm. This video may be transmitted from the PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

The user writes by hand on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has, for example, a light emitting part at its tip that is turned on and emits light when the user presses it against the whiteboard 413 for handwriting. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures an image of the light emitting unit, analyzes the image, and identifies the direction of the electronic pen 2501. Further, the electronic pen 2501 emits sound waves along with light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The position of the electronic pen 2501 can be specified based on the direction and distance. A stroke is drawn (projected) at the position of the electronic pen 2501.

Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button based on the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the stroke (set of coordinates) handwritten by the user is saved by the projector 411. The projector 411 stores handwritten information in a predetermined server 412, USB memory 2600, or the like. Handwritten information is saved for each page. The coordinates are saved as they are, not the image data, so the user can re-edit them. However, in this embodiment, the menu 430 does not need to be displayed because operation commands can be called by hand.

<<Another example 2 of the configuration of handwriting input device>>
FIG. 35 is a diagram showing another example of the configuration of the handwriting input device. In the example of FIG. 35, the handwriting input device includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

The terminal device 600 is connected to the image projection device 700A and the pen motion detection device 810 by wire. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

The pen motion detection device 810 communicates with the electronic pen 820 and detects the motion of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating the point indicated by the electronic pen 820 on the screen 800 and transmits it to the terminal device 600.

The terminal device 600 generates image data of a stroke image input by the electronic pen 820 based on the coordinate information received from the pen motion detection device 810. A stroke image is drawn on the screen 800 by the image projection device 700A.

Further, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

<<Another example 3 of the configuration of handwriting input device>>
FIG. 36 is a diagram illustrating a configuration example of a handwriting input device. The example in FIG. 36 includes a terminal device 600, a display 800A, and a pen motion detection device 810 as a handwriting input device.

Pen motion detection device 810 is placed near display 800A. Pen motion detection device 810 detects coordinate information indicating the point indicated by electronic pen 820A on display 800A, and transmits it to terminal device 600. Note that in the example of FIG. 36, the electronic pen 820A may be charged by the terminal device 600 via the USB connector.

Terminal device 600 generates image data of a stroke image input by electronic pen 820A based on the coordinate information received from pen motion detection device 810. Terminal device 600 displays the stroke image on display 800A.

<<Another example 4 of the configuration of handwriting input device>>
FIG. 37 is a diagram illustrating a configuration example of a handwriting input device. In the example of FIG. 37, the handwriting input device includes a terminal device 600 and an image projection device 700A.

The terminal device 600 performs wireless communication (such as Bluetooth (registered trademark)) with the electronic pen 820B and receives coordinate information of the point indicated by the electronic pen 820B on the screen 800. Then, the terminal device 600 generates image data of a stroke image input by the electronic pen 820B based on the received coordinate information. The terminal device 600 causes the image projection device 700A to project a stroke image.

Further, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

As described above, each of the embodiments described above can be applied in various system configurations.

In this embodiment, a handwriting input system in the form of a system in which an information processing system on a network performs processing such as handwriting recognition and returns the processing results to the handwriting input device 2 will be described.

In the explanation of this embodiment, the components with the same reference numerals or the contents of the figures in Embodiment 1 have the same functions, so the explanation of the components that have already been explained will be omitted or only the differences will be explained. There are cases where

FIG. 38 is an example of a system configuration diagram of the handwriting input system 100. The handwriting input system 100 includes a handwriting input device 2 and an information processing system 10 that can communicate via a network N.

The handwriting input device 2 is placed in a facility such as a company, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 10 is located, for example, in a data center. The handwriting input device 2 is connected to the Internet i via a firewall 8, and the information processing system 10 is also connected to the Internet i via a high-speed LAN in a data center.

The handwriting input device 2 may be connected to the Internet i using wireless communication such as a telephone network. In this case, the wireless communication is 3G (3rd Generation), 4G (4rd Generation), 5G (5rd Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

The information processing system 10 includes one or more information processing devices, and the one or more information processing devices serve as a server and provide services to the handwriting input device 2. A server is a computer or software that functions to provide information and processing results in response to requests from clients. As described later, the information processing system 10 receives pen coordinates from the handwriting input device 2, and transmits information necessary for displaying the operation guide 500 shown in FIG. 17 to the handwriting input device 2.

The server side system is sometimes called a cloud system. A cloud system is a system that uses cloud computing, and cloud computing refers to a usage form in which resources on a network are used without being aware of specific hardware resources. Cloud systems are not necessarily located on the Internet. Although the information processing system is placed on the Internet in FIG. 38, it may be placed on a local network (referred to as on-premises in this case).

Further, in some embodiments, information handling system 10 includes multiple computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

The configuration of the handwriting input device 2 may be the same as that in the first embodiment, but in this embodiment, it is sufficient that it has a touch panel, a display, and a communication function. Handwriting input device 2 may include multiple computing devices configured to communicate with each other.

In this embodiment, a general information processing device such as a PC or a tablet can execute a web browser or a dedicated application. A web browser or a dedicated application communicates with the information processing system 10. When the web browser operates, the user inputs or selects the URL of the information processing system 10 to connect the handwriting input device to the information processing system 10. The handwriting input device 2 executes a web application provided by the information processing system 10 using a web browser. A web application refers to software or its mechanism that is executed by a program written in a programming language (e.g. JavaScript (registered trademark)) that runs on a web browser and a program on a web server that cooperates and is executed on a web browser. .

When a dedicated application operates, it connects to the URL of the information processing system 10 registered in advance. Since the dedicated application has a program and a user interface, the program sends and receives necessary information to and from the information processing system 10 and displays it on the user interface.

The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, WebSocket, or a dedicated communication protocol.

<Hardware configuration example>
The hardware configuration of the handwriting input device 2 may be the same as that shown in FIG. 5. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

FIG. 39 is a hardware configuration diagram of the information processing system 10. As shown in FIG. 39, the information processing system 10 is constructed by a computer, and includes a CPU 601, ROM 602, RAM 603, HD 604, HDD (Hard Disk Drive) controller 605, It is equipped with a display 606, an external device connection I/F (Interface) 608, a network I/F 609, a bus line 610, a keyboard 611, a pointing device 612, a DVD-RW (Digital Versatile Disk Rewritable) drive 614, and a media I/F 616. .

Among these, the CPU 601 controls the operation of the entire information processing system 10. The ROM 602 stores programs used to drive the CPU 601 such as IPL. RAM 603 is used as a work area for CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls reading and writing of various data to the HD 604 under the control of the CPU 601. The display 606 displays various information such as a cursor, menu, window, characters, or images. External device connection I/F 608 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. The network I/F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 601 shown in FIG. 39.

Further, the keyboard 611 includes a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 614 controls reading and writing of various data on a DVD-RW 613, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may be DVD-R or the like. The media I/F 616 controls reading or writing (storage) of data to a recording medium 615 such as a flash memory.

<About device functions>
Next, the functions of the handwriting input system 100 will be described using FIG. 40. FIG. 40 is an example of a functional block diagram showing the functions of the handwriting input system 100 in block form. Note that in the explanation of FIG. 40, differences from FIG. 6 will be mainly explained. Note that the function of the pen 2500 may be the same as in the first embodiment.

In this embodiment, the handwriting input device 2 includes a display section 22 , a display control section 44 , a handwriting input section 21 , and a communication section 42 . Each function that the handwriting input device 2 has is a function that is realized when any of the components shown in FIG. It is a means.

The function of the handwriting input section 21 of this embodiment may be the same as that of the first embodiment. The handwriting input unit 21 converts the user's pen input d1 into pen operation data da (pen up, pen down, or pen coordinate data) and sends it to the display control unit 44.

The display control unit 44 controls the display of the handwriting input device 2. First, the display control unit 44 performs interpolation calculation of the coordinates between the discrete values of the pen coordinate data, which are discrete values, and transmits the pen coordinate data from pen down to pen up as one stroke db to the display unit 22.

Further, the display control unit 44 transmits pen operation data dc to the communication unit 42 and obtains various display data dd from the communication unit 42. The display data includes information for displaying the operation guide 500 in FIG. 17. The display control unit 44 transmits display data de to the display unit 22.

The communication unit 42 transmits pen operation data dc to the information processing system 10, receives various display data dd from the information processing system 10, and transmits it to the display control unit 44 (an example of a first communication unit). The communication unit 42 transmits and receives data in, for example, JSON format or XML format.

The function of the display section 22 may be the same as that in the first embodiment. The display section 22 displays the stroke db and display data de. The display unit 22 converts the stroke db and display data de written into the video memory by the display control unit 44 into data according to the characteristics of the display 220, and transmits the converted data to the display 220.

<<Functions of information processing system>>
The information processing system 10 includes a communication unit 43, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, and a character string conversion control unit 28. It includes a column conversion dictionary section 29, a predictive conversion control section 30, a predictive conversion dictionary section 31, an operation command recognition control section 32, an operation command definition section 33, and a pen ID control data storage section 36. Each of the functions of the information processing system 10 is realized when any of the components shown in FIG. or means.

The communication unit 43 receives the pen operation data dc from the handwriting input device 2 and transmits the pen operation data df to the handwriting input display control unit 23 . The communication unit 43 receives the display data dd from the handwriting input display control unit 23 and transmits it to the handwriting input device 2 (an example of a second communication unit). The communication unit 43 transmits and receives data in, for example, JSON format or XML format.

The other functions are the same as those in the first embodiment, or even if they are different, there is no problem in explaining the present embodiment.

<Operating procedure>
The operation of the handwriting input system 100 will be explained using the above configuration and FIGS. 41 to 48. 41 to 48 are sequence diagrams illustrating an example of a process in which the handwriting input device 2 displays character string candidates and operation command candidates. The process in FIG. 41 starts when the handwriting input device 2 is activated (a web browser or a dedicated application is activated) and communication with the information processing system 10 is established. Note that the overall flow of FIGS. 41 to 48 may be the same as that of FIGS. 30 to 36.

S1: When communication is established, the handwriting input display control unit 23 transmits a handwriting object start signal to the handwriting input storage unit 25 in order to secure the memory area of the handwriting input device 2. The handwritten input storage unit 25 secures a handwritten object area (memory area for storing handwritten objects). The handwritten object area may be secured after the user contacts the handwriting input section 21 with a pen.

S2a: Next, the user touches the handwriting input section 21 with a pen. The handwriting input section 21 detects pen down and transmits the detected pen down to the display control section 44 .

S2b: The display control unit 44 transmits pen down to the communication unit 42 in order to notify the information processing system 10 that the pen is down.

S2c: The communication unit 42 transmits pen down to the information processing system 10.

S2d: The communication unit 43 of the information processing system 10 receives the pen down message and transmits it to the handwriting input display control unit 23.

S3: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 secures a stroke area.

S4a: When the user moves the pen while keeping it in contact with the handwriting input section 21, the handwriting input section 21 transmits the pen coordinates to the display control section 44.

S4b: The display control unit 44 transmits the pen coordinates to the communication unit 42 in order to notify the information processing system 10 of the pen coordinates.

S4c: The communication unit 42 transmits the pen coordinates to the information processing system 10.

S4d: The communication unit 43 of the information processing system 10 receives the pen coordinates and transmits them to the handwriting input display control unit 23.

S5: The display control unit 44 transmits pen coordinate complementary display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 interpolates pen coordinates using the pen coordinate complementary display data and displays a line. The process in step S6 is the same as in the first embodiment.

S8a: When the user releases the pen from the handwriting input section 21, the handwriting input section 21 transmits a pen-up message to the display control section 44.

S8b: The display control unit 44 transmits the pen-up to the communication unit 42 in order to notify the information processing system 10 of the pen-up.

S8c: The communication unit 42 transmits the pen up to the information processing system 10.

S8d: The communication unit 43 of the information processing system 10 receives the pen-up and transmits it to the handwriting input display control unit 23.

The subsequent steps S9 to S17 and steps S18 to S41 are the same as in the first embodiment.

S42a: If the user touches the handwriting input section 21 with a pen before the timer times out, the handwriting input section 21 sends a pen down (the same event as step S2) to the display control section 44. The processing in steps S42b to S42d may be the same as steps S2b to S2d. Further, the processing in steps S43 to S51 is the same as in the first embodiment.

S52a: The handwriting input display control unit 23 creates selectable candidate display data including each character string candidate, operation command candidate, each selection probability, and estimated writing direction shown in FIG. Selectable candidate display data consisting of command candidates is transmitted to the communication unit 43.

S52b: The communication unit 43 transmits selectable candidate display data to the handwriting input device 2.

S52c: The communication unit 42 of the handwriting input device 2 receives the selectable candidate display data and transmits it to the display control unit 44.

S52d: Since the display control unit 44 has received the selectable candidate display data, it transmits it to the display unit 22 to display it.

S53a: Further, the handwritten input display control unit 23 transmits rectangular area display data (rectangular frame) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 17) to the communication unit 43.

S53b: The communication unit 43 transmits the rectangular area display data to the handwriting input device 2.

S53c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data and transmits it to the display control unit 44.

S53d: Since the display control unit 44 has received the rectangular area display data, it transmits it to the display unit 22 to display it. The process in step S54 is the same as in the first embodiment.

S55a: When the user deletes the selectable candidates or additionally writes by hand on the handwritten object, the handwriting input unit 21 transmits the occurrence of deletion of the selectable candidate display or change in the handwritten object to the display control unit 44.

S55b: The display control unit 44 sends a message to the communication unit 42 in order to notify the information processing system 10 of the occurrence of deletion of the selectable candidate display or change of the handwritten object.

S55c: The communication unit 42 transmits the occurrence of deletion of the selectable candidate display or change of the handwritten object to the information processing system 10.

S55d: The communication unit 43 of the information processing system 10 receives the occurrence of selectable candidate display deletion or change in handwritten object, and transmits it to the handwritten input display control unit 23. The processing in steps S56, S57, and S58 is the same as in the first embodiment.

S59a: The handwritten input display control unit 23 transmits the selectable candidate display data deletion to the communication unit 43.

S59b: The communication unit 43 transmits the selectable candidate display data deletion to the handwriting input device 2.

S59c: The communication unit 42 of the handwriting input device 2 receives the selectable candidate display data deletion and transmits it to the display control unit 44.

S59d: Since the display control unit 44 has received the selectable candidate display data deletion request, it transmits it to the display unit 22 to erase the selectable candidates.

S60a: The handwritten input display control unit 23 transmits to the communication unit 43 a request to erase the rectangular area display data of the handwritten object and the selected object.

S60b: The communication unit 43 transmits to the handwriting input device 2 an instruction to erase the rectangular area display data of the handwritten object and the selected object.

S60c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data deletion of the handwritten object and the selected object, and transmits it to the display control unit 44.

S60d: Since the display control unit 44 has received the request to erase the rectangular area display data of the handwritten object and the selected object, it transmits the request to the display unit 22 to erase the rectangular area of the handwritten object and the selected object. Therefore, if the display of the operation command candidate is deleted under a condition other than that the operation command candidate is selected, the handwritten object remains displayed.

If the user selects a selectable candidate while the selectable candidate deletion timer is starting, steps S61 to S79 are executed.

S61a: If the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the selection of character string candidates or operation command candidates to the display control unit 44.

S61b: The display control unit 44 sends a message to the communication unit 42 in order to notify the information processing system 10 of the selection of character string candidates or operation command candidates.

S61c: The communication unit 42 transmits the selection of character string candidates or operation command candidates to the information processing system 10.

S61d: The communication unit 43 of the information processing system 10 receives the selection of character string candidates or operation command candidates and transmits it to the handwritten input display control unit 23. The processing in steps S62 to S66 is the same as in the first embodiment.

S67a: Next, the handwritten input display control unit 23 transmits a selectable candidate display data deletion command to the communication unit 43.

S67b: The communication unit 43 transmits the selectable candidate display data deletion to the handwriting input device 2.

S67c: The communication unit 42 of the handwriting input device 2 receives the selectable candidate display data deletion and transmits it to the display control unit 44.

S67d: Since the display control unit 44 receives the selectable candidate display data deletion command, it causes the display unit 22 to delete the selectable candidates.

S68a: The handwritten input display control unit 23 transmits to the communication unit 43 a request to erase the rectangular area display data of the handwritten object and the selected object.

S68b: The communication unit 43 transmits rectangular area display data deletion to the handwriting input device 2.

S68c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data deletion and transmits it to the display control unit 44.

S68d: Since the display control unit 44 receives the request to erase the rectangular area display data, it causes the display unit 22 to erase the rectangular area.

S69a: The handwritten input display control unit 23 transmits handwritten object display data deletion to the communication unit 43.

S69b: The communication unit 43 transmits handwritten object display data deletion to the handwritten input device 2.

S69c: The communication unit 42 of the handwriting input device 2 receives the handwritten object display data deletion and transmits it to the display control unit 44.

S69d: Since the display control unit 44 has received the handwritten object display data deletion command, it causes the display unit 22 to delete the handwritten object and the pen coordinate complementary display data. The process in step S70 may be the same as in the first embodiment.

If a character string candidate is selected, steps S71 to S73 are executed. The processing in steps S71 and S72 may be the same as in the first embodiment.

S73a: Next, the handwritten input display control unit 23 uses the defined font received from the handwritten input storage unit 25 to transmit character string object display data to be displayed at the same position as the handwritten object to the communication unit 43.

S73b: The communication unit 43 transmits the character string object display data to the handwriting input device 2.

S73c: The communication unit 42 of the handwriting input device 2 receives the character string object display data and transmits it to the display control unit 44.

S73d: Since the display control unit 44 has received the character string object display data, it causes the display unit 22 to display the character string object.

If an operation command candidate is selected, steps S74 to S78 are executed. Furthermore, if there is a selected object, steps S74 to S76 are executed.

S74a: If a candidate for the operation command for the selected object is selected (if the selected object exists), the handwritten input display control unit 23 transmits a command to erase the selected object display data to the communication unit 43. This is to temporarily erase the original selected object.

S74b: The communication unit 43 transmits selected object display data deletion to the handwriting input device 2.

S74c: The communication unit 42 of the handwriting input device 2 receives the selected object display data deletion and transmits it to the display control unit 44.

S74d: Since the display control unit 44 receives the selected object display data deletion command, it causes the display unit 22 to delete the selected object.

S75: Next, the handwritten input display control unit 23 transmits an instruction to execute an operation command for the selected object to the handwritten input storage unit 25. The handwritten input storage section 25 transmits the display data (edited or modified display data) of the new selected object to the handwritten input display control section 23 .

S76a: Next, the handwritten input display control section 23 transmits the selected object display data to the communication section 43.

S76b: The communication unit 43 transmits the selected object display data to the handwriting input device 2.

S76c: The communication unit 42 of the handwriting input device 2 receives the selected object display data and transmits it to the display control unit 44.

S76d: Since the display control unit 44 has received the selected object display data, it causes the display unit 22 to redisplay the selected object after the operation command has been executed. The processing in steps S77 to S79 may be the same as in the first embodiment.

In this way, even in a system configuration in which the handwriting input device 2 and the information processing system 10 communicate, the same effects as in the first embodiment can be achieved. Note that the processing flows shown in FIGS. 41 to 48 are just examples, and may include processing caused by communication between the handwriting input device 2 and the information processing system 10, or may be omitted. good. Further, the handwriting input device 2 may perform part of the processing performed by the information processing system 10. For example, the handwriting input device 2 may perform processing related to deletion.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications can be made without departing from the gist of the present invention. and substitutions can be added.

For example, although this embodiment has been described using an electronic whiteboard as an example, any information processing device having a touch panel can be suitably applied. Information processing devices equipped with touch panels include, for example, projectors (PJs), output devices such as digital signage, HUD (head up display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network appliances, It may be a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

Further, in this embodiment, the coordinates of the pen tip are detected by a method of detecting the coordinates of the pen tip using a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. Further, the pen emits ultrasonic waves as well as emits light, and the handwriting input device 2 calculates the distance based on the arrival time of the ultrasonic waves. The location of the pen can be determined by direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

Furthermore, in the present embodiment, candidates for editing and modification operation commands are displayed when there is a selected object, and candidates for input/output operation commands are displayed when there is no selected object. However, candidates for editing and modification operation commands and candidates for input/output operation commands may be displayed simultaneously.

Further, in the configuration example shown in FIG. 7 and the like, in order to facilitate understanding of the processing by the handwriting input device 2, the system is divided according to main functions. The present invention is not limited by the method of dividing the processing units or the names thereof. The processing of the handwriting input device 2 can also be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Moreover, each function of the embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" as used herein refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function explained above. This includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

The pen ID control data storage section 36 is an example of control data storage means. The handwritten input display control section 23 is an example of display control means. The handwriting recognition control unit 26 is an example of handwriting recognition control means. The communication unit 42 is an example of first communication means. The communication unit 43 is an example of second communication means.

2 Handwriting input device 21 Handwriting input section 22 Display section 23 Handwriting input display control section 24 Candidate display timer control section 25 Handwriting input storage section 26 Handwriting recognition control section 27 Handwriting recognition dictionary section 28 Character string conversion control section 29 Character string conversion dictionary section 30 Predictive conversion control section 31 Predictive conversion dictionary section 32 Operation command recognition control section 33 Operation command definition section 36 Pen ID control data storage section

Japanese Patent Application Publication No. 2018-026185

Claims (10)

A handwriting input device that displays handwritten stroke data based on the position of an input means in contact with a touch panel,
control data storage means for storing control data of the input means in association with identification information of the input means received from the input means;
A display control means for displaying an operation guide including text data on a display section at a position corresponding to a position of stroke data,
display control means that causes the display unit to display information based on the stroke data by reflecting the control data associated with the identification information of the input means received from the input means on the stroke data;
Correlating angle information determined based on an angle between a straight line handwritten on the operation guide and a predetermined direction with identification information of the input means received from the input means when the stroke data of the straight line was handwritten. handwriting recognition control means stored in the control data storage means;
The handwriting recognition control means inputs the angle information when a straight line is handwritten using the operating guide, and recognizes the straight line as a character when the straight line is handwritten using a method other than the operating guide. input device. The control data storage means stores, as the control data, the angle information of the user using the input means relative to the handwriting input device with a predetermined direction as a reference;
The handwriting input device according to claim 1 , wherein the display control means reflects the angle information on the stroke data and displays information based on the stroke data. The handwriting recognition control means recognizes the stroke data and converts it into text data,
The handwriting recognition control means rotates the stroke data based on the angle information, and then recognizes and converts the stroke data into one or more text data;
The handwriting input device according to claim 2 , wherein the display control means displays one or more of the text data as information based on the stroke data. 4. The handwriting input device according to claim 3 , wherein the display control means displays one or more of the text data rotated based on the angle information as information based on the stroke data. The display control means displays an operation header for accepting user operations together with the text data,
5. The handwriting input device according to claim 4 , wherein the display control means displays the text data rotated based on the angle information and the operation header as information based on the stroke data. The operation header that accepts a user's operation that is displayed together with the text data by the display control means has a rotation operation button that accepts the angle information,
Any one of claims 3 to 5, wherein the angle information is stored in the control data storage means in association with identification information of the input means received from the input means when the rotation operation button is pressed. The handwriting input device described in . The handwriting input device according to claim 3, wherein the handwriting recognition control means sets the angle information in units of 90 degrees or 45 degrees. The display control means displays one text data selected by the user from the one or more text data displayed in a rotated manner,
Display multiple text data selected by multiple users,
8. The handwriting input device according to claim 3 , wherein when a predetermined operation is detected, a plurality of text data are displayed together in alignment in the same direction. A handwriting input method in which a handwriting input device displays handwritten stroke data based on the position of an input means in contact with a touch panel, the method comprising:
a step in which the display control means causes the display unit to display an operation guide including text data at a position corresponding to the position of the stroke data;
acquiring control data for the input means that is associated with identification information of the input means from a control data storage means;
display control means causes the display unit to display information based on the stroke data by reflecting the control data associated with the identification information of the input means received from the input means on the stroke data; ,
The handwriting recognition control means transmits the angle information determined based on the angle between the straight line handwritten on the operation guide and the predetermined direction to the input means received from the input means when the stroke data of the straight line is handwritten. storing the control data in the control data storage means in association with the identification information;
the handwriting recognition control means inputting the angle information when a straight line is handwritten using the operating guide, and recognizing the straight line as a character when the straight line is handwritten using a method other than the operating guide;
A handwriting input method characterized by having the following. A handwriting input device that displays handwritten stroke data based on the position of the input means in contact with the touch panel,
communicating with said input means;
displaying an operation guide including text data on a display unit at a position corresponding to the position of the stroke data;
Obtaining control data of the input means associated with identification information of the input means from the control data storage means,
reflecting the control data associated with identification information of the input means received from the input means on the stroke data and displaying information based on the stroke data on the display unit;
Correlating angle information determined based on an angle between a straight line handwritten on the operation guide and a predetermined direction with identification information of the input means received from the input means when the stroke data of the straight line was handwritten. Storing the control data in the control data storage means;
inputting the angle information when a straight line is handwritten using the operating guide, and recognizing the straight line as a character when the straight line is handwritten using a method other than the operating guide;
A program characterized by executing.

Images