JP2020190892A - Display device, program and display method - Google Patents

Abstract

To provide a display device enabling an addition of a character, etc., without displaying a character string input frame.SOLUTION: A display device 2 that displays handwriting data handwritten on the basis of the positions of input means in contact with a touch panel includes: a handwriting recognition control unit 26 that executes a control on a recognition of the handwriting data; a display unit 22 that displays defined data converted from the handwriting data; and a character string addition control unit 41 that adds, when second handwriting data satisfies a predetermined condition relative to the defined data, character string candidates converted from the second handwriting data by the handwriting recognition control unit at the head of the defined data or the tail thereof. The display unit displays, at the head of the defined data or the tail thereof, additional character string candidates obtained by adding the character string candidates converted from the second handwriting data by the handwriting recognition control unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a display device, a program, and a display method.

Some computer-controlled whiteboard devices or applications capable of handwriting input can not only handwrite characters, but also recognize the handwritten characters and convert them into text data for display. For example, the user can select a menu to switch from the handwriting mode to the handwriting character recognition mode and convert the handwritten characters or the like into text data.

While using such a display device, a technique for adding characters or the like to a character string already converted into text data has been devised (see, for example, Patent Document 1). In Patent Document 1, when a user touches a position where a character string displayed by voice recognition is desired to be inserted, a character string input frame is displayed at a predetermined position on a display terminal and input to the character string input frame. A voice recognition error correction device that inserts a character string at a position touched by a user is disclosed.

However, in the conventional technique, there is a problem that it is necessary to display a character string input frame in addition to characters and the like.

An object of the present invention is to provide a display device capable of adding characters and the like without displaying a character string input frame.

In view of the above problems, the present invention is a display device that displays handwritten data handwritten based on the position of the input means in contact with the touch panel, and is a handwritten recognition control unit that controls the recognition of the handwritten data. The display unit that displays the confirmed data converted from the handwritten data by the handwritten recognition control unit and the second handwritten data handwritten around the confirmed data satisfy a predetermined condition with respect to the confirmed data. In this case, the second handwritten data has a character string addition control unit that adds a character string candidate converted by the handwriting recognition control unit at the beginning or the end of the finalized data, and the display unit has the display unit. It is characterized in that an additional character string candidate to which a character string candidate to which the second handwritten data is converted by the handwriting recognition control unit is added is displayed at the beginning or the end of the finalized data.

It is possible to provide a display device capable of adding characters and the like without displaying a character string input frame.

It is a figure explaining the method of adding a character string performed by a display device. It is an overall block diagram of an example of a display device. This is an example of a perspective view of a pen. It is a hardware block diagram of an example of a display device. It is a functional block diagram of an example which shows the function which a display device has in a block shape. It is a functional block diagram of an example which shows the function about the user authentication which a display device has in the form of a block. It is a figure which shows an example of the defined control data. It is a figure which shows an example of the dictionary data of the handwriting recognition dictionary part. It is a figure which shows an example of the dictionary data of the character string conversion dictionary part. It is a figure which shows an example of the dictionary data of the predictive conversion dictionary part. It is a figure which shows an example of the operation command definition data and system definition data held by the operation command definition part. It is a figure which shows an example of the operation command definition data when there is selection data selected by handwriting data. It is a figure which shows an example of the operation guide and the selectable candidate which the operation guide displays when there is no selection data. It is an example of the figure explaining the specification example of the selection data. It is a figure which shows the display example of the candidate of the operation command based on the operation command definition data when there is handwriting data. It is a figure which shows the display example of the candidate of the operation command based on the operation command definition data when there is handwriting data. It is a figure explaining the determination whether or not to display the additional character string candidate which added the character string based on the handwritten data to the confirmed data in the operation guide. It is a figure explaining the use example registration determination by a predictive conversion control unit. It is a figure explaining the position of the additional character string when the character string based on the handwritten data is added to the head of the confirmed data. It is a sequence diagram of an example explaining the process of displaying the character string candidate and the operation command candidate by a display device (the 1). It is a sequence diagram of an example explaining the process which the display device displays a character string candidate and a candidate of an operation command (the 2). It is a sequence diagram of an example explaining the process of displaying the character string candidate and the operation command candidate by a display device (No. 3). It is a sequence diagram of an example explaining the process of displaying the character string candidate and the operation command candidate by a display device (the 4). It is a sequence diagram of an example explaining the process of displaying the character string candidate and the operation command candidate by a display device (the 5). It is a sequence diagram of an example explaining the process which the display device displays the character string candidate and the operation command candidate (the 6). It is a figure explaining the method of adding a character string performed by a display device. This is an example of a flowchart showing a procedure in which a display device adds the other confirmed data to one confirmed data as the confirmed data moves. It is a figure which shows other configuration example of a display device. It is a figure which shows other configuration example of a display device. It is a figure which shows other configuration example of a display device. It is a figure which shows other configuration example of a display device.

Hereinafter, as an example of the embodiment for carrying out the present invention, the display device and the display method performed by the display device will be described with reference to the drawings.

<Outline of operation of display device>
FIG. 1 is a diagram illustrating a method of adding a character string performed by the display device of the present embodiment. In FIG. 1, a case where the character string "memo" is added to the end of the finalized data 573 "today's proceedings" to display "today's proceedings memo" will be described. The finalized data 573 is the data after being converted by character recognition.
(1) Confirmed data 573 "Today's proceedings" has already been displayed. Around the confirmed data 573 (near the end), the user adds handwritten data 504 (an example of the second handwritten data) of "me". Handwritten data is data in a handwritten state.
(2) The display device determines whether or not the handwritten data 504 satisfies a predetermined condition with respect to the confirmed data 573. The predetermined condition is whether or not the distance between the handwritten data 504 and the confirmed data is equal to or less than the threshold value. When the predetermined condition is satisfied, it is determined that the additional character string candidate 580 is included in the operation guide 500 and displayed. The additional character string candidate is a character string in which a character string candidate based on handwritten data is added to the finalized data 573. In addition, the display device displays the character string candidates obtained by the recognition of "me" in the same manner as the normal handwriting input. Therefore, the display device has an additional character string candidate 580 called "today's proceedings memo", a handwritten recognition character string candidate 506 called "me" which is a recognition result of the handwritten data 504, and a converted character string "memo" and "mechanism". Candidate 507 is displayed as a selectable candidate.
(3) When the user selects "Today's proceedings memo", the confirmed data 573 is replaced with the additional character string candidate 580 "Today's proceedings memo". The character string after being replaced is called an additional character string 581.
(4) When the user does not select "Today's proceedings memo" but selects, for example, "memo", the character string "memo" is displayed in place of the handwritten data 504 "me".
(5) If the user does not select anything, the handwritten data 504 called "me" is displayed as it is.

As described above, since the display device of the present embodiment determines whether or not the handwritten data 504 satisfies the predetermined condition with respect to the confirmed data 573, the addition of the character string can be performed without any special operation. Further, since the handwritten recognition character string candidate 506 and the conversion character string candidate 507 are displayed in a selectable manner together with the additional character string candidate 580, the user can add the handwritten data to the finalized data 573, or from the handwritten data. The converted character string can also be displayed independently.

<Terminology>
The input means may be any means capable of handwriting on the touch panel. For example, there are pens, human fingers and hands, rod-shaped members, and the like. In addition, line-of-sight input may be possible.

Confirmed data is a character string that has been converted into text data by character recognition. It doesn't matter if the conversion is correct or not. The characters include numerical values, symbols, alphabets, and the like.

Handwritten data refers to one or more handwritten strokes.

Addition means adding more later.

<Overall configuration of the device>
The overall configuration of the display device 2 according to the present embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an overall configuration diagram of the display device 2. FIG. 2A shows a display device 2 used as a horizontally long electronic blackboard suspended from a wall as an example of the display device 2.

The display device 2 is a device that displays handwritten data handwritten based on the position of the input means in contact with the touch panel. It is also called a handwriting input device because it allows users to input data written by hand.

As shown in FIG. 2A, a display 220 is installed on the upper part of the display device 2. The user U can handwrite (also referred to as input or drawing) characters or the like on the display 220 using the pen 2500.

FIG. 2B shows a display device 2 used as a vertically long electronic blackboard suspended from a wall.

FIG. 2C shows a display device 2 placed flat on the desk 230. Since the display device 2 has a thickness of about 1 cm, it is not necessary to adjust the height of the desk even if it is placed flat on a general desk. Also, it can be easily moved.

<Example of pen appearance>
FIG. 3 shows an example of a perspective view of the pen 2500. FIG. 3 shows an example of a multifunctional pen 2500. A pen 2500 that has a built-in power supply and can send commands to the display device 2 is called an active pen (a pen that does not have a built-in power supply is called a passive pen). The pen 2500 in FIG. 3 has one physical switch on the pen tip, one on the pen bottom, and two on the side of the pen. The pen tip is for writing, the pen bottom is for erasing, and the pen side is for assigning user functions. Is. The pen 2500 of the present embodiment has a non-volatile memory and stores a pen ID that does not overlap with other pens.

If the pen has a switch, it is possible to reduce the operation procedure of the display device 2 by the user. A pen with a switch mainly refers to an active pen, but in the electromagnetic induction method, a passive pen that does not have a built-in power supply can generate electric power only by an LC circuit, so it includes not only an active pen but also an electromagnetic induction passive pen. Pens with optical, infrared, and capacitive switches other than electromagnetic induction are active pens.

It is assumed that the hardware configuration of the pen 2500 is the same as that of a general control method including a communication function and a microcomputer. The coordinate input method of the pen 2500 includes an electromagnetic induction method, an active electrostatic coupling method, and the like. Further, the pen 2500 may have functions such as pen pressure detection, tilt detection, and hover function (displays a cursor before the pen touches it).

<Hardware configuration of the device>
Subsequently, the hardware configuration of the display device 2 will be described with reference to FIG. The display device 2 has an information processing device or a computer configuration as shown in the figure. FIG. 4 is an example of a hardware configuration diagram of the display device 2. As shown in FIG. 4, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204. ing.

Of these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores a program used to drive the CPU 201 such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as a work area of the CPU 201. The SSD 204 stores various data such as a program for the display device 2.

The display device 2 includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221 and a wireless communication device 222, and an infrared I / F 223. , Power control circuit 224, AC adapter 225, and battery 226.

The display controller 213 controls and manages the screen display in order to output the output image to the display 220 or the like. The touch sensor 216 detects that the pen 2500 or the user's hand (the pen or the user's hand serves as an input means) touches the display 220. In addition, the touch sensor 216 receives the pen ID.

The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs coordinates and detects coordinates. As for the method of inputting the coordinates and detecting the coordinates, for example, in the case of an optical type, two light receiving / receiving devices installed at both upper ends of the display 220 emit a plurality of infrared rays in parallel with the display 220 to display the display. This is a method of receiving light that is reflected by a reflecting member provided around 220 and returns on the same optical path as the light path emitted by the light receiving element. The touch sensor 216 outputs infrared position information emitted by two light emitting / receiving devices blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies a coordinate position which is a contact position of the object. Further, the touch sensor controller 215 has a communication unit 215a and can wirelessly communicate with the pen 2500. For example, when communicating according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate without setting the connection for communicating the pens 2500 with the display device 2.

The power switch 227 is a switch for switching ON / OFF of the power supply of the display device 2. The tilt sensor 217 is a sensor that detects the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is used in the installation state of FIG. 2A, FIG. 2B, or FIG. 2C, depending on the installation state. The thickness of characters etc. can be changed automatically.

The serial interface 218 is a communication interface with the outside such as USB. It is used for inputting information from the outside. The speaker 219 is used for audio output and the microphone 221 is used for audio input. The wireless communication device 222 communicates with a terminal carried by the user and relays a connection to the Internet, for example. The wireless communication device 222 communicates by Wi-Fi, Bluetooth (registered trademark), or the like, but the communication standard does not matter. The wireless communication device 222 forms an access point, and when the SSID (Service Set Identifier) and the password obtained by the user are set in the terminal carried by the user, the access point can be connected.

It is preferable that the wireless communication device 222 is provided with two access points.
a. Access point → Internet
b. Access point-> internal network-> Internet a The access point is for external users, and the user cannot access the internal network, but can use the Internet. The access point of b is for internal users, and the users can use the internal network and the Internet.

The infrared I / F 223 detects a display device 2 arranged adjacent to the display device 2. Only the display devices 2 arranged adjacent to each other can be detected by utilizing the straightness of infrared rays. It is preferable that one infrared ray I / F 223 is provided on each side, and it is possible to detect in which direction of the display device 2 the other display device 2 is arranged. As a result, the screen is expanded, and handwritten information handwritten in the past (handwritten information on another page with the size of one display 220 as one page) and the like can be displayed on the adjacent display device 2.

The power supply control circuit 224 controls the AC adapter 225 and the battery 226, which are the power supplies of the display device 2. The AC adapter 225 converts alternating current shared by commercial power sources into direct current.

If the display 220 is so-called electronic paper, it can also be driven by the battery 226 because it consumes little or no power to maintain the display of the image. This makes it possible to use the display device 2 for applications such as digital signage even in places where it is difficult to connect a power source such as outdoors.

Further, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG.

The touch sensor 216 is not limited to the optical type, but is not limited to the optical type, but is a capacitance type touch panel that specifies the contact position by detecting the change in capacitance, and the contact position is specified by the voltage change of two opposing resistance films. Various detection means such as a resistive touch panel and an electromagnetic induction type touch panel that detects the electromagnetic induction generated by the contact object coming into contact with the display unit and specifies the contact position may be used. The touch sensor 216 may be of a method that does not require an electronic pen to detect the presence or absence of touch of the pen tip. In this case, you can use your fingertips or a pen-shaped stick to perform touch operations. The pen 2500 does not have to be an elongated pen type.

<About the function of the device>
Next, the function of the display device 2 will be described with reference to FIG. FIG. 5 is an example of a functional block diagram showing the functions of the display device 2 in a block shape. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, and a character string conversion control. It includes a unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, and a character string additional control unit 41. Each function of the display device 2 is a function realized by operating any of the components shown in FIG. 4 by an instruction from the CPU 201 according to a program developed on the RAM 203 from the SSD 204. It is a means.

The handwriting input unit 21 is realized by a touch sensor 216 or the like, and accepts handwriting input by the user. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and the coordinates between the discrete values are complemented.

The display unit 22 is realized by a display 220 or the like, and displays handwritten data and an operation menu. The display unit 22 converts the drawing data d3 written by the handwritten input display control unit 23 into the video memory into data according to the characteristics of the display 220, and transmits the drawing data d3 to the display 220.

The handwriting input display control unit 23 performs overall control regarding handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 to display it.

The candidate display timer control unit 24 is a display control timer for selectable candidates. Generates the timing to start or stop the timer to start the display of selectable candidates and the timing to delete the display. The candidate display timer control unit 24 receives the timer start request d4 (which may be a timer stop request) from the handwriting input display control unit 23, and transmits the timeout event d5 to the handwriting input display control unit 23.

The handwriting input storage unit 25 has a storage function for storing user data (handwriting data / character string data). The handwriting input storage unit 25 receives the user data d6-1 from the handwriting input display control unit 23, saves it in the handwriting input storage unit 25, receives the acquisition request d6-2 from the handwriting input display control unit 23, and saves the handwriting input. The user data d7 stored in the unit 25 is transmitted. The handwriting input storage unit 25 transmits the position information d36 of the confirmed data to the operation command recognition control unit 32.

The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike general OCR (Optical Character Reader), characters (multilingual languages such as English as well as Japanese), numbers, symbols (%, $, &, etc.) and figures (lines) are operated in parallel with the user's pen operation. , Circles, triangles, etc.) Various algorithms have been devised for the recognition method, but in the present embodiment, the details are omitted because a known technique can be used.

The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, executes handwriting recognition, and holds a handwriting recognition character string candidate. Further, the handwriting recognition control unit 26 uses the handwriting recognition dictionary unit 27 to hold the language character string candidate converted from the handwriting recognition character string candidate d12. When the acquisition request d8-2 is separately received from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the held handwriting recognition character string candidate and language character string candidate d9 to the handwriting input display control unit 23. ..

The handwriting recognition dictionary unit 27 is dictionary data for language conversion of handwriting recognition. The handwriting recognition dictionary unit 27 receives the handwriting recognition character string candidate d12 from the handwriting recognition control unit 26, converts it into a linguistically certain language character string candidate d13, and transmits it to the handwriting recognition control unit 26. For example, in the case of Japanese, Hiragana is converted to Kanji or Katakana.

The character string conversion control unit 28 controls conversion of the conversion character string candidate into a character string. The converted character string is a character string that is likely to be generated including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives the handwriting recognition character string and the language character string candidate d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and holds them. When the acquisition request d14 is separately received from the handwriting input display control unit 23, the held conversion character string candidate d15 is transmitted to the handwriting input display control unit 23.

The character string conversion dictionary unit 29 is dictionary data for character string conversion. The character string conversion dictionary unit 29 receives the handwriting recognition character string and the language character string candidate d17 from the character string conversion control unit 28, and transmits the conversion character string candidate d18 to the character string conversion control unit 28.

The predictive conversion control unit 30 receives the handwritten recognition character string and the language character string candidate d10 from the handwritten recognition control unit 26, receives the conversion character string candidate d16 from the character string conversion control unit 28, and uses the predictive conversion dictionary unit 31 for each of them. Use to convert to predictive string candidates. The predictive conversion character string is a character string that is likely to be generated including a handwriting recognition character string, a language character string, or a conversion character string. When the acquisition request d19 is separately received from the handwriting input display control unit 23, the predicted character string candidate d20 is transmitted to the handwriting input display control unit 23.

The predictive conversion dictionary unit 31 is dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives the handwriting recognition character string, the language character string candidate, and the conversion character string candidate d21 from the predictive conversion control unit 30, and transmits the predictive character string candidate d22 to the predictive conversion control unit 30.

The operation command recognition control unit 32 receives the handwritten recognition character string and the language character string candidate d30 from the handwriting recognition control unit 26, receives the conversion character string candidate d28 from the character string conversion control unit 28, and predicts the predicted character string candidate d29. Received from the conversion control unit 30. Then, the operation command conversion request d26 is transmitted to the operation command definition unit 33 for each, and the operation command candidate d27 is received from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidate d27.

When the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 transmits the operation command candidate d27 to the operation command recognition control unit 32.

Further, the operation command recognition control unit 32 receives the pen operation data d24-1 from the handwritten input display control unit 23, and transmits the position information acquisition request d23 of the confirmed data input and confirmed in the past to the handwritten input storage unit 25. The confirmed data specified by the pen operation data is retained as selection data (including position information). The operation command recognition control unit 32 specifies the position of the pen operation data d24-1 and selection data satisfying a predetermined criterion. When the acquisition request d24-2 is separately received from the handwriting input display control unit 23, the selection data d25 identified as a candidate for the held operation command is transmitted to the handwriting input display control unit 23.

The character string addition control unit 41 receives the pen operation data d43 from the handwriting input display control unit 23, transmits the acquisition request d37 of the confirmed data input and confirmed in the past to the handwriting input storage unit 25, and the pen operation data of the handwriting data. And confirm data (including position information) is retained. The character string addition control unit 41 compares the positions of the pen operation data of the handwritten data and the confirmation data, and determines whether or not the addition determination condition 408 is satisfied.

Further, the character string addition control unit 41 receives the distance determination result acquisition d38 from the handwriting input display control unit 23, and transmits the distance determination result d39 to the handwriting input display control unit 23. When the distance determination result is added, the handwriting input display control unit 23 transmits the acquisition request d40 of the confirmed data input and confirmed in the past to the handwriting input storage unit 25, and acquires the confirmed data (including the position information). The handwriting input display control unit 23 transmits the usage example registration determination request d41 to the predictive conversion control unit 30, and acquires the usage example character string d42 from the predictive conversion control unit 30.

<About user authentication>
In the present embodiment, since control is performed using the result of user authentication, it is preferable that the display device 2 has a function of user authentication. Therefore, the function related to user authentication will be described with reference to FIG.

FIG. 6 is a block-shaped diagram showing the user authentication function of the display device 2. Although only the handwriting input display control unit 23 is shown in FIG. 6 as a function related to the user authentication unit 34 (user authentication means), each function shown in FIG. 5 may be used as the user authentication result.

The authentication information acquisition unit 35 acquires the authentication information d31 from the user U. The authentication information d31 may be the card number of the IC card, the user ID and password, or biometric information such as a fingerprint. The user authentication unit 34 acquires the authentication information d32 from the authentication information acquisition unit 35, and searches the authentication information d33 in the user information DB 36. If there is a user who matches the search, the user information d34 is acquired from the user information DB 36. The user information may be any information representing the attributes of the user, for example, a user name, a password, a computer name of the user, a department, an authority, and the like.

Since the user authentication unit 34 transmits the user information d35 to the handwriting input display control unit 23, the handwriting input display control unit 23 can execute an operation command using the user information. The operation command using the user information will be described with reference to FIG.

In addition to the display device 2, the authentication function may be possessed by an external authentication server or the like. In this case, the display device 2 transmits the authentication information to the authentication server and acquires the authentication result and the user information.

<About defined control data>
Next, the defined control data used by the display device 2 for various processes will be described with reference to FIG. 7. FIG. 7 shows an example of defined control data. In the example of FIG. 7, control data is shown for each control item.

The selectable candidate display timer 401 defines the time until the selectable candidate is displayed (an example of the first time). This is because the selectable candidates are not displayed during handwriting. In FIG. 7, it means that selectable candidates are displayed if pen down does not occur within TimerValue = 500 [ms] from pen up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used at the start of the selectable candidate display timer in step S18 of FIG. 22, which will be described later.

The selectable candidate erase timer 402 defines the time until the displayed selectable candidate is erased (an example of the second time). This is to delete the selectable candidates when the user does not select the selectable candidates. In FIG. 7, it means that the selectable candidate display is deleted if the selectable candidate is not selected within TimerValue = 5000 [ms] from the display of the selectable candidate. The selectable candidate erase timer 402 is held by the candidate display timer control unit 24. The selectable candidate erase timer 402 is used at the start of the selectable candidate display erase timer in step S57 of FIG.

The handwritten data neighborhood rectangular area 403 defines a rectangular area considered to be near the handwritten data. In the example of FIG. 7, the rectangular area 403 near the handwritten data is a rectangle in which the rectangular area of the handwritten data is horizontally enlarged by 50% (Horizontal) of the estimated character size and vertically expanded by 80% (Vertical) of the estimated character size. It becomes an area. In the example of FIG. 7, the ratio of the estimated character size (% designation) is used, but if the unit is "mm" or the like, the length can be fixed. The handwritten input storage unit 25 holds the rectangular area 403 near the handwritten data. The estimated character size 405 is used in step S9 of FIG. 21 to determine the overlapping state of the rectangular area near the handwritten data and the rectangular area of the stroke.

The estimated writing direction / character size determination condition 404 defines a constant for determining the writing direction and the measurement direction of the character size. In the example of FIG. 7, the difference between the time when the first stroke is added and the time when the stroke is added last in the handwritten data rectangular area is MinTime = 1000 [ms] or more, and the horizontal distance (width) of the handwritten data rectangular area. If the difference between and the vertical distance (height) is MinDiff = 10 [mm] or more and the horizontal distance is longer than the vertical distance, it means that the estimated writing direction is "horizontal writing" and the estimated character size is the vertical distance. To do. When the horizontal distance is shorter than the vertical distance, it means that the estimated writing direction is determined as "vertical writing" and the estimated character size is determined as the horizontal distance. If the above conditions are not satisfied, the estimated writing direction is determined to be "horizontal writing" (Default Dir = "Horizontal"), and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The handwriting input storage unit 25 holds the estimated writing direction / character size determination condition 404. The estimated writing direction / character size determination condition 404 is used in the acquisition of the estimated writing direction in step S51 of FIG. 23 and the acquisition of the character string data font in steps S75 and S81 of FIG.

The estimated character size 405 defines data for estimating the size of characters and the like. In the example of FIG. 7, the estimated character size determined by the estimated writing direction / character size determination condition 404 is a small character 405a (hereinafter referred to as a minimum font size) having an estimated character size 405 and a large character 405c (hereinafter referred to as a minimum font size). It means that it is compared with the maximum font size). If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Other than that, it is determined that the character size is medium character 405b. The estimated character size 405 is held by the handwriting input storage unit 25. The estimated character size 405 is used in the character string data font acquisition in steps S75 and S81 of FIG.

Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the Font Size of the estimated character size 405, and uses the font having the closest size. For example, if the estimated character size is 25 [mm] (small character Font Size) or less, it is estimated as "small character", and if the estimated character size is more than 25 mm and 50 mm (medium character Font Size) or less, it is estimated as "medium character". If the character size exceeds 100 mm (Font Size for large characters), it is judged as "large characters". "Small characters" 405a is Mincho 25mm font (Font Style = "Mincho" Font Size = "25mm"), "Medium characters" 405b is Mincho 50mm font (Font Style = "Mincho" Font Size = "50mm"), "Large characters" 405c uses Gothic 100mm font (Font Style = "Gothic" Font Size = "100mm"). If you want to increase the types of font sizes or styles, you can increase the types of estimated character size 405.

The straddle line determination condition 406 defines data used for determining whether or not a plurality of confirmed data have been selected. The handwritten data is a single stroke, and in the example of FIG. 7, the length of the long side of the handwritten data is 100 [mm] or more (MinLenLongSide = "100 mm"), and the length of the short side is 50 [mm] or less. If there is definite data (MaxLenShortSide = "50mm") and the overlap rate of the long side direction and short side direction with the handwritten data is 80 [%] or more (MinOverLapRate = "80%"), there are multiple definite data. Judged as selected (selected data). The operation command recognition control unit 32 holds the straddle line determination condition 406. The straddle line determination condition 406 is used in the straddle line determination for determining the selection data in step S39 of FIG.

The boxing line determination condition 407 defines data used for determining whether or not the handwritten data is a boxing line. In the example of FIG. 7, the operation command recognition control unit 32 determines that the definite data in which the overlap ratio of the handwritten data in the long side direction and the short side direction is 100% or more (MinOverLapRate = "100%") is selected data. The surrounding line determination condition 407 is held by the operation command recognition control unit 32. The surrounding line determination condition 407 is used in the surrounding line determination of the selected data in step S39 of FIG. 22.

Which of the straddle line determination condition 406 and the surrounding line determination condition 407 may be preferentially determined. For example, when the straddle line judgment condition 406 is made loose (when the straddle line is easy to select) and the boxing line judgment condition 407 is made strict (when only the boxing line can be selected), the operation command is recognized. The control unit 32 should give priority to the surrounding line determination condition 407 for determination.

The additional determination condition 408 defines a threshold value used for determining whether or not the handwritten data is added to the finalized data. In the example of FIG. 7, when the shortest distance between the handwritten data and the confirmed data is within "2 mm", the character string addition control unit 41 determines that the handwritten data may be added to the confirmed data. The threshold value is an example.

<Example of dictionary data>
The dictionary data will be described with reference to FIGS. 8 to 10. FIG. 8 is an example of the dictionary data of the handwritten recognition dictionary unit 27, FIG. 9 is an example of the dictionary data of the character string conversion dictionary unit 29, and FIG. 10 is an example of the dictionary data of the predictive conversion dictionary unit 31. These dictionary data are used in steps S20 to S30 of FIG. 22, respectively.

In the present embodiment, the conversion result by the dictionary data of the handwritten recognition dictionary unit 27 of FIG. 8 is a language character string candidate, the conversion result of the dictionary data of the character string conversion dictionary unit 29 of FIG. 9 is a conversion character string candidate, and the prediction of FIG. The conversion result by the dictionary data of the conversion dictionary unit 31 is called a predicted character string candidate. "Before conversion" of each dictionary data is a character string for searching dictionary data, "after conversion" is a character string after conversion corresponding to the character string to be searched, and "probability" is a probability selected by the user. The probability is calculated from the result of the user selecting each character string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised as a method for calculating the probability, but the calculation may be performed by an appropriate method as appropriate, and details are omitted. The present embodiment is characterized in that character string candidates are displayed in descending order of selection probability from the estimated writing direction.

In the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 8, the handwritten "gi" has a probability of 0.55 as "discussion", the probability of 0.4 as "skill", and the handwritten "gishi" has a probability of 0.5 as "engineer". It is shown that it is converted to "engineer" with a probability of 0.45. The same applies to other "before conversion" character strings. In FIG. 8, the character string "before conversion" is a handwritten hiragana, but other than the hiragana may be registered in "before conversion".

In the dictionary data of the character string conversion dictionary unit 29 of FIG. 9, it is shown that the character string "discussion" is converted into "minutes" with a probability of 0.95, and the character string "skill" is converted into "skill test" with a probability of 0.85. The same applies to other "before conversion" character strings.

In the dictionary data of the predictive conversion dictionary unit 31 of FIG. 10, the character string "minutes" is converted to "destination of minutes" with a probability of 0.65, and the character string "skill test" is converted to "decision of skill test" with a probability of 0.75. Indicates that it will be done. In the example of FIG. 10, all the character strings before conversion are in Chinese characters, but characters other than Chinese characters may be registered.

The dictionary data is language-independent, and any character string may be registered before and after conversion.

<Operation command definition data held by the operation command definition section>
Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to FIGS. 11 and 12. FIG. 11 shows an example of the operation command definition data and the system definition data held by the operation command definition unit 33.

FIG. 11A shows an example of operation command definition data. The operation command definition data of FIG. 11A is an example of operation command definition data when there is no selection data selected by handwritten data, and all operation commands that operate the display device 2 are targeted. The operation command of FIG. 11A has an operation command name (Name), a character string (String) that partially matches the character string candidate, and an operation command character string (Command) to be executed. “% ~%” In the operation command character string are variables, and are associated with the system definition data as shown in FIG. 11 (b). That is, "% to%" is replaced with the system-defined data shown in FIG. 11B.

First, in the operation command definition data 701, the operation command name is "read the minutes template", the character string that partially matches the character string candidate is "minutes" or "template", and the operation command character string to be executed is "Read File https". //% username%:% password%@server.com/template/minutes.pdf ". In this example, it is shown that the system definition data of "% to%" is included in the operation command character string to be executed, and "% username%" and "% password%" are replaced with the system definition data 704 and 705, respectively. Therefore, the final operation command string to be executed is "ReadFile https://taro.tokkyo:x2PDHTyS@server.com/template/minutes.pdf", and "https://taro.tokkyo:x2PDHTyS@ Indicates to read a file called "server.com/template/minutes.pdf".

In the operation command definition data 702, the operation command name is "save in the minutes folder", the character string that partially matches the character string candidate is "minutes" or "save", and the operation command character string to be executed is "Write File https: //% username%:% password%@server.com/minutes/%machinename%_%yyyy−mm−dd% .pdf ”. Similar to the operation command definition data 701, the operation command character strings "% username%", "% password%", and "% machinename%" are replaced with the system definition data 704 to 706, respectively. In addition, "% yyyy-mm-dd%" indicates that it is replaced with the current date. For example, if the current date is September 26, 2018, it will be replaced with "2018-09-26". The final operation command to be executed is "Write File https://taro.tokkyo:x2PDHTyS@server.com/minutes/%My-Machine_2018-09-26.pdf" and the minutes are "https://taro.tokkyo". : x2PDHTyS@server.com/minutes/%My−Machine_2018−09−26.pdf ”indicates to save to a file (Write File).

In the operation command definition data 703, the operation command name is "print", the character string that partially matches the character string candidate is "print" or "print", and the operation command character string to be executed is "Print File https: //% username%". :% password%@server.com/print/%machinename%-"% yyyy-mm-dd% .pdf". If the operation command character string is replaced in the same way as the operation command definition data 702, the operation command to be finally executed is "Print File https://taro.tokkyo: x2PDHTyS@server.com/print/%My-Machine_2018-09-26". It becomes ".pdf" and indicates that the file "https://taro.tokkyo:x2PDHTyS@server.com/print/%My-Machine_2018-09-26.pdf" is printed (Print File). That is, the file is sent to the server. When the user communicates the printer with the server and specifies a file, the printer prints the contents of the file on paper.

In this way, since the operation command definition data 701 to 703 can be specified from the character string candidates, the operation command can be displayed by handwriting by the user. In addition, if the user authentication is successful, the operation command definition data such as "% username%" and "% password%" are replaced with the user information, so that the file can be input / output in association with the user.

When the user is not authenticated (including the case where the authentication fails but the user can use the display device 2), the display device 2 is set to the preset "% username%" of the display device 2. "Replace with"% password% "etc. Therefore, it is possible to input / output a file in association with the display device 2 without user authentication.

Next, the operation command definition data when there is handwritten data, that is, the operation command definition data of the editing system and the modification system will be described. FIG. 12 shows an example of operation command definition data when there is selection data selected by handwriting data. The operation command definition data of FIG. 12 has an operation command name (Name), an operation command candidate group name (Group), and an operation command character string (Command) to be executed.

The operation command definition data 707 defines the operation commands of the editing system (Group = "Edit"), and is an example of the definition data of the operation command names of the editing system "erase", "move", "rotate", and "select". That is, these operation commands are displayed for the selected data, and the user can select the desired operation command.

The operation command definition data 708 defines the operation commands of the modification system (Group = "Decorate"), and the definition data examples of the modification system operation command names "thick", "thin", "large", "small", and "underline" are used. is there. These operation commands are displayed for the selected data, and the user can select the desired operation command. In addition, color operation commands may be displayed.

Therefore, since the operation command definition data 707 and 708 are specified by the user selecting the selected data by handwriting data, the operation command can be displayed by the user handwriting.

<Display example of selectable candidates>
FIG. 13 is an example of the operation guide 500 and the selectable candidate 530 displayed by the operation guide 500. When the user handwrites the handwritten data 504 (due to the timeout of the selectable candidate display timer), the operation guide 500 is displayed. The operation guide 500 includes an operation header 520, an operation command candidate 510, an additional character string candidate 580, a handwritten recognition character string candidate 506, a language character string candidate 571, a conversion character string candidate 507, a character string / predictive conversion candidate 508, and , Has a handwritten data rectangular area display 503. The selectable candidates 530 are operation command candidates 510, additional character string candidates 580, handwriting recognition character string candidates 506, language character string candidates 571, conversion character string candidates 507, and character string / predictive conversion candidate 508. Further, the selectable candidate 530 excluding the operation command candidate 510 is referred to as a character string candidate 539.

The operation header 520 has buttons 501, 509, 502, 505. Button 501 accepts a switching operation between predictive conversion and kana conversion. In the example of FIG. 13, when the user presses the button 509 displayed as "prediction", the handwriting input unit 21 receives it and notifies the handwriting input display control unit 23 to that effect, and the display unit 22 says "kana". Change the display to button 509. After the change, the character string candidates 539 are arranged in descending order of probability of "kana conversion".

Button 502 operates the page of candidate display. In the example of FIG. 13, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts the deletion of the operation guide 500. When the user presses the button 505, the handwriting input unit 21 accepts and notifies the handwriting input display control unit 23 to that effect, and the display unit 22 erases the display other than the handwriting data. Button 509 accepts batch display deletion. When the user presses the button 509, the handwriting input unit 21 accepts and notifies the handwriting input display control unit 23 to that effect, and the display unit 22 erases all the displays shown in FIG. 13, including the handwriting data. , Allows the user to rewrite from the beginning.

In FIG. 13, the handwritten data 504 is the character “me” handwritten by the user. The handwritten data rectangular area display 503 that surrounds the handwritten data 504 with a dotted line frame is displayed. These display procedures will be described with reference to the sequence diagrams of FIGS. 20 to 25.

The additional character string candidate 580 is a character string candidate in which the handwriting recognition character string candidate 506, the language character string candidate 571, the conversion character string candidate 507, or the character string / predictive conversion candidate 508 is added to the finalized data 573. A plurality of additional character string candidates 580 may be displayed. The handwriting recognition character string candidate 506, the language character string candidate 571, the conversion character string candidate 507, or the character string / predictive conversion candidate 508 may be referred to as a “character string based on the handwriting data 504”.

Each character string candidate is arranged in descending order of probability in the handwriting recognition character string candidate 506, the language character string candidate 571, the conversion character string candidate 507, and the character string / predictive conversion candidate 508. The "me" of the handwriting recognition character string candidate 506 is a candidate for the handwriting recognition result. In this example, "me" is correctly recognized. The language character string candidate 571 is a character string candidate in which the handwriting recognition candidate is converted into kana. In this example, "me" is converted to "eye". The conversion character string candidate 507 is a conversion character string candidate converted from the handwriting recognition character string candidate and the language character string candidate 571. In this example, "mechanism" and "memo" converted from the language character string candidate "me" are displayed. The character string / predictive conversion candidate 508 is a predictive character string candidate converted from the handwriting recognition character string candidate 506, the language character string candidate 571, or the conversion character string candidate 507. In this example, "Take a note" is displayed.

The operation command candidate 510 is a candidate for the operation command selected based on the operation command definition data 701 to 703 of FIG. 11A. In the example of FIG. 13, the bullet ">>" 511 indicates that it is a candidate for an operation command. If there is operation command definition data that partially matches "me", "eye", "mechanism", and "memo", it is displayed as operation command candidate 510. In this way, the operation command candidates are displayed when the operation command definition data including the converted character string is found, and therefore are not always displayed.

As shown in FIG. 13, since the character string candidate 539 and the operation command candidate 510 are displayed at the same time (both), the user can arbitrarily select either the character string candidate or the operation command that he / she tried to input.

<Example of specifying selected data>
The display device 2 of the present embodiment can specify the selected data by the user selecting the confirmed data by handwriting. The selected data is subject to editing or modification.

FIG. 14 is an example of a diagram illustrating an example of designating selection data. In FIG. 14, the handwritten data 11 is displayed with a solid black line, the handwritten data rectangular area 12 is shaded with gray, the confirmed data 13 is displayed with a black line, and the selected data rectangular area 14 is displayed with a dotted line. In addition, when distinguishing each, a lowercase alphabet is added to the code. Further, as the judgment condition (whether or not there is a predetermined relationship) for judging the definite data as the selection data, the straddle line judgment condition 406 or the box line judgment condition 407 of the defined control data shown in FIG. 7 is used. ing.

FIG. 14A is an example in which the user specifies two horizontal written definite data 13a and 13b with a straddle line (handwritten data 11a). In this example, the short side length H1 and the long side length W1 of the handwritten data rectangular area 12a satisfy the condition of the straddle line determination condition 406, and the overlap rate with the confirmed data 13a and 13b is the straddle line determination condition. Since the condition of 406 is satisfied, the finalized data 13a and 13b of both the "minutes" and the "giji" are designated as the selection data.

FIG. 14B is an example in which the horizontal writing confirmation data 13c is designated by a surrounding line (handwriting data 11b). In this example, only the definite data 13c called "minutes" in which the overlap rate of the definite data 13c and the handwritten data rectangular area 12c satisfies the condition of the box line determination condition 407 is designated as the selection data.

FIG. 14C is an example in which a plurality of vertically written fixed data 13d and 13e are designated by a straddle line (handwritten data 11c). In this example, as in FIG. 14A, the short side length H1 and the long side length W1 of the handwritten data rectangular area 12d satisfy the condition of the straddle line determination condition 406, and are referred to as “minutes”. Since the overlap rate with each of the two confirmed data 13d and 13e called "Giji" satisfies the condition of the straddle line determination condition 406, both the confirmed data 13d and 13e of "Minutes" and "Giji" are selected data. It is specified.

FIG. 14D is an example in which the vertical writing confirmation data 13f is designated by a surrounding line (handwritten data 11d). In this example, as in FIG. 14B, only the definite data 13f called “minutes” is designated as the selection data.

<Display example of operation command candidates>
FIG. 15 shows a display example of operation command candidates based on the operation command definition data when there is the handwritten data shown in FIG. FIG. 15A is a candidate for an editing operation command, and FIG. 15B is a candidate for a modification operation command. Further, FIG. 15A shows an example in which the selected data is designated by the handwritten data 11a of FIG. 14A.

As shown in FIGS. 15A and 15B, the main menu 550 is a candidate for the operation command displayed following the bullet ">>" 511. In the main menu 550, the name of the last executed operation command or the name of the first operation command in the operation command definition data is displayed. The bullet ">>" 511a on the first line is a candidate for an editing operation command, and the bullet ">>" 511b on the second line is a candidate for a modification operation command.

The ">" 512 at the end of the line indicates that there is a submenu (an example of a submenu button). The ">" 512a on the first line displays the submenu of candidate operation commands for editing (the last selected), and the ">" 512b on the second line shows the remaining submenus for the candidate operation commands for modifiers. Display the menu. When the user presses ">" 512, the submenu 560 is displayed on the right side. The submenu 560 displays all the operation commands defined in the operation command definition data. In the display example of FIG. 15A, the submenu 560 corresponding to ">" 512a on the first line from the time when the main menu is displayed is also displayed. It may be displayed by pressing ">" 512a on the first line.

When the user presses any operation command name with the pen, the handwritten input display control unit 23 executes the Command of the operation command definition data associated with the operation command name for the selected data. That is, "Delete" when "Delete" 521 is selected, "Move" when "Move" 522 is selected, "Rotate" when "Rotate" 523 is selected, and "Select" 524 when "Select" 524 is selected. Execute "Select" respectively.

For example, if the user presses "Erase" 521 with a pen, "Minutes" and "Giji" can be erased, and if "Move" 522, "Rotate" 523, and "Select" 524 are pressed, the bounding box (circumscribed rectangle of selected data) ) Is displayed, and "Move" 522 and "Rotate" 523 can be moved or rotated by dragging the pen, and "Select" 524 can perform other bounding box operations.

Character string candidates other than operation command candidates, "one" 541, "one," 542, "~" 543, "→" 544, and "⇒" 545, are recognition results of straddle lines (handwritten data 11a). If the user intends to enter a string instead of an operation command, he can select a string candidate.

In FIG. 15B, pressing “>” 512b on the second line displays the submenu of FIG. 15B. In the display example of FIG. 15B, the main menu 550 and the submenu 560 are displayed as in FIG. 15A. Based on the operation command definition data in FIG. 14, "Thick" is selected when "thick" 531 is selected, "Thin" is selected when "thin" 532 is selected, and "Large" is selected when "large" 533 is selected. When "small" 534 is selected, "Small" is executed, and when underline 535 is selected, "Underline" is executed by the handwriting input display control unit 23 for the selected data.

In addition, how thick it should be when "thick" 531 is selected, how thin it should be when "thin" 532 is selected, how large it should be when "large" 533 is selected, and "small". Fixed values are separately defined for how small the size should be when 534 is selected, the line type when the underline 535 is selected, and the like. Alternatively, it is even better that when the submenu of FIG. 15B is selected, a separate selection menu is opened so that the user can make adjustments.

When the user presses "thick" 531 with a pen, the handwriting input display control unit 23 thickens the lines constituting the finalized data 13a and 13b of "minutes" and "giji". When "thin" 532 is pressed with a pen, the handwriting input display control unit 23 can make the lines constituting "minutes" and "giji" thinner, and when "large" 533 is pressed with a pen, the handwriting input display control unit 23 becomes larger. When the "small" 534 is pressed with a pen, the handwriting input display control unit 23 can be made smaller, and when the "underline" 535 is pressed with a pen, the handwriting input display control unit 23 can add an underline.

FIG. 16 shows a display example of operation command candidates based on the operation command definition data when there is the handwritten data shown in FIG. The difference from FIGS. 15 (a) and 15 (b) is that FIGS. 16 (a) and 16 (b) show an example in which the selected data is specified by the handwritten data 11b (enclosed line) of FIG. 14 (b). As can be seen by comparing FIGS. 15 and 16, there is no difference in the operation command candidates displayed depending on whether the handwritten data is a line or a box. This is because the handwriting input display control unit 23 displays the operation command candidates on the display unit 22 when the selection data is specified. However, the candidate operation command that recognizes the handwritten data and displays it according to the handwritten data may be changed. In this case, the recognized handwritten data (such as one or ○) is associated with the operation command definition data as shown in FIG.

In FIG. 16, character string candidates other than the operation command candidates, “○” 551, “∞” 552, “0” 553, “00” 554, and “B” 555, are the recognition results of the surrounding line (handwritten data 11b). Yes, you can select a string candidate if the user intends to enter a string instead of an operation command.

<Distance judgment>
FIG. 17 is a diagram for explaining whether or not to display the additional character string candidate 580 in which the character string based on the handwritten data is added to the finalized data in the operation guide 500. FIG. 17A shows an example of the confirmed data 573 and the handwritten data 504, and FIG. 17B shows an enlarged view of the closest portion of the confirmed data 573 and the handwritten data 504.

When the distance D1 between the closest unit 601 of the handwritten data 504 and the closest unit 602 of the confirmed data 573 is equal to or less than the value defined by the additional determination condition 408, the character string additional control unit 41 is handwritten. Add a character string based on the data 504 (converted text data such as handwritten recognition character string candidate 506, language character string candidate 571, conversion character string candidate 507, or character string / predictive conversion candidate 508) (additional) ). As a result, it is determined whether or not there is a usage example of the additional character string candidate.

The position to be added is the beginning or the end of the finalized data 573. The handwriting input display control unit 23 acquires whether the confirmation data 573 is vertical writing or horizontal writing from the handwriting input storage unit 25, and is based on the handwriting data 504 whichever is closer to the handwriting data 504 than the beginning or the end of the confirmation data 573. Add a string.

FIG. 17C is a diagram illustrating a method of determining whether to add to the beginning or the end in the case of horizontal writing. In the case of horizontal writing, the handwriting input display control unit 23 generates the circumscribing rectangle 610 of the confirmed data 573, and sets the distances D2 and D3 between the center of gravity 611 of the handwritten data 504 and the two short sides of the circumscribing rectangle 610 of the confirmed data 573, respectively. calculate. The character string addition control unit 41 determines that the character string based on the handwritten data 504 is added to the shorter side of the distances D2 and D3.

FIG. 17D is a diagram illustrating a method of determining whether to add to the beginning or the end in the case of vertical writing. In the case of vertical writing, the handwriting input display control unit 23 generates the circumscribing rectangle 610 of the confirmed data 573, and sets the distances D4 and D5 between the center of gravity 611 of the handwritten data 504 and the two short sides of the circumscribing rectangle 610 of the confirmed data 573. Calculate each. The character string addition control unit 41 determines that a character string based on the handwritten data 504 is added to the shorter side of the distances D4 and D5.

<Usage example registration judgment>
When the distance determination result is added, the character string addition control unit 41 requests the predictive conversion control unit 30 to make a usage example registration determination.

FIG. 18 is a diagram illustrating a usage example registration determination by the predictive conversion control unit 30. Confirmed data 573, handwriting recognition character string candidate 506, language character string candidate 571, conversion character string candidate 507, and character string / predictive conversion candidate 508 are transmitted to the predictive conversion control unit 30. The predictive conversion control unit 30 sets the handwritten recognition character string candidate 506, the language character string candidate 571, the conversion character string candidate 507, and the character string / predictive conversion candidate to the one determined to be added at the beginning or the end of the confirmed data 573. When 508 is added, it is determined whether those usage examples are registered.

In the example of FIG. 18, "Today's proceedings" + "Me", "Today's proceedings" + "Eyes", "Today's proceedings" + "Memo", "Today's proceedings" + "Mecha", "Today's proceedings" + Determine if the usage example of "Take a note" is registered, and if so, how likely it is (frequency). The predictive conversion control unit 30 determines a usage example character string having a probability equal to or higher than the threshold value. For example, "Today's Minutes" + "Eyes" and "Today's Minutes" + "Mecha" have no registered usage examples and the probability is zero, but the probability of "Today's Minutes" + "Me" is Output a usage example character string such as 0.1, the probability of "today's proceedings" + "memo" is 0.8, and the probability of "today's minutes" + "take a memo" is 0.5.

The handwriting input display control unit 23 displays a part or all of the usage example character strings as additional character string candidates 580 based on the number and probability of other character string candidates 539.

<Position of additional character string when adding to the beginning>
When a character string based on the handwritten data 504 is added to the beginning of the finalized data 573, the display device 2 may or may not maintain the position of the beginning of the line.

FIG. 19 is a diagram illustrating the position of the additional character string 581 when the character string based on the handwritten data 504 is added to the head of the finalized data 573. As shown in FIG. 19A, the handwritten data "previous" was handwritten near the beginning of the finalized data 573 "today's proceedings". In this case, the character string addition control unit 41 determines that the character string based on the handwritten data is added to the beginning of the finalized data.

The handwriting input display control unit 23 can make the position of the beginning of the line of the additional character string 581 the same as the position of the beginning of the confirmation data 573 as shown in FIG. 19 (b), or as shown in FIG. 19 (c). The position of the beginning of the line of the additional character string 581 can be adjusted to the handwritten data 504. The user may set in advance which method is to be processed, or the user may select each time from the operation header 520. Alternatively, the handwriting input display control unit 23 may automatically select the wider space between the beginning of the confirmation data 573 and the frame of the display 220 and the space from the end of the confirmation data 573 to the frame of the display 220.

<Operation procedure>
The operation of the display device 2 will be described with reference to the above configuration and FIGS. 20 to 25. 20 to 25 are sequence diagrams of an example for explaining a process in which the display device 2 displays a character string candidate and an operation command candidate. The process of FIG. 20 starts when the display device 2 starts (when the application starts). In FIGS. 20 to 25, the functions of FIG. 5 are indicated by reference numerals due to space limitations.

S1: First, the handwriting input display control unit 23 transmits the start of handwriting data to the handwriting input storage unit 25. The handwriting input storage unit 25 secures a handwriting data area (a memory area for storing handwriting data). The handwriting data area may be secured after the user brings the pen into contact with the handwriting input unit 21.

S2: Next, the user brings the pen into contact with the handwriting input unit 21. The handwriting input unit 21 detects the pen down and transmits it to the handwriting input display control unit 23.

S3: The handwriting input display control unit 23 transmits a stroke start to the handwriting input storage unit 25, and the handwriting input storage unit 25 secures a stroke area.

S4: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

S5: The handwriting input display control unit 23 transmits pen coordinate complementary display data (data that interpolates discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates using the pen coordinate complementary display data and displays the line.

S6: The handwriting input display control unit 23 transmits the pen coordinates and the reception time thereof to the handwriting input storage unit 25. The handwriting input storage unit 25 adds pen coordinates to the stroke. While the user is moving the pen, the handwriting input unit 21 periodically repeats the transmission of the pen coordinates to the handwriting input display control unit 23, so that the processes of steps S3 to S6 are repeated until the pen is up.

S7: When the user releases the pen from the handwriting input unit 21, the handwriting input unit 21 transmits the pen-up to the handwriting input display control unit 23.

S8: The handwriting input display control unit 23 transmits the end of the stroke to the handwriting input storage unit 25, and the handwriting input storage unit 25 determines the pen coordinates of the stroke. After the pen coordinates of the stroke are fixed, the pen coordinates cannot be added to the stroke.

S9: Next, the handwriting input display control unit 23 transmits the acquisition of the overlapping state of the handwriting data neighborhood rectangle area and the stroke rectangle area to the handwriting input storage unit 25 based on the handwriting data neighborhood rectangle area 403. The handwriting input storage unit 25 calculates the overlap situation and transmits the overlap situation to the handwriting input display control unit 23.

Subsequent steps S10 to S15 are executed when the handwritten data neighborhood rectangular area and the stroke rectangular area do not overlap.

S10: When the rectangular area near the handwriting data and the rectangular area of the stroke do not overlap, one handwriting data is determined, so the handwriting input display control unit 23 transmits the holding data clear to the handwriting recognition control unit 26.

S11 to S13: The handwriting recognition control unit 26 transmits the retained data clear to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32, respectively. The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data related to the character string candidates and the operation command candidates held so far. At the time of clearing, the last handwritten stroke is not added to the handwritten data.

S14: The handwriting input display control unit 23 transmits the end of the handwriting data to the handwriting input storage unit 25. The handwriting input storage unit 25 confirms the handwriting data. Confirmation of handwritten data means that one handwritten data is completed (no more strokes are added).

S15: The handwriting input display control unit 23 transmits the start of handwriting data to the handwriting input storage unit 25. The handwriting input storage unit 25 secures a new handwriting data area in preparation for the start of handwriting (pen down) of the next handwriting data.

S16: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting input storage unit 25 with respect to the stroke completed in step S8. When steps S10 to S15 are executed, the stroke to be added is the first stroke of the handwriting data, and the handwriting input storage unit 25 adds the stroke data to the starting handwriting data. If steps S10 to S15 are not executed, the added stroke is added to the handwritten data that is already being handwritten.

S17: Subsequently, the handwriting input display control unit 23 transmits the stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds stroke data to the stroke data holding area in which character string candidates are stored.

S19: The handwriting recognition control unit 26 executes handwriting recognition for this stroke data holding area.

S20: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate, which is the execution result, to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits a linguistically certain language character string candidate to the handwriting recognition control unit 26.

S21: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

S22: The character string conversion control unit 28 transmits the handwriting recognition character string candidate and the language character string candidate to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the conversion character string candidate to the character string conversion control unit 28.

S23: The character string conversion control unit 28 transmits the received conversion character string candidate to the predictive conversion control unit 30.

S24: The predictive conversion control unit 30 transmits the received conversion character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predictive character string candidate to the predictive conversion control unit 30.

S25: The predictive conversion control unit 30 transmits the received predictive character string candidate to the operation command recognition control unit 32.

S26: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits a candidate operation command to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire operation command candidates corresponding to the operation command definition data having a character string (String) that matches the predicted character string candidate.

After that, the same processing is performed until the transmission of the operation command candidates in steps S27 to S35.
S27: The character string conversion control unit 28 transmits the received conversion character string candidate to the operation command recognition control unit 32.

S28: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits a candidate operation command to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidates corresponding to the operation command definition data having the character string (String) that matches the conversion character string candidate.

S30: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the language character string candidate to the predictive conversion control unit 30.

S31: The predictive conversion control unit 30 transmits the handwriting recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predictive character string candidate to the predictive conversion control unit 30.

S32: The predictive conversion control unit 30 transmits the received predictive character string candidate to the operation command recognition control unit 32.

S33: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits a candidate operation command to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire operation command candidates corresponding to the operation command definition data having a character string (String) that matches the predicted character string candidate.

S34: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command recognition control unit 32.

S35: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits a candidate operation command to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire operation command candidates corresponding to the operation command definition data having a character string (String) that matches the language character string candidate.

S36: Next, the handwriting recognition control unit 26 transmits the stroke addition to the operation command recognition control unit 32.

S37: The handwriting recognition control unit 26 transmits the stroke addition to the character string addition control unit 41. By adding strokes, strokes that compose handwritten data can be obtained.

S38: The operation command recognition control unit 32 transmits the acquisition of the position information of the confirmed data to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the position information of the confirmed data to the operation command recognition control unit 32.

S39: In order to determine the selected data, the operation command recognition control unit 32 receives the position information of the stroke received from the handwriting recognition control unit 26 by adding the stroke in step S38 and the position information of the confirmed data received from the handwriting input storage unit 25. Whether or not there is a predetermined relationship is determined based on the crossing line determination condition 406 and the surrounding line determination condition 407, and the definite data that can be determined to be selected is saved as selection data. Further, in this case, since the selected data is specified, candidates for input / output system operation commands are acquired from the operation command definition unit 33.

S40: The character string addition control unit 41 transmits the confirmation data acquisition to the handwriting input storage unit 25. Since a plurality of confirmed data are usually displayed, the handwritten input storage unit 25 specifies the confirmed data closest to the handwritten data. The handwriting input storage unit 25 transmits this confirmed data to the character string addition control unit 41.

S41: The character string addition control unit 41 determines whether or not the handwriting data received from the handwriting recognition control unit 26 and the confirmed data received from the handwriting input storage unit 25 satisfy the addition determination condition 408 by adding the stroke in step S37. If it is determined that the condition is satisfied, an additional position indicating whether to add at the beginning or the end is saved. In addition, the identification information of the confirmed data satisfying the additional determination condition 408 is retained (to specify the confirmed data of the addition destination).

Further, the handwritten recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 are handwritten recognition character string candidates, language character string candidates, conversion character string candidates, and predictive character strings, respectively. The data related to the candidates, the operation command candidates, and the selected data are retained so that they can be acquired in the subsequent steps S47 to S50. The character string addition control unit 41 holds the distance determination result, the additional position, and the identification information of the confirmed data for step S52 in the subsequent stage.

S18: Immediately after the handwriting input display control unit 23 transmits the stroke addition to the handwriting recognition control unit 26 in step S17, the selectable candidate display timer start is transmitted to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

Subsequent steps S43 to S45 are executed if a pendown occurs before a certain period of time elapses (before the timer times out).

S43: If the user touches the handwriting input unit 21 with the pen before the timer times out, the handwriting input unit 21 transmits a pen down (the same event as in step S2) to the handwriting input display control unit 23.

S44: The handwriting input display control unit 23 transmits a stroke start (same as step S3) to the handwriting input storage unit 25. The sequence after this is the same as after step S3.

S45: Further, the handwriting input display control unit 23 transmits the selectable candidate display timer stop to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer. This is because the timer is unnecessary because the pen down is detected.

Steps S46 to S87 are executed if no pendown occurs before a certain amount of time has elapsed (before the timer times out). Therefore, the operation guide 500 shown in FIG. 13 is displayed.

S46: If the user does not touch the pen to the handwriting input unit 21 while the selectable candidate display timer is starting, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

S47: The handwriting input display control unit 23 transmits the handwriting recognition / language character string candidate acquisition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the currently held handwriting recognition / language character string candidate to the handwriting input display control unit 23.

S48: The handwriting input display control unit 23 transmits the conversion character string candidate acquisition to the character string conversion control unit 28. The character string conversion control unit 28 transmits the currently held conversion character string candidate to the handwriting input display control unit 23.

S49: The handwriting input display control unit 23 transmits the prediction character string candidate acquisition to the prediction conversion control unit 30. The predictive conversion control unit 30 transmits the currently held predictive character string candidate to the handwriting input display control unit 23.

S50: The handwriting input display control unit 23 transmits the acquisition of operation command candidates to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the currently held operation command candidates and selection data to the handwriting input display control unit 23.

S51: Further, the handwriting input display control unit 23 transmits the estimated writing direction acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 determines from the stroke addition time, the horizontal distance, and the vertical distance of the handwriting data rectangular area, and transmits the estimated writing direction to the handwriting input display control unit 23.

S52: The handwriting input display control unit 23 transmits the distance determination result acquisition to the character string addition control unit 41. The handwriting input display control unit 23 acquires the distance determination result, the additional position if the distance determination result is added, and the identification information of the confirmed data from the character string addition control unit 41.

If the distance determination result is added, steps S53 and S54 are executed.
S53: When the distance determination result is added, the handwriting input display control unit 23 specifies the identification information of the confirmed data and transmits the confirmed data acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the confirmed data to the handwriting input display control unit 23. This confirmed data is confirmed data in which it is determined that the distance from the handwritten data 504 satisfies the additional determination condition 408.

S54: Further, when the distance determination result is added, the handwriting input display control unit 23 uses the usage example registration determination request (confirmed data, handwriting recognition character string candidate, language character string candidate, conversion character string candidate, character string / prediction). The conversion candidate) is transmitted to the predictive conversion control unit 30. As a result, the handwriting input display control unit 23 receives the usage example character string having the usage example together with the probability.

S55: The handwritten input display control unit 23 uses these handwritten recognition character string candidates (“me” in FIG. 13), language character string candidates (for example, “eyes” in FIG. 13), and conversion character string candidates (“mechanism” in FIG. 13). "Memo"), predicted character string candidate ("take note" in FIG. 13), operation command candidate ("XX in XX" in FIG. 13), each selection probability, and estimated writing direction. The display data of the operation guide 500 such as 13 is created, and the operation guide 500 composed of the character string candidate and the operation command candidate is transmitted to the display unit 22 to be displayed. Further, the usage example character string is displayed as an additional character string candidate 580 by being transmitted to the display unit 22.

S56: Further, the handwriting input display control unit 23 transmits the handwriting data and the rectangular area display data (rectangular frame) of the selection data (handwriting data rectangular area display 503 in FIG. 13) to the display unit 22.

S57: The handwritten input display control unit 23 transmits the start of the selectable candidate display erase timer to the candidate display timer control unit 24 in order to erase the display of the operation guide 500 after a certain period of time. The candidate display timer control unit 24 starts this timer.

In steps S58 to S62, when the user erases the selectable candidate display displayed on the display unit 22 or a change occurs in the handwritten data (that is, addition of a stroke of the handwritten data) while the selectable candidate erase timer is started. (If deleted, moved, transformed or split), or if no candidates are selected by the timeout.

Further, steps S58 to S59 are executed when the candidate display is deleted or a change in the handwritten data occurs.

S58: The handwriting input unit 21 transmits the occurrence of the selectable candidate display deletion or the change of the handwriting data to the handwriting input display control unit 23.

S59: The handwriting input display control unit 23 transmits the selectable candidate erasure timer stop. The candidate display timer control unit 24 stops the timer. This is because the timer is not required because the handwritten data is operated within a certain period of time.

S61: The handwriting input display control unit 23 deletes the display by transmitting the deletion of the operation guide 500 to the display unit 22.

S62: The handwritten input display control unit 23 erases the display by transmitting the rectangular area display data erasure of the handwritten data and the selected data to the display unit 22. Therefore, when the display of the operation command candidate is deleted under the condition other than the operation command candidate being selected, the handwritten data is maintained as it is.

S60: On the other hand, if the selectable candidate display erase or the handwritten data is not changed (when the user does not operate the pen) while the selectable candidate erase timer is starting, the candidate display timer control unit 24 times out. Is transmitted to the handwriting input display control unit 23.

Similarly, after the time-out of the selectable candidate display erase timer, the handwriting input display control unit 23 executes steps S61 and S62. This is because the operation guide 500, the handwritten data, and the rectangular area display data of the selected data may be erased after a certain period of time has elapsed.

If the user selects a selectable candidate while the selectable candidate erase timer is starting, steps S63 to S87 are executed.

S63: When the user selects a selectable candidate while the selectable candidate erasure timer is starting, the handwriting input unit 21 transmits a character string candidate or a candidate selection of an operation command to the handwriting input display control unit 23.

S64: The handwriting input display control unit 23 transmits the selectable candidate display erase timer stop to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

S65: Next, the handwriting input display control unit 23 transmits the retained data clear to the handwriting recognition control unit 26.

S66: The handwriting recognition control unit 26 transmits the retained data clear to the character string conversion control unit 28.

S67: The handwriting recognition control unit 26 transmits the retained data clear to the predictive conversion control unit 30.

S68: The handwriting recognition control unit 26 transmits the retained data clear to the operation command recognition control unit 32.

S69: The handwriting recognition control unit 26 transmits the retained data clear to the character string addition control unit 41.

The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, the operation command recognition control unit 32, and the character string addition control unit 41 are candidates for the character string candidates and operation commands that have been held so far. Clear the relevant data.

S70: Next, the handwriting input display control unit 23 erases the display of the operation guide 500 by transmitting the erase of the operation guide 500 to the display unit 22.

S71: The handwritten input display control unit 23 erases the handwritten data rectangular area display 503 by transmitting the rectangular area display data erase of the selected data to the display unit 22.

If the additional character string candidate 580 is selected, steps S72 to S77 are executed.

S72: The handwriting input display control unit 23 deletes the handwriting data 504 by transmitting the handwriting data display data erasure and the pen coordinate complementary display data erasure transmitted in step S5 to the display unit 22. This is because the handwritten data 504 becomes unnecessary because the additional character string candidate 580 is selected.

S73: The handwriting input display control unit 23 transmits the deletion of the handwriting data to the handwriting input storage unit 25. Further, since the confirmed data is not required, the confirmation data deletion is transmitted to the handwriting input storage unit 25.

S74: The handwriting input display control unit 23 transmits the addition of additional data of the additional character string candidate 580 to the handwriting input storage unit 25. The additional data is, for example, "today's proceedings memo" when the finalized data 573 is "today's proceedings" and the handwritten data 504 is "me".

S75: The handwriting input display control unit 23 outputs the additional data font acquisition to the handwriting input storage unit 25. The additional data font is font data for displaying the additional character string candidate 580 displayed in the operation guide 500.

S76: When the additional character string candidate 580 is selected, the confirmed data 573 is replaced with the additional character string candidate 580. Therefore, the handwriting input display control unit 23 requests the display unit 22 to delete the confirmed data 573.

S77: The handwriting input display control unit 23 uses the additional data font to send the additional data display data to the display unit 22 to display the additional data (additional character string 581) at the same position as the erased confirmed data 573. , Display. The additional character string 581 is, for example, "today's proceedings memo". For example, the positions of the first character of the additional character string 581 and the first character of the erased confirmed data 573 are displayed so as to match.

If the additional character string candidate 580 is not selected (when a character string candidate 539 other than the additional character string candidate is selected), steps S78 to S82 are executed.
S78: The handwriting input display control unit 23 deletes the handwriting data 504 by transmitting the handwriting data display data erasure and the pen coordinate complementary display data erasure transmitted in step S5 to the display unit 22.

S79: The handwriting input display control unit 23 transmits the handwriting data deletion to the handwriting input storage unit 25. Confirmed data 573 is not deleted.

S80: Since the character string candidate 539 other than the additional character string candidate 580 is selected, the handwriting input display control unit 23 transmits the addition of the character string data of the selected character string candidate 539 to the handwriting input storage unit 25. For example, when "memo" is selected, the "memo" is saved in the handwriting input storage unit 25.

S81: Further, the handwriting input display control unit 23 transmits the acquisition of the character string data font of the selected character string candidate to the handwriting input storage unit 25. The handwriting input storage unit 25 selects a defined font from the estimated character size of the handwriting data and transmits it to the handwriting input display control unit 23.

S82: Next, the handwriting input display control unit 23 uses the defined font received from the handwriting input storage unit 25 to display the character string data display data (for example, “memo”) to be displayed at the same position as the handwriting data 504. It is displayed by transmitting to 22.

If a candidate operation command is selected, steps S83 to S86 are executed. Further, if there is selection data, steps S83 to S85 are executed.

S83: When a candidate for an operation command for the selected data is selected (when the selected data exists), the handwriting input display control unit 23 deletes the display by transmitting the selected data display data deletion to the display unit 22. .. This is to erase the original selected data once. Also, the handwritten data is deleted.

S84: Next, the handwriting input display control unit 23 transmits the execution of the operation command to the selected data to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the display data (edited or modified display data) of the new selection data to the handwriting input display control unit 23.

S85: Next, the handwriting input display control unit 23 transmits the selection data display data to the display unit 22 to redisplay the selection data after executing the operation command.

If there is no selection data (when an input / output operation command is selected), step S86 is executed.

S86: When an input / output operation command is selected, the handwritten input display control unit 23 executes an operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user. When the user authentication unit 34 succeeds in authenticating the user, the handwriting input display control unit 23 sets the information about the user who has succeeded in the authentication to% to% of the operation command and executes it.

S87: The handwriting input display control unit 23 transmits the start of the handwriting data to the handwriting input storage unit 25 for the next handwriting data. The handwriting input storage unit 25 secures a handwriting data area. After that, the processes of steps S2 to S87 are repeated.

<Summary>
As described above, since the display device 2 of the present embodiment determines whether or not the handwritten data 504 satisfies a predetermined condition with respect to the confirmed data 573, the addition of the character string can be performed without any special operation. it can. Further, since the handwritten recognition character string candidate 506 and the conversion character string candidate 507 are displayed in a selectable manner together with the additional character string candidate 580, the user can add the handwritten data to the finalized data 573, or from the handwritten data. The converted character string can also be displayed independently.

In this embodiment, a display device that combines two confirmed data that have already been confirmed will be described.

<Outline of operation of display device>
FIG. 26 is a diagram illustrating a method of adding a character string performed by the display device 2 of the present embodiment.
(1) Confirmed data 573a (first confirmed data) called "Today's proceedings" and confirmed data 573b (second confirmed data) called "memo" are already displayed. The user specifies the confirmation data 573b with the pen 2500 and moves near the confirmation data 573a.
(2) The display device determines whether or not the handwritten data 504 satisfies a predetermined condition with respect to the confirmed data 573. The predetermined condition is whether or not the distance between the handwritten data 504 and the confirmed data is equal to or less than the threshold value. When the predetermined condition is satisfied, it is determined that the additional character string candidate 580 is included in the operation guide 500 and displayed. The additional character string candidate is a character string in which a character string candidate based on handwritten data is added to the finalized data 573. In addition, the display device displays the character string candidates obtained by the recognition of "me" in the same manner as the normal handwriting input. Therefore, the display device has an additional character string candidate 580 called "today's proceedings memo", a handwritten recognition character string candidate 506 called "me" which is a recognition result of the handwritten data 504, and a converted character string "memo" and "mechanism". Candidate 507 is displayed as a selectable candidate.
(3) When the user selects "Today's proceedings memo", the confirmed data 573a and 573b are replaced with the additional character string 581 "Today's proceedings memo".
(4) If the user does not select anything, the "memo" which is the moved confirmed data 573b remains displayed.
(5) When the user selects "mechanism", the confirmed data 573b is replaced with "mechanism" of the reconversion character string candidate 582.

As described above, in addition to the effect of the first embodiment, the display device 2 of the present embodiment determines whether or not the confirmed data 573a and 573b satisfy the predetermined conditions even when the user moves the confirmed data 573. , Character strings can be added without any special operation.

<About the configuration>
In this embodiment, the hardware configuration diagram of FIG. 4 and the functional block diagram shown in FIG. 5 described in the first embodiment will be described as being applicable.

<Processing procedure>
In the case of movement, handwriting is not input, so the processing procedure is different from that of the first embodiment.

FIG. 27 is an example of a flowchart showing a procedure in which the display device 2 adds the other confirmed data to one confirmed data as the confirmed data moves. FIG. 27 starts with two or more confirmed data being displayed.

First, the user moves the confirmed data 573b. The handwriting input unit 21 accepts the movement operation, and the handwriting input display control unit 23 displays the confirmed data 573b at the movement destination (S101).

The handwriting input display control unit 23 determines whether or not there is another confirmed data near the moved confirmed data 573b (S102). This determination method may be the same as the method described with reference to FIG. 17 of the first embodiment.

If the determination in step S102 is No, the handwriting input display control unit 23 displays the moved confirmed data 573b as it is (S108).

When the determination in step S102 is Yes, the handwriting input display control unit 23 transmits a usage example registration determination request to the predictive conversion control unit 30, and acquires a usage example character string acquired from the predictive conversion control unit 30 (S103). Further, the handwritten input display control unit 23 acquires the language conversion character string candidate, the conversion character string candidate, and the predicted character string candidate based on one or more of the characters possessed by the moved confirmed data in the same manner as in the first embodiment. Displayed on the operation guide 500.

Next, the handwriting input display control unit 23 sets the selectable candidate display erasure timer T1 to zero (S104).

The handwriting input display control unit 23 determines whether or not the character string candidate 539 is selected based on the signal from the handwriting input unit 21 (S105).

When the character string candidate 539 is selected (Yes in S105), the handwriting input display control unit 23 causes the display unit 22 to display the selected character string candidate 539 (S106). When the additional character string candidate 580 is selected, the handwriting input display control unit 23 deletes the unmoved confirmed data 573a and the moved confirmed data 573b from the display unit 22 and the handwritten input storage unit 25, respectively, and adds the additional character string. The candidate 580 is recorded in the handwriting input storage unit 25 and displayed on the display unit 22. It is displayed so that the beginning of the line of the additional character string candidate 580 matches the beginning of the confirmation data 573a. When another character string candidate 539 is selected, the handwriting input display control unit 23 deletes the moved confirmed data 573b from the display unit 22 and the handwriting input storage unit 25, and saves the selected character string candidate 539 by handwriting input. It is recorded in the unit 25 and displayed on the display unit 22.

When the character string candidate 539 is not selected (No in S105) and the selectable candidate display erase timer T1 exceeds the threshold value Ta and times out (Yes in S107), the handwriting input display control unit 23 keeps the moved confirmed data 537b as it is. Display (S108).

<Summary>
As described above, in addition to the effect of the first embodiment, the display device of the present embodiment determines whether or not the confirmed data 573a and 573b satisfy the predetermined conditions even when the confirmed data is moved. Columns can be added without any special operation.

<< Another example of display device configuration 1 >>
Although the display device of the present embodiment is described as having a large touch panel, the display device is not limited to the one having a touch panel.

FIG. 28 is a diagram showing another configuration example of the display device. In FIG. 28, the projector 411 is installed on the upper side of the normal whiteboard 413. This projector 411 corresponds to a display device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard that the user directly handwrites with a marker. The whiteboard may be a blackboard, and may be a flat surface large enough to project an image.

The projector 411 has an ultra-short focus optical system, and can project an image with little distortion from about 10 cm onto the whiteboard 413. This image may be transmitted from the PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

The user writes on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has, for example, a tip portion that emits light when the user presses it against the whiteboard 413 for handwriting to turn on the switch. Since the wavelength of light is near infrared or infrared, it is invisible to the user's eyes. The projector 411 has a camera, images a light emitting unit, analyzes the image, and identifies the direction of the electronic pen 2501. Further, the electronic pen 2501 emits sound waves together with light emission, and the projector 411 calculates the distance from the arrival time of the sound waves. The position of the electronic pen 2501 can be specified by the direction and the distance. A stroke is drawn (projected) at the position of the electronic pen 2501.

Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the button pressed by the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the stroke (set of coordinates) handwritten by the user is saved by the projector 411. The projector 411 stores handwritten information in a predetermined server 412, a USB memory 2600, or the like. Handwritten information is stored page by page. Since the coordinates are saved instead of the image data, the user can re-edit them. However, in the present embodiment, since the operation command can be called by hand, the menu 430 does not have to be displayed.

<< Another example of display device configuration 2 >>
FIG. 29 is a diagram showing another configuration example of the display device. In the example of FIG. 29, the display device includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

The terminal device 600 is wiredly connected to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects the image data input by the terminal device 600 onto the screen 800.

The pen motion detection device 810 communicates with the electronic pen 820 and detects the motion of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 and transmits the coordinate information to the terminal device 600.

The terminal device 600 generates image data of a stroke image input by the electronic pen 820 based on the coordinate information received from the pen motion detection device 810, and causes the image projection device 700A to draw the stroke image on the screen 800.

Further, the terminal device 600 generates superimposed image data showing a superimposed image in which the background image projected on the image projection device 700A and the stroke image input by the electronic pen 820 are combined.

<< Another example of display device configuration 3 >>
FIG. 30 is a diagram showing a configuration example of the display device. In the example of FIG. 30, the display device includes a terminal device 600, a display 800A, and a pen motion detection device 810.

The pen motion detection device 810 is arranged in the vicinity of the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A, and transmits the coordinate information to the terminal device 600. In the example of FIG. 30, the electronic pen 820A may be charged by the terminal device 600 via the USB connector.

The terminal device 600 generates image data of a stroke image input by the electronic pen 820A based on the coordinate information received from the pen motion detection device 810, and displays the image data on the display 800A.

<< Another example of display device configuration 4 >>
FIG. 31 is a diagram showing a configuration example of the display device. In the example of FIG. 31, the display device includes a terminal device 600 and an image projection device 700A.

The terminal device 600 performs wireless communication (Bluetooth (registered trademark), etc.) with the electronic pen 820B, and receives the coordinate information of the point indicated by the electronic pen 820B on the screen 800. Then, the terminal device 600 generates image data of the stroke image input by the electronic pen 820B based on the received coordinate information, and causes the image projection device 700A to project the stroke image.

Further, the terminal device 600 generates superimposed image data showing a superimposed image in which the background image projected on the image projection device 700A and the stroke image input by the electronic pen 820 are combined.

As described above, each of the above-described embodiments can be applied in various system configurations.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to Examples, but the present invention is not limited to these Examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

The finalized data 573 is stored as a character code, and the handwritten data is stored in the display device 2 as coordinate point data. Further, it can be saved in various storage media or stored in a storage device on the network, and can be later downloaded from the display device 2 and reused. The display device 2 to be reused may be any display device or a general information processing device. Therefore, the user can reproduce the handwritten content on a different display device 2 and continue the conference or the like.

For example, in the present embodiment, the electronic blackboard has been described as an example, but any information processing device having a touch panel can be suitably applied. Information processing devices equipped with a touch panel include, for example, PJs (Projectors), output devices such as digital signage, HUD (Head Up Display) devices, industrial machines, imaging devices, sound collectors, medical devices, network home appliances, etc. It may be a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

Further, in the present embodiment, the coordinates of the pen are detected by the method of detecting the coordinates of the pen tip with the touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. Further, the pen emits ultrasonic waves together with light emission, and the display device 2 calculates the distance from the arrival time of the ultrasonic waves. The position of the pen can be specified by the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

Further, in the present embodiment, candidates for editing system and modification system operation commands are displayed when there is selection data, and candidates for input / output system operation commands are displayed when there is no selection data. However, candidates for editing and modifying operation commands and candidates for input / output operation commands may be displayed at the same time.

Further, the configuration examples shown in FIG. 6 and the like are divided according to the main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the method of dividing the processing unit or the name. The processing of the display device 2 can be divided into more processing units according to the processing content. Further, one processing unit can be divided so as to include more processing.

Further, each function of the embodiment described above can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

2 Display device

Japanese Unexamined Patent Publication No. 2014-149612

Claims (10)

A display device that displays handwritten data handwritten based on the position of the input means in contact with the touch panel.
A handwriting recognition control unit that controls the recognition of handwriting data,
A display unit that displays confirmed data that has been converted and confirmed from the handwritten data by the handwriting recognition control unit.
When the second handwritten data handwritten around the confirmed data satisfies a predetermined condition for the confirmed data, the second handwritten data is converted to the beginning or end of the confirmed data by the handwriting recognition control unit. It has a character string addition control unit that adds a character string candidate, and
The display unit displays an additional character string candidate to which a character string candidate obtained by converting the second handwriting data by the handwriting recognition control unit is added at the beginning or end of the finalized data. apparatus. When the character string addition control unit determines that the predetermined condition is satisfied,
The display unit
Claim 1 is characterized in that the second handwritten data handwritten around the confirmed data displays a character string candidate converted by the handwriting recognition control unit and the additional character string candidate in a selectable manner. The display device described in. When the selection of the additional character string candidate is accepted by the input means,
The display according to claim 2, wherein the display unit erases the confirmed data and the second handwritten data, and displays the additional character string candidate at a position where the confirmed data is displayed. apparatus. When the selection of the character string candidate is accepted by the input means,
The display unit does not erase the confirmed data, but erases the second handwritten data.
The display device according to claim 2, wherein the character string candidate is displayed at a position where the second handwritten data is displayed. The predetermined condition is that the distance between the confirmed data and the second handwritten data handwritten around the confirmed data is within the threshold value.
If the above-mentioned predetermined conditions are met
The character string addition control unit adds the character string candidate obtained by converting the second handwriting data by the handwriting recognition control unit to the head or the end of the finalized data that is closer to the handwriting data. The display device according to any one of claims 2 to 4, wherein the additional character string candidate is generated. The predetermined condition is that the distance between the confirmed data and the second handwritten data is within the threshold value, and the character string candidate obtained by converting the second handwritten data by the handwriting recognition control unit is confirmed. The usage example of the character string added to the data is registered,
If the above-mentioned predetermined conditions are met
The display device according to claim 5, wherein the character string addition control unit generates the additional character string candidate in which a usage example is registered. When the movement of the second confirmed data is accepted while the display unit is displaying a plurality of the confirmed data.
When the second confirmed data satisfies a predetermined condition with respect to the first confirmed data different from the second confirmed data, the character string addition control unit is added to the beginning or the end of the first confirmed data. The display device according to claim 1, wherein the second finalized data is added. When the character string addition control unit determines that the predetermined condition is satisfied,
The display unit
Additional character string candidates to which the second confirmed data is added to the beginning or end of the first confirmed data, and
The display device according to claim 7, wherein a character string candidate converted based on at least a part of the second finalized data is selectively displayed. A display device that displays handwritten data handwritten based on the position of the input means in contact with the touch panel.
A handwriting recognition control unit that controls the recognition of handwriting data,
A display unit that displays confirmed data converted from the handwritten data by the handwriting recognition control unit and confirmed.
When the second handwritten data handwritten around the confirmed data satisfies a predetermined condition for the confirmed data, the second handwritten data is converted by the handwriting recognition control unit at the beginning or the end of the confirmed data. It functions as a character string addition control unit that adds the character string candidates that have been created.
The display unit is a program characterized in that an additional character string candidate to which a character string candidate to which the second handwriting data is converted by the handwriting recognition control unit is added is displayed at the beginning or the end of the finalized data. .. It is a display method performed by a display device that displays handwritten data handwritten based on the position of the input means in contact with the touch panel.
A step in which the handwriting recognition control unit controls the recognition of the handwriting data,
A step in which the display unit displays confirmed data converted from the handwritten data by the handwriting recognition control unit and confirmed.
When the second handwritten data handwritten around the confirmed data satisfies a predetermined condition for the confirmed data, the character string addition control unit puts the second handwritten data at the beginning or the end of the confirmed data. The step of adding the character string candidate converted by the handwriting recognition control unit, and
A step in which the display unit displays an additional character string candidate to which a character string candidate obtained by converting the second handwriting data by the handwriting recognition control unit is added to the beginning or end of the finalized data.
A display method characterized by having.

Images