JP7494506B2 - Display device, display method, and program - Google Patents

Description

The present invention relates to a display device, a display method, and a program.

There is a known technology that uses handwriting recognition technology to convert handwritten data into characters and display them on a display. In addition, the user can correct the characters displayed on the display using a predefined operating method.

Technology has been devised for correcting character strings using symbols (see, for example, Patent Document 1). Patent Document 1 discloses a method for correcting character strings by combining replacement symbols and handwritten character recognition.

However, conventional technology has the problem that it does not have a proofreading function that supports multiple languages. For example, the proofreading symbols for character replacement are different in Japanese and Chinese. As a result, when various users use the display device, each user cannot proofread characters efficiently.

In view of the above problems, the present invention aims to provide a display device with a proofreading function that supports multiple languages.

In view of the above problems, the present invention is a display device capable of converting handwritten data into characters of a different language, comprising: a proofreading symbol recognition unit that recognizes proofreading symbols corresponding to a language set in the display device; a proofreading processing unit that proofreads one or more characters with the proofreading symbols recognized by the proofreading symbol recognition unit; a character recognition unit that converts handwritten data into one or more characters; a display control unit that selectably displays the one or more characters recognized by the character recognition unit and the one or more proofreading symbols recognized by the proofreading symbol recognition unit; and a plurality of proofreading symbol recognition dictionaries corresponding to each language, wherein the proofreading symbol recognition unit recognizes handwritten proofreading symbols with all of the proofreading symbol recognition dictionaries; and a suggestion unit that acquires a first proofreading symbol having a recognition probability of a certain level or higher in the proofreading symbol recognition dictionary different from the proofreading symbol recognition dictionary corresponding to the language set in the display device, and a second proofreading symbol having the same content from the proofreading symbol recognition dictionary corresponding to the language set in the display device, and the display control unit selectably displays the first proofreading symbol and the second proofreading symbol acquired by the suggestion unit .

It is possible to provide a display device with a proofreading function that supports multiple languages.

11A and 11B are diagrams illustrating a proofreading method for replacing a part of a character string to be corrected with a different character. FIG. 2 is an example of a perspective view of a pen. FIG. 1 is an example of an overall configuration diagram of a display device. FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. FIG. 2 is an example of a functional block diagram for explaining functions of the display device by dividing them into blocks. FIG. 1 is a list of proofreading symbols commonly used in Japan. FIG. 1 shows a list of commonly used proofreading symbols in China. FIG. 13 is a diagram showing an example of a proofreading symbol list screen on which proofreading symbols corresponding to each language are displayed. FIG. 11 is a flowchart illustrating an example of an overall flow of operations performed by the display device when a handwritten input by a user is detected. 11A and 11B are diagrams illustrating an example of a manipulation guide and selectable candidates displayed by the manipulation guide. 1 is a diagram illustrating an example of a method for identifying the position of each character in a character string. 1 is a diagram for explaining an example of a method for replacing characters; FIG. 11 is a flowchart illustrating an example of a method for replacing characters. 10 is a diagram illustrating an example of a method for setting the language of characters to be recognized; FIG. 13 is a diagram illustrating an example of a country setting screen. FIG. 11 is a flowchart illustrating an example of a dictionary switching procedure. FIG. 11 is a diagram for explaining a method for replacing characters. 11A and 11B are diagrams for explaining an operation of leaving a proofreading mark without replacing characters; FIG. 11 is a flowchart illustrating an example of an operation of the display device when a user presses a "do not calibrate" button. FIG. 11 is a functional block diagram for explaining functions of the display device by dividing them into blocks (Example 2); FIG. 13 is a diagram showing an operation in which proofreading symbols are suggested when a user handwrites a proofreading symbol that is not used in the set language. 13 is a diagram showing an example of an operation guide in which proofreading symbols proposed by a proposing unit are displayed. FIG. 11 is a flowchart showing an example of a procedure for calibrating the display device by a method in which the calibration processing unit calibrates according to the content of the calibration symbol selected by the user in the operation guide. FIG. 11 is an example of a flowchart showing a procedure for calibrating the display device by a method in which the calibration symbol recognition unit recognizes the replaced calibration symbol again and the calibration processing unit calibrates the display device using the recognized calibration symbol. FIG. 11 is a functional block diagram for explaining functions of the display device by dividing them into blocks (third embodiment); 11A and 11B are diagrams illustrating an example of a method for determining a language by a handwritten data language determining unit; 1 is a diagram for explaining an example of a method for replacing characters; 11 is a flowchart illustrating an example of a procedure for a handwritten data language determination unit to determine a language. FIG. 11 is a functional block diagram for explaining functions of the display device by dividing them into blocks (Fourth Embodiment); 10A and 10B are diagrams illustrating an example of a method for determining a language by a voice data language determination unit; 1 is a diagram for explaining an example of a method for replacing characters; 11 is a flowchart illustrating an example of a procedure for a voice data language determination unit to determine a language. FIG. 13 is a diagram showing another configuration example of the display device. FIG. 13 is a diagram showing another configuration example of the display device. FIG. 13 is a diagram showing another configuration example of the display device. FIG. 13 is a diagram showing another configuration example of the display device.

Below, a display device as an example of an embodiment of the present invention and a display method performed by the display device will be described with reference to the drawings.

<Summary of string correction>
First, the present embodiment will be described together with a comparative example for reference. Note that the comparative example is not limited to the prior art or publicly known technology.

Figure 1 is a diagram explaining a proofreading method for replacing part of a character string 302 to be corrected with a different character. Figure 1(a) shows a proofreading method using proofreading symbols, and Figure 1(b) explains the inconvenience that occurs when proofreading symbols used in a language different from the language of the characters to be recognized are handwritten.

A calibration method using calibration symbols will be described with reference to FIG.
(1) The user handwrites a replacement correction symbol 320 (a correction symbol of two or more characters in accordance with JIS Z 8208) for the character to be replaced in the character string 302 to be corrected.
(2) The user handwrites replacement characters 305 above the replacement proofreading symbols 320. The display device changes the characters to be replaced 307 designated by the proofreading symbols 320 to the replacement characters 305.

The character string to be corrected 302 is not limited to a character string input by handwriting, but may be a character string read from a file or downloaded from a network.

In this way, for example, when the display device is set to recognize Japanese and the user handwrites correction symbols used in Japanese, the display device can correctly recognize the correction symbols. However, when the display device is set to recognize Chinese and the user handwrites correction symbols used in Japanese, the display device cannot always correctly recognize the correction symbols. This will be described with reference to FIG. 1(b).
(1) When a user handwrites “Shunya Elementary School Hangzhou” (Chinese) into the display device, the display device converts the handwritten data into a string of characters and displays it.
(2) Next, the user handwrites the proofreading symbols 320 for replacing characters used in Japanese to change "Elementary school" to "School." Examples of such situations include a Chinese person staying in Japan who is not familiar with proofreading and who imitates and learns the proofreading symbols written by a Japanese person. Another example is a situation in which a Japanese person proofreads characters handwritten by a Chinese person.
(3) However, since the display device is designed to recognize Chinese, it cannot recognize the proofreading symbols 320 that are not in the Chinese dictionary. As a result, the handwritten data remains displayed, or only low-precision character conversion candidates are displayed.

In order for a user to make corrections using proofreading symbols, they must either remember or look up and handwrite proofreading symbols commonly used in Chinese. Thus, if a display device cannot recognize proofreading symbols appropriate to the language, there is a risk that operability will be reduced.

Therefore, in this embodiment, the user can select the language of the proofreading symbols to be recognized (the language of the proofreading symbols to be recognized). Since the user can select the language of the proofreading symbols to be recognized, even if the user writes by hand in various languages, the text can be proofread using the proofreading symbols used in that language. Furthermore, even if the user only remembers the proofreading symbols of a certain country, by setting the language of the proofreading symbols, the text can be proofread using the proofreading symbols that the user remembers.

<Terminology>
Handwritten data is data that displays a sequence of coordinate points on a display where a user moves an input tool continuously, as a trajectory. A series of operations in which a user presses an input tool against a display, moves it continuously, and then removes it from the display is called a stroke, and data handwritten using strokes is called stroke data. Handwritten data has one or more stroke data.

Replacement means changing one or more characters with one or more other characters. Transposition means swapping one character for another in a string of two or more characters.

Dragging is the act of holding down the mouse button and moving it, and dropping is the act of releasing the mouse button at the desired location. Dragging moves the text. Dropping determines the position of the text.

Proofreading refers to correcting defects and errors in characters. Proofreading symbols are symbols that can easily and accurately describe instructions to correct errors. Proofreading symbols are specified in JISZ 8208. In this embodiment, Japanese proofreading symbols are described, but proofreading symbols appropriate to the user's country may be used. The user may be able to select which country's proofreading symbols to use. Proofreading symbols other than the standardized proofreading symbols may also be used. Proofreading content indicates how the characters are proofread. The same proofreading content may include not only a perfect match but also similar cases. For example, the proofreading content of "replacement" is similar if even one character is replaced.

The language set on the display device is a setting for determining into which language handwritten data will be converted. When set from a specific screen, it may be set based on the handwritten data or voice data.

The destination language refers to the language's character code into which the handwritten data is converted. For example, the proofreading symbols used in Chinese are proofreading symbols that are standardized in Chinese and are known by more than half of proofreaders. In this embodiment, the proofreading symbols are those registered in the proofreading symbol recognition dictionary 32, which will be described later. The same applies to other languages.

<Example of pen appearance>
Fig. 2 shows an example of a perspective view of the pen 2500. Fig. 2 shows an example of a multi-function pen 2500. The pen 2500 that has a built-in power source and can send commands to the display device 2 is called an active pen (a pen without a built-in power source is called a passive pen). The pen 2500 in Fig. 2 has one physical switch at the tip, one at the end, and two on the sides of the pen, with the tip being for writing, the end being for erasing, and the sides being for user function allocation. The pen 2500 of this embodiment has a non-volatile memory and stores a pen ID that is unique to other pens.

If the pen has a switch, it is possible to reduce the number of steps the user must take to operate the display device 2. Although the term "pen with a switch" primarily refers to an active pen, the electromagnetic induction method includes passive pens that do not have a built-in power source but can generate power using only an LC circuit, and so includes not only active pens but also passive pens that use the electromagnetic induction method. Pens with switches that use optical, infrared, and capacitive methods other than the electromagnetic induction method are all active pens.

The hardware configuration of the pen 2500 is assumed to be similar to a general control system equipped with a communication function and a microcomputer. The input method of the coordinates of the pen 2500 includes an electromagnetic induction method and an active electrostatic coupling method. The pen 2500 may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

<Overall configuration of the device>
The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing the overall configuration of the display device 2. Fig. 3(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

As shown in FIG. 3(a), a display 220 as an example of a display device is provided on the top of the display device 2. A user U can handwrite (also called input or drawing) characters, etc. on the display 220 using a pen 2500.

Figure 3(b) shows a display device 2 used as a vertically long electronic whiteboard hung on a wall.

Figure 3(c) shows the display device 2 placed flat on a desk 230. The thickness of the display device 2 is about 1 cm, so there is no need to adjust the height of the desk even if it is placed flat on a regular desk. In addition, it can be easily moved.

<Hardware configuration of the device>
Next, the hardware configuration of the display device 2 will be described with reference to Fig. 4. As shown in the figure, the display device 2 has a configuration of an information processing device or a computer. Fig. 4 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 4, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

Of these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the display device 2.

The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I/F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand serve as input means) have come into contact with the display 220. The touch sensor 216 also receives the pen ID.

The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. For example, in the case of an optical type, the input of coordinates and the detection of coordinates are performed by two light receiving/emitting devices installed at both ends of the upper side of the display 220 emitting a plurality of infrared rays parallel to the display 220, and receiving the light reflected by a reflecting member provided around the display 220 and returning on the same optical path as the optical path of the light emitted by the light receiving element. The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving/emitting devices that are blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 specifies the coordinate position that is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, when communication is performed using a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate without having to configure the connection settings to allow the pen 2500 to communicate with the display device 2.

The power switch 227 is a switch for switching the power of the display device 2 ON/OFF. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state shown in FIG. 3(a), FIG. 3(b), or FIG. 3(c), and can automatically change the thickness of characters, etc. depending on the installation state.

The serial interface 218 is an external communications interface such as a USB. It is used for inputting information from the outside. The speaker 219 is used for audio output, and the microphone 221 is used for audio input. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi or Bluetooth (registered trademark), but any communication standard can be used. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) and password they have obtained in the terminal they carry, they can connect to the access point.

It is preferable that the wireless communication device 222 is provided with two access points.
a. Access point → Internet
b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the internal network but can use the Internet. Access point b is for users inside the company, and users can use the internal network and the Internet.

The infrared I/F 223 detects the display device 2 arranged adjacently. By utilizing the linearity of infrared rays, it is possible to detect only the display device 2 arranged adjacently. It is preferable that one infrared I/F 223 is provided on each side, and it is possible to detect in which direction of a display device 2 another display device 2 is arranged. This expands the screen, and it is possible to display handwritten information that was previously handwritten on the adjacent display device 2 (handwritten information on another page, with the area of one display 220 being one page) etc.

The power supply control circuit 224 controls the AC adapter 225 and battery 226, which are the power sources for the display device 2. The AC adapter 225 converts the AC shared by the commercial power source into DC.

When the display 220 is so-called electronic paper, it consumes little or no power to maintain the image display, so it can also be driven by a battery 226. This makes it possible to use the display device 2 for applications such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 201 shown in FIG. 4.

The touch sensor 216 is not limited to an optical type, and may use various detection means such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive film touch panel that identifies the contact position by voltage changes in two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the tip of the pen is touching. In this case, a fingertip or a pen-shaped stick can be used to perform the touch operation. The pen 2500 does not need to be a long, thin pen type.

<About the function>
Next, the functions of the display device 2 will be described with reference to Fig. 5. Fig. 5 is an example of a functional block diagram for explaining the functions of the display device 2 by dividing them into blocks. The display device 2 has a contact position detection unit 21, a drawing data generation unit 22, a character recognition unit 23, a proofreading symbol recognition unit 24, a proofreading processing unit 25, a display control unit 26, a data recording unit 27, a network communication unit 28, and an operation reception unit 29. Each function of the display device 2 is a function or means realized by any of the components shown in Fig. 4 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

The contact position detection unit 21 detects the coordinates of the position where the pen 2500 touches the touch sensor 216. The drawing data generation unit 22 acquires the coordinates where the tip of the pen 2500 touches from the contact position detection unit 21. This sequence of coordinate points is connected by interpolation to generate stroke data.

The character recognition unit 23 performs character recognition processing on one or more stroke data (handwritten data) handwritten by the user, and converts them into character codes. During character recognition, a dictionary corresponding to the type of language registered in the character recognition dictionary 31 is used. The character recognition dictionary 31 corresponds to each language to which the handwritten data is converted. In one embodiment, the user sets which dictionary to use on the screen. In FIG. 5, the Japanese dictionary 31a, Chinese dictionary 31b, English dictionary 31c, French dictionary 31d, and Korean dictionary 31e are shown, but these are merely examples. In this embodiment, it is described that when the language of the characters to be recognized is selected, the proofreading symbol recognition dictionary 32 also switches, but the switching may be performed separately.

The character recognition unit 23 recognizes characters (not only Japanese but also multiple languages such as English), numbers, symbols (%, $, &, etc.), shapes (lines, circles, triangles, etc.), etc. in parallel with the user's pen operation. Various algorithms have been devised for the recognition method, but details will be omitted as this embodiment can use known technology.

The proofreading symbol recognition unit 24 detects proofreading symbols from the user's handwritten data. The proofreading symbol recognition unit 24 can detect different proofreading symbols depending on the language set in the display device 2. When recognizing proofreading symbols, a dictionary corresponding to the type of language registered in the proofreading symbol recognition dictionary 32 is used. The proofreading symbol recognition unit 24 refers to the proofreading symbol recognition dictionary 32 for each language to recognize proofreading symbols that have different shapes depending on the language of the converted characters but the same proofreading content. The proofreading symbol recognition dictionary 32 corresponds to each language to which the handwritten data is converted. In FIG. 5, the Japanese dictionary 32a, the Chinese dictionary 32b, the English dictionary 32c, the French dictionary 32d, and the Korean dictionary 32e are shown, but these are only examples. From the computer's perspective, proofreading symbols have the same stroke shape as characters, etc., so if a developer provides the coordinates of proofreading symbols as training data for machine learning, the proofreading symbols can be detected in the same way as characters, etc.

Although shown in the form of a dictionary in FIG. 5, each dictionary may be, for example, a neural network type classifier such as deep learning or CNN (Convolution Neural Network). The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

In addition, machine learning techniques include perceptrons, deep learning, support vector machines, logistic regression, naive Bayes, decision trees, random forests, etc., and are not limited to the techniques described in this embodiment.

The proofreading processor 25 determines the proofreading content, such as character replacement, substitution, or insertion, based on the detected proofreading symbol, and places one or more characters (which may be handwritten or present in the character string) at the position of the character string specified by the proofreading symbol. Details will be described later.

The display control unit 26 displays handwritten data, character strings converted from the handwritten data, and operation menus for user operation on the display. The data recording unit 27 stores handwritten data handwritten on the display device 2, converted character strings, proofreading history (image data described later), and PC screens, files, etc. in the memory unit 30. The network communication unit 28 connects to a network such as a LAN and transmits and receives data to and from other devices via the network.

The storage unit 30 is constructed in the SSD 204 or RAM 203 shown in FIG. 4, and stores the above information recorded by the data recording unit 27.

表１は記憶部３０が記憶する文字列情報を模式的に示す。文字列情報は、識別情報、修正対象の文字列、校正記号、差し替え用文字、差し替え対象文字、修正された文字列、画像データ１、画像データ２、及び、画像データ３の各項目を有している。
識別情報…例えば、変換単位（１回の変換で変換された文字列）で各文字列を識別する識別情報である。
修正対象の文字列…修正される文字列である。修正対象の文字列の左上コーナーと右下コーナーの座標も保存されている。
校正記号…ユーザーが手書きした校正記号の校正内容や識別情報である。
差し替え用文字…差し替えに使用される文字である。少なくとも修正対象の文字列における位置が保存されている。
差し替え対象文字…差し替えられる文字である。少なくとも修正対象の文字列における位置が保存されている。
修正された文字列…差し替え用文字で差し替えられた文字列である。修正された文字列の左上コーナーと右下コーナーの座標も保存されている。
画像データ１、画像データ２、及び、画像データ３…校正履歴として保存される文字列の画像データである。画像データ１は、差し替え対象文字３０７に校正記号が手書きされた修正対象文字列３０２の画像データである。画像データ２は手書き状態の差し替え用文字３０５の画像データである。画像データ３は修正された文字列３３０の画像データである。

Table 1 shows a schematic diagram of the character string information stored in the storage unit 30. The character string information has the following items: identification information, a character string to be corrected, proofreading symbols, replacement characters, characters to be replaced, corrected character string, image data 1, image data 2, and image data 3.
Identification information: For example, identification information for identifying each character string in a conversion unit (a character string converted in one conversion).
String to be modified: The string to be modified. The coordinates of the upper left and lower right corners of the string to be modified are also saved.
Proofreading symbol: Proofreading contents and identification information of the proofreading symbol handwritten by the user.
Replacement characters: These are characters used for replacement. At least their position in the string to be corrected is preserved.
Character to be replaced: This is the character to be replaced. At least the position in the character string to be corrected is preserved.
Modified string: The string replaced with the replacement character. The coordinates of the top left and bottom right corners of the modified string are also saved.
Image data 1, image data 2, and image data 3 are image data of character strings saved as proofreading history. Image data 1 is image data of a character string 302 to be corrected, in which proofreading symbols have been handwritten on characters to be replaced 307. Image data 2 is image data of replacement characters 305 in a handwritten state. Image data 3 is image data of a corrected character string 330.

<Example of proofreading symbol>
Figure 6 is a list of proofreading symbols used in Japan. The code at the beginning of each proofreading symbol will be used to explain the proofreading content of each symbol.
1 to 3: Proofreading symbols for replacing characters. The user handwrites the proofreading symbols and then the characters to be corrected.
4: A proofreading symbol for canceling a correction made by the proofreading symbol. The user handwrites a diagonal line through the replacement character 305 and writes "iki" near the character 307 to be replaced.
5. This is the proofreading symbol for deleting a character. The user writes a diagonal line through the character to be deleted and writes "Toru" near the character to be deleted.
6, 7... are correction symbols for inserting a character. The user handwrites a "V" shaped correction symbol at the insertion location and then handwrites the character to be inserted near the "V" shaped correction symbol.

As described above, in any of the proofreadings 1 to 7, the user does not need to drag and drop the replacement characters 305.
8, 9... are proofreading symbols for replacing characters in a string. The user handwrites the proofreading symbols shown in the figure.

Similarly, when using replacement proofreading symbols, the user does not need to drag and drop the characters to be replaced.
10: A proofreading symbol for starting a new paragraph. The user writes the proofreading symbol shown in the figure by hand.
11: A proofreading symbol for stopping a line break. The user writes the proofreading symbol shown in the figure by hand.
12: Proofreading symbol for moving to the next line. The user handwrites the proofreading symbol shown in the figure.
13: This is a proofreading symbol for correcting to lowercase. The user encircles the character that he/she wants to make lowercase, draws a horizontal line above the character that he/she wants to make lowercase, and writes the character "小" by hand.
14: This is a proofreading symbol for correcting to uppercase. The user encircles the character that he/she wants to make into an uppercase letter, draws a horizontal line above the character that he/she wants to make into an uppercase letter, and writes the character "大" by hand.
15: Proofreading symbols for indentation. The user writes the proofreading symbols shown in the figure by hand.
16: Proofreading symbol for indentation. The user handwrites the proofreading symbol shown in the figure.
17 to 20 are proofreading symbols for increasing character spacing, decreasing character spacing, increasing line spacing, and decreasing line spacing, respectively. The user handwrites the proofreading symbols shown in the figure.
21: A correction symbol for correcting a consonant sound (stopping a consonant sound). The user writes the correction symbol shown by hand.
22: Proofreading symbol for correcting a moraic syllable (stopping a moraic syllable). The user writes the proofreading symbol as shown by hand.
23: Proofreading symbols for correcting superscripts and subscripts. The user handwrites the proofreading symbols shown in the figure.
24: Proofreading symbols for adding ruby. The user handwrites the proofreading symbols shown in the figure to the characters to which he/she wants to add ruby, and then writes the ruby by hand.
25: Proofreading symbols for correcting the font. The user handwrites the proofreading symbols shown in the figure on the characters for which the font is to be changed, and then handwrites the corrected font.

Note that the proofreading symbols shown are just an example, and other proofreading symbols may exist. Also, proofreading symbols indicating the content of a configuration may be of a different shape.

Figure 7 is a list of proofreading symbols used in China. The correction content of each proofreading symbol is explained using the symbols in the speech bubbles.
31...Proofreading symbol for deletion.
32: Proofreading symbols for replacement. The user handwrites the replacement characters by handwriting the proofreading symbols shown in the figure on the characters to be replaced.
33: Proofreading symbol for inserting a character. The user handwrites the proofreading symbol shown in the figure between the characters to be inserted and then handwrites the characters to be inserted.
34: Proofreading symbols for replacing characters in a character string. The user handwrites the proofreading symbols shown in the figure.
35: A proofreading symbol for canceling the correction made by the proofreading symbol. The user handwrites the proofreading symbol shown below the deletion proofreading symbol.
36, 37... Proofreading symbols for emphasizing characters. The user writes the proofreading symbols shown in the figure below the characters he or she wishes to emphasize by hand.
38: This is a proofreading symbol that joins divided paragraphs together. The user connects the end of the destination paragraph and the beginning of the source paragraph with the proofreading symbol shown in the figure.

Figure 7 shows only one example of Chinese proofreading symbols; others are omitted. Proofreading symbols have also been defined for English, French, Korean, etc.

As the proofreading symbols differ for each language, they can be difficult for users to remember. Therefore, it is preferable that the display device 2 has a function to display the proofreading symbols used in each language.

Figure 8 shows an example of a proofreading symbol list screen 440 that displays proofreading symbols that correspond to each language. A list 441 of proofreading symbols for the proofreading content that corresponds to each language is displayed on the proofreading symbol list screen 440. The user can operate the scroll bar 445 to display the remaining proofreading symbols that are not displayed. Also, although only Japanese and Chinese are displayed in the figure, it is possible to display proofreading content and proofreading symbols for English, French, Korean, etc. The user can also select one or more languages to display.

For example, when the user selects a proofreading symbol on the proofreading symbol list screen 440, the proofreading symbol list screen 440 is cleared and the display device 2 displays the selected proofreading symbol on the display. The user can move the proofreading symbol onto any character to proofread it.

The user can also search for proofreading symbols from the proofreading symbol list screen 440. The user inputs the proofreading content of the proofreading symbol they wish to search for in the proofreading content field 442, and presses the search button 443. This displays the proofreading symbols in each language that correspond to the input proofreading content.

In this way, the proofreading symbols that correspond to each language are displayed, allowing users to check the proofreading symbols for the language they are writing by hand.

<Operation flow of the display device>
The flow of operations of the display device 2 in response to handwritten input by the user will be described with reference to Fig. 9. Fig. 9 is an example of a flowchart showing the overall flow of operations performed by the display device 2 when handwritten input by the user is detected. This flowchart is common regardless of the type of proofreading symbol.

First, the user inputs handwritten data with the pen 2500. As a result, the contact position detection unit 21 detects the contact position, and the drawing data generation unit 22 generates handwritten data. The display control unit 26 displays the handwritten data on the display 220 (S1001).

Next, the character recognition unit 23 or the proofreading symbol recognition unit 24 performs character recognition or proofreading symbol recognition on the handwritten data (S1002). In addition to characters, the handwritten data may be converted into numbers, symbols, or alphabets.

If the recognition result is a proofreading symbol (S1003), the proofreading processing unit 25 determines whether a character string is necessary based on the type of proofreading symbol (S1004). Whether a proofreading symbol requires a character string is determined by the proofreading symbol as shown in FIG. 6. For example, replacement, insertion, and ruby annotation require a character string. Other than that, no character string is necessary.

If a character string is required, the proofreading processing unit 25 waits for input of the character string (S1005). Therefore, the process returns to step S1001.

If the character string is unnecessary, the proofreading processing unit 25 executes proofreading processing (S1006). Details will be described later, but the processing specified by the proofreading symbol is performed on one or more characters of the character string specified by the proofreading symbol in the character string to be corrected 302. The processing uses characters in the character string to be corrected 302 that are detected according to the proofreading symbol. The display control unit 26 displays the character string after proofreading.

If the recognition result is a character string (S1003), the proofreading processing unit 25 determines whether it is waiting for a character string (S1007). In other words, it determines whether it has passed step S1005.

When in the character string waiting state, the proofreading processing unit 25 executes proofreading processing (S1006). As will be described in detail later, one or more characters in the correction target character string 302 that are identified by the proofreading symbol are changed to a newly handwritten character string. Alternatively, newly handwritten characters are inserted in the location of the character string identified by the proofreading symbol. Alternatively, handwritten characters are added as ruby to the character string identified by the proofreading symbol. The display control unit 26 displays the proofread character string.

If the system is not waiting for a character string, the display control unit 26 displays the character string converted from the handwritten data (S1008).

<Example of selectable candidates>
Next, an operation guide 500 displayed when converting handwritten data will be described with reference to Fig. 10. Fig. 10 shows an example of the operation guide 500 and selectable candidates 530 displayed by the operation guide. The user handwrites handwritten data 504, and the operation guide 500 is displayed. The operation guide 500 has an operation header 520, operation command candidates 510, handwriting recognition character string candidates 506, conversion character string candidates 507, character string/prediction conversion candidates 508, and a handwritten data rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, the handwriting recognition character string candidates 506, the conversion character string candidates 507, and the character string/prediction conversion candidates 508. The selectable candidates 530 excluding the operation command candidates 510 are called character string candidates 539.

The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch between predictive conversion and kana conversion. In the example of FIG. 10, when the user presses button 509 displaying "Predictive", the operation acceptance unit 29 accepts it, and the display control unit 26 changes the display to button 509 displaying "Kana". After the change, the character string candidates 539 are arranged in descending order of the probability of "Kana conversion".

Button 502 operates the page of the candidate display. In the example of Figure 10, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts the deletion of the operation guide 500. When the user presses button 505, the operation acceptance unit 29 accepts it, and the display control unit 26 erases the display except for the handwritten data. Button 509 accepts the deletion of all displays. When the user presses button 509, the operation acceptance unit 29 accepts it, and the display control unit 26 erases all displays shown in Figure 10, including the handwritten data, allowing the user to start over with handwriting.

Handwritten data 504 is the character "gi" handwritten by the user. Handwritten data rectangular area display 503 is displayed surrounding handwritten data 504. In FIG. 10, operation guide 500 is displayed when one character is entered, but operation guide 500 is displayed when the user stops writing. Therefore, the number of characters in handwritten data 504 can be any number. In the example of FIG. 10, handwritten data rectangular area display 503 is displayed in a dotted line frame.

The handwritten recognition character string candidates 506, the converted character string candidates 507, and the character string/predictive conversion candidates 508 are arranged in descending order of probability. The "gi" in the handwritten recognition character string candidate 506 is a candidate for the recognition result. In this example, "gi" is correctly recognized.

The conversion character string candidate 507 is a conversion character string candidate (e.g., a phrase containing "technique") converted from the result of the kana-kanji conversion of "gi" (e.g., "technique"). In this example, "technique trial" is an abbreviation for "technical mass production trial." The character string/predictive conversion candidate 508 is a predicted character string candidate converted from the conversion character string candidate 507. In this example, "Approve the technique trial" and "Address to send the minutes of the meeting" are displayed.

The operation command candidates 510 are predefined operation command candidates (commands for file operations, character editing, etc.) that are displayed according to the recognized characters. In the example of FIG. 10, the first character of the line "》" 511 indicates that it is an operation command candidate. In FIG. 10, the character string candidate for "ぎ", "meiji", partially matches the definition data and is therefore displayed as an operation command candidate 510.

When the user selects "Load minutes template", the operation command defined in the definition data is executed. In this way, operation command candidates are displayed only when operation command definition data containing the converted string is found, so they may not always be displayed.

As shown in FIG. 10, character string candidates and operation command candidates are displayed simultaneously (together), allowing the user to select either the character string candidate or the operation command that he or she wishes to input.

Although not shown in FIG. 10, proofreading symbols may also be included in the selectable candidates 530. If the probability of a proofreading symbol is above a certain level, the proofreading symbol recognition unit 24 recognizes it as a proofreading symbol and the operation guide 500 is not displayed. The operation guide 500 including the proofreading symbol may be displayed regardless of the probability. If the probability of a proofreading symbol is below a certain level, a proofreading symbol candidate is displayed in the operation guide 500. For example, in a neural network type classifier such as deep learning or CNN, the nodes in the output layer output the probability of the proofreading symbol corresponding to the node. The higher this probability, the more likely it is that the user has handwritten it.

<How to identify the position of each character in a string>
A method for identifying the position of each character in a character string will be described with reference to FIG. 11. FIG. 11 is an example of a diagram for explaining a method for identifying the position of each character in a character string. In FIG. 11, a character string "red small flower" 310 is displayed. The storage unit 30 stores the coordinate P1 of the upper left corner and the coordinate P2 of the lower right corner of the character string. The font and character size used are also known. If the font is known, the character size of each character is also known. For example, the storage unit 30 stores a table in which the vertical and horizontal sizes are registered in correspondence with general kanji, hiragana, numbers, etc. Therefore, the proofreading processing unit 25 can use this information to calculate the coordinates of each character (for example, the coordinates of the upper left corner and the lower right corner of a square of a one-character sentence).

In FIG. 11, the x coordinates of each character are indicated by A to E (the y coordinate is y1 or y2). The proofreading processor 25 can easily calculate the coordinates A to E. Therefore, the proofreading processor 25 can compare the coordinates A to E with the coordinates of the proofreading symbols to determine the character 307 to be replaced, etc.

<Example of correction>
An example of correction of character replacement using proofreading marks will be described with reference to Fig. 12. Fig. 12 is a diagram for explaining a method of replacing characters. Fig. 13 is a flow chart showing the procedure of the method of replacing characters.

S1: When the user handwrites "little red flower" on the display 220, the display control unit 26 displays the above operation guide 500. When the user selects a character string from the character string candidates 539, the display control unit 26 erases the handwritten data and displays the converted character string (character string to be corrected 302) on the display.

S2: The user handwrites the character replacement proofreading symbol 320 (drawing a correction symbol of two or more characters according to JIS Z 8208) to change "small" to "big." The proofreading symbol recognition unit 24 detects this proofreading symbol. Note that if a proofreading symbol is detected (if the probability is above a certain level), the operation guide 500 is not displayed. If the operation guide 500 is displayed, the proofreading symbols are displayed as options so the user can select one. The operation guide 500 may be displayed at all times regardless of the probability. Detection of a proofreading symbol causes the display device 2 to enter correction mode.

S3: When the proofreading symbol recognition unit 24 detects this proofreading symbol 320, it sends a process to the proofreading processing unit 25 to change the character 307 to be replaced at the position specified by the proofreading symbol 320 to a space (character code of a space). That is, it detects the coordinates of the left and right ends of the proofreading symbol 320 and sends them to the proofreading processing unit 25. The proofreading processing unit 25 changes the character 307 to be replaced specified by the two coordinates to a space (character code of a space). Therefore, in the example of Figure 12, each character of "small" becomes blank. This space is called a replacement area. The proofreading processing unit 25 stores the string 302 to be corrected and the character 307 to be replaced in association with identification information.

S4: Next, the user handwrites the character "big" in any position. Since the correction mode is active, the user may handwrite anywhere. The display control unit 26 displays the operation guide 500. When the user selects a character string from the character string candidates 539, the display control unit 26 erases the handwritten data and displays "big", which is the converted replacement character 305, on the display 220. The proofreading processing unit 25 stores the replacement character 305 in association with the same identification information as the character string to be corrected 302 and the replacement character 307.

S5: Because it is in correction mode, the proofreading processing unit 25 places the replacement character 305 in the replacement area. Therefore, the display control unit 26 displays "big" between "red" and "flower".

In this way, by the replacement process using the proofreading symbols, "small red flower" is corrected to "big red flower" 330, without the need for a drag-and-drop operation. The correction mode ends after the correction. The corrected character string 330 is stored in the memory unit 30 in association with the above-mentioned identification information.

If the number of characters in the replacement characters 305 is smaller than the number of characters in the characters to be replaced 307, the proofreading processor 25 deletes (closes) spaces so that no spaces remain after the replacement characters 305. If the number of characters in the replacement characters 305 is greater than the number of characters in the characters to be replaced 307, the proofreading processor 25 shifts the character string following the replacement characters 305 so that it is not overwritten. Instead of deleting or shifting spaces in this way, the proofreading processor 25 may delete all of "red" and "flower" in step S4 and display the entire "big red flower."

<Setting the language of the characters to be recognized (the language of the proofreading symbols to be recognized)>
A method for setting the language of characters to be recognized will be described with reference to Fig. 14. In this embodiment, when the language of characters to be recognized is set, the language of proofreading symbols to be recognized is also set. This can improve user convenience. However, the language of characters to be recognized and the language of proofreading symbols to be recognized may be set independently. Fig. 14(a) is an example of an in-operation screen 401 displayed on the display device 2.

The operation screen 401 displays several icons. One of them is a settings icon 410. The user presses the settings icon 410 with the pen 2500. When the operation reception unit 29 receives the press, the display control unit 26 displays a language setting screen, which will be described below.

FIG. 14(b) is an example of a language setting screen 402. The language setting screen 402 has a current setting field 403 and a pull-down button 406. Currently, Japanese is the language of the characters to be recognized. When the user presses the pull-down button 406, the display control unit 26 displays a list 404 of languages of recognizable characters supported by the display device 2. The list 404 of languages of recognizable characters is the dictionary languages registered in the character recognition dictionary 31. In FIG. 14(b), Chinese, English, French, and Korean are displayed. For example, the user selects Japanese and presses the setting button 405 at the bottom right. When the operation reception unit 29 accepts the press, the display control unit 26 displays "Japanese" in the current setting field 403. FIG. 14(c) shows the language setting screen 402 when Japanese is set.

When the language is set on the language setting screen 402, the character recognition unit 23 switches the dictionary to be used to the Japanese dictionary 31a, and the proofreading symbol recognition unit 24 switches the dictionary to be used to the Japanese dictionary 32a, which contains proofreading symbols used in Japanese-speaking countries.

As shown in FIG. 15, the display device 2 can accept a similar setting by selecting a country or region. FIG. 15 is an example of a country setting screen 420. When the user presses a pull-down button 421, the display control unit 26 displays a list 422 of countries or regions that use recognizable languages supported by the display device 2. Japan, China, the United States, France, and Korea are displayed in FIG. 15. In this way, the user can change the dictionary by selecting a country or region rather than a language. This is convenient when the language used in the country in question is unknown.

Figure 16 is an example of a flowchart showing the procedure for switching dictionaries. The operation reception unit 29 receives the language setting on the language setting screen 402 (S101).

Next, the character recognition unit 23 switches to the character recognition dictionary 31 corresponding to the language, and the proofreading symbol recognition unit 24 switches to the proofreading symbol recognition dictionary 32 corresponding to the language (S102).

<Example of proofreading when Chinese is selected>
A correction example of character replacement using proofreading symbols used in Chinese will be described below when the user selects Chinese on the language setting screen 402. Fig. 17 is a diagram for explaining a method of character replacement. The flow chart may be the same as that shown in Fig. 13.

S11: When the user handwrites "Shunmei Elementary School Hangzhou" on the display 220, the display control unit 26 displays the above operation guide 500. When the user selects a character string from the character string candidates 539, the display control unit 26 erases the handwritten data and displays the converted character string (character string to be corrected 302) on the display.

S12: The user handwrites the proofreading symbol 340, which is used in Chinese to replace characters, to change "elementary school" to "school." The proofreading symbol recognition unit 24 recognizes this proofreading symbol 340 using the Chinese dictionary 32b. The operation guide 500 may or may not be displayed. Upon detection of the proofreading symbol 340, the display device 2 enters correction mode.

S13: When the proofreading symbol recognition unit 24 detects this proofreading symbol 340, it sends a process to the proofreading processing unit 25 to change the character 307 to be replaced at the position specified by the proofreading symbol 340 to a space (character code of a space). That is, it detects the coordinates of the left and right ends of the proofreading symbol 320 and sends them to the proofreading processing unit 25. The proofreading processing unit 25 changes the character 307 to be replaced specified by the two coordinates to a space (character code of a space). Therefore, in the example of Figure 17, each character of "elementary school" becomes blank. This space is called a replacement area. The proofreading processing unit 25 stores the character string 302 to be corrected and the character 307 to be replaced in association with identification information.

S14: Next, the user handwrites the characters "school" at any position. Since the correction mode is active, the user may handwrite at any position. The display control unit 26 displays the operation guide 500. When the user selects a character string from the character string candidates 539, the display control unit 26 erases the handwritten data and displays "school" as the converted replacement character 305 on the display 220. The proofreading processing unit 25 stores the replacement character 305 in association with the same identification information as the character string to be corrected 302 and the replacement character 307.

S15: Because it is in correction mode, the proofreading processing unit 25 places the replacement character 305 in the replacement area. Therefore, the display control unit 26 displays "school" between "spring buds" and "hangzhou".

In this way, when Chinese is recognized, the user can correct the character string by handwriting Chinese correction symbols. Once corrected, the correction mode ends. The corrected character string 330 is stored in the memory unit 30 in association with the above-mentioned identification information.

A further note about FIG. 1(b): If the proofreading symbols 320 handwritten by the user are not recognized, for example, the user temporarily deletes the proofreading symbols 320 (for example, instructs UNDO). The user then selects Japanese on the language setting screen 402. This allows the proofreading symbol recognition unit 24 to recognize proofreading symbols used in Japanese, so that when the user handwrites the proofreading symbols 320 used in Japan, the display device 2 can proofread Chinese with the Japanese proofreading symbols 320.

In the language setting screen 402, the character recognition dictionary 31 and the proofreading symbol recognition unit 24 are switched as a set, but the language setting screen 402 may also accept the character recognition dictionary 31 and the proofreading symbol recognition unit 24 separately. In this case, the display device 2 can recognize Japanese proofreading symbols while converting the user's handwritten data into Chinese. The user does not need to switch the language setting between character input and proofreading.

<Save handwritten proofreading symbols>
1B, a case where the handwritten proofreading symbols are not recognized has been described, but there may be cases where the user does not want to replace characters. For example, there may be cases where the user specifies only the area to be proofread and then proofreads it later. To deal with such cases, in the correction mode, the display device 2 asks the user whether or not to perform proofreading.

Figure 18 is a diagram that illustrates the operation of leaving the proofreading marks without replacing characters.

S31, S32: Same as steps S1, S2 in Figure 12.

S33: When the proofreading symbol recognition unit 24 detects this proofreading symbol 320, the display control unit 26 displays a "No proofreading" button 350, which accepts "No proofreading." The "No proofreading" button 350 is displayed for a certain period of time and then automatically disappears.

S34: When the user presses the "Do not proofread" button 350, the operation reception unit 29 accepts it. As a result, the proofreading processing unit 25 does not perform proofreading, and the data recording unit 27 can save the state in which the proofreading symbol is handwritten on the character string.

Figure 19 is an example of a flow chart illustrating the operation of the display device 2 when the user presses the "Do not calibrate" button 350. The explanation of Figure 19 will explain the differences from Figure 13.

Steps S41 and S42 may be similar to steps S1 and S2 in FIG. 13. Next, when the proofreading symbol recognition unit 24 detects this proofreading symbol 320, the display control unit 26 displays a "No proofreading" button 350 that accepts "No proofreading" (S43).

The operation reception unit 29 determines whether or not pressing of the "Do not calibrate" button 350 has been received (S44).

If the "Do not calibrate" button 350 is not pressed, steps S45 to S47 are executed, but this process may be the same as steps S3 to S5 in FIG. 13.

When the "Do not proofread" button 350 is pressed, the operation reception unit 29 accepts it. In this case, the proofreading processing unit 25 does not proofread, and the state in which the proofreading symbol is handwritten on the character string can be saved (S48).

<Major Effects>
As described above, the display device 2 of this embodiment allows the user to select the language of the proofreading symbols to be recognized. Since the user can select the language of the proofreading symbols to be recognized, even if the user handwrites in various languages, the text can be proofread using the proofreading symbols used in the languages. Furthermore, even if the user only remembers the proofreading symbols of a certain country, by setting the language of the proofreading symbols, the text can be proofread using the proofreading symbols that the user remembers.

In this embodiment, we will describe a display device 2 that uses a proofreading symbol recognition dictionary 32 of another language to suggest the proofreading symbol intended by the user if the proofreading symbol handwritten by the user is not registered in the proofreading symbol recognition dictionary 32 currently set in the display device 2.

<About the function>
Fig. 20 is an example of a functional block diagram for explaining the functions of the display device 2 in blocks. In this embodiment, components with the same reference numerals as in Fig. 5 perform similar functions, so that only the main components of this embodiment may be mainly explained.

The display device 2 of this embodiment has a suggestion unit 33. The suggestion unit 33 recognizes proofreading symbols not only in the proofreading symbol recognition dictionary 32 of the language set by the user, but also in the proofreading symbol recognition dictionary 32 of all languages, and searches for proofreading symbols with a certain probability or higher. In this way, even if the user is writing by hand in Chinese but accidentally writes proofreading symbols in Japanese, the suggestion unit 33 will suggest Chinese proofreading symbols, so the user can proofread without having to change the language setting. There is no need to remember the proofreading symbols used in Chinese.

<Example of correction>
An example of a correction symbol suggestion will be described with reference to Fig. 21. Fig. 21 is a diagram showing an operation in which a correction symbol is suggested when a user handwrites a correction symbol that is not used in the set language. Note that the user has set Chinese as the target language for character recognition and the target language for correction symbol recognition on the language setting screen 402.

S41: Same as step S11 in Figure 17.

S42: The user handwrites the proofreading symbol 320 to replace the character used in Japanese to change "elementary school" to "school." The proofreading symbol recognition unit 24 detects this proofreading symbol 320 using the Chinese proofreading symbol recognition dictionary 32. However, the Chinese proofreading symbol recognition dictionary 32 does not register the proofreading symbol 320 used in Japanese (no proofreading symbol with a certain degree of probability can be found in the proofreading symbol recognition dictionary 32).

S43: In this case, Chinese is set as the target language for proofreading symbol recognition on the display device 2, but the proofreading symbol recognition unit 24 recognizes the proofreading symbols 320 in the proofreading symbol recognition dictionary 32 that supports all other languages. If there is a proofreading symbol that has a certain or higher probability of being obtained by recognition, the suggestion unit 33 suggests the language and content of that proofreading symbol recognition dictionary 32. In FIG. 21, it is assumed that a proofreading symbol (an example of a first proofreading symbol) has been found in the Japanese proofreading symbol recognition dictionary 32. The suggestion unit 33 also obtains a proofreading symbol (an example of a second proofreading symbol) with the same content as the proofreading symbol that has a certain or higher probability of being obtained by recognition from the currently set proofreading symbol recognition dictionary 32 (here, the Chinese proofreading symbol recognition dictionary 32). An example of a suggestion by the operation guide 500 is shown in FIG. 22.

S44: The display control unit 26 replaces the proofreading symbol selected in the operation guide 500 with the proofreading symbol handwritten by the user and displays it. In other words, the display control unit 26 deletes the proofreading symbol 320 and displays, for example, the Chinese proofreading symbol 340. Since step S44 is the same as step S12 in FIG. 17, the subsequent process is the same as in FIG. 17. Note that there are two methods for the subsequent process, which will be described later.

Figure 22 shows an operation guide 500 in which the proofreading symbols proposed by the suggestion unit 33 are displayed. The operation guide 500 is displayed because there is a high possibility (the probability is below a certain level) that the proofreading symbols 320 used in Japanese will not be found in the Chinese proofreading symbol recognition dictionary 32. The operation guide 500 displays the contents of the proofreading symbols in a selectable list 540 of proofreading symbols that have been recognized using the proofreading symbol recognition dictionary 32 for each language. In addition, because the proofreading symbol for Japanese has been recognized as the character "H", character string candidates 539 beginning with "H" are displayed as selectable characters. The user can select the proofreading symbol 540b of the language currently set in the display device 2. Alternatively, even if the user selects the proofreading symbol 540a of a language not set (for example, Japanese), there is no inconvenience because the contents of the proofreading symbol are specified.

<<Two processes executed after replacing proofreading symbols>>
After step S44 in FIG. 21, the following methods are available.
1. The proofreading processing unit 25 performs proofreading based on the content of the proofreading symbol selected by the user on the operation guide 500.
2. The replaced proofreading symbol (proofreading symbol 340 in FIG. 21) is recognized again by the proofreading symbol recognition unit 24, and the proofreading processing unit 25 proofreads the text using the recognized proofreading symbol.

Figure 23 is a flowchart showing the procedure for calibrating the display device 2 using a method in which the calibration processing unit 25 calibrates using the content of the calibration symbol selected by the user in the operation guide.

The processing of steps S201 and S202 may be the same as steps S1 and S2 in FIG. 13. The proofreading symbol recognition unit 24 recognizes proofreading symbols in the Chinese proofreading symbol recognition dictionary 32. If the probability obtained by recognition is less than a certain value (No in S203), the proofreading symbol recognition unit 24 recognizes proofreading symbols in all proofreading symbol recognition dictionaries 32 (S204). Note that step S204 may be executed regardless of the probability.

Next, if there is a proofreading symbol with a certain probability or higher, the suggestion unit 33 acquires the language of that proofreading symbol recognition dictionary 32 and the content of the proofreading symbol (replacement, interchange, etc.) (S205). The suggestion unit 33 also acquires a proofreading symbol with the same content as the proofreading symbol with a certain probability or higher obtained by recognition from the currently set proofreading symbol recognition dictionary 32 (here, the Chinese proofreading symbol recognition dictionary 32). Meanwhile, the character recognition unit 23 performs character recognition since there is a possibility that the proofreading symbol 320 is a character.

The display control unit 26 displays proofreading symbols and character string candidates 539 with a certain probability or higher in the operation guide 500 (S206).

The operation reception unit 29 determines whether or not a selection of a proofreading symbol has been received from the operation guide 500 (S207). If the character string candidate 539 is selected, the display control unit 26 displays the selected character and the process ends. Alternatively, if the operation guide 500 is left alone, the handwritten data (in this case, the proofreading symbol) remains.

If a proofreading symbol is selected (Yes in S207), the display control unit 26 deletes the original handwritten proofreading symbol 320 and displays the selected proofreading symbol in the same location as the handwritten proofreading symbol (S208). The processing up to this point corresponds to steps S41 to S44 in FIG. 21.

The proofreading processing unit 25 changes one or more characters (characters to be replaced) that are displayed by overlapping the proofreading symbol 340 to a space (S209).

The user handwrites the replacement character 305, and the display control unit 26 displays the replacement character 305 (S210).

Then, the proofreading processing unit 25 places the replacement character 305 in the replacement area according to the content of the proofreading symbol 340 selected in the operation guide 500 (S211).

Figure 24 is a flow chart showing the procedure for the display device 2 to calibrate the replaced calibration symbol again by the calibration symbol recognition unit 24, and the calibration processing unit 25 calibrates it using the recognized calibration symbol. The explanation of Figure 24 will explain the differences from Figure 23.

In FIG. 24, step S208-2 has been added. In step S208-2, the proofreading symbol displayed by the proofreading symbol recognition unit 24 is recognized again. Since the proofreading symbol displayed by the display device 2 has an ideal shape, it is recognized as a proofreading symbol with a certain degree of probability or higher. In this way, the proofreading symbol can be recognized in the same way as in the normal processing flow.

<Major Effects>
According to the display device 2 of this embodiment, if the proofreading symbols handwritten by the user are not registered in the currently set proofreading symbol recognition dictionary 32, the proofreading symbols intended by the user can be suggested using the proofreading symbol recognition dictionary 32 of other languages. Therefore, even if the user does not remember the proofreading symbols of each language, if the user remembers the proofreading symbols of one language, the characters of other languages can be proofread using the proofreading symbols of the language that the user knows.

In the first embodiment, the user sets the language from a language setting screen, but in this embodiment, a display device 2 that determines the language based on handwritten data will be described.

<About the function>
Fig. 25 is an example of a functional block diagram for explaining the functions of the display device 2 in blocks. In this embodiment, components with the same reference numerals as in Fig. 5 perform similar functions, so that only the main components of this embodiment may be mainly explained.

The display device 2 of this embodiment has a handwritten data language determination unit 34. The handwritten data language determination unit 34 determines the language of the characters handwritten by the user based on the handwritten character string, and sets it on the display device.

Figure 26 is a diagram explaining a method of language determination by the handwritten data language determination unit 34. The handwritten data language determination unit 34 refers to character dictionaries 361-363 for each language, which are prepared in advance and divided into words or a fixed number of characters, and determines that the character to be determined is written in the language of the character dictionary 361-363 in which the character to be determined is most frequently included. Words or a fixed number of characters are registered as coordinate data, and the handwritten data language determination unit 34 divides the handwritten data into words or a fixed number of characters and performs pattern matching with the coordinate data registered in the dictionary. The characters registered in the dictionary may be text data (electronic data).

Alternatively, the handwritten data language determination unit 34 may use the recognition result of the character recognition unit 23. Since the character recognition unit 23 outputs the character recognition result together with the probability, the character recognition unit 23 recognizes characters using the character recognition dictionary 31 of all languages, and the handwritten data language determination unit 34 determines that the character was handwritten in the language of the character recognition dictionary 31 with the highest probability.

<Example of correction>
Fig. 27 is a diagram for explaining a method for replacing characters. It is assumed that the handwritten data language determination unit 34 has determined the language before the processing of Fig. 27. Therefore, the processing flow may be the same as that of Fig. 17.

Figure 28 is an example of a flow chart showing the procedure for the handwritten data language determination unit 34 to determine the language. First, the contact position detection unit 21 accepts the input of handwritten data (S301). Handwritten data that the user has freely started may be used, or the display control unit 26 may display a menu or the like so that the user can handwrite a specific sentence. For example, the user may input a sentence equivalent to "Good morning" in the language he or she uses.

The handwritten data language determination unit 34 compares the character strings obtained by dividing the handwritten data into a certain number of characters with the character dictionaries 361 to 363 for each language (S302).

The handwritten data language determination unit 34 determines that the character dictionary 361-363 with the largest number of registered character strings was handwritten in the corresponding language (S303).

<Major Effects>
According to this embodiment, the language can be determined based on the handwritten data without the user having to consciously set the language, thereby improving convenience.

In the first embodiment, the user sets the language from a language setting screen, but in this embodiment, a display device 2 that determines the language based on voice data will be described.

<About the function>
Fig. 29 is an example of a functional block diagram for explaining the functions of the display device 2 in blocks. In this embodiment, components with the same reference numerals as in Fig. 5 perform similar functions, so that only the main components of this embodiment may be mainly explained.

The display device 2 of this embodiment has a voice data language determination unit 35. The voice data language determination unit 35 determines the language of characters handwritten by the user based on the voice spoken by the user, and sets it in the display device 2.

Figure 30 is a diagram explaining how the voice data language determination unit 35 determines the language. The voice data language determination unit 35 receives voice recognition results, including scores, from two or more voice recognizers 36. The two or more voice recognizers 36 are voice recognition devices corresponding to the respective languages. The score may be called "likelihood" or "certainty," and the content of the score is not important as long as it is a measure of the validity of the recognition result. The voice data language determination unit 35 determines that the language corresponds to the voice recognizer 36 with the highest score.

<Example of correction>
Fig. 31 is a diagram for explaining a method for replacing characters. It is assumed that the handwritten data language determination unit 34 has determined the language before the processing of Fig. 31. Therefore, the processing flow may be the same as that of Fig. 17.

Figure 32 is an example of a flow chart showing the procedure for the voice data language determination unit 35 to determine the language. First, the voice data language determination unit 35 accepts input of voice data from the microphone 221 (S401). Voice data freely spoken by the user may be used, or the voice data language determination unit 35 may output "Good morning. How are you?" or the like from the speaker 219 as if the user were speaking. It is preferable to output in several different languages.

The voice data language determination unit 35 performs voice recognition on the input voice data using each voice recognizer 36 (S402).

The voice data language determination unit 35 determines that the language corresponding to the voice recognizer 36 that returns the largest score is used for handwriting (S403).

<Major Effects>
According to this embodiment, the language can be determined based on the user's speech without the user having to consciously set the language, thereby improving convenience.

In the following example, another configuration example of the display device 2 is described.

<<Another Configuration Example 1 of the Display Device>>
Although the display device 2 of the present embodiment has been described as having a large touch panel, the display device 2 is not limited to having a touch panel.

Figure 33 is a diagram showing another example of the configuration of the display device 2. In Figure 33, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to the display device 2. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user directly writes by hand with a marker. The whiteboard may be a blackboard, and may be any flat surface large enough to project an image.

The projector 411 has an ultra-short focal length optical system and can project an image with little distortion onto the whiteboard 413 from a distance of about 10 cm. This image may be transmitted from the PC 400-1 connected wirelessly or via a wired connection, or may be stored in the projector 411.

The user writes by hand on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at the tip, that is switched on and emits light when the user presses it against the whiteboard 413 to write. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eye. The projector 411 has a camera that captures the light-emitting part, analyzes the image, and identifies the direction of the electronic pen 2501. The electronic pen 2501 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The position of the electronic pen 2501 can be identified based on the direction and distance. The handwritten data is drawn (projected) at the position of the electronic pen 2501.

The projector 411 projects the menu 430, so when the user presses a button with the electronic pen 2501, the projector 411 identifies the position of the electronic pen 2501 and the pressed button by the switch ON signal. For example, when the save button 431 is pressed, the handwritten data (coordinate point sequence) handwritten by the user is saved in the projector 411. The projector 411 saves the handwritten information in a predetermined server 412 or USB memory 2600, etc. The handwritten information is saved for each page. Since the coordinates are saved instead of image data, the user can re-edit it. However, in this embodiment, the menu 430 does not need to be displayed because the operation command can be called up by handwriting.

<<Another Configuration Example 2 of the Display Device>>
34 is a diagram showing another example of the configuration of the display device 2. In the example of FIG. 34, the display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

The terminal device 600 is connected by wire to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating the point that the electronic pen 820 is pointing at on the screen 800 (the detection method may be the same as that in FIG. 33 ) and transmits it to the terminal device 600.

Based on the coordinate information received from the pen operation detection device 810, the terminal device 600 generates image data of the handwritten data input by the electronic pen 820, and causes the image projection device 700A to draw an image of the handwritten data on the screen 800.

In addition, the terminal device 600 generates superimposed image data showing a superimposed image obtained by combining the background image projected by the image projection device 700A with the image of the handwritten data input by the electronic pen 820.

<<Another Configuration Example 3 of the Display Device>>
Fig. 35 is a diagram showing an example of the configuration of the display device 2. In the example of Fig. 35, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810.

The pen motion detection device 810 is placed near the display 800A, detects coordinate information indicating the point indicated by the electronic pen 820A on the display 800A (the detection method may be the same as that in FIG. 33), and transmits it to the terminal device 600. In the example of FIG. 35, the electronic pen 820A may be charged by the terminal device 600 via the USB connector.

Based on the coordinate information received from the pen operation detection device 810, the terminal device 600 generates image data of the image of the handwritten data input by the electronic pen 820A and displays it on the display 800A.

<<Another Configuration Example 4 of the Display Device>>
Fig. 36 is a diagram showing a configuration example of the display device 2. In the example of Fig. 36, the display device 2 includes a terminal device 600 and an image projection device 700A.

The terminal device 600 performs wireless communication (Bluetooth (registered trademark), etc.) with the electronic pen 820B to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute positional information formed on the screen 800, or the terminal device 600 may receive the coordinate information from the screen 800.

Then, based on the received coordinate information, the terminal device 600 generates image data of the image of the handwritten data input by the electronic pen 820B, and causes the image projection device 700A to project the image of the handwritten data.

In addition, the terminal device 600 generates superimposed image data showing a superimposed image that combines the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820.

As described above, each of the above-mentioned embodiments can be applied to various system configurations.

<Other application examples>
The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

The character string is stored as character codes, and the handwritten data is stored as coordinate point data on the display device 2. It can also be stored on various storage media or on a storage device on a network, and later downloaded from the display device 2 for reuse. The display device 2 to be reused can be any display device, or even a general information processing device. Therefore, the user can reproduce the handwritten content on a different display device 2 and continue a meeting, etc.

For example, although an electronic whiteboard has been described as an example in this embodiment, an electronic whiteboard may also be called an electronic whiteboard, an electronic information board, or the like. This embodiment can also be suitably applied to any information processing device having a touch panel. Information processing devices equipped with a touch panel may be, for example, a PJ (Projector), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, a network home appliance, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

In this embodiment, the coordinates of the pen are detected by detecting the coordinates of the pen tip using a touch panel, but the coordinates of the pen tip may also be detected using ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as stroke data.

The configuration examples in FIG. 5 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into even more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processes.

In addition, some of the processing performed by the display device 2 may be performed by a server connected to the display device 2 via a network.

Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), and conventional circuit modules designed to execute each function described above.

2 Display device

JP 2014-14961 A

Claims (10)

1. A display device capable of converting handwritten data into characters of different languages, comprising:
a proofreading symbol recognition unit that recognizes proofreading symbols corresponding to a language set on the display device;
a proofreading processing unit that proofreads one or more characters using the proofreading symbol recognized by the proofreading symbol recognition unit;
a character recognition component for converting the handwritten data into one or more characters;
a display control unit that selectively displays one or more characters recognized by the character recognition unit and one or more proofreading symbols recognized by the proofreading symbol recognition unit;
A plurality of proofreading symbol recognition dictionaries corresponding to respective languages ;
the proofreading symbol recognition unit recognizes handwritten proofreading symbols using all of the proofreading symbol recognition dictionaries;
a suggestion unit that acquires a first proofreading symbol having a certain or higher probability of recognition in the proofreading symbol recognition dictionary different from the proofreading symbol recognition dictionary corresponding to the language set in the display device, and a second proofreading symbol having the same content from the proofreading symbol recognition dictionary corresponding to the language set in the display device;
The display device according to claim 1, wherein the display control unit displays the first proofreading symbol and the second proofreading symbol acquired by the suggestion unit in a selectable manner . The display device according to claim 1, characterized in that the proofreading symbol recognition unit recognizes different proofreading symbols depending on the language set in the display device. the display control unit displays a language setting screen for accepting a setting of a language of characters to which the handwritten data is to be converted;
an operation receiving unit that receives a language setting on the language setting screen ;
The display device according to claim 2 , wherein the proofreading symbol recognition unit recognizes proofreading symbols corresponding to a language setting accepted by the operation acceptance unit. a handwritten data language determination unit that determines the language of characters to be converted based on handwritten data handwritten by a user and sets the determined language on the display device;
The display device according to claim 2 , wherein the proofreading symbol recognition unit recognizes proofreading symbols corresponding to a language set by the handwritten data language determination unit. a voice data language determination unit that determines a language used by a user based on voice data uttered by the user and sets the language on the display device;
3. The display device according to claim 2, wherein the proofreading symbol recognition unit recognizes proofreading symbols corresponding to a language set by the voice data language determination unit. A display device as described in any one of claims 3 to 5, characterized in that the proofreading symbol recognition unit recognizes proofreading symbols handwritten by the user using the proofreading symbol recognition dictionary corresponding to the language set in the display device. The display device according to claim 1, characterized in that, when the selection of the second proofreading symbol is accepted, the proofreading processing unit proofreads one or more characters specified by the handwritten proofreading symbol with the second proofreading symbol. The handwritten data is subjected to character recognition by a character recognition unit or recognition by the proofreading symbol recognition unit;
the display control unit causes a character string converted from the handwritten data to be displayed on a display unit when a character is recognized by the character recognition unit;
The display device according to claim 1, characterized in that when a proofreading symbol is recognized by the proofreading symbol recognition unit, one or more characters of the string of characters displayed on the display unit that are identified by the proofreading symbol recognized by the proofreading symbol recognition unit are proofread . A display method performed by a display device capable of converting handwritten data into characters of different languages, comprising the steps of:
A step in which a proofreading symbol recognition unit recognizes proofreading symbols corresponding to a language set in the display device;
a proofreading processor proofreading one or more characters with the proofreading symbol recognized by the proofreading symbol recognition unit;
a character recognition unit converting the handwritten data into one or more characters;
a step of causing a display control unit to selectively display one or more characters recognized by the character recognition unit and one or more proofreading symbols recognized by the proofreading symbol recognition unit;
a step of the proofreading symbol recognition unit recognizing handwritten proofreading symbols using all of the proofreading symbol recognition dictionaries based on a plurality of proofreading symbol recognition dictionaries corresponding to respective languages;
A step in which a suggestion unit acquires a first proofreading symbol having a certain or higher probability of recognition in a proofreading symbol recognition dictionary different from the proofreading symbol recognition dictionary corresponding to the language set in the display device, and a second proofreading symbol having the same content from the proofreading symbol recognition dictionary corresponding to the language set in the display device;
a step of displaying the first proofreading symbol and the second proofreading symbol acquired by the suggestion unit in a selectable manner by the display control unit ;
A display method comprising the steps of: A display device capable of converting handwritten data into characters of different languages,
a proofreading symbol recognition unit that recognizes proofreading symbols corresponding to a language set on the display device;
a proofreading processing unit that proofreads one or more characters using the proofreading symbol recognized by the proofreading symbol recognition unit;
a character recognition component for converting the handwritten data into one or more characters;
a display control unit that selectively displays one or more characters recognized by the character recognition unit and one or more proofreading symbols recognized by the proofreading symbol recognition unit;
The proofreading symbol recognition unit recognizes handwritten proofreading symbols using all of the proofreading symbol recognition dictionaries based on a plurality of proofreading symbol recognition dictionaries corresponding to each language,
a suggestion unit that acquires a first proofreading symbol having a certain or higher probability of recognition in the proofreading symbol recognition dictionary different from the proofreading symbol recognition dictionary corresponding to the language set in the display device, and a second proofreading symbol having the same content from the proofreading symbol recognition dictionary corresponding to the language set in the display device;
The display control unit is a program for selectively displaying the first proofreading symbol and the second proofreading symbol acquired by the suggestion unit .

Images