JP2022119463A - Display, display method, and program - Google Patents

Abstract

To improve operability in display of characters through voice recognition.SOLUTION: A display comprises: a voice processing unit that, when a first touch operation to a screen is detected, receives input of voice and converts the input voice into characters through voice recognition; and a display control unit performs control of displaying the characters obtained through the conversion at a position on the screen where the first touch operation is received. When a second touch operation to the characters displayed through the voice recognition is detected, the voice processing unit receives the second touch operation as an operation to correct the characters.SELECTED DRAWING: Figure 4

Description

The present invention relates to a display device, display method and program.

A display device is disclosed that accepts voice input, converts the input voice into characters by voice recognition, and displays the characters.

For example, Patent Literature 1 discloses an input display device that determines display attributes of characters related to text data of voice received by a voice receiving means on a display unit, and displays characters based on the determined display attributes. there is The display attributes are, for example, display start and display end positions of characters or character strings to be displayed, font size, vertical writing or horizontal writing, and the like.

In the conventional technology, in character display by voice recognition, operations such as specifying the timing to start voice recognition, specifying the position to display characters, and correcting the displayed characters are complicated, and the operability is low. be.

An object of the technology disclosed herein is to improve operability in character display by voice recognition.

The disclosed technology includes a voice processing unit that receives voice input and converts the input voice into characters by voice recognition when a first touch operation on the screen is detected; a display control unit that performs control to display the converted character at the position where the touch operation is received, and the voice processing unit detects a second touch operation on the character displayed by the voice recognition. Then, the display device accepts the second touch operation as the character correction operation.

It is possible to improve operability in character display by voice recognition.

It is a figure which shows an example of the whole structure of a display apparatus. It is a figure which shows an example of the hardware constitutions of a display apparatus. It is a figure which shows an example of the perspective view of a pen. It is a figure which shows an example of the function of a display apparatus. FIG. 4 is a sequence diagram showing an example of the flow of audio processing; FIG. 4 is a first diagram showing an example of a display screen related to audio processing; FIG. 11 is a second diagram showing an example of a display screen related to audio processing; FIG. 11 is a third diagram showing an example of a display screen related to audio processing; FIG. 11 is a first diagram showing another configuration example of the display device; FIG. 11 is a second diagram showing another configuration example of the display device; FIG. 13 is a third diagram showing another configuration example of the display device; FIG. 14 is a fourth diagram showing another configuration example of the display device;

Embodiments of the present invention will be described below with reference to the drawings.

First, the terms used in this embodiment will be explained. Handwritten data is data in which a sequence of coordinate points is displayed as a trajectory on a display by continuously moving the input means by the user. A series of operations in which the user presses the input means against the display, moves it continuously, and then releases it from the display is called a stroke, and data handwritten by the stroke is called stroke data. The stroke data can also be said to be a coordinate group representing a coordinate point sequence.

Handwritten data has one or more stroke data. Specifically, the handwritten data is a plurality of characters whose time from the end of a stroke to the start of the next stroke, that is, the time from when the input means is separated from the display to when the input means is brought into contact with the display is shorter than a predetermined time. This data is a collection of stroke data. Therefore, handwritten data includes a group of coordinates indicated by stroke data of the handwritten data.

The input means is means for inputting handwritten data by designating the coordinates of the display. For example, a pen, a human finger or hand, a rod-shaped member, etc. are examples of input means.

Next, the configuration of the display device will be described.

FIG. 1 is a diagram showing an example of the overall configuration of a display device.

(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic blackboard hung on a wall.

(b) shows the display device 2 used as a vertically long electronic blackboard hung on a wall.

(c) shows the display device 2 laid flat on the desk 230 . Since the display device 2 has a thickness of about 1 cm, it is not necessary to adjust the height of the desk even if the display device 2 is placed flat on a general desk. Also, it can be easily moved.

The display device 2 is provided with a display 220 as an example of a display section. The user U can use the pen 2500 to handwrite characters or the like on the display 220 (also referred to as input or drawing).

FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device;

The display device 2 has a configuration of an information processing device or a computer. The display device 2 includes a CPU (Central Processing Unit) 201 , ROM (Read Only Memory) 202 , RAM (Random Access Memory) 203 , and SSD (Solid State Drive) 204 .

The CPU 201 controls the operation of the display device 2 as a whole. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). A RAM 203 is used as a work area for the CPU 201 . The SSD 204 stores various data such as programs for the display device 2 .

The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, and an infrared I/F 223. , a power control circuit 224 , an AC adapter 225 and a battery 226 .

A display controller 213 controls and manages screen display in order to output an output image to the display 220 or the like. The touch sensor 216 detects that the pen 2500 , the user's hand, or the like (the pen or the user's hand serves as input means) touches the display 220 . Also, the touch sensor 216 receives the pen ID.

A touch sensor controller 215 controls the processing of the touch sensor 216 . The touch sensor 216 performs coordinate input and coordinate detection. Specifically, for example, in the case of an optical system, two light emitting/receiving devices installed at both ends of the upper side of the display 220 radiate a plurality of infrared rays in parallel with the display 220 . The two light emitting/receiving devices receive the light reflected by the reflecting members provided around the display 220 and returning on the same optical path as the light emitted by the light receiving element.

The touch sensor 216 outputs to the touch sensor controller 215 the positional information of the infrared rays emitted by the two light emitting/receiving devices blocked by the object. The touch sensor controller 215 identifies the coordinate position, which is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can wirelessly communicate with the pen 2500 .

For example, when communicating according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate without performing connection settings for communicating the pens 2500 with the display device 2. FIG.

The power switch 227 is a switch for switching ON/OFF of the power of the display device 2 . The tilt sensor 217 is a sensor that detects the tilt angle of the display device 2 . The display device 2 can automatically change the thickness of characters and the like according to the installation state measured by the tilt sensor 217 .

A serial interface 218 is a communication interface with the outside, such as a USB, and inputs information from the outside. Speaker 219 outputs sound. Microphone 221 inputs voice. The wireless communication device 222 communicates with a terminal carried by the user, and relays connection to the Internet, for example. The wireless communication device 222 communicates according to standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), but may communicate according to any standard. The wireless communication device 222 forms an access point, and when the SSID (Service Set Identifier) and password obtained by the user are set in the terminal carried by the user, the terminal can access the wireless communication device 222 forming the access point. can connect to

The wireless communication device 222 has, for example, the following two access points.
a. Access point→Internet b. Access point→internal network→Internet The access point a is for users outside the company. A terminal connected to the access point a cannot access the company network, but can use the Internet. The access point b is for in-house users. The terminal connected to the access point b allows the user to use the company network and the Internet.

The infrared I/F 223 detects the display device 2 arranged next to it. Only the display device 2 arranged next to it can be detected by using the straightness of infrared rays. It is preferable that one infrared I/F 223 is provided on each side, and it is possible to detect in which direction of the display device 2 another display device 2 is arranged. As a result, the screen is widened, and handwritten information handwritten in the past (handwritten information on another page with the size of one display 220 as one page) or the like can be displayed on the adjacent display device 2 .

A power control circuit 224 controls an AC adapter 225 and a battery 226 that are power sources for the display device 2 . The AC adapter 225 converts alternating current shared by commercial power supplies into direct current.

If the display 220 is so-called electronic paper, it can be driven by a battery 226 as it consumes little or no power to maintain the image display. This makes it possible to use the display device 2 for applications such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

Furthermore, the display device 2 has a bus line 210 . A bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG.

Note that the touch sensor 216 is not limited to an optical type, and may be a capacitive detection means that identifies a contact position by detecting a change in capacitance, or a contact position that is identified by a voltage change of two opposing resistive films. Various detection means may be provided, such as a resistance film type detection means that detects the contact position, or an electromagnetic induction type detection means that detects the electromagnetic induction caused by the contact of the contact object with the display section and specifies the contact position. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touched. In this case, a fingertip or a pen-shaped stick can be used for touch operation. It should be noted that the pen 2500 does not have to have an elongated pen shape.

FIG. 3 is a diagram showing an example of a perspective view of a pen.

A pen 2500 that incorporates a power supply and can transmit commands to the display device 2 is called an active pen (a pen that does not incorporate a power supply is called a passive pen). The pen 2500 of FIG. 3 has physical switches, one on the tip of the pen, one on the back of the pen, and two on the sides of the pen. A switch on the tip of the pen is for writing, a switch on the bottom of the pen is for erasing, and a switch on the side of the pen is for assigning user functions. The pen 2500 of this embodiment has a non-volatile memory and stores a pen ID that does not overlap with other pens.

It should be noted that if a pen with a switch is used, it is also possible to reduce the user's operation procedure of the display device 2 . A pen with a switch mainly refers to an active pen, but in the electromagnetic induction system, even a passive pen that does not have a built-in power supply can generate power only with an LC circuit, so it includes not only an active pen but also an electromagnetic induction passive pen. Pens with optical, infrared, and capacitance switches other than electromagnetic induction switches are active pens.

Note that the hardware configuration of the pen 2500 is the same as a general control system having a communication function and a microcomputer. Methods for inputting the coordinates of the pen 2500 include an electromagnetic induction method, an active electrostatic coupling method, and the like. The pen 2500 may also have functions such as pen pressure detection, tilt detection, or a hover function (displaying a cursor before the pen touches it).

FIG. 4 is a diagram illustrating an example of functions of the display device.

The display device 2 includes a contact position detection unit 21, a drawing data generation unit 22, a character recognition unit 23, a contact processing unit 24, an audio processing unit 25, a display control unit 26, a data recording unit 27, a network A communication unit 28 , an operation reception unit 29 , and a storage unit 30 are provided.

Each part provided in the display device 2 is realized by operating one of the components shown in FIG.

The contact position detection unit 21 detects the coordinates of the position where the pen tip of the pen 2500 contacts the display 220 . The contact position detection unit 21 is implemented by the touch sensor controller 215 and the touch sensor 216 . The contact position detection unit 21 is an example of receiving means for receiving input of handwritten data by the input means.

The drawing data generation unit 22 acquires coordinates at which the pen tip of the pen 2500 touches the display 220 from the contact position detection unit 21 a plurality of times at short time intervals, and connects by interpolating the acquired coordinates. Generates rendering data representing the rendered image. That is, the drawing data generator 22 generates drawing data representing an image based on handwritten data.

The character recognition unit 23 performs character recognition processing on one or more stroke data (handwritten data) handwritten by the user and converts them into character codes. The character recognition unit 23 recognizes characters (not only Japanese but also multiple languages such as English), numbers, symbols (%, $, &, etc.), figures (lines, circles, triangles, etc.), etc., in parallel with the user's pen operation. to recognize. Various algorithms have been devised for the recognition method, but in the present embodiment, the details are omitted on the assumption that a known technique can be used.

The contact processing unit 24 executes processing specified according to the operation mode of each function when the contact position detection unit 21 detects the position where the pen tip of the pen 2500 contacts the display 220 . Specifically, the operation mode of the voice recognition function is switched between a voice recognition ON mode and a voice recognition OFF mode. Further, as an operation mode for a correction function for correcting characters by voice recognition, a correction ON mode and a correction OFF mode are switched. When the touch processing unit 24 detects a touch operation on the display 220 in the voice recognition OFF mode and the correction OFF mode, the contact processing unit 24 switches to the voice recognition ON mode and the correction ON mode.

Here, the touch operation is an operation in which the pen tip of the pen 2500 is brought into contact with the display 220 for a short period of time equal to or shorter than the reference time. On the other hand, an operation of touching the display 220 with the pen tip of the pen 2500 for a long time exceeding the reference time is called a long press operation. A reference time that serves as a criterion for determining whether a touch operation or a long-press operation is set in advance, for example, 1 second. This reference time may be changeable.

When the operation mode for the correction function is the correction ON mode, the touch processing unit 24 requests the voice processing unit 25 to start correcting the characters when detecting a touch operation on characters by voice recognition. When the touch processing unit 24 detects a touch operation on the correction candidate displayed on the screen under the control of the voice processing unit 25, the contact processing unit 24 notifies the voice processing unit 25 of completion of selection of the correction candidate.

Further, the contact processing unit 24 switches the operation mode of the correction function to the correction OFF mode after a certain period of time (hereinafter referred to as Tb) has elapsed from the start of the correction ON mode.

Note that the contact processing unit 24 may manage the operation mode of the speech recognition function or the correction function separately for each speech recognition. That is, if the correction ON mode continues after switching to the voice recognition OFF mode, and if the voice recognition ON mode is subsequently switched to by a touch operation on another position, the operation modes for the two correction functions are simultaneously switched. Correction ON mode is entered. In this way, the contact processing unit 24 manages each time a touch operation for starting speech recognition is performed as an individual speech recognition function, and also associates the operation mode of the corresponding correction function with the speech recognition function. managed individually.

The voice processing unit 25 executes various processes related to voice input according to the operation mode of each function. Specifically, when the operation mode of the voice recognition function is the voice recognition ON mode, the voice processing unit 25 receives voice input via the microphone 221, performs voice recognition processing, and converts it into characters. , the converted characters are transferred to the display control unit 26 .

Further, when the contact processing unit 24 requests the start of character correction, the voice processing unit 25 passes the correction candidate to the display control unit 26 .

When the operation mode of the speech recognition function is the speech recognition ON mode, the speech processing unit 25 changes the operation mode of the speech recognition function after a certain period of time (hereinafter referred to as Ta) has elapsed after the start of the speech recognition ON mode. Switch to voice recognition OFF mode. When the operation mode of the voice recognition function is the voice recognition OFF mode, the voice processing unit 25 does not perform any particular processing even if voice input is received via the microphone 221 .

The display control unit 26 displays, on the display 220, an image based on handwritten data, an operation menu for user operation, characters to be handed over to the voice processing unit 25, correction candidates, and the like.

The data recording unit 27 records, in the storage unit 30, handwritten data handwritten on the display device 2, character strings converted by character recognition or voice recognition, and information indicating segmentation positions of phrases in voice recognition.

The network communication unit 28 connects to a network such as a LAN (Local Area Network) and transmits/receives data to/from other devices via the network.

The operation reception unit 29 receives an operation of an operation button or the like displayed on the display 220 by an input means such as the pen 2500 .

The storage unit 30 is constructed in the SSD 204, the RAM 203, or the like, and stores the above information recorded by the data recording unit 27. FIG.

FIG. 5 is a sequence diagram showing an example of the flow of audio processing.

When the pen tip of the pen 2500 touches the display 220 of the display device 2 by the user's operation (step S101), the contact processing unit 24 performs a touch operation (first touch operation) at the contact position detected by the contact position detection unit 21. ) is detected. Then, the contact processing unit 24 switches the operation mode of the voice recognition function to the voice recognition ON mode.

The contact processing unit 24 notifies the voice processing unit 25 that the operation mode of the voice recognition function has been switched to the voice recognition ON mode (that is, the start of voice recognition) together with information indicating the contact position (step S102). Upon receiving the voice recognition start notification, the voice processing unit 25 stores information indicating the contact position in the storage unit 30 via the data recording unit 27 as start coordinates for starting character display (step S103).

Next, the contact processing unit 24 switches the operation mode of the correction function to the correction ON mode (step S104). Then, the contact processing unit 24 starts a timer in order to detect the lapse of a certain period of time (Ta and Tb) from the start of the speech recognition ON mode and the correction ON mode (step S105).

Next, upon receiving a voice input from the user (step S106), the voice processing unit 25 performs voice recognition processing and converts it into characters. Here, the speech processing unit 25 causes the storage unit 30 to store, through the data recording unit 27, the segmentation positions of phrases or words and candidate characters to be converted into each phrase or word by speech recognition. It is desirable that the candidates for the characters to be stored include not only the first candidate, but also the second candidate, the third candidate, and the like.

Then, the voice processing unit 25 hands over the converted characters to the display control unit 26 and requests drawing (step S107). The display control unit 26 controls the display 220 to display the transferred character at the position of the start coordinates stored in the storage unit 30 (step S108).

The voice processing unit 25 and the display control unit 26 perform the same processing as steps S106 to S108 each time voice input from the user is received (steps S109 to S111).

When the timer detects that the predetermined time (Ta) has elapsed, the speech processing unit 25 switches the operation mode of the speech recognition function to the speech recognition OFF mode (step S112). It should be noted that Ta and Tb are set in advance such that Ta<Tb. Therefore, here the operating mode for the correction function has not been switched to the correction OFF mode.

Next, when the pen tip of the pen 2500 touches the character drawn on the display 220 by the user's operation (step S113), the contact processing unit 24 detects the touch operation (first two touch operations). Then, the contact processing unit 24 requests the voice processing unit 25 to start correcting the character drawn at the contact position detected by the contact position detection unit 21 (step S114).

The speech processing unit 25 sets the word including the character displayed at the position touched by the pen tip of the pen 2500 as the correction part. It should be noted that the audio processing unit 25 may specify the corrected portion by another method. For example, the word displayed at the position of the line (strikethrough) drawn by the pen tip of the pen 2500 may be the correction part. Also, the word displayed in the area surrounded by the line drawn by the pen tip of the pen 2500 may be the correction part.

It should be noted that the speech processing unit 25 determines breaks between words or phrases based on the position of the phrases or words that are stored in the storage unit 30 at the time of speech recognition. For example, for the speech input "Jyuuyoujikou no Setsumei", during speech recognition, the division positions of the words "Juyou", "Jikou", "No" and "Setsumei" are memorized. If so, the word ``jyuuyo'' is corrected based on the stored delimiter position.

Note that the speech processing unit 25 may determine the breaks between words or phrases by other methods. For example, the voice processing unit 25 determines whether or not a character string including, for example, one to four characters centering on the touched character is appropriate as a word, thereby determining the break of the word. Also good. For example, when "yo" in "jyuuyou" is the touched character, the voice processing unit 25 selects "you", "uyo", "uyou", "jyuuyo" among the candidates for the cut. Therefore, it is also possible to determine a break in which the appropriate word "jyuyo" is one word.

Next, the voice processing unit 25 hands over the correction candidate to the display control unit 26 and requests display (step S115). Note that the speech processing unit 25 selects candidates (for example, second candidates, third candidates) other than the currently displayed character candidates (usually the first candidate) among the character candidates stored in the storage unit 30 at the time of speech recognition. etc.) are candidates for correction. For example, at the time of speech recognition, information indicating that "heavy" was determined as the first candidate, "important" as the second candidate, "moderate" as the third candidate, and "medium" as the fourth candidate. is stored in the storage unit 30, the voice processing unit 25 corrects "important", "sub-volume" and "medium", except for the first candidate "weight" to be corrected. Candidate. Alternatively, the speech processing unit 25 may determine correction candidates by other methods. For example, the speech processing unit 25 may use a speech recognition engine to output words or phrases close to the word or phrase for which the break has been determined.

The display control unit 26 performs control to display the handed over correction candidate (step S116).

Next, when the pen tip of the pen 2500 touches the drawn correction candidate on the display 220 by the user's operation (step S117), the contact processing unit 24 detects the touch operation ( third touch operation). The contact processing unit 24 notifies the audio processing unit 25 of the completion of selection of the correction candidate drawn at the contact position detected by the contact position detection unit 21 (step S118).

The voice processing unit 25 hands over the corrected character string to the display control unit 26 and requests drawing (step S119). The display control unit 26 performs control to draw the transferred character string (step S120).

When the timer detects that the predetermined time (Tb) has elapsed, the contact processing unit 24 switches the operation mode of the correction function to the correction OFF mode (step S121).

FIG. 6 is a first diagram showing an example of a display screen relating to audio processing.

FIG. 6(a) shows the display 220 in a state where the tip of the pen 2500 touches the voice input start position (FIG. 5; step S101). FIG. 6(b) shows the display 220 in a state in which the voice input "Explanation of important matters" is input (FIG. 5; step S106). Note that the display 220 displays "weight" instead of "important" as a result of voice recognition processing.

FIG. 6(c) shows the display 220 in a state where the tip of the pen 2500 touches the correction location (FIG. 5; step S113). FIG. 6(d) shows the display 220 in a state where correction candidates are displayed (FIG. 5; step S116). FIG. 6(e) shows the display 220 in a state where the correction is completed (FIG. 5; step S120).

According to the display device 2 according to the present embodiment, when the first touch operation on the screen is detected, the voice processing unit 25 receives voice input and converts the input voice into characters by voice recognition. do. This makes it possible to start speech recognition with a simple operation. Further, the display control unit 26 performs control to display the converted character at the position on the screen where the first touch operation is received. With this, it is possible to specify the position where the result of speech recognition is to be displayed by a simple operation.

Furthermore, according to the display device 2 according to the present embodiment, when the second touch operation on the displayed character is detected by voice recognition, the voice processing unit 25 converts the second touch operation into a character correction operation. accept as This makes it possible to correct the result of speech recognition with a simple operation.

When distinguishing a touch operation from a long press operation, the touch operation is an operation that provides a quicker response than the long press operation, and each operation described above is an intuitive operation. Therefore, it is possible to improve the operability in character display by voice recognition.

It should be noted that the correction ON mode ends after a certain period of time (Tb) elapses. This makes it possible to limit the time during which the above-described touch operation is treated as a correction to the time when the operation is highly likely to be performed. If such a limitation is not applied, there is a high possibility that even if a new character input is desired, it will be treated as a correction. This can further improve the operability.

Note that the voice processing unit 25 may switch the operation mode of the voice recognition function to the voice recognition OFF mode when a segment break is detected by voice recognition, or when a certain period of time elapses after the voice is interrupted.

Further, the contact processing unit 24 changes the operation mode of the correction function when a certain period of time elapses after switching from the voice recognition ON mode to the voice recognition OFF mode or when the time corresponding to the amount of characters converted by voice recognition has elapsed. You may make it switch to correction OFF mode.

Further, the display device 2 may display a menu or the like indicating switching to the correction OFF mode or switching to the speech recognition OFF mode, and switch by an explicit operation of the user.

FIG. 7 is a second diagram showing an example of a display screen relating to audio processing.

As shown in FIG. 7(d), the voice processing unit 25 causes the display control unit 26 to display a frame line for accepting handwritten input instead of displaying correction candidates in the process of step S115 in FIG. You can ask the In this case, upon receiving a handwritten input, the character recognition unit 23 recognizes characters from the input handwritten data and requests the display control unit 26 to perform control to display the corrected characters.

FIG. 8 is a third diagram showing an example of a display screen relating to audio processing.

As shown in FIG. 8(d), the voice processing unit 25 accepts voice input instead of displaying correction candidates in the process of step S115 of FIG. In this case, upon receiving input by voice, the voice processing unit 25 executes voice recognition processing on the input voice, requests the display control unit 26 to control display as corrected characters, and also requests the storage unit Store in 30.

FIG. 9 is a first diagram showing another configuration example of the display device.

In FIG. 9, a projector 411 is installed on the upper side of a normal whiteboard 413 . This projector 411 corresponds to the display device 2 . The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user directly writes with a marker. Note that the whiteboard may be a blackboard, and may be a flat surface that is large enough to project an image.

The projector 411 has an ultra-short focal length optical system, and can project an image with less distortion onto the whiteboard 413 from about 10 cm. This image may be transmitted from the PC 400 - 1 connected wirelessly or by wire, or may be stored in the projector 411 .

The user writes by hand on the whiteboard 413 using the dedicated electronic pen 2501 . The electronic pen 2501 has a light-emitting portion, for example, at its tip, which is turned on when the user presses it against the whiteboard 413 for handwriting. Since the wavelength of light is in the near infrared or infrared, it is invisible to the user's eyes. The projector 411 has a camera, captures an image of the light emitting unit, analyzes the image, and identifies the direction of the electronic pen 2501 . Also, the electronic pen 2501 emits light and emits sound waves, and the projector 411 calculates the distance from the arrival time of the sound waves. The position of the electronic pen 2501 can be identified by the direction and distance. Handwritten data is drawn (projected) at the position of the electronic pen 2501 .

Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button based on the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the handwritten data (coordinate point sequence) handwritten by the user is saved in the projector 411 . Projector 411 stores the handwritten information in predetermined server 412, USB memory 2600, or the like. Handwritten information is stored page by page. Since the coordinates are saved as they are instead of the image data, the user can re-edit them. However, in this embodiment, the menu 430 does not have to be displayed because the operation command can be called by handwriting.

FIG. 10 is a second diagram showing another configuration example of the display device.

The display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810. FIG.

The terminal device 600 is connected to the image projection device 700A and the pen motion detection device 810 by wire. The image projection device 700A projects image data input from the terminal device 600 onto the screen 800 .

The pen motion detection device 810 communicates with the electronic pen 820 and detects motion of the electronic pen 820 in the vicinity of the screen 800 . Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 (the detection method may be the same as in FIG. 9), and transmits the information to the terminal device 600 .

Terminal device 600 generates image data of handwritten data input by electronic pen 820 based on the coordinate information received from pen motion detection device 810, and causes image projection device 700A to draw an image of the handwritten data on screen 800. FIG.

Terminal device 600 also generates superimposed image data indicating a superimposed image obtained by synthesizing the background image projected by image projection device 700A and the image of the handwritten data input by electronic pen 820 .

FIG. 11 is a third diagram showing another configuration example of the display device.

In the example of FIG. 11, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810. In the example of FIG.

Pen motion detection device 810 is arranged near display 800A and detects coordinate information indicating a point indicated by electronic pen 820A on display 800A (the detection method may be the same as in FIG. 26). Send to Note that in the example of FIG. 28, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

Based on the coordinate information received from pen motion detection device 810, terminal device 600 generates image data representing an image of handwritten data input by electronic pen 820A, and displays it on display 800A.

FIG. 12 is a fourth diagram showing another configuration example of the display device.

In the example of FIG. 12, the display device 2 includes a terminal device 600 and an image projection device 700A.

Terminal device 600 performs wireless communication (Bluetooth (registered trademark), etc.) with electronic pen 820B to receive coordinate information of a point indicated by electronic pen 820B on screen 800 . As for the coordinate information, minute position information formed on the screen 800 may be read by the electronic pen 820</b>B, or the coordinate information may be received from the screen 800 .

Based on the received coordinate information, terminal device 600 generates image data of the image of the handwritten data input by electronic pen 820B, and causes image projection device 700A to project the image of the handwritten data.

Terminal device 600 also generates superimposed image data indicating a superimposed image obtained by synthesizing the background image projected by image projection device 700A and the image of the handwritten data input by electronic pen 820 .

As described above, each of the above-described embodiments can be applied in various system configurations.

Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

Character strings are stored in the display device 2 as character codes, and handwritten data are stored in the display device 2 as coordinate point data. Also, it can be saved in various storage media or saved in a storage device on a network, and can be downloaded from the display device 2 and reused later. The display device 2 to be reused may be any display device or a general information processing device. Therefore, the user can reproduce the handwritten content on a different display device 2 and continue the meeting.

Further, in the present embodiment, the coordinates of the pen tip are detected by the method of detecting the coordinates of the pen tip with the touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. The pen also emits light and emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The position of the pen can be specified by direction and distance. The projector draws (projects) the trajectory of the pen as stroke data.

Further, the configuration examples in FIG. 4 and the like are divided according to main functions in order to facilitate understanding of the processing by the display device 2 . The present invention is not limited by the division method or name of the unit of processing. The processing of the display device 2 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Also, part of the processing performed by the display device 2 may be performed by a server connected to the display device 2 via a network.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. Devices such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array) and conventional circuit modules are included.

Each function of the embodiment described above can be realized by an application program that runs on an OS (Operating System) that defines the operation of the CPU 201 provided in the display device 2 .

Note that the device to which the present embodiment is applied is not limited to an electronic blackboard as long as it is a display device having a function of operating an object by touch operation. Display devices to which the present embodiment is applied include, for example, PJs (projectors), output devices such as digital signage, HUD (Head Up Display) devices, network appliances, connected cars, notebook PCs (personal computers). , a mobile phone, a smart phone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a wearable PC, or the like.

The present invention has been described above based on each embodiment, but the present invention is not limited to the requirements shown in the above embodiments. These points can be changed within the scope of the present invention, and can be determined appropriately according to the application form.

2 display device 21 contact position detection unit 22 drawing data generation unit 23 character recognition unit 24 contact processing unit 25 voice processing unit 26 display control unit 27 data recording unit 28 network communication unit 29 operation reception unit 30 storage unit

JP 2012-088969 A

Claims (7)

a voice processing unit that receives voice input and converts the input voice into characters by voice recognition when a first touch operation on the screen is detected;
a display control unit that performs control to display the converted character at a position on the screen where the first touch operation is received;
When a second touch operation on the character displayed by the voice recognition is detected, the voice processing unit accepts the second touch operation as a correction operation of the character.
display device. The display control unit displays a correction candidate for the character when a second touch operation to the character displayed by the voice recognition is detected.
The display device according to claim 1. When the voice processing unit detects a second touch operation on the characters displayed by the voice recognition, the voice processing unit accepts handwritten input as a correction of the characters.
The display device according to claim 1. When the second touch operation to the character displayed by the voice recognition is detected, the voice processing unit receives voice input as correction of the character and converts it to the corrected character by voice recognition.
The display device according to claim 1. When the first touch operation on the screen is detected, the audio processing unit switches the operation mode of the correction function to the correction ON mode, and when the operation mode of the correction function is the correction ON mode, When the second touch operation on the character displayed by the voice recognition is detected, the second touch operation is accepted as a correction operation on the character, and when a certain time elapses from the start of the correction ON mode, the correction is performed. Switching the operating mode for the function to Correction OFF mode;
The display device according to any one of claims 1 to 4. A computer implemented method comprising:
a step of receiving voice input and converting the input voice into characters by voice recognition when a first touch operation on the screen is detected;
a step of controlling to display the converted character at a position on the screen where the first touch operation is received;
When a second touch operation on the character displayed by the voice recognition is detected, accepting the second touch operation as a correction operation of the character;
Display method. to the computer,
a step of receiving voice input and converting the input voice into characters by voice recognition when a first touch operation on the screen is detected;
a step of controlling to display the converted character at a position on the screen where the first touch operation is received;
when a second touch operation on the displayed character is detected by the voice recognition, accepting the second touch operation as a correction operation on the character;
program to run the

Images