KR20130102702A - Multi-modal input device using handwriting and voice recognition and control method thereof - Google Patents

Abstract

PURPOSE: A multi-modal input device using writing and voice recognition functions and a method for controlling thereof supply desired input information exactly as a result by modifying and grasping a misrecognition part of initial writing automatically when voice information is provided after a writing recognition result is confirmed. CONSTITUTION: A multi-modal input device using writing and voice recognition functions and a method for controlling thereof comprise the following steps: a writing recognition module (110) recognizes writing information, and converts it into first conversion information including first result information; a voice recognition module (120) is activated automatically for a predetermined time right after a transform operation, recognizes voice information, and converts it into a second conversion information including a second result information; a determining unit (132) grasps a misrecognition part of the first result information by using the second conversion information; a correction part (134) obtains a third result information by modifying the misrecognition part; and a display device (140) indicates the first and the third result information. [Reference numerals] (110) Writing recognition module; (120) Voice recognition module; (132) Determining unit; (134) Correction part; (136) Language option unit; (138) Notifying unit; (140) Display device; (150) Dictionary DB; (AA, EE) Writing information; (BB,GG,KK) Word information; (CC,DD) First conversion information; (FF) Conversion information determined by a user signal; (HH) Language model information; (II) First and second conversion information; (JJ) Third result information; (LL,MM) Second conversion information; (NN) Voice information

Description

Multi-modal input device using handwriting and voice recognition and control method

The present invention relates to a multi-modal input device and a method of controlling the input device, and more particularly, a handwriting recognition module using voice information received from a voice recognition module that is automatically activated for a predetermined time immediately after a conversion operation of the handwriting recognition module. The present invention relates to a multi-modal input device using handwriting and speech recognition for identifying and correcting misidentified parts and displaying a result thereof, and a method of controlling the input device.

Recently, with regard to a means of communicating between an electromechanical device such as a computer or a mobile terminal device and a human, a technology of utilizing a multimodal (multimode) type text input interface using a plurality of inputs such as a keyboard, a pen, and a voice has been integrated. It is attracting attention.

Among the methods using the multi-modal technology, methods for correcting and correcting the misrecognition portion of the result information (text) obtained through the first input method through the second input method have been disclosed in the past.

In US Patent No. 7,506,271, which is related with the present invention, when the initial result information obtained through the handwriting recognition method is displayed, the user selects a misrecognition portion and corrects the misrecognition portion by providing a speech recognition, a keyboard utilization, or a replaceable word list. And how to modify it.

In addition, US Patent No. 7,941,316, which is related to the present invention, displays initial result information obtained through a voice recognition module activated by a user, and provides a keyboard or a list of replaceable words to check for misrecognition one word at a time. It discloses how to correct and correct the misrecognition.

However, in order to correct and correct the misrecognition part in the conventionally disclosed multi-modal technology, the user has to select the misrecognition part directly and use a keyboard to input a key or select from a replaceable list. Even in the case of the user, there is an inconvenience that a separate operation of the user is required for activating the speech recognition engine.

The present invention, in order to improve the above problems, the voice recognition module is automatically activated for a predetermined time immediately after the conversion operation of the handwriting recognition module even if the user does not specify by a separate operation, if there is received voice information based on this To provide a multi-modal input device and a control method of the input device using handwriting and voice recognition that can automatically and correctly identify and correct the first recognition information of the handwriting information. There is this.

The multi-modal input device using the handwriting and the voice recognition as described above comprises: a handwriting recognition module for recognizing handwriting information and converting the handwritten information into a plurality of first conversion information including first result information; A voice recognition module which is automatically activated for a predetermined time immediately after a conversion operation of the handwriting recognition module to recognize voice information and convert the voice information into a plurality of second conversion information including second result information; A determination unit for identifying a misrecognition portion of the first result information received from the handwriting recognition module using the plurality of second conversion information received from the speech recognition module, and correcting the misrecognition portion of the first result information by modifying a third A controller including a correction unit for obtaining result information; And a display device displaying at least one of the first result information and the third result information.

On the other hand, the control method of the multi-modal input device using the handwriting and speech recognition of another aspect of the present invention, the first result information of the plurality of first conversion information converted by recognizing the handwriting information in the handwriting recognition module to display on the display device A first step of doing; A second step of automatically activating the voice recognition module immediately after the conversion to the first conversion information and converting the voice recognition module into a plurality of second conversion information including second result information when the voice information is recognized during the predetermined time; ; A third step of identifying a misrecognition portion of the first result information by using the plurality of second transformation information; And a fourth step of correcting the identified misrecognition portion to display third result information on the display device instead of the first result information, or to simultaneously display the first and third result information on the display device. It is composed.

When using the multi-modal input device using the handwriting and speech recognition and the control method according to the present invention as described above, after confirming the handwriting recognition result and providing only the voice information without the user's separate operation, the first recognition information of the first handwriting information is automatically Since it can be identified and corrected, there is an advantage that the user can accurately obtain the desired input information as a result.

In addition, in the case of the prior art, unlike a separate user operation is required to activate or switch the voice recognition module to a state in which voice recognition is possible, the voice recognition module of the present invention automatically for a predetermined time immediately after the conversion operation of the handwriting recognition module. By activating, there is an advantage in that the timing at which the voice input is applied is determined in advance so that the user's operation is not necessary, and thus, the false recognition rate can be lowered while reducing unnecessary power consumption.

In addition, using the multi-modal input device using the handwriting and speech recognition and the control method according to the present invention, by providing an option to select a language that can be recognized, anyone using any language can easily use, and display a notification window Even the first person to display on the device has an advantage that it is possible to know that the false recognition portion can be corrected through voice recognition for a predetermined time after handwriting recognition conversion.

In addition, in the present invention, by automatically detecting the misrecognition part from the first handwriting recognition result and providing various modification methods such as a matching method, a language model method, and a direct selection method, which are determined according to the accuracy value for the misrecognition part, the false recognition rate There is an advantage that can be reduced to excellent.

1 is an internal schematic diagram of a multi-modal input device using handwriting and speech recognition according to an embodiment of the present invention.
2 is an external schematic diagram of a multi-modal input apparatus using handwriting and speech recognition according to an embodiment of the present invention.
3 is a flowchart illustrating a control method of a multi-modal input apparatus using handwriting and voice recognition according to an embodiment of the present invention.
4 is a flowchart illustrating a control method of a multi-modal input apparatus using handwriting and voice recognition according to another embodiment of the present invention.
5A is a schematic diagram illustrating comparing first and second converted information in a process of identifying a misrecognition portion of first result information;
FIG. 5B is a schematic diagram illustrating searching for second result information matching each other among the first and second transform information in FIG. 5A; FIG.
6A is a diagram illustrating an example of a result displayed on the display apparatus of FIG. 2.
6B is a diagram illustrating another example of a result displayed on the display apparatus of FIG. 2.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will now be described in more detail with reference to the accompanying drawings. The terms and words used in the present specification and claims are to be construed in accordance with the technical idea of the present invention based on the principle that the inventor can properly define the concept of a term in order to explain its invention in the best way It should be interpreted as meaning and concept.

1 is an internal schematic diagram of a multi-modal input device using handwriting and speech recognition according to an embodiment of the present invention.

Multi-modal input device 100 using handwriting and speech recognition according to the present invention, the handwriting recognition module 110 for recognizing the handwriting information and converting it into a plurality of first conversion information including the first result information; A voice recognition module 120 which is automatically activated for a predetermined time immediately after the conversion operation of the handwriting recognition module 110 to recognize voice information and convert the voice information into a plurality of second conversion information including second result information; A determination unit 132 for identifying a misrecognition portion of the first result information received from the handwriting recognition module 110 using the plurality of second conversion information received from the voice recognition module 120, and the first result information. A controller 130 including a correction unit 134 for correcting a misrecognition portion to obtain third result information; And a display device 140 displaying at least one of the first result information and the third result information.

The multi-modal input device 100 using handwriting and speech recognition according to the present invention to be described later receives the handwriting information and displays the result, and recognizes and corrects the misrecognition part of the result displayed through the voice information and then returns the result. The main characteristic of the display is that the writing information or voice information input by the user is one or more words or one or more sentences composed of a plurality of words.

In the present invention, the handwriting recognition module 110 is a device for recognizing handwriting information and converting the handwritten information into a plurality of first conversion information, and typically, a digitizer, which is an input device that reads coordinates, which are input analog data, and delivers them in a digital format. Recognizing handwriting information input by a user using an electronic pen or a finger with respect to a separate device or a touch screen display device 140 used, and converts the handwriting information into a plurality of first conversion information according to a predetermined algorithm. To convert.

At this time, the plurality of first conversion information includes the first result information displayed on the display device 140 to be described later, the first result information selection criteria may vary according to the above algorithm, the algorithm is known in the art The method may be applied, and the handwriting recognition module 110 may determine, for example, first conversion information that is most likely similar to the handwriting information as the first result information.

The voice recognition module 120 is a device for recognizing voice information and converting the voice information into a plurality of second conversion information. The voice recognition module 120 is automatically activated for a predetermined time without a user's special operation immediately after the above-described conversion operation of the handwriting recognition module 110. The present invention is characterized in that it recognizes voice information applied by a user through a microphone or the like and converts the voice information into a plurality of second conversion information according to a predetermined algorithm.

In this case, the time at which the voice recognition module 120 is activated may be adjusted by a user, and the plurality of second conversion information includes second result information. The search for the second result information will be described later. do.

The controller 130 identifies and corrects the misrecognition portion of the first result information received from the handwriting recognition module 110 using the plurality of second conversion information received from the voice recognition module 120 as described above, and thus the third result information. In particular, the apparatus for converting the data into an apparatus includes a determination unit 132 for identifying a misrecognition portion of the first result information, and a correction unit 134 for correcting the identified misrecognition portion to obtain third result information.

Determination unit 132 is a device for identifying the misrecognition portion, the first matching information received from the handwriting recognition module 110 and the second matching information received from the plurality of second conversion information received from the voice recognition module 120 2 It is desirable to be programmed or controlled to search for and use the result information to automatically identify misrecognition parts.

The correction unit 134 is a device that is programmed or controlled to automatically correct the misrecognition portion by receiving information on the misrecognition portion identified by the determination unit 132, and various correction methods or algorithms may be stored in advance. It is preferable to correct the misrecognition portion of the first result information by using a correction method determined by comparing the accuracy value for the partial grasp and the predetermined reference value.

Detailed descriptions related to a method of identifying a misrecognition portion of the determination unit 132 and a method of correcting a misrecognition portion of the correction unit 134 will be described later.

In addition, the determination unit 132 and the correction unit 134 described above may receive information including word information and language model information stored in the dictionary database 150. In particular, the determination unit 132 may include a language model. Receiving the information can be used independently or auxiliary to identify the misrecognition portion, and the correction unit 134 can be used independently or auxiliary to receive the language model information to correct the misrecognition portion.

The dictionary database 150 may be an internal type mounted in the multi-modal input device 100 according to the present invention or an external type interlocked with an external server through a wired or wireless communication network, and may be continuously updated, and the handwriting recognition module 110 described above. ) And the voice recognition module 120 may also transmit information including word information.

Here, the language model information may be used independently or auxiliary to identify and correct the misrecognition portion of the first result information, and for example, the misrecognition portion of the first result information that is a sentence in English. In the case of grasping the first result information, each word constituting the first result information can be analyzed according to the position of the verb, the object, the bore, and the like. You can analyze which conversion information is syntactically replaceable in.

In addition, the controller 130 may further include a language option unit 136 and a notification unit 138, in addition to the determination unit 132 and the correction unit 134, the language option unit 136 is a voice recognition Simultaneously with the activation of the module 120, a notification window, for example, a notification window containing a content that can be corrected by voice and / or a microphone symbol and the like, may be controlled to be displayed on the display device 140 to be described later.

Here, the display format of the notification window may be changed in various ways.

In addition, the notification unit 138 may control the display device 140 to display a selectable language option in relation to the handwriting information and the voice information during the operation of the multi-modal input device 100 according to the present invention.

The language option unit 136 preferably operates by receiving a user signal to be described below before handwriting recognition or voice recognition is performed, and the notification unit 138 preferably operates immediately after handwriting recognition is made. It is adjustable.

In the present invention, the display device 140 is a device for displaying at least one of the first result information and the third result information, the present invention can be applied without particular limitation to conditions such as form, type, size, material, etc. When the touch screen is applied to the display device 140, the handwriting information may be transmitted to the handwriting recognition module 110 by operating as a single input interface rather than a device that unilaterally displays the result.

Furthermore, according to a specific misrecognition portion correcting method of the correction unit 134, which will be described later, the display device 140 may display the first transformation information and the second transformation information corresponding to the misrecognition portion, and another input method such as a virtual keyboard. May be displayed, and the conversion information determined by the user signal applied through the display device 140 may be transmitted to the correction unit 134 of the controller 130.

That is, the display device 140 may directly receive and display first conversion information and second conversion information, respectively, from the handwriting recognition module 110 and the voice recognition module 120 in some cases, and from the controller 130. First and second conversion information and third result information may be received and displayed.

Referring to FIG. 2, which is an external schematic diagram of the multi-modal input device 200 using handwriting and speech recognition according to an embodiment of the present invention, a user writes in handwriting using the electronic pen 220 on the digitizer 210. The information 250 is input, and the display apparatus 230 displays the first result information 260 converted from the handwriting information 250.

However, the user may confirm that some of the input writing information 250 is 252 is misrecognized and displayed on the display device 230, and the user may use voice information (eg, to correct and correct the misrecognition portion 262). 272 is applied near the microphone 240.

As described above, since the voice recognition module of the present invention is automatically activated only for a predetermined time immediately after the conversion operation of the handwriting recognition module, the user should apply the voice information 272 to be corrected within a predetermined time after checking the result of the writing recognition. I can understand that.

Through the multi-modal input device 100 using such handwriting and speech recognition, after confirming the result of the handwriting recognition, and providing the voice information, the timing of not only detecting and correcting the misrecognition portion of the initial handwriting information but also the timing of the voice recognition is possible. It can be decided in advance to reduce the extra user operation or unnecessary power consumption.

Meanwhile, a control method of a multi-modal input device using handwriting and voice recognition, which is another aspect of the present invention, will be described with reference to FIGS. 3 and 4, and in particular, a method of correcting and identifying a misrecognition portion of first result information. This will be described with reference to FIGS. 5A, 5B and 6A, 6B.

3 is a flowchart illustrating a control method of a multi-modal input apparatus using handwriting and voice recognition according to an embodiment of the present invention.

A control method of a multi-modal input apparatus using handwriting and voice recognition according to the present invention includes a first step of displaying first result information on a display device among a plurality of first converted information converted by recognizing handwriting information by a handwriting recognition module. ; A second step of automatically activating the voice recognition module immediately after the conversion to the first conversion information for a predetermined time, and converting the voice recognition module into a plurality of second conversion information including second result information when the voice information is recognized during the time; A third step of identifying a misrecognition portion of the first result information by using the plurality of second transformation information; And a fourth step of displaying the third result information on the display device instead of the first result information by correcting the identified misrecognition part or simultaneously displaying the first and third result information on the display device. There is a characteristic.

In other words, when the control of the multi-modal input apparatus using handwriting and voice recognition according to the present invention starts as shown in FIG. 3 (S300), first, the handwriting recognition module recognizes handwriting information and embeds the recognized information into an algorithm. In accordance with the conversion (S302), this algorithm can be applied to any conventionally known.

The handwriting conversion result information determined by the above algorithm among the converted information is displayed on the display device, and the voice recognition module is automatically activated for a predetermined time without a user's special operation immediately after the conversion operation of the handwriting recognition module (S304). It is preferable that the time to be changed varies depending on the user's setting or the like.

Here, after the voice recognition is automatically activated, the voice information recognition step may further include a waiting step and a writing input waiting step.

The reason is that after the handwriting recognition is completed, the voice recognition function is activated. However, if the handwriting input is activated during the activation, the voice recognition function needs to be deactivated and the process needs to be moved to the step of recognizing and converting the handwriting information (S302).

Subsequently, it is determined whether the voice information is recognized during the activation time (S306), and the voice recognition module automatically changes to the inactive state after a predetermined time regardless of whether the voice information is recognized.

If the voice information is recognized by the user, the recognized information is converted according to the applied algorithm, and the misrecognition portion of the handwriting conversion result information displayed on the display apparatus is grasped using the voice conversion information (S308).

The misrecognition part detected by the controller is corrected through one of various correction methods, and the corrected result information is displayed on the display device (S310). Instead of the existing handwriting conversion result information, the modified result information is displayed or both result information is displayed. Display at the same time.

If, during the activation time, the voice recognition module does not recognize the voice information (S312), it is determined that there is no misrecognition portion, only the handwriting conversion result information is displayed, and the control is terminated (S314).

In addition, it is preferable that a language model is applied together when identifying and correcting a misrecognition part, and a decision on a method of correcting a misrecognition part is preferably made after comparing an accuracy value for identifying a misrecognition part with a predetermined reference value.

4 is a flowchart illustrating a control method of a multi-modal input apparatus using handwriting and voice recognition according to another embodiment of the present invention. In particular, FIG.

When the control is started by the control method of the multi-modal input apparatus using handwriting and speech recognition according to the present invention (S400), the language option of the controller is operated first to select a language option that can be selected in relation to the input writing information and voice information. The display may be displayed on the display device (S402).

The operation time of the language option unit can be selected by the user at any time during the whole process. When one of the languages provided through the operation of the language option unit is selected, the conversion operation of the handwriting recognition module and the voice recognition module is performed according to the selected language. Is done.

Users can select one of the languages, such as Korean, English, Chinese, and Japanese, which are available as language options, for increased convenience, and continuous updates can be made through the dictionary database. Can be seen.

Thereafter, the handwriting recognition module recognizes the handwriting information and converts the recognized information into a plurality of first conversion information according to an embedded algorithm (S404), and displays first result information determined by an algorithm among the plurality of first conversion information. It is displayed on the device, and immediately after the conversion operation of the handwriting recognition module, the voice recognition module is automatically activated for a predetermined time without user's special operation (S406).

At this time, it is preferable to display a notification window on the display device that the notification unit of the controller operates to inform the user that the mistake can be corrected by voice (S408). Even a user who does not check the misrecognition portion of the first result information and correct the misrecognition portion by voice.

Subsequently, it is determined whether the voice information is recognized during the activation time (S410), and the voice recognition module automatically changes to the inactive state after a predetermined time regardless of whether the voice information is recognized.

If the voice information is recognized, the voice recognition module in the activated state converts the voice information into a plurality of second conversion information according to the applied algorithm (S412), and determines the controller that has received the first conversion information and the second conversion information. The step of comparing the two pieces of conversion information with each other (S414), searching for the second result information matching each other of the first and second conversion information (S416), and using the second result information to identify the misrecognition portion Step S418 may be performed.

In this case, it is preferable to use a plurality of second transformation information and also apply a language model in the misrecognition part identification process.

Hereinafter, a process of identifying a misrecognition portion of the first result information when the writing information 250 of 'I like that boy' is input by the user will be described in detail with reference to FIGS. 5A and 5B.

FIG. 5A is a schematic diagram illustrating comparing first and second conversion information in a process of determining a misrecognition portion of first result information by a determination unit of a controller, wherein the handwriting recognition module is 'I like that boy'. ) Is converted into a plurality of first conversion information, the voice recognition module converts the voice information 272 called 'boy' into a plurality of second conversion information, and the determination unit that has received the first conversion information and the second conversion information Each second transformation information may be compared according to a predetermined priority with respect to the first transformation information.

That is, in the process of obtaining the first result information 260 of 'I like that bag', first conversion information, which is a conversion value for each of 'I', 'like', 'that', and 'bag', is present. The second conversion information 'boi', 'boy', 'voy', 'voit', etc. can be listed in the 1st, 2nd, 3rd and 4th ranks, respectively, and compared with the plurality of the first conversion information according to the ranking. have.

For example, when the first rank 'boi' is compared with the first transformation information, no matching value is searched. When the second rank 'boy' is compared with the first transformation information, the matching value is searched as shown in FIG. 5B. This is defined as second result information 270, and the determination unit may determine a misrecognition part by using second result information 270 matching the first transform information among the plurality of second transform information.

If first conversion information matching 'boi', which is the first rank among the second conversion information, is searched, 'bay', 'bey', and 'bog' corresponding to the 'bag' part of the first result information 260 may be used. It is most likely to be one of the conversion values such as', 'boy', and so on. In this case, the determination unit may also identify the misrecognition part by using the second transformation information, and apply the language model information received from the dictionary database together. It can increase the accuracy of grasping for.

Here, it is preferable to select the second result information 270 by determining that the matching information is exactly the same, but the second result information that is matched by being determined to be conversion information having a similarity or higher than a predetermined probability according to the internal setting of the determination unit. 270).

A series of procedures for correcting the misrecognition portion of the first result information that is identified is carried out in the correction unit of the controller, the correction unit may be stored in various ways.

In determining the correction method of the misrecognition portion in the correction part, it is preferable to determine through the step (S420) comparing the accuracy value for identifying the misrecognition portion with a predetermined reference value.

The accuracy value for misrecognition is determined according to the algorithm or program applied to the correction unit, and it is a value that can determine the reliability of the multi-modal input device using handwriting and speech recognition according to the present invention together with the reference value. It is desirable to make decisions based on repeated test results and continuous update information.

For example, if the accuracy of the comparison result is equal to or higher than the predetermined reference value, the correction unit modifies the misrecognition portion into second result information matching each other among the first and second conversion information (S422), and the second result information is displayed on the display device. Control to display the reflected third result information (S424).

That is, as shown in FIG. 5B, 'bag', which is a misrecognition part, is corrected using 'boy', which is matched second result information 270, and 'I like that boy', which is third result information, is obtained.

As another example, if the accuracy value is equal to or higher than the predetermined reference value, the correction unit modifies the misrecognition portion into language model conversion information determined by applying a language model to each second conversion information (S426), and sets the language model to the display device. In operation S428, the third result information reflecting the conversion information may be displayed.

 As described above, the misrecognition part may be modified by applying language model information independently. However, in the above-described example, the misrecognition part may be modified by assisting the language model information.

As another example, when the comparison result shows that the accuracy value is lower than the predetermined reference value, the correction unit converts the misrecognition portion into the conversion information determined by the user signal among the first and second transformation information corresponding to the misrecognition portion displayed on the display apparatus (S430). In operation S432, the display apparatus may control to display the third result information reflecting the determined conversion information in operation S424.

That is, if the accuracy value is lower than the predetermined reference value, the correction unit may control the display device to display a list or a virtual keyboard of the first and second conversion information corresponding to the misrecognition portion, and the user tries to write the original in the list. You can select the conversion information you have done or enter it directly using the virtual keyboard.

The user signal generated by the user's selection or input is transmitted to the correction unit of the controller, and the correction unit may correct the misrecognition part with the conversion information determined by the user signal.

As such, when the accuracy value is lower than the predetermined reference value, the case may be a proper noun such as a person name or a local name or a word not registered in the dictionary database.

After correcting the misrecognition part identified as described above in various ways to convert the first result information to the third result information, the third result information is displayed on the display device instead of the first result information or the first and third results are displayed. Information is displayed at the same time (S424), and the control ends (S436).

In addition, when displaying the first and third result information at the same time, if there is no special input, the controller may determine that the displayed third result information is the information that the user wanted to input and control to display only the third result information. If necessary, the controller may control one of the above-described modification methods to be provided through the display apparatus.

Alternatively, if it is determined whether the voice information is recognized (S410), if the voice recognition module does not recognize the voice information during the activation time (S434), it is determined that there is no misrecognition portion, and only the first result information is displayed (S438), and the control ends. (S436).

Hereinafter, when the control method of the series of multi-modal input devices described so far is used, the results displayed on the display device of FIG. 2 will be described with reference to FIGS. 6A and 6B.

Referring to the exemplary embodiment illustrated in FIG. 6A, although the user inputs the writing information 250 such as 'I like that boy' into the handwriting using the electronic pen 220 on the digitizer 210, the display device 230 Looking at the first result information 260 by the handwriting recognition of 'I like that bag' shown in FIG. 2, the user may know that the bag is recognized as a bag 262 instead of the boy 252.

That is, the handwriting recognition module according to the present invention, when determined by the applied algorithm or program, 'bag' rather than 'bog', 'bey', 'bay', 'bou', 'boy', etc. among the plurality of first conversion information. It can be seen that the first result information 260 including 'bag' is determined to be similar to the most handwritten information.

Subsequently, the notification unit controls the display device 230 to display a notification window 232 indicating that the voice recognition module can be corrected within a predetermined time with the activation of the voice recognition module at the same time, and the user originally intended to input ' Voice information 272 called 'boy' may be applied.

The determination unit receives a plurality of second conversion information including the second result information 'boy' from the speech recognition module to determine that 'bag' of the first result information 260 is a misrecognition portion 262, and the correction unit The third result information 280 obtained by modifying the bag '262 to' boy '282 is controlled to be displayed on the display apparatus 230.

In this case, as illustrated in FIGS. 5A and 5B, the determination unit compares the plurality of first and second conversion information with each other and uses the 'boy', which is the matching second result information, to identify a misrecognition portion of the first result information. It is preferable to check the language model information stored in the dictionary database.

In addition, referring to another embodiment illustrated in FIG. 6B, although the user inputs the writing information 250 such as 'I like that boy, Luna' on the digitizer 210 by using the electronic pen 220, the display is displayed. Looking at the first result information 260 by the handwriting recognition of 'I like that bag, Lune' displayed on the device 230, the 'bag' 262 instead of the 'boy' 252 and the 'Luna' 254 Instead, the user may know that it is misrecognized as 'Lune' 264.

That is, when the handwriting recognition module according to the present invention is determined by the applied algorithm or program, among the first conversion information related to the 'boy' 252 which is the handwriting information, 'bog', 'bey', 'bay', 'bou' 'bag' is judged to be the most similar to ',' 'boy', and 'Lune' rather than 'Lana', 'Lane', 'Luna' among the first converted information related to 'Luna' 254, which is handwriting information. The first result information 260 including the misrecognition portions 262 and 264 is displayed by determining that the most similarity is found.

Subsequently, the user checks the misrecognition portions 262 and 264 from the first result information 260 and sequentially orders the 'boy' and 'Luna' voice information 272 and 274 originally intended to be input through handwriting recognition within a predetermined time. I can say it.

The determination unit receives a plurality of second conversion information including the second result information 'boy' and 'Luna' from the speech recognition module, and 'bag' and 'Lune' of the first result information 260 are misrecognized. 264), and the correction unit controls the display apparatus 230 to display the third result information 280 obtained by modifying the bag 262 with the boy 282.

However, in case of 'Luna', a proper noun corresponding to a person's name may be determined to be lower than a predetermined reference value.

That is, in this case, the display device may include 'Lana', 'Lane', 'Luna', which are first conversion information 292 corresponding to 'Lune', which is identified as a misrecognition portion, and 'runa', which is second conversion information 294. , 'lunar', 'luna', etc. can be displayed, and the user can select (298) or input directly through the virtual keyboard (296) method of the conversion information 290, which was originally intended to be recognized as handwriting. Can be.

The user signal generated by the user's input has one conversion information, and the conversion information determined by the user signal is transmitted to the controller of the controller.

As a result, the correction unit controls the display apparatus 230 to display the third result information 280 obtained by modifying the 'Lune' 264 to the 'Luna' 284 as the conversion information determined by the user signal.

Through the control method of the multi-modal input device using handwriting and voice recognition, if only the voice information is provided after checking the result of the handwriting recognition, the false recognition part of the initial handwriting information can be automatically recognized and corrected, and the timing at which the voice input is applied. It is possible to reduce a user's operation or unnecessary power consumption by deciding in advance, and to reduce a false recognition rate by providing various correction methods that are determined according to accuracy figures for identifying misidentification parts.

In the above, a specific preferred embodiment according to the present invention has been described. However, it is to be understood that the present invention is not limited to the above-described embodiments, and the above-described embodiments merely represent a part of various embodiments to which the principles of the present invention are applied. Those skilled in the art to which the present invention pertains may make various changes without departing from the spirit of the technical idea of the present invention described in the claims below.

100: multi-modal input device 110: handwriting recognition module
120: voice recognition module 130: controller
132: judgment unit 134: government
140: display device 150: dictionary database

Claims (15)

In a multi-modal input device,
A handwriting recognition module recognizing handwriting information and converting the handwriting information into a plurality of first conversion information including first result information;
A voice recognition module which is automatically activated for a predetermined time immediately after a conversion operation of the handwriting recognition module to recognize voice information and convert the voice information into a plurality of second conversion information including second result information;
A determination unit for identifying a misrecognition portion of the first result information received from the handwriting recognition module using the plurality of second conversion information received from the speech recognition module, and correcting the misrecognition portion of the first result information by modifying a third A controller including a correction unit for obtaining result information; And
And a display device configured to display at least one of the first result information and the third result information.
The method of claim 1, wherein the determination unit searches and uses the second result information that matches each other among a plurality of first conversion information received from the handwriting recognition module and a plurality of second conversion information received from the voice recognition module. Multimodal input device using handwriting and speech recognition, characterized in that the grasp portion.
The multimodal input apparatus of claim 1, wherein the correction unit modifies the misrecognition portion using a correction method determined by comparing an accuracy value for identifying a misrecognition portion with a predetermined reference value. .(newly open)
The multimodal input apparatus of claim 1, wherein the determination unit and the correction unit receive language model information stored in the dictionary database to identify and correct the misrecognition portions, respectively.
The apparatus of claim 1, wherein the controller further comprises a notification unit configured to display, on the display device, a notification window indicating that the voice recognition module can be corrected at the same time as the voice recognition module is activated. Modal input device.
The apparatus of claim 1, wherein the controller further comprises a language option unit configured to display, on the display apparatus, a language option selectable in relation to the handwriting information and the voice information during the input device operation. Multi modal input device.
In the control method of the multi-modal input device,
A first step of displaying the first result information on the display apparatus among the plurality of first conversion information converted by recognizing the writing information by the writing recognition module;
A second step of automatically activating the voice recognition module immediately after the conversion to the first conversion information and converting the voice recognition module into a plurality of second conversion information including second result information when the voice information is recognized during the predetermined time; ;
A third step of identifying a misrecognition portion of the first result information by using the plurality of second transformation information; And
And correcting the identified misrecognition portion to display third result information on the display device instead of the first result information, or to simultaneously display the first and third result information on the display device. A control method of a multi-modal input apparatus using handwriting and speech recognition.
The method according to claim 7, wherein in each step, the language option selectable in relation to the handwriting information and the voice information is displayed on the display device, wherein the handwriting recognition module and the voice recognition module are converted according to the selected language. Control method of multi-modal input device using speech recognition.
The method of claim 7, wherein when the voice recognition module is activated in the second step, a notification window is displayed on the display device to notify the display device that a notification can be corrected by voice. Control method of multi modal input device.
The method of claim 7, wherein the third step is
Comparing the plurality of first and second transformation information with each other, and identifying the misrecognition portion by using second result information that is matched with each other among the first and second transformation information. And controlling a multi-modal input device using speech recognition.
The multimodal input apparatus of claim 7 or 10, wherein in the third step, the misrecognition portion is determined by using a language model and simultaneously applying the plurality of second transformation information. Control method.
The multimodal input using handwriting and speech recognition according to claim 7 or 10, wherein in the fourth step, a method of correcting the misrecognition portion is determined by comparing an accuracy value for identifying the misrecognition portion with a predetermined reference value. Control method of the device. (newly open)
The method according to claim 12, wherein if the accuracy value is equal to or higher than a predetermined reference value, the misrecognition portion is corrected with second result information that matches each other among the first and second conversion information, and the second result information is displayed on the display device. The control method of the multi-modal input apparatus using handwriting and speech recognition, characterized in that for displaying the third result information reflected.
The method of claim 12, wherein when the accuracy value is equal to or higher than a predetermined reference value, the misrecognition portion is corrected to language model transformation information determined by applying a language model to each second transformation information, and the language model is applied to the display device. And a third result information reflecting the converted information.
The method according to claim 12, wherein when the accuracy value is lower than a predetermined reference value, the misrecognition portion is corrected to the conversion information determined by the user signal among the first and second conversion information corresponding to the misrecognition portion displayed on the display device. And displaying third result information reflecting the determined conversion information on the display device.

Images