KR20070008993A - Method of data acquisition using collaboration between input modalities - Google Patents

Abstract

A method for obtaining data with collaboration between input modalities is provided to flexibly and efficiently offer a service wanted by a user by using diverse input modalities according to an environment and using the collaboration between the input modalities. Data input is waited in an input standby state(201). The data inputted through one input modality is obtained if recognition reliability of the inputted data is over the first reference threshold and switchover to other input modalities having the higher reliability than one input modality is guided if not. The data inputted through one other input modality is obtained if the recognition reliability of the inputted data is over the corresponding reference threshold and the switchover to other input modalities having the higher reliability than input modality is guided if not. The integrated input data is obtained by integrating the input data obtained from each step(209). The obtained integrated input data is verified(210).

Description

Method of data acquisition using collaboration between input modalities}

1 is a configuration diagram of an embodiment of a multi-modal system using speech recognition and pen recognition to which the present invention is applied;

2 is a flowchart illustrating a method of acquiring voice data using a complementary relationship between voice recognition and pen input modality according to the present invention.

* Explanation of symbols for the main parts of the drawings

10: multimodal system 100: multimodal manager

101: multimodal browser 102: pen recognition module

103: speech recognition module 104: conversation manager

105: Dialog Engine 106: Web Server

The present invention relates to a data acquisition method using a complementary relationship between input modalities in a multi-modal system, and a computer-readable recording medium recording a program for realizing the method. In the modeled multi-modal system, when the input and output of the voice is transferred to the next state by using input means in different complementary relations (cooperative relations), data that failed to acquire is collected (collected) to improve the performance and reliability of the entire system. A data acquisition method using a complementary relationship between input modalities and a computer readable recording medium having recorded thereon a program for realizing the method.

In general, a multimodal system is a system for providing a service desired by a user by modeling two or more human sensory organs. Examples of the multi-modal input include voice input, keyboard input, mouse input, pen input, touch screen input, gesture input, eye movement input, and the like. In the present invention, for example, voice input and pen (or keyboard) input will be described for convenience. Shall be. However, the present invention is not limited thereto and may be implemented by using two or more different inputs.

Multimodal input methods are currently being developed in various ways, but the simplest and most convenient method is to assume one modal input at a time. Such a sequential multi-modal input method can avoid technical difficulties such as collision or synchronization between different inputs. Therefore, the present invention will be described by taking the case of implementing the sequential input method as an example.

In conventional multimodal systems, all multimodal input patterns are viewed independently and no attempt is made to use or infer relationships between them. In general, voice input does not require a different input medium, but has low reliability, and an interface through the input device has high reliability but low portability and convenience. In this case, preferences among various input methods change according to the environment, and different input methods have complementary or cooperative relationships.

Nevertheless, the conventional multimodal system considers each other's inputs to be different independent inputs, and causes a user to repeat an inefficient input until a correct input is received.

The present invention has been proposed to solve the above problems, and it is possible to flexibly and efficiently provide a service (input method) desired by the user by using various complementary input methods (cooperative relationship) according to the environment. It is an object of the present invention to provide a data acquisition method using a complementary relationship between input modalities and a computer-readable recording medium recording a program for realizing the method.

In other words, the present invention is a multi-modal system modeling two or more sensory organs of the human body to transfer the data that has previously failed to acquire while transitioning to the next state by using input means in a different complementary relationship (cooperative relationship) during voice input and output The present invention provides a data acquisition method using a complementary relationship between input modalities and a computer-readable recording medium recording a program for realizing the method, by which acquisition (acquisition) can improve the performance and reliability of the entire system. There is this.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

According to an aspect of the present invention, there is provided a data acquisition method comprising: waiting for data input in an input standby state; Obtaining input data when the recognition reliability of the data input through the one input means is greater than or equal to the first reference threshold, and inducing conversion to another input means having higher reliability than the one input means when the recognition reliability of the data input through the one input means is less than the first reference threshold; 1 treatment step; Acquiring input data when the recognition reliability of the data input through the other input means is equal to or greater than a corresponding reference threshold, and inducing switching to another input means having higher reliability than the other input means as the recognition reliability is lower than the corresponding reference threshold. A second processing step of repeatedly performing the processing up to the input means having the highest reliability; And integrating the input data obtained in each processing step to obtain integrated input data.

The method further includes a verifying step of verifying the obtained integrated input data.

In addition, the method of the present invention further includes the step of calculating the reference threshold value using the input data finally obtained by performing the verification step and reflecting the next service.

On the other hand, the present invention, a multi-modal system having a processor, the function to wait for data input in the input standby state; A function of acquiring input data when the recognition reliability of the data input through the one input means is greater than or equal to the first reference threshold, and inducing a switch to another input means having higher reliability than the one input means when the recognition reliability of the data input through the one input means is less than or equal to the first reference threshold. ; Acquiring input data when the recognition reliability of the data input through the other input means is equal to or greater than a corresponding reference threshold, and inducing switching to another input means having higher reliability than the other input means as the recognition reliability is lower than the corresponding reference threshold. Repeating the process up to the input means having the highest reliability; And a computer-readable recording medium having recorded thereon a program for realizing a function of acquiring integrated input data by integrating the obtained input data.

The present invention also provides a computer-readable recording medium having recorded thereon a program for further realizing a verification function for verifying the acquired integrated input data.

The present invention also provides a computer-readable recording medium having recorded thereon a program for further realizing a function of calculating a corresponding reference threshold value to reflect the next service using the input data finally obtained by performing the verification function. do.

As described above, the present invention allows data to be acquired using a complementary relationship (cooperative relationship) between input modalities in a multi-modal system, and can be used to increase the recognition performance. That is, the present invention can improve the reliability of an overall multi-modal system by examining whether an input of higher reliability exists if the input of a low reliability comes in and if recognition fails repeatedly. have. For example, in the present invention, even if the speech recognition fails, by presenting another modal input (pen input), it is possible to increase the reliability of the multi-modal system by enabling the acquisition and reuse of the voice data and improving the performance of the system. .

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, to help understand the present invention, the reliability of the input schemes and the concept of a multimodal system will be described.

The recognition result of the pen input or voice input input by the user differs depending on the pattern. For example, in the case of the touch screen, the recognition result of the input signal is only two cases when a signal is detected and not, so the reliability is always 1 or 0, whereas the reliability of the pen recognition result or the speech recognition result is the surrounding environment. It is between 0 and 1 because of its influence or technical difficulties. In the speech recognition system, since there is no other input means other than the speech interface, there is a disadvantage in that it cannot transfer to the next process if the recognition fails continuously. However, in the multi-modal system, not only various input means are provided, but also complementary means are provided between input devices which present different inputs when a certain input fails. This modeled a situation that is common in the human world, for example, when two people don't understand each other's names when they talk on the phone.

1 is a block diagram of an embodiment of a multi-modal system using speech recognition and pen recognition according to the present invention.

As shown in FIG. 1, the multimodal system using voice recognition and pen recognition to which the present invention is applied includes a multimodal manager 100 and a conversation manager 104, and the two managers 100 and 104. ) Is connected through the multimodal manager interface. The multi-modal manager 100 includes a multi-modal browser 101, a pen recognition module (server) 102, a voice recognition module (server) 103, and the conversation manager 104 includes a conversation engine. It comprises a 105 and a web server 106, the conversation manager 104 is connected to other external agents through a conversation technology interface.

In this case, multimodal situations, such as speech recognition and pen (writing) recognition, generally require a considerable level of computing power. Therefore, after the terminal performs a predetermined level of preprocessing on the input signal, the feature extraction result is sent to the speech recognition module 103 or the pen recognition module 102, and thus the speech recognition module 103 or the pen recognition module 102 is performed. ) Is the way to get the final result.

In a multimodal system, for the interaction between the system and the user, the conversation manager 104 monitors the flow of the conversation until the input condition for the specific task is satisfied. For a more general approach, two important input modalities can be integrated using both Voice (eXtensible Markup Language) XML and Web Markup Language (HyperText Markup Language).

The multi-modal browser 101 enables a user to have an equal representation between input devices. In the current system, the multimodal browser 101 binds the computer environment of the voice dialog and the existing graphic interface environment. On the other hand, the conversation manager 104 receives the user's input and stores it in its own storage device, and captures the necessary information from the web server 106 and delivers it to the user. For reference, in the multi-modal system to which the present invention is applied, a voice output module (server) such as speech synthesis is not shown in the drawings for the convenience of focusing on the input device.

FIG. 2 is a flowchart illustrating a method of acquiring voice data using a complementary relationship between voice recognition and pen input modality according to the present invention, and illustrates a process of integrating / verifying data input from an input device and proceeding to the next step. It is a flow chart.

As illustrated in FIG. 2, in a multi-modal system, when a user attempts to input through speech recognition, when the voice data whose reliability does not exceed a threshold is input more than a predetermined number of times (eg, two or more times), the conversation manager 104 is input. After the user requests input with a pen and checks the validity of the result, the voice data and text data are collected and stored in the storage device. Extending this to a general multimodal input, if a low reliability input comes in and if the recognition fails repeatedly, examine whether there is an input that expects a higher reliability recognition result, and if so, let the user adopt the input. It can improve the recognition performance of modal system. In general, the order of low reliability input to high input is voice input, handwriting input, keyboard input, touch screen (pen, hand), and mouse.

Referring to FIG. 2, a method of obtaining voice data using a complementary relationship between voice recognition and pen input modality is described in detail as follows.

First, when the service is started, a voice input or pen input waiting state is initially waited for the user's input (201).

When the user selects a voice input and inputs a voice in the voice input or pen input standby state 201, the voice is received and the end point is detected to secure a voice data segment (202).

The speech recognition reliability is then obtained for the speech input data using the recognition grammar 205 representing the speech recognition engine and the pen recognition list (204). Here, for example, the speech recognition reliability may be calculated as a ratio between the phoneme sequence of the target model and the phoneme sequence of the anti-model. One anti-model is created by merging all the data except the target model. Since the size of the model becomes larger than the original data, it is randomly proportional to the target data volume to balance the data volume of the target model. Random data is generated to generate an appropriate amount of anti-data.

Subsequently, when primary speech recognition fails by comparing the obtained speech recognition reliability with a reference (allowed) threshold (207), the process proceeds to the speech input process 202 to perform the following processes, and when the secondary speech recognition fails. Next, the process proceeds to the pen input process 203 and performs the following processes. If the speech recognition is successful, the process proceeds to the multimodal input integration process 209. Here, referring to a specific example of setting the reference threshold value, the reference threshold value is calculated by using the input data finally obtained by performing a subsequent verification process 210 and reflected in the next service. The processing procedure for each case of the input recognition success and failure will be described later.

Meanwhile, when the user selects a pen input and inputs data through the pen in the voice input or pen input standby state 201, the pen recognizes the corresponding data to secure pen input data (203).

The pen recognition reliability is then obtained for the pen input data using the recognition grammar 205 representing the speech recognition engine and the pen recognition list (206). Here, the pen recognition reliability calculates the recognition reliability similarly to the method used in the above speech recognition after size normalizing the handwritten data first.

Subsequently, when the first pen recognition fails by comparing the obtained pen recognition reliability with a reference (allowed) threshold (208), the process proceeds to the pen input process 203 to perform the following processes, and when the second pen recognition fails Next, the process proceeds to the voice input or pen input standby state 201 and performs the following processes. If the pen recognition is successful, the process proceeds to the multimodal input integration process 209. Here, referring to a specific example of setting the reference threshold value, the reference threshold value is calculated by using the input data finally obtained by performing a subsequent verification process 210 and reflected in the next service. The processing procedure for each case of the input recognition success and failure will be described later.

Thereafter, the recognized voice input data and the pen input data are integrated to obtain all input data (209). For example, the results of Hong Gil-dong's voice recognition and pen recognition are stored in the storage media as shown in Table 1 below with the following additional information. In general, the input data is stored in the order of high input data from low input reliability data.

Thereafter, in operation 210, an error of the obtained entire input data may be verified to finally obtain high purity input data.

In the comparison process 207 and 208, the case where the input data is successfully recognized and transferred to the next process is as follows.

1.Initial state (pen or voice input standby) → Voice (success) → Next process

2. Initial state → Pen (success) → Next step

3. Initial state → Pen (failure) → Input request (write one more time) → Pen (success) → Next step

4. Initial state → Voice (failure) → Input request (speak one more time) → Voice (failure) → Input request (write with pen this time) → Pen (success) → Next step

5. Initial state → Voice (failure) → input request (speak one more time) → Voice (failure) → input request (write with pen this time) → Pen (failure) → input request (write one more time) → Pen (success) → Next course

In the comparison process 207 or 208, the recognition of the input data fails and returns to the initialization state as follows.

1.Initial state → Voice (failure) → Timeout → Initial state

2. Initial state → Voice (failure) → Input request (speak again) → Timeout → Initial state

3. Initial state → voice (failure) → input request (speak one more time) → voice (failure) → input request (write with a pen this time) → time out → initial state

4. Initial state → voice (failure) → input request (speak one more time) → voice (failure) → input request (write with a pen this time) → pen (failure) → input request (write one more time) → time out → initial stage condition

5. Initial state → voice (failure) → input request (speak one more time) → voice (failure) → input request (write with a pen this time) → pen (failure) → input request (write one more time) → pen (failure) → Initial state

6. Initial state → Pen (failure) → Input request (write one more time) → Timeout → Initial state

7. Initial state → Pen (failure) → Input request (write one more time) → Pen (failure) → Initial state

The data finally obtained is limited to the case where recognition is successful and transferred to the next process. As a result, in the case of speech data that satisfies the reference reliability, text information is absent, but the input finally accepted by the alternative input means of the pen input because it does not satisfy the reference reliability includes not only the speech data but also the text information. High value for use, such as being used as learning data for analysis.

As described above, the present invention sets up a relationship between input devices from low-reliability input to high input devices in advance in an increasingly diversified user input environment, without regard to each input means independently or independently. If unsuccessful, the conversation manager continuously attempts an input request from the alternate device to the user until the reliability reaches an acceptable threshold, providing a more user-centric, robust and intelligent service.

That is, the present invention defines a sequence between input devices for which the final recognition is successful in advance, and then allows the conversation manager to use the speech recognition process and the pen input process to recognize the speech defined in the recognition grammar more than once. By replacing the target word with a pen input and requesting the user, the user can provide a seamless service to the user and obtain textual voice data. As a result, it is possible to reduce the probability of misperception by setting the reference threshold somewhat strict if an alternative input means is present. Finally, in order to increase the reliability of the data, the third-party verification between the speech data and the text data which has undergone the recognition process is eliminated, thereby obtaining high quality speech data. As a result, this may contribute to the gradual quality improvement of the speech recognition system.

In the above embodiment, it is assumed that two types of input devices are speech recognition and pen recognition. However, the embodiment is just an example. In the present invention, the input device for the multi-modal system is a voice input, a keyboard input, a mouse input, a pen. All types of input devices may be used, such as input, touch screen input, gesture input, and eye movement input.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, ROM, RAM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

The present invention as described above, in the multi-modal system modeling two or more sensory organs of the human being obtained while the transition to the next state by using the input means in a different complementary relationship (cooperative relationship) during voice input and output By acquiring (collecting) failed data, the performance and reliability of the entire system can be improved.

In other words, in an increasingly diversified user input environment, the present invention pre-sets the relationship between input devices from low-reliability input to high input devices without having to view each input means separately or independently, and any input device is used for recognition. If unsuccessful, the conversation manager continuously attempts an input request to the user through an alternative device until the reliability reaches an allowable threshold, thereby providing a more user-oriented robust and intelligent service.

In addition, the present invention, after defining the sequence between the input devices that the final recognition is successful in advance to the table, the conversation manager uses a speech recognition process and a pen input process, if the speech recognition fails more than once, the recognition target defined in the recognition grammar By replacing words with pen input and asking the user, it provides the user with a seamless service and the advantage of obtaining textualized voice data. It is possible to reduce the probability of misrecognition by setting up more strictly, and finally, to remove the error data through the third-party verification between the speech data and the text data which have been recognized to increase the reliability of the data, and finally the high quality speech Acquire the data and accordingly the points of the speech recognition system There is an effect that can contribute to quality improvement.

Claims (9)

In the data acquisition method, Waiting for data input in an input waiting state; Obtaining input data when the recognition reliability of the data input through the one input means is greater than or equal to the first reference threshold; 1 treatment step; Acquiring input data when the recognition reliability of the data input through the other input means is equal to or greater than a corresponding reference threshold, and inducing switching to another input means having higher reliability than the other input means as the recognition reliability is lower than the corresponding reference threshold. A second processing step of repeatedly performing the processing up to the input means having the highest reliability; And Acquiring integrated input data by integrating the input data acquired in each processing step; Data acquisition method using a complementary relationship between the input modality comprising a. The method of claim 1, A verification step of verifying the acquired integrated input data Data acquisition method using a complementary relationship between the input modality further comprising. The method of claim 2, Performing the verification step to calculate the corresponding threshold value using the finally obtained input data to reflect the next service Data acquisition method using a complementary relationship between the input modality further comprising. The method according to any one of claims 1 to 3, The first processing step, A first input step of receiving data through the first input means; A first comparing step of recognizing the data input in the first input step, measuring a recognition reliability, and comparing the measured reliability with a first reference threshold; Obtaining input data according to a comparison result of the first comparing step, as the recognition reliability is equal to or greater than a first reference threshold; A first re-input processing step of repeating the first input step and the first comparison step by requesting re-input as the recognition result is less than a first reference threshold as a result of the comparison in the first comparison step; Acquiring input data according to a comparison result in the first re-input processing step, as the recognition reliability is equal to or greater than a first reference threshold; And A first switch derivation step of inducing a switch to the other input means having higher reliability than the first input means as a result of the comparison in the first re-input processing step, as the recognition reliability is less than the first reference threshold; Data acquisition method using a complementary relationship between the input modality comprising a. The method of claim 4, wherein The processing of the second processing step, A second input step of receiving data through the other input means; A second comparing step of recognizing data input in the second input step, measuring a recognition reliability, and comparing the received data with a second reference threshold; Obtaining input data according to a comparison result of the second comparing step, the recognition reliability being greater than or equal to a second reference threshold; A second re-input processing step of repeating the second input step and the second comparison step by requesting re-input as the recognition result is less than a second reference threshold as a result of the comparison in the second comparison step; Obtaining input data as a result of the comparison in the second re-input processing step, as the recognition reliability is greater than or equal to a second reference threshold; And A second switch derivation step of inducing a switch to another input means having higher reliability than the other input means as the recognition reliability is less than a second reference threshold as a result of the comparison in the second re-input processing step. Data acquisition method using a complementary relationship between the input modality comprising a. The method of claim 5, The first input means is a voice input means, and the other input means is a pen input means, using a complementary relationship between input modalities, wherein the text data or the text data and the voice data are obtained through one conversion induction process. Data acquisition method. In a multimodal system with a processor, Waiting for data input in an input standby state; A function of acquiring input data when the recognition reliability of the data input through the one input means is greater than or equal to the first reference threshold, and inducing a switch to another input means having higher reliability than the one input means when the recognition reliability of the data input through the one input means is less than or equal to the first reference threshold. ; Acquiring input data when the recognition reliability of the data input through the other input means is equal to or greater than a corresponding reference threshold, and inducing switching to another input means having higher reliability than the other input means as the recognition reliability is lower than the corresponding reference threshold. Repeating the process up to the input means having the highest reliability; And A function of integrating the obtained input data to obtain integrated input data A computer-readable recording medium having recorded thereon a program for realizing this. The method of claim 7, wherein Verification function for verifying the acquired integrated input data A computer-readable recording medium that records a program for further realization. The method of claim 8, A function of calculating the corresponding threshold value using the finally obtained input data by performing the verification function and reflecting it to the next service A computer-readable recording medium that records a program for further realization.

Images