KR20200109833A - A Coputer Program for Reducing Waiting Time in Automatic Speech Recognition - Google Patents

Abstract

A program for executing a program in a computer to execute a method for reducing waiting time in automatic voice recognition in a computer can improve fast response time and shorten overall waiting time by monitoring voice buffer to predict the ″end of speech″ of a received voice and improving user experience of natural language-based human computer interaction, classify incoming sounds, remove a noise, detect dissatisfied words and phrases to determine EOS time limit, call EOS based on a keyword to which a weight is assigned, and shorten voice recognition waiting time by varying and controlling EOS time.

Description

A Coputer Program for Reducing Waiting Time in Automatic Speech Recognition, for executing a program on a computer for executing a method of reducing waiting time in automatic speech recognition.

The present invention monitors the voice buffer to predict the "end of speech" of the received voice, thereby improving the user experience of human-computer interaction based on natural language, thereby reducing the overall waiting time and faster response time. To a program for running on a computer how to reduce the waiting time in

Automatic speech recognition (ASR) and natural language understanding (NLU) receive a user's voice and go through an interpretation process to find the most similar data in text format. Deep Neural Networks (DNNs) are used to create probabilistic state diagrams that search for text data closest to an established linguistic model, and take vast amounts of data consisting of dictionary data, morphors, training data, etc. Is learned using.

The ASR system provides the ability for the ASR system itself to indicate the end of speech (EOS), but this calling method can start too early or make it unnecessary too late to get meaningful contextual data. Disfluency (disfluency) expression often leads to silence, so the ASR system returns meaningless data or discards it entirely due to lack of context, adding latency to the overall user experience.

Korean Patent Laid-Open Patent 10-2009-0054642 (Name of invention: voice recognition method and apparatus)

Among the speech data, silence due to the disfluency of the voice may be followed, but the current ARS system judges that silence caused by the disfluency is terminated and returns it as meaningless data as it is, or vice versa. Since the time point cannot be determined normally, there is a problem that unnecessary waiting time is increased.

Due to such a problem, there are problems in that meaningful text data cannot be extracted normally or that user convenience is impaired due to an increase in waiting time.

The present invention was devised to solve the above-described problems, and by monitoring the voice buffer to predict the "end of speech" of the received voice, the user experience of human-computer interaction based on natural language was improved, resulting in faster response time and It is an object of the present invention to provide a program for executing a method of reducing the waiting time in a computer in automatic speech recognition that can shorten the overall waiting time.

An embodiment of the present invention, which is preferred as a means for solving the problems of the present invention described above, is a program for executing a method of reducing waiting time in automatic speech recognition on a computer, comprising: receiving a speech as an audio input; Performing noise analysis on the incoming sound data; Determining whether the received sound data has characteristics of actual speech data; Performing automatic speech recognition to convert the nearest equivalent data into text format; Determining a value of the ASR output, a speech category, and a speech disfluent; Determining an end period of the speech end time window based on the value of the ASR output, the speech category, and the speech discomfort; Determining an end moment of speech based on the value of the subsequent ASR output, speech category and speech imbalance; It is characterized by calling and controlling the initial EOS based on the EOS timeout and intermediate ASR results prior to the final ASR result.

In addition, the noise analysis step,

Observing the duration of the continuous level of the input sound to determine the noise level; Includes.

And observing the ratio of the silent and non-noisy portions of the sound sample to determine the value of the input data.

Also, maintaining a list of offensive words to predict the high weighted keywords that will follow; Includes.

It may also include a list of known speech imbalances for adjusting the EOS time frame window.

In addition, it may include; determining the EOS based on the weighted keyword.

Also, observing the pattern of the silence and data burst to determine the likelihood that the input data can be regarded as speech data; may include.

Further, the method may further include calculating the ratio of the silence to the sound level, and determining a probability that the input data can be considered as speech data.

Also, it may include the step of determining the requested query based on the shape of the word (in this case, whether the word is an object of the sentence).

It can also provide a longer introduction to help first-time callers prepare to talk to artificial intelligence based on caller records.

According to an embodiment of the present invention, by adaptively variable determination of the speech end timer window according to the characteristics of speech data according to analysis of a speech signal, it is possible to detect a more rapid and accurate speech end (EOS) time point. .

In addition, the characteristics of the speech data are determined according to at least one of the value information of the speech data detected according to the analysis of the speech signal, the classification information of the speech data, and the disfluency information of the speech data. While maintaining the accuracy of, the waiting time for a voice recognition response can be shortened, and accordingly, user convenience can be greatly improved.

For a more complete understanding of the invention, reference is now made to the following description taken in connection with the accompanying drawings.
Figure 1 shows the latency of the ASR experienced by the user after the command is issued.
2 illustrates how voice input data is perceived and how EOS calls should be detected according to an aspect of the present specification.
3 shows a local database for finding potential matches of previously issued commands to reduce waiting time according to an aspect of the present specification.
4 shows how an EOS call is handled in the presence of insensitivity according to an aspect of the present specification.
5 illustrates how the dynamic EOS window size can reduce latency according to an aspect of the present specification.
6 shows the waiting time that can be reduced for utterances including disfluency according to an aspect of the present specification.
FIG. 7 illustrates an automatic speech recognition method with variable EOS timeout context dependent calling improved by speech activity detection and continuous characteristics of speech data according to an aspect of the present specification.
8 illustrates a method of filtering non-speech data prior to ASR processing according to an aspect of the present specification.
9 illustrates dynamic word weight allocation according to an aspect of the present specification
10 illustrates the overall latency reduction that can be achieved according to an aspect of the present specification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present invention is interpreted as a voice input while maintaining a high level of accuracy through a call EOS based on an intermediate result analysis and senseless voice (speech opacity) presence detection while performing noise analysis based on the presence of speech, sequence, and continuity. It improves the user experience by reducing the latency that appears in the speech recognition response.

Since most client-side systems do not have the hardware specifications required to host the complete language model locally, the language recognition model is usually hosted on a remote server. The customer then sends a voice to receive the text (STT) results voiced. ASR can return intermediate results and decide when to call EOS. Calling EOS returns the final machine analysis dialog box.

The timeout mechanism is usually used by the ASR system to call EOS. After observing a predetermined number of silent time frames, the ASR system calls EOS, processes the obtained speech data, and interprets it as text data. The silent time frame defined by the ASR system for determining EOS is based on the latency user experience. While speaking the command, the user can pause the sentence for any reason, so an erroneous EOS call can lead to unwanted end results.

Fig. 1 shows that the user has experienced the overall waiting time to receive the intended result with the logarithmic linear model. The total waiting time can be expressed by the equation.

log (Yi) = μ + α + βXi + υi

Here, the response variable is the network delay (μ), 103, the independent variable is the ASR processing time (α), 102, the blocking variable is the EOS detection (β), 101, the ASR process factor due to, and the ratio of the expected result of the wrong user (е ). X is the speech length 104 determined by the time of the EOS call. If the utterance is repeated due to unexpected results, the overall X may increase.

When the ASR detects silence on the audio input, the EOS timeout begins. If audio is detected before the EOS time limit (100 seconds), the time limit is reset. If audio is not detected within the EOS timeout time frame 101, EOS is called. When the ASR system processes the audio, the result is sent back to the user (105).

2 shows a method of checking voice input data in an ASR system. As shown in 201, 208, 202, 209, bursts of data and patterns of silence may occur, but continuous data may occupy multiple time frames as observed in the following.

In most ASR systems, EOS will be called at 207 after following a full sentence and several time frames of silence. Calling the initial EOS at point 211 can perform the desired action much faster, reducing the waiting time the user feels. In addition to the server ASR, a local ASR solution can reside within the client to access the cached interpretation to compensate for the misinterpretation caused by the initial EOS.

Calling EOS early can significantly reduce the wait time the user feels, so the model is closer to the log log model. However, doing so creates a potential misinterpretation, causing the overall latency experienced by the user. You can run the server to log previous queries and cache answers to the most frequently asked questions so that the final result is not the desired probability with an incorrect result. This introduces an additional probabilistic model to the ASR design, reducing the possibility of misinterpretation.

3 shows how locally cached data processing intermediate ASR results calls initial EOS based on a probability model. Once the user 300 provides the utterance 301, the speech data 304 uses the local audio analysis module 302 to find a match within the local operational database 307, and the required text data or speech data 308 If matched by providing the audio/text data 303 is provided to the user terminal. When the speech data 304 does not match, it is transmitted to the remote ASR system 305 to provide a response processing 306 according to the voice recognition of the remote ASR system 305 to the user terminal. Accordingly, it is possible to generate the desired result 303 much faster by processing it in the local audio analysis module than when retrieving data from the remote ASR system 305. The locally cached data is updated based on the incoming speech data, accumulates command data, updates the probability of the command, and improves the accuracy of the probability model.

Delay can be increased when dissatisfaction is introduced into the ASR system. The ASR system either waits for more meaningful data or simply calls EOS. In both cases, there is an additional delay in returning meaningless data and the user may have to repeat the voice command. In this specification, disfluency, a meaningless sound, is used to expect contextual data and prepare the system to increase the importance of the next utterance. Korean follows the format of "subject-object-verb", and in most cases, the subject can be omitted completely and the desired context can be provided. This sentence format is ideal for invoking initial EOS detection while anticipating the expected request and yielding accurate results. For this specification, three months of data were collected, and 90% of user input is in the form of a request that follows the "subject-object-verb" or "object-verb" format.

Figure 4 shows how this specification handles the complaints of traditional ASR systems that are directly related to the timing of EOS calls. Most ASR systems either eliminate the defect or go through a matching process to produce an analysis that is not relevant to the situation. In most cases, the ASR system can call EOS at 401 or 403 depending on the preset size of the time frame. While the input is being processed and the final result is returned, the ASR system often does not accept additional input until the interpretation work is complete. This often causes the user to repeat the desired utterance, prolonging the overall waiting time felt by the user. This specification extends the EOS time window (EOS time window) and increases the weight of the next utterance after a non-fluency sound. Instead of reaching EOS at 403, this specification suggests a time window to disable EOS at 401 and 403. When data from 404 and 405 are received, the EOS time window is reset until no more sound data is visible in successive time frames. EOS is called in the time zone within 408.

As shown in FIG. 5, since non-fluency speech data with meaningful context data may follow irregularities, the EOS timeout window may be dynamically changed according to the utterance length following the irregularities. A typical EOS timeout window expires at a fixed location. However, by changing the size of the EOS timeout window 411, the total waiting time is reduced (412).

6 shows the waiting time that can be reduced for utterances including opacity according to this specification. 500 and 501 are the opacity following 502, which is the actual utterance the user wants to process. (A)-In conventional technology, EOS is called at 503 after the first EOS timeout occurs at 515. 505 represents the data transfer time and process time added after calling EOS. 506 is the returned result from ASR system A. (A)-If contextual data is ignored in the prior art, the user repeats the command after recognizing that the intended speech was ignored after receiving the data from non-volatile (506). 508 and 509 are irregular, followed by 510, which contains contextual data that ASR must process. 518 is the moment of the EOS call after 511, which is the EOS timeout. 512 represents the data transfer time and processing time of EOS. Calling EOS on the 518 later may result in more delay than calling the intermediate EOS on the 503, but the actual latency shown at 506 increases by 517 as further data entry is blocked while the ASR interprets and returns the data. 506 is in progress, which is affected by both 515 and 505, and the user repeats the utterance at 504 as the utterance 502 cannot be sent to the ASR system. By extending the time frame of the EOS timeout, 513 hours can be obtained compared to the case where disfluency is not handled for exception handling.

To compensate for potential errors that could misinterpret noise and cause unwanted initial EOS, all sound data can first be subjected to noise analysis and voice activity detection to ensure that non-contextual data is filtered and not passed to the STT and ASR systems. . The detection of incoming sound data can first be analyzed to form an initial determination of whether the input data is classified as noise or speech data through sound level monitoring. This method can optimize resource usage by local filtering out unnecessary transactions on the ASR server. Speech data often follows data in a burst pattern, but noise tends to represent a continuous level of data. The variety and range of sound in the frequency domain can also be used to determine if the input sound can be considered an incoming "human speech". The amount of consecutive frames with voice presence can be used to detect voice activity.

7 illustrates an automatic speech recognition method with variable EOS timeout context dependent calling improved by speech activity detection and continuous characteristics of speech data according to an aspect of the present specification. The incoming speech data 600 first performs a noise analysis (S101) from a voice signal, and undergoes a noise analysis (S101) that undergoes a voice activity detection (S103) and a speech data continuity check (S105). It is determined whether speech data (speech data) is classified as speech data based on a preset category (S107), and only when it is classified as speech data, the speech data enters a local automatic speech recognition (ASR) process for the STT process ( S109). Then, the speech data is classified (S111), it is determined whether the data is non-fluency (S113), it is determined whether the data is valid (S115), and the data retrieves a cached match in the database (S117). ), if matching data is found, EOS is called (S121), that is, it is determined that the sentence ends, and appropriate data processing corresponding to the sentence response is performed. If there is non-wheezing data and the speech data does not have validity, or if the matching cannot be confirmed, the EOS time window is extended (S125) and the speech data of the next frame is received until the extended EOS time expires (S127). It returns to the step of determining whether classification is possible by performing noise analysis, activity information detection, and continuity check.

8 illustrates a method of filtering non-speech data prior to ASR processing according to an aspect of the present specification. Before sending the data to the ASR system, incorrect speech data can affect EOS calls. To avoid this scenario, the data read from the audio buffer S201 is a check of the voice activity detection S203, where the presence of voice is detected based on a fixed time frame. Since speech data comes in a continuous waveform format, examining voice activity within a frame (S205) gives you the insight needed to set expectations for the next frame configuration. If the voice presence is low at the end of the time frame, there is a possibility that the next frame contains no data. Since speech data tends to have quiet time frames between time frames with the speech data, checking the time frame pattern with speech activity can determine if the speech data is consistent noise or contains speech data.

9 shows dynamic word weight allocation. When the audio is received (S301), the audio is recognized as speech if it can be classified by the speech data classification (S303), and the EOS timer is started (S305), and equivalent text data is obtained (S307). If the text data includes influency (S309), the EOS timer is extended (S311), and the importance of the next word increases (S313). The weight of the importance word can be adjusted based on the aforementioned method, such as a locally stored database or EOS time frame size. If the influency is not detected, the text data is analyzed and matched in a database based on a word importance weight (S315), and it is determined whether it is predicted as an unnecessary word (S317). If it is not predicted as an unnecessary word, it is determined whether the EOS timer is ended before the additional audio (S319), and when it is terminated, an output corresponding to a response result and command data processing are performed (S321). If it is predicted as an unnecessary word or if the EOS timer has not expired, the speech data reception (S391) is returned.

Referring to FIG. 10A, it can be seen that the terminal device of the prior art transmits the end speech data according to the end of speech, and the speech response time is delayed according to the EOS detection of the ASR system.

On the other hand, referring to FIG. 10(B), the terminal device 100 according to an embodiment of the present invention enables forced EOS processing by the local recognition system even when the same speech end point arrives. Processing of the final recognition result by varying the time window is performed early, so that a speech response can be provided to the talker faster than in the prior art.

As EOS has a database stored locally to find the command system, the processed data is analyzed as speech data, and EOS is initially called as 811 (forced EOS) compared to the existing system 808 (EOS detection). Reduction in waiting time, that is, reduction in response time 809 becomes possible. As such, the present invention adjusts the EOS timeout timer based on the type of voice. When the ending speech data 807 of the prior art and the ending speech data 812 of the present invention are ended at the same time, it can be seen that (B) the speech response 810 of the present technology is faster than (A) the speech response 806 of the prior art.

Meanwhile, the above-described method according to various embodiments of the present invention may be implemented as a program and provided to each server or devices while being stored in various non-transitory computer readable media. Accordingly, the user terminal 100 can access the server or device and download the program.

The non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, and ROM.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

Claims (10)

As a program for executing a method of reducing waiting time in automatic speech recognition on a computer,
Receiving a voice as an audio input;
Performing noise analysis on the incoming sound data;
Determining whether the received sound data has characteristics of actual speech data;
Performing automatic speech recognition to convert the nearest equivalent data into text format;
Determining an output value, a speech category, and speech inflexibility by the automatic speech recognition;
Variably determining an end time of the speech time out window (EOS) based on a value of the ASR output, a speech category, and speech incompatibility;
Determining a speech timeout moment based on the value of the subsequent ASR output, speech category, and speech incompatibility;
Initial EOS call based on EOS timeout and intermediate ASR results prior to the final ASR result
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that in a computer.
In claim 1,
The noise analysis step,
Checking the continuity of speech data by observing the duration of the continuous level of the input sound to determine a noise level;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Observing the ratio of the silent and non-noisy portions of a sound sample to determine whether or not it has a characteristic of the speech data;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Maintaining a list of non-fluency words to predict the high weighted keywords that will follow;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
A list of known speech disproportions to adjust the EOS time frame window;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Determining the EOS based on the weighted keyword;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Observing the pattern of silence and data bursts to determine the likelihood that the input data can be considered speech data;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Calculating the silent to sound level ratio and determining a likelihood that the input data can be considered as speech data;
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Determining the requested query based on the shape of the word (in this case, whether the word is an object of the sentence);
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that it comprises a computer.
In claim 1,
Based on the caller's record, it provides a longer introduction to help first-time callers prepare to talk to artificial intelligence.
A program for executing a method for reducing a waiting time in automatic speech recognition, characterized in that in a computer.

Images