KR102076212B1 - Method of operating a speech recognition device - Google Patents

Abstract

According to an embodiment of the present invention, an operation method of a speech recognition device for more quickly and accurately detecting the end of speech comprises the steps of: receiving a voice signal of a speaker from a terminal device; obtaining speech data according to analysis of the voice signal; and automatically recognizing the speech data while a speech end timer window is continued to convert the speech data into text information and output the text information. The speech end timer window is adaptively varied according to at least one of value information of the speech data, classification information of the speech data, and disfluency information of the speech data detected according to the voice signal analysis.

Description

Method of operating a speech recognition device

The present invention relates to a method of operating a speech recognition device. More specifically, the present invention relates to a method of operating a speech recognition apparatus capable of shortening a response time and a waiting time through a speech end time detection process.

Voice signals are the most common, simple, easy and fast means or media used for communicating between users.

However, there is a problem that medium-to-long distance communication in a natural state is difficult, and automatic speech recognition (ASR) technology has been proposed for automated recognition processing of such speech signals.

This automatic speech recognition (ASR) technology detects speech sections from audio signals, acquires and processes contextual data for communication from speech sections, thereby enabling automated responses.

Recently, such ASR technology has been built as a separate server-based learning system, and has been provided as a service of a method of outputting natural language analysis and similar text data according to a user's voice input. For example, the learning system may be constructed by a probabilistic proximity text search diagram using a deep neural network (DNN) based language model, and various databases and training data are used for this purpose.

However, the current ASR system may be useful for accurately extracting context data, but there is an inefficient problem in detecting an end-of-speech point.

More specifically, current ASR systems are detecting their own end-of-speech (EOS) time points within the automated speech recognition process, which has been a factor in determining the EOS time point too early or too late.

For example, silence due to speech fluency may be followed, but the current ARS system judges this as the end of speech and outputs it as meaningless data, or conversely, it is unnecessary to determine the end of speech normally. There is a problem that increases the waiting time.

Due to these problems, meaningful text data may not be extracted normally or problems may occur that impair user convenience due to increased waiting time.

Korean Laid-Open Patent No. 10-2009-0054642 (Name of the invention: Speech recognition method and apparatus)

The present invention has been made to solve the above-described problems, and independent of the ASR system, according to the characteristics of the speech data according to the analysis of the speech signal, by adaptively determining the end point of speech (EOS) more It is an object of the present invention to provide a method of operating a speech recognition device for detecting a quick and accurate speech end.

According to an aspect of the present invention, there is provided a method of operating a speech recognition device, the method comprising: receiving a voice signal of a talker from a terminal device; Acquiring speech data according to the analysis of the speech signal; And automatically recognizing speech data while the speech end timer window continues, and converting the speech data into text information, wherein the speech end timer window includes value information of the speech data detected according to the speech signal analysis. And adaptively vary according to at least one of classification information of the speech data and disfluency information of the speech data.

On the other hand, speech recognition apparatus according to an embodiment of the present invention for solving the above problems, the voice receiving unit for receiving a voice signal of the talker from the terminal device; An audio analysis module for obtaining speech data according to the analysis of the speech signal; An automatic speech recognition processing unit for automatically recognizing speech data while the speech end timer window continues, and converting the speech data into text information; And adaptively vary the speech end timer window according to at least one of value information of the speech data detected according to the speech signal analysis, classification information of the speech data, and disfluency information of the speech data. The EOS matching unit determines the speech end point.

On the other hand, the method according to an embodiment of the present invention for solving the above problems can be implemented as a program for executing the method on a computer and a computer-readable recording medium on which the program is recorded.

According to an exemplary embodiment of the present invention, the speech end timer window may be adaptively and variably determined according to characteristics of speech data according to analysis of a voice signal, so that a more accurate and accurate speech end point may be detected. .

In addition, the characteristic of the speech data is determined according to at least one of the value information of the speech data detected according to the speech signal analysis, the classification information of the speech data, and the disfluency information of the speech data, thereby providing a high level. It is possible to reduce the waiting time of the voice recognition response while maintaining the accuracy of the user, thereby greatly improving the user convenience.

1 is a block diagram schematically illustrating an entire system according to an exemplary embodiment of the present invention.
2 and 6 are diagrams for explaining speech recognition and waiting time reduction effects according to sections of voice data according to an exemplary embodiment of the present invention.
7 to 9 are flowcharts for describing a speech recognition method according to an embodiment of the present invention in more detail for each process.
FIG. 10 is a ladder diagram illustrating a difference in waiting time between a speech recognition process and an existing process according to an exemplary embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art, although not explicitly described or illustrated herein, may embody the principles of the present invention and invent various devices that fall within the spirit and scope of the present invention. In addition, all conditional terms and embodiments listed herein are in principle clearly intended for the purpose of understanding the concept of the invention and are not to be limited to the specifically listed embodiments and states. Should be.

In addition, it is to be understood that all detailed descriptions, including the principles, aspects, and embodiments of the present invention, as well as listing specific embodiments, are intended to include structural and functional equivalents of these matters. In addition, these equivalents should be understood to include not only equivalents now known, but also equivalents to be developed in the future, that is, all devices invented to perform the same function regardless of structure.

Thus, for example, the block diagrams herein should be understood to represent a conceptual view of example circuitry embodying the principles of the invention. Similarly, all flowcharts, state transitions, pseudocodes, etc., are understood to represent various processes performed by the computer or processor, whether the computer may be substantially represented on a readable medium and whether the computer or processor is clearly shown. Should be.

In addition, the explicit use of terms presented in terms of processor, control, or similar concept should not be interpreted exclusively as a citation of hardware capable of executing software, and without limitation, ROM for storing digital signal processor (DSP) hardware, software. (ROM), RAM, and non-volatile memory are to be understood to implicitly include. Other hardware for the governor may also be included.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically showing an entire system according to an embodiment of the present invention.

Referring to FIG. 1, a system according to an embodiment of the present disclosure may include a terminal device 100, an audio analysis module 120, a local index module 130, a matching module 140, and an automatic speech recognition (ASR) system ( 200).

As illustrated in FIG. 1, the terminal device 100 may be a user terminal including one or more voice receivers for receiving a spoken voice from a talker and outputting a voice signal, and outputting a voice signal corresponding to the input spoken voice. Examples of the device to be delivered to the audio analysis module 120 include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, and the like. It may be a variety of devices capable of user input and information display.

In addition, the speech recognition apparatus according to an exemplary embodiment of the present invention includes an audio analysis module 120, a local index module 130, a local database 110, and a matching module 140, which are locally connected to the terminal device 100. It can be configured as a local recognition system.

Here, the local recognition system may be provided separately from the automatic speech recognition system 200 provided at a remote location, and obtains audio or text information that can be matched with a voice signal through the local database 110 to the terminal device 100. Or wired or wirelessly connected to the automatic speech recognition system 200 through a network to provide speech speech data extracted from a speech signal to the automatic speech recognition system 200, and recognized from the automatic speech recognition system 200. The result data may be provided to the terminal device 100.

In particular, by the local recognition system according to an exemplary embodiment of the present disclosure, matchable audio and text information may be obtained according to local audio analysis and conversion, and may be provided to the terminal device 100 first, and further, speech termination (EOS). Forced EOS processing for processing may be performed, thereby delaying response processing of the terminal device 100 and improving user convenience.

On the other hand, speech speech data that does not match in the local recognition system may be delivered to the automatic speech recognition system 200, and the automatic speech recognition system 200 performs a normal speech recognition process based on learning data on the speech data. In response to the recognition result, speech recognition data may be transmitted to the terminal device 100. Speech recognition data may include, for example, text based contextual data.

The terminal device 100 receives the matched audio / text information or speech recognition data of the automatic speech recognition system 200 from the matching module 140 or the automatic speech recognition system 200, and responds to the received information. Can be performed.

For example, the terminal device 100 identifies the instruction information corresponding to the user's voice input from the matched audio / text information or speech recognition data, processes the user interface output corresponding to the instruction information, or processes other users. Messages can be sent to the phone, a telephone connection can be processed, or a response voice can be output.

For example, the terminal device 100 may determine a query requested from the talker based on the word information of the speech data and provide a corresponding process.

For example, the terminal device 100 may provide the caller with initial guidance information for the speech data based conversation service when there is no existing usage history based on the call log information recorded in the local database 110. It may be.

As described above, the local recognition system connected to the terminal device 100 may be connected to the automatic speech recognition system 200 by wire or wirelessly through a network. The modules of each terminal device 100 and the local recognition system and the automatic speech recognition system 200 for communication between networks are connected to the Internet network, LAN, WAN, PSTN (Public Switched Telephone Network) and Public Switched Data Network (PSDN). It can transmit and receive data through cable TV network, WIFI, mobile communication network and other wireless communication networks. In addition, each terminal device 100 and automatic speech recognition system 200 may include a respective communication module for communicating in a protocol corresponding to each communication network.

In addition, the local database 110, the audio analysis module 120, the local index module 130, and the matching module 140 of the local recognition system according to an exemplary embodiment of the present disclosure are internal to the terminal device 100. The speech recognition apparatus according to an embodiment of the present invention may be configured as an external module that is provided as an internal module, or may be implemented as an external module that is locally connected to the terminal device 100 by an external interface. In addition, the system may be configured as an external extension-connected local recognition system or an entire system including the automatic speech recognition system 200, but is not limited thereto.

More specifically, the audio analysis module 120 of the speech recognition apparatus according to an embodiment of the present invention, when a voice signal is output from the spoken voice received from the voice receiver of the terminal device 100, according to the analysis of the voice signal Speech data can be obtained.

Here, the speech data may include speech data classified as speech speech by a local classification process of the speech signal.

The audio analysis module 120 may first request a local conversion of voice data to the index module 130, and the index module 130 may use the local database 110 to obtain audio or text information that matches the voice data. Index and pass the index result to the matching module 140.

When the matching audio / text information exists according to the index result of the voice data, the matching module 140 may provide the matched audio / text to the terminal device 100 as a response result corresponding to the voice signal. have.

On the other hand, if there is no matching audio / text information as a result of the index, the matching module 140 may transmit the speech data classified as the speech voice to the automatic speech recognition system 200.

Accordingly, the automatic speech recognition system 200 may provide the terminal device 100 with automatic speech recognition data that has been subjected to normal automated speech recognition processing.

Meanwhile, the speech data obtained by analyzing the voice signal in the audio analysis module 120 may include speech analysis information. Here, the speech analysis information may include at least one of value information of the speech data detected according to the speech signal analysis, category information of the speech data, and non-fluency information of the speech data. Each analysis information will be described later in detail.

Accordingly, the audio analysis module 120 may perform the speech processing according to the speech speech classification and the analysis process of the speech signal for generating the speech analysis information by diversifying, in particular the speech according to the embodiment of the present invention. The data may be used in the determination process of the speech end (EOS) timer window in the matching module 140.

Here, the EOS timer window according to an embodiment of the present invention may represent time interval information set and managed by the matching module 140 to detect the local speech end independently of the remote automatic speech recognition system 200. have.

Accordingly, the matching module 140 automatically recognizes the speech data while the speech end timer window continues through the local index module 130 or the automatic speech recognition system 200 to process the response to the terminal device 100. When the speech end timer window expires, however, the forced EOS information is generated and provided to the terminal device 100, thereby enabling local speech termination independent of the automatic speech recognition system 200.

In particular, the matching module 140 may determine the speech end timer window according to at least one of validity information of the speech data detected according to the speech signal analysis, classification information of the speech data, and disfluency of the speech data. By adaptively varying to determine the speech end point, it can bring about a quick and effective response delay shortening effect compared to the existing system using only the automatic speech recognition system 200.

In addition, the local database 110 may store a probability model based on previous query and response information of the terminal device 100, and the index module 130 may respond to the text information extracted from the audio analysis module 120. The index may be delivered to the matching module 140.

Here, such a local database 110 may accumulate input voice data and response command data and update the probability model to further increase accuracy.

In addition, the matching module 140 may provide the terminal apparatus 100 with response information when the matching is confirmed, so that the voice queried by the user may not be delivered to the remote automatic speech recognition system 200. The response result may be provided, and the EOS according to the response result matching may be determined early.

Therefore, according to an embodiment of the present invention, the command processing is much faster than the indexing process using only the remote automatic speech recognition system 200.

2 to 6 are diagrams for explaining a speech recognition and waiting time reduction process according to an interval of voice data according to an exemplary embodiment of the present invention.

For example, since a language model is not easily processed only by hardware specifications of a typical user terminal, an automatic speech recognition system provided as a remote server is used. Such a general automatic speech recognition system provides an intermediate result corresponding to speech voice data according to a request of a terminal, but the speech recognition system determines a speech end point (EOS) call time and provides final sentence information according to an EOS call.

In general, an automatic speech recognition system is set to call EOS when a predetermined silence time frame is identified, and the automatic speech recognition system can interpret voice data until text is called as text data. Delay time due to the processing time of the automatic speech recognition system for EOS determination is inevitably generated, and user inconvenience may be increased. There is also the possibility of a false EOS being called as the voice may be suspended for any reason while the user is speaking the command.

For example, referring to FIG. 2, the overall waiting time according to the user experience of the conventional automatic speech recognition system may be represented by an equation based on a log linear model.

Here, (μ) may be a response variable which is a network delay 13, (α) may be an independent variable which is an automatic speech recognition (ASR) processing time 12, and (β) may be an ASR process for EOS detection. It may be a blocking variable that is a time 11, and (e) may represent an error rate that will result in a result that does not match the user's prediction. And, (X) may represent the utterance length 14 determined by the EOS call time. For example, if an utterance is repeated due to an unexpected result, the overall X may increase.

And, in such a conventional automatic speech recognition system, EOS time-out may be started upon detecting silence of the audio input. For example, if audio is detected before the EOS time limit (100 seconds, Fig. 2 section 11), the time limit is reset. If audio is not detected within the EOS time limit section 11, EOS can be called. Thereafter, a time point 15 at which the result of processing the audio by the automatic speech recognition system is provided to the terminal is formed.

3 illustrates a method of verifying voice input data in a conventional automatic speech recognition system, which may include burst and silence patterns 21, 28, 22, and 29, and includes a plurality of time frames. It can be configured as continuous data according to. In an automatic speech recognition system, a speech end point (EOS) may be called at a time point 27 after a plurality of time frames ends for complete sentence construction.

However, the matching module 140 according to an embodiment of the present invention can quickly call the initial EOS in the middle silent section 31 according to the analysis process of the audio analysis module 120, and thus the user experience waiting time accordingly. This can be shortened. In addition, the matching module 140 may compensate for the interpretation error due to initial EOS detection by providing the local database 110 index result of the index module 130. To this end, the local recognition system may be provided as a module residing inside the terminal device 100, and the local database 110 may be a cache database or a local TTS / STT conversion engine that stores driving history information of the terminal device 100. It can include a database.

As such, when the initial EOS is quickly called by the local process, the waiting time felt by the user due to the delay reduction can be greatly reduced, and the ideal effect based on the above Equation 1 model can be derived.

However, since initial EOS detection may cause a potential misinterpretation, it may take additional waiting time due to material, and therefore, the local database 110 according to an embodiment of the present invention may pre-index the desired result. In addition, the previous query information QUERY of the terminal device 100 may be recorded, and response information corresponding to a frequently asked query may be cached and stored in advance in the local database 110.

Accordingly, the local database 110 may construct a probabilistic model to reduce the probability of a false response, and provide the matching module 140 with indexed response information through the index module 130.

On the other hand, the audio analysis module 120 may process filtering of non-context data through noise analysis and voice activity detection to compensate for potential errors that may misinterpret noise and cause unwanted initial EOS. Accordingly, voice data not classified as speech data may be blocked before being transmitted to the index module 130 or the automatic speech recognition system 200. By this local filtering process, unnecessary data transmission to the automatic speech recognition system 200 can be reduced and resource usage can be optimized.

For this processing, the audio analysis module 120 may first process noise level monitoring of the input voice data to initially identify the noise section and the voice section, detect a burst pattern of the voice section, or continuously By detecting the noise level data, only speech speech data can be classified and extracted from the speech signal.

In addition, the audio analysis module 120 may classify and extract only speech data that may be regarded as human speech by setting a range of voice variables in the frequency domain.

4 and 5 illustrate voice data waveforms when a non-fluent period is included.

In the speech recognition system in accordance with an embodiment of the present invention, the matching module 140 confirms the existence of a non-fluency interval so that an additional delay that returns meaningless data or repeats a voice command does not occur. If the non-fluentness is confirmed, a process of increasing the importance weight of the next word to be recognized may be performed.

In particular, in the case of Korean, it follows the format of "subject-object-verb", and the subject is omitted entirely. Therefore, the non-fluent information can be used as an ideal information for detecting the initial EOS while providing proper text and context recognition. have. In the three months of data collection and test processing based on the system according to an embodiment of the present invention, it was confirmed that more than 90% was in the form of "subject-object-verb" or "object-verb".

More specifically, referring to FIG. 4, in the case of a typical automatic speech recognition system, when a non-fluent interval 400, 402 such as stuttering repeating 'a', 'he', and silence is generated, an interpretation that is not relevant to the situation is insufficient. Will be calculated. In addition, the wrong EOS may be called in the silent sections 401 and 403 according to a preset size of the time frame. In addition, there are cases where the automatic speech recognition system does not allow further input while wrong input or EOS is processed and the final result is returned. Therefore, the user repeats the desired speech, and there is a problem that the overall waiting time is increased.

In order to solve this problem, the matching module 140 according to an embodiment of the present invention may locally set an EOS time window for EOS detection and change a window size corresponding thereto, and in particular, an EOS time window when a non-fluent interval is detected. While extending the can increase the recognition weight of the next word. For example, the matching module 140 according to an embodiment of the present invention may extend the EOS time windows in the silent sections 401 and 403 due to non-fluent speech in FIG. Can be deactivated.

In addition, the matching module 140 may reset the end time of the EOS time window to the time point at which the voice data of the continuous frame ends, and thus the EOS call extension time point 408 may be reset.

In addition, FIG. 5 illustrates a reduction in time delay due to the dynamic size change of the EOS time window. When the speech module includes burst data according to non-fluency, the matching module 140 dynamically adjusts the EOS time window according to the speech length. The change can be processed.

Referring to FIG. 5, in the case of general EOS detection, the fixed length time frame section 410 must pass after the end of the utterance, but in the case of the matching module 140 according to an exemplary embodiment of the present invention, voice data including non-fluency With respect to, by processing the dynamic reduction change 411 of the EOS time window corresponding to the utterance length, the user can receive the effect of reducing the waiting time 412 by the difference.

Meanwhile, FIG. 6 is a diagram for explaining a reduction in processing delay time of speech data including non-fluence according to an embodiment of the present invention, compared to the related art.

As shown in FIG. 6 (A), when the voice sections 500 and 501 and the silence section 503 including non-fluentness are generated in the voice of the talker in the prior art, EOS by the silence section in the speech recognition system is generated. Interval 515 is detected, and after recognition processing time 505 passes, an incorrect context recognition result 506 is provided.

As a result, the user does not get a response to the actual question voice 502 and needs to speak 504 again. Accordingly, additional EOS interval 516 detection and recognition time 514 are required, resulting in additional delay time ( After 517 is taken, the response result is provided.

On the other hand, referring to FIG. 7B by a process according to an embodiment of the present invention, even if a voice input including non-fluent intervals 508 and 509 is received from the talker, the matching module 140 may generate an EOS time window. By setting the local time, the time window ending time according to the non-fluency detection is extended to wait until the actual query voice 510 is received.

In addition, the recognition result may be provided to the user after early EOS according to the EOS window expiration 511 is detected and only the processing time 512 is appropriately taken.

As such, it can be seen that the waiting time difference 513 can be clearly shortened compared to the prior art, which can be a major factor in improving user convenience.

7 to 9 are flowcharts for describing a speech recognition method according to an embodiment of the present invention in more detail for each process.

7 is a flowchart illustrating an entire process of local speech recognition according to an exemplary embodiment of the present invention.

Referring to FIG. 7, first, the audio analysis module 120 processes noise analysis from a voice signal received from the terminal device 100 (S101).

The audio analysis module 120 detects voice activity information according to the noise analysis (S103).

Thereafter, the audio analysis module 120 checks speech data continuity (Succesivity) (S105).

Accordingly, the audio analysis module 120 determines whether speech data can be classified according to the comprehensive analysis result (S107), and if possible, the audio analysis module 120 transmits the speech signal to the index module 130 to the local automatic speech recognition process. Enter (S109).

Here, the matching module 140 may set a local EOS time window for the automatic speech recognition process.

In addition, when it is determined in step S107 that the speech data cannot be classified according to the noise detection, the audio analysis module 120 may perform noise analysis on the voice signal of the next time frame.

In the local automatic speech recognition process, the index module 130 performs speech data pre-category processing (S111). The pre-classification may include a classification process of matching the speech data with preset classification values.

In addition, the matching module 140 determines whether there is a disfluency section in the speech data from the analysis result of the audio analysis module 120 (S113), and determines whether there is a content validity of the speech data (S113). S115) If non-fluency does not exist and validity exists, data analysis based matching processing of the local database 110 is performed through the index module 130 (S117).

Here, the matching process may be based on history information cached in the local database 110 or may be processed by the TTS and STT engines mounted in the local database 110, and the matching module 140 may be configured as the index module 130. By checking the response result, it may be checked whether or not matching is performed (S119).

On the other hand, if there is a non-fluent interval in the speech data in step S113, the content validity of the speech data does not exist in step S115, or whether matching is not confirmed in step S119, the matching module 140 advances the EOS time window in advance. It extends for a set time period (S125).

Thereafter, the matching module 140 checks whether the time window expires (S127), calls EOS at the time when it expires, determines the sentence termination according to the matching check (S121), and when there is output data according to the sentence termination, Provided to the terminal device 100, the appropriate data processing corresponding to the sentence response is performed (S123).

8 is a flowchart illustrating a speech data classification process according to an exemplary embodiment of the present invention in more detail.

Referring to FIG. 8, the audio analysis module 120 first reads data from a voice buffer (S201).

The audio analysis module 120 detects voice activity information for each data frame (S203), determines activity information for each frame (S205), and determines continuous activity accordingly (S207). By judging the existence of the continuous feature corresponding to the speech pattern from (S209), it is possible to determine the possibility of classifying into speech data (S211).

According to the classification possibility determination, the audio analysis module 120 classifies the speech data into speech data (S213), and if it is not classifiable, ignores the data (S215).

According to the processing of the audio analysis module 120 as described above, it is possible to minimize the impact of EOS calls due to erroneous audio data before entering the local automatic speech recognition process for voice data or passing it to the automatic speech recognition system 200. have. To this end, the audio analysis module 120 may check the voice activity information of the data read from the voice buffer, detect voice activity based on a fixed time frame, and detect continuous activity for setting expectations of the next frame configuration. It can be determined.

For example, the audio analysis module 120 may predict the presence of voice in the next frame according to the detection of voice activity in the current frame according to the continuous waveform. Therefore, according to the time frame pattern analysis of the voice activity corresponding thereto, it is possible to detect whether the voice data includes noise or speech data.

More specifically, the audio analysis module 120 detects the silence pattern and the burst speech data corresponding to the speech prediction data from the speech signal, and analyzes the speech prediction data according to the silent-to-sound level ratio calculation of the speech prediction data. It can be classified into the speech data.

In addition, the audio analysis module 120 determines a noise level according to continuous level continuous detection of the speech signal, and verifies the content validity of the speech data according to the ratio of the non-silence section and the non-noise section to the speech signal sample. ), The matching module 140 may determine whether to perform data analysis based matching according to the content validity analysis result.

9 is a flowchart illustrating a speech recognition process based on non-fluentness according to an embodiment of the present invention.

Referring to FIG. 9, when voice data is received from the audio analysis module 120 (S301), and speech data is classified (S303), the matching module 140 sets a speech end (EOS) timer window. The EOS timer is started (S305).

The matching module 140 obtains text information corresponding to the speech data through the cache data of the local database 110 or the STT / TTS engine (S307).

Here, the matching module 140 determines the existence of disfluency of speech data according to the analysis of the audio analysis module 120 (S309), and if present, extends an EOS timer (S311), The corresponding word importance weight is increased (S313).

On the other hand, when the non-fluency is not confirmed, the matching module 140 performs a response matching analysis based on the importance weight of the text information through the index module 130 (S315), and determines whether unnecessary words are predicted (S315). In step S317, if it is not an unnecessary word, it is determined whether the EOS timer expires before the additional audio (S319), and an output corresponding to the response result is returned, and the corresponding command data processing is processed in the terminal device 100 ( S321).

If unnecessary words are predicted in step S317 or the EOS timer does not expire in step S319, the audio analysis module 120 receives and processes the next voice data again.

In addition, the matching module 140 may store and manage the non-fluent word list and the prediction keyword weight information for obtaining the non-fluency information in the local database 110.

Accordingly, the matching module 140 sets the speech end timer window variable value corresponding to the non-fluent information based on previously stored information, and the degree of variation according to the non-fluent information is based on the prediction keyword weight information. Can be set to be determined.

FIG. 10 is a ladder diagram illustrating a difference in waiting time between a speech recognition process and an existing process according to an exemplary embodiment of the present invention.

Referring to FIG. 10A, the terminal device according to the related art may confirm that the end speech data according to the end of the speech is transmitted, and the speech response time is delayed according to the EOS detection of the ASR system.

On the contrary, referring to FIG. 10B, the terminal device 100 according to an embodiment of the present invention enables the forced EOS processing by the local recognition system even when the same speech end point arrives. Processing of the final recognition result by the time window variable is performed early so that the speech response can be provided to the talker faster than in the prior art.

Meanwhile, the aforementioned method according to various embodiments of the present disclosure may be implemented as a program and provided to each server or devices in a state of being stored in various non-transitory computer readable mediums. Accordingly, the user terminal 100 may access the server or the device and download the program.

A non-transitory readable medium refers to a medium that stores data semi-permanently and is read by a device, not a medium that stores data for a short time such as a register, a cache, or a memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

Claims (12)

A method of operating a speech recognition device provided in a terminal device connected to a remote automatic speech recognition system through a network,
Receiving a talker's voice signal;
Acquiring speech data according to the analysis of the speech signal; And
And automatically recognizing speech data during the duration of the speech end timer window by a local index module or the remote automatic speech recognition system to provide response information corresponding to the speech data.
The speech end timer window may be configured according to at least one of validity information of the speech data detected according to the voice signal analysis, classification information of the speech data, and disfluency information of the speech data. Adaptively variable,
The speech end timer window is varied by a matching module driven by a separate local process inside the terminal device that is distinct from the remote automatic speech recognition system,
The matching module classifies speech data for the duration of the speech end timer window from the voice signal, outputs response information matching the speech data indexed from a local database via the local index module, and does not match. A module for processing the response information matched by the automatic recognition of the speech data transmitted to the automatic speech recognition system to be received and outputted to the terminal device;
The matching module determines the forced speech recognition end of the local automatic speech recognition process when the speech end timer expires, or transmits the forced speech end signal to the remote automatic voice system so that the output processing of the response information is performed early. Make sure,
The matching module, in the local automatic speech recognition process using the local index module, when the non-fluent interval is identified from the classified speech data, expands the speech end timer window and increases the importance weight of the next word to be recognized. Characterized in that,
Providing the response information,
Determining a query requested from a talker based on the word information of the speech data;
Method of operation of speech recognition device. delete delete The method of claim 1,
Acquiring the speech data,
Determining a noise level for detecting continuous level duration from the speech signal;
Method of operation of speech recognition device. The method of claim 1,
Determining validity information of the speech data according to a ratio of a non-silence section and a non-noise section to a sample of the speech signal;
Method of operation of speech recognition device. The method of claim 1,
And storing and managing the non-fluent word list and the predicted keyword weight information for obtaining the non-fluency information.
Method of operation of speech recognition device. The method of claim 6,
Presetting the speech end timer window variable value corresponding to the non-fluent information to the local database;
Method of operation of speech recognition device. The method of claim 7, wherein
The non-fluent information is determined based on the prediction keyword weight information.
Method of operation of speech recognition device. The method of claim 1,
Acquiring the speech data,
Detecting a silence pattern and burst speech data corresponding to speech prediction data from the speech signal;
Method of operation of speech recognition device. The method of claim 9,
Acquiring the speech data,
Classifying the speech prediction data into the speech data according to a silence-to-sound level ratio calculation of the speech prediction data.
Method of operation of speech recognition device. delete The method of claim 1,
Providing initial guide information for the speech data based conversation service to the caller based on call log information;
Method of operation of speech recognition device.

Images