KR100576803B1 - Speech Recognition Method and Device by Integrating Audio, Visual and Contextual Features Based on Neural Networks - Google Patents

Abstract

The present invention efficiently converges voice and video information based on neural network for robust voice recognition in noisy environment, and integrates voice, video and context using context information and post-processing method which are command usage patterns in mobile terminal. The present invention relates to a neural network-based speech recognition apparatus and method capable of further improving speech recognition rate.

Integrated voice recognition method of the present invention, the feature extraction step of extracting a feature vector from the input voice and video signal; A dual mode neural network recognition step of recognizing user voice by fusing voice and image information based on neural networks; A context information recognizing step of recognizing a user command pattern in the mobile terminal; And a post-processing step of outputting a final recognition result by integrating the dual mode neural network recognition result and the context information recognition result.

Speech Recognition, Dual Mode Recognition, Neural Network Recognizer, BMNN, Backpropagation Algorithm, Context Information Recognition

Description

Speech Recognition Method and Device by Integrating Audio, Visual and Contextual Features Based on Neural Networks}

1 is a view showing the configuration and processing of the dual mode speech recognition device according to the present invention.

2 is a diagram illustrating a structure of a neural network based integrated speech recognizer according to the present invention.

3 is a diagram illustrating a dual mode speech recognizer incorporating audio and video information according to the present invention.

4 illustrates a contextual information recognizer in accordance with the present invention.

<Description of Symbols for Main Parts of Drawings>

101: feature extractor 102: dual mode neural network recognizer

103: context information recognizer 104: post-processing unit

The present invention relates to a dual mode speech recognition apparatus and method based on a neural network, and more particularly, by efficiently combining voice and image information of a dual mode speech recognizer using a neural network and integrating user command usage pattern information in a mobile terminal. An apparatus and method for integrated speech recognition of speech, video, and context based on neural networks to provide improved speech recognition rates in a noisy environment and to clearly understand user intentions.

In recent years, as the society becomes more and more multimedia, multi-modal forms using facial expressions, directions, lip shapes, gaze tracking, hand gestures, and voices in order to realize human and machine interfaces more easily and clearly. Cognitive research is actively underway.

In particular, such research has been actively studied as a dual mode speech recognition method, a speech recognition method that is robust to a noisy environment with the recent development of mobile terminal technology. The dual mode speech recognition method improves the speech recognition rate by simultaneously considering the video information that can supplement the voice information sensitive to the noise environment. For example, when talking in a noisy environment such as a factory, people may think of a meaning using not only the voice of each other but also image information such as the shape of the mouth or the gesture.

The performance of such a dual mode speech recognition method depends on how efficiently the heterogeneous voice and image information are fused. Conventional fusion methods are divided into feature fusion and decision fusion. Feature fusion refers to a method of fusing information before recognition, and decision fusion refers to a method of performing final recognition by fusing the recognized result.

Feature fusion must solve the synchronization problem of speech and video information for efficient fusion, while decision fusion must satisfy the assumption that speech and video information must be independent of each other and lack association. Therefore, crystal fusion is easier to apply than feature fusion, but its performance is inferior, and feature fusion is good, but it is difficult to apply it because the synchronization of input information should be considered.

As such a fusion method, HMM (Hidden Markov Model) and neural networks are typically used. The fusion method using HMM is difficult to determine the number of states and Gaussian mixture, which are learning variables that are sensitive to the result, and the commonly used Continuous Density Hidden Markov Model (CDMM) is an input feature. Are difficult to apply because of the constraints that must satisfy stochastic independence conditions.

On the other hand, a fusion method based on neural networks can efficiently fuse heterogeneous information. However, conventionally proposed methods using neural networks have an inefficient problem in that the computational amount is increased as the learning method and the model are complicated or the size of the model is large.

In particular, to realize speech recognition in a mobile terminal, not only a robust speech recognition method is required in a noisy environment, but also a method for efficiently generating a recognizer.

Accordingly, the present invention is to solve the above-mentioned problems and necessity, the object of the present invention is to efficiently converge voice and image information using neural network for robust voice recognition in a noisy environment, The present invention provides an integrated speech recognition apparatus and method for speech, video, and context based on neural networks that can further improve speech recognition rate by applying post-processing techniques.

An integrated speech recognition apparatus for speech, video, and context based on a neural network for achieving the object of the present invention includes: a feature extractor for extracting features of an input speech and video signal; A dual mode neural network recognizer for recognizing a user's voice by fusing feature information of the voice and video signal based on a neural network; A context information recognizer for recognizing a command pattern of a user; And a post-processing unit for integrating the output value of the dual mode neural network recognizer and the output value of the context information recognizer and outputting a final recognition result.

In addition, the integrated speech recognition method of speech, video, and context based on a neural network for achieving the object of the present invention, the feature extraction step of extracting a feature vector from the input voice and video signal; A dual mode neural network recognizing step of recognizing a user voice by fusing the feature extracted voice and image information based on a neural network; A context information recognizing step of recognizing a user command pattern in the mobile terminal; And a post-processing step of integrating an output value of the dual mode neural network recognition and an output value of the context information recognition to output a final recognition result.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing the configuration and processing of the neural network-based dual mode speech recognition device of the present invention.

Referring to FIG. 1, the dual mode speech recognition apparatus of the present invention includes a feature extractor 101, a bimodal neural network (BMNN) recognizer 102, a context information recognizer 103, and a post processor 104. Is made of.

The feature extractor 101 stores an image captured by the camera in a video buffer with time information and system ticks, and segments audio signals simultaneously input through a microphone into isolated words through an endpoint detection process. do. At this time, a time tick representing the same time as the endpoint detection time is calculated and images input at the same time are received from the image buffer. Then, feature vectors are extracted from the synchronized video and audio signal using a feature extraction method.

The feature vectors of the extracted video and audio signals are used as inputs of the dual mode neural network recognizer 102.

On the other hand, Figure 2 shows the structure of the neural network-based voice, video, and context information integrated voice recognizer according to the present invention. 3 is a diagram illustrating the structure of the dual mode neural network recognizer 102 in which voice and image information are fused, and FIG. 4 is a diagram illustrating the structure of the context information recognizer 103.

As shown in FIG. 3, the dual mode neural network recognizer 102 is composed of four layers of an input layer, a hidden layer, a fusion layer, and a coupling layer, and has an omnidirectional neural network structure. In addition, the learning algorithm of the present invention uses a backpropagation algorithm and learning and recognition are composed of isolated words.

In the input layer, the window concept is applied to perform the abstraction function of the input feature considering the implicit tense position of the input information, and the overlap zone structure having high performance in the isolated word recognition is used.

The fusion layer performs a fusion function using neural network weights obtained by appropriately adjusting voice and video features by neural network learning process to compensate for loss of speech information due to noise.

Here, when the connection structure of the recognizer, the number of frames and the number of nodes of each layer are examined, the nodes of the upper layer frame corresponding to the nodes of all the frames included in the window are completely connected, and the fusion layer is Since there is no window, it is completely connected to the output layer.

As described above, the dual mode neural network recognizer 102 uses a backpropagation learning algorithm as an omnidirectional neural network structure. The basic neural network, provides a nonlinear mapping function between the input (I) and the output (O), as shown in the following equation (1).

(1)

(One)

Here, W is the weight of the neural network, and A means the neural network structure.

The neural network learning process is to properly adjust the weight of the neural network to generate a target output value from the neural network when the input is presented.

The purpose of this backward propagation learning is to determine the weights using gradient descent to minimize the error E in a recursive manner with respect to the weights of neural networks.

(2)

(2)

는 kn번째 목표 출력 값,

은 입력패턴으로부터 신경망에 의해 계산된 실제 출력 값을 의미한다. From here,

Is the kn th target output value,

Means the actual output value calculated by the neural network from the input pattern.

In addition, the weight change amount at any point in the learning is expressed by Equation 3 below.

(3)

(3)

, 학습률(learning rate)을

라고 할 때, 이중모드 신경망 인식기(102)에서 각 계층의 가중치 변경에 대한 일반식은 다음의 수학식 4내지 수학식 6과 같다.Therefore, the number of learning

, Learning rate

In this case, the general equation for the weight change of each layer in the dual mode neural network recognizer 102 is as shown in Equation 4 to Equation 6 below.

Weight change between the output layer (k) and the fusion layer (c)

(4)

(4)

Weight change between the fusion layer (c) and the hidden layer (h)

(5)

(5)

Weight change between the hidden layer h and the input layer i

(6)

(6)

은

번째 입력층 프레임의

번째 노드 출력 값을 의미하고,

은

번째 은닉층 프레임의

번째 노드에 대한 오차를 의미한다. 또한,

은

번째 은닉층 프레임과 대응되는 융합층 프레임들의 집합을 의미한다. From here,

silver

Of the first input layer frame

Means the output value of the first node,

silver

Of the first hidden frame

The error for the first node. Also,

silver

Means a set of fused layer frames corresponding to the first hidden layer frame.

Meanwhile, the contextual information recognizer 103 also has a multi-layer perceptron structure capable of learning sequential information and uses a backpropagation learning algorithm.

Existing neural networks do not have sequential information, which is information about commands that are sequentially input, in the input node of the input layer, and thus cannot recognize the command pattern including the sequential information. Accordingly, the present invention can know the sequential information on the input command pattern by representing the input value in binary form. FIG. 4 illustrates the structure of the context information recognizer 103 and illustrates a case in which the isolated words to be recognized are four of "zero," "one," "two," and "three."

Here, the number of input nodes is set equal to the number of words to be recognized multiplied by the number of preceding words, and the number of output nodes is set equal to the number of words to be recognized. The number of preceding words refers to the number of words considered in order to predict the current word, and refers to the number of words most recently recognized among the previously recognized words. If the input value is entered as "Japanese-English: 0100 1000 0010", the highest value is output to the word node used after "Japanese-English" at the output node. In FIG. 4, since "three" has the highest value, in this case, the word to be used after "Japanese-English" is predicted as "three".

In this case, if the number of the preceding words is set too large, the output prediction value tends to be low because the occurrence frequency is low according to the pattern. On the contrary, if the setting is too small, the prediction value tends to be biased into the minimum pattern. You need to set the appropriate number of.

On the other hand, the post-processing unit 104 combines the results of the dual mode neural network recognizer 102 and the results of the context information recognizer 103 in a sequential combining method as shown in Table 1 below.

TABLE 1

): 이중모드 신경망 인식기에서 i번째 출력노드의 출력 값, Con(

): 문맥정보 인식기에서 i번째 출력노드의 출력 값,

: 임계값. IF (

＜ BMNN(

)) i = max _i (BMNN(

)) 번째 고립단어 인식, ELSE IF (

＜ Con(

)) i = max _i (Con(

)) 번째 고립단어 인식, ELSE IF ((BMNN(

)

) and (Con(

)

)) i = max _i (BMNN(

)ㆍCon(

)) 번째 고립단어 인식. BMNN (

): Output value of the i th output node in the dual mode neural network recognizer, Con (

): Output value of the i th output node in the context information recognizer,

: Threshold. IF (

<BMNN (

)) i = max _i (BMNN (

Recognition of the second isolated word, ELSE IF (

<Con (

)) i = max _i (Con (

Recognize the second isolated word, ELSE IF ((BMNN (

)

) and (Con (

)

)) i = max _i (BMNN (

) Con (

Recognize the second isolated word.

)보다 작을 경우 문맥정보 인식기(103)의 인식결과를 고려하는 방법이다. In the sequential coupling method, a threshold value set by a user is determined by a recognition result of the dual mode neural network recognizer 102.

If smaller than), the recognition result of the context information recognizer 103 is considered.

If the results of both recognizers are smaller than the threshold value, the difference between the recognition result of the dual mode neural network recognizer and the context information recognizer is small, that is, the final recognition result is that the two recognizers have similar recognition results. To multiply the recognition results of two recognizers, select the largest value. Here, the threshold set by the user is the minimum threshold value at which the user can trust the results of the recognizer.

The integrated voice recognition method of voice, video, and context based on the neural network of the present invention may be stored in a computer-readable recording medium. Such a recording medium includes all kinds of recording media in which programs and data are stored so that they can be read by a computer system. Examples of the recording medium include read only memory, random access memory, and compact disk. -Rom, DVD (Digital Video Disk) -Rom, magnetic tape, floppy disk, optical data storage device. In addition, these recording media can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

As described above, the integrated speech recognition apparatus and method for speech, video, and context based on the neural network according to the present invention efficiently improve the speech recognition rate by applying a post-processing technique using a user's command pattern and efficiently fused speech and video information. It can be improved.

In addition, the dual-mode neural network recognizer can reduce the size of the model by applying a window to perform abstraction function on input information, and as a result, can efficiently generate the model, and also has high performance in isolated word recognition. By applying the method and feature fusion method, the efficiency is improved. The post-processing method using the context information can greatly improve the speech recognition rate by using the user's behavior pattern when it is difficult to recognize only the voice and image information.

What has been described above is only one embodiment for implementing the integrated speech recognition apparatus and method of speech, video, and context based on the neural network according to the present invention, and the present invention is not limited to the above-described embodiments. Without departing from the gist of the invention as claimed in the claims, anyone of ordinary skill in the art will have the technical spirit of the present invention to the extent that various modifications can be made.

Claims (15)

A feature extractor which extracts features of an input audio and video signal; It consists of a multi-layer perceptron structure including an input layer, a hidden layer, a fusion layer, and a bonding layer, and has an omnidirectional neural network structure, and based on the neural network structure, the neural network weights between the layers are adjusted through a backpropagation learning algorithm for the input pattern. A target output value is generated, and an input function and an abstraction layer apply a window concept to perform an abstraction function of an input feature, and in the fusion layer, a window concept using the adjusted neural network weights for the features of the voice and video signals. A dual mode neural network recognizer that fuses to an implicit tense position and recognizes a user's voice as an isolated word, thereby compensating for loss of voice information due to noise; A context information recognizer having a multi-layer perceptron structure capable of learning sequential information using a backpropagation learning algorithm, and recognizing the sequential information by representing input information in binary form and recognizing the sequential information; And The recognition result value of the dual mode neural network recognizer is considered first and the recognition result value of the contextual information recognizer is sequentially considered only when the recognition result value of the dual mode neural network recognizer is smaller than a threshold value. If the result value is less than the threshold value, the post processing unit for multiplying the recognition result value of the two recognizers by selecting the difference between the recognition result values of the two recognizers as the final recognition result; Integrated speech recognition device for voice, video, and context. The method of claim 1, wherein the feature extraction unit, The input speech signal is divided into isolated words through an endpoint detection process to synchronize the audio signal and the video signal in isolated words, and extract feature vectors for each of the inputted audio signals. Integrated speech recognition device. delete According to claim 1, The dual mode neural network recognizer, In the input layer and the hidden layer, the speech recognition apparatus of the speech, video, and context based on the neural network, characterized in that to perform the abstraction function of the input feature by applying a window concept. delete delete delete delete A feature extraction step of extracting a feature vector from an input audio and video signal; It has an omnidirectional neural network structure of multi-layered perceptron including an input layer, a hidden layer, a fusion layer, and a coupling layer, and in the fusion layer, the characteristics of the voice and video signals are fused to a implicit tense position through a window concept based on a neural network. A dual mode neural network recognition step of compensating loss of speech information due to noise by recognizing speech as an isolated word; Recognizing user command patterns sequentially used through a multi-layer perceptron structure that can learn sequential information using a back-propagation learning algorithm, and expressing input information in binary form to recognize sequential information, thereby recognizing the user command pattern in a mobile terminal. Context information recognizing step of recognizing; And The recognition result value of the dual mode neural network recognition step is considered first and the recognition result value of the context information recognition step is sequentially considered only when the recognition result value is smaller than a threshold value, and both recognition result values are critical. If the value is smaller than the value, the two recognition result values are multiplied, and a post-processing step of selecting and outputting a small difference between the two recognition result values as the final recognition result; Neural network-based voice, image, and context integration Voice recognition method. The method of claim 9, wherein the feature extraction step, Integrated speech recognition based on neural network based on neural network, characterized by synchronizing the audio signal and the video signal in units of isolated words divided by the endpoint detection process, and extracting the feature vector for each voice and video signal Way. delete The method of claim 9, wherein the dual-mode neural network recognition step, An integrated speech recognition method of speech, video, and context based on a neural network, which applies an abstraction function of an input feature by applying a window concept to each layer. delete delete delete

Images