KR20140093459A - Method for automatic speech translation - Google Patents

Abstract

An automatic translation method of the present invention may comprise the steps of authenticating a facial image of a user through the comparison of multiple facial feature vectors and facial image templates which are related to the preset facial image after obtaining the facial image of the user; extracting voice features after separation of the voice segment from the input audio signal; converting the extracted voice features to text sentences by using an acoustic model constructed by speaker adaptation; converting the converted text sentence to target language sentences by using an automatic translation engine; and converting the converted target language sentence to target language voice by using a voice synthesis engine and outputting the target language voice.

Description

{METHOD FOR AUTOMATIC SPEECH TRANSLATION}

The present invention relates to an automatic interpretation method, and more particularly, to an automatic interpretation service in a portable terminal by applying a face image authentication and a speaker adaptation of a user and a voice interval checking method by a lip motion image tracking The present invention relates to an automatic interpretation method that is suitable for providing an automatic interpretation method.

As is well known, automatic interpretation is a technology implemented for communication between people using different languages, and realized by a combination of element technologies such as speech recognition, automatic translation, and speech synthesis.

Recently, such an automatic interpretation technology has been installed in portable terminals such as smart phones and smart devices, and is exposed to acoustic noise environment, so that there is a need to utilize image information in speech recognition processes that are susceptible to acoustic noise. This is because the image information is vulnerable to illumination and the amount of computation is large, so that it can be acquired and processed irrespective of acoustic noise, despite the disadvantage of real-time processing restriction.

Speaker adaptation is a technique that can obtain performance close to speaker dependent speech recognition by modifying a speech recognition model to fit a user's speech to a speaker independent speech recognition model targeting a plurality of speakers. Furthermore, the speech recognition model can be continuously improved by the new pronunciation by the online speaker adaptation technique.

Here, the speech interval setting specifies the beginning and end of the speech with respect to the acoustic signal, which determines the success or failure of the speech recognition. However, it is a reality that the accurate setting of the voice interval under the noisy environment remains a very difficult task.

Korean Patent Publication No. 2005-0015585 (published on February 21, 2005)

The present invention utilizes image information in an automatic interpretation system. An environment in which image information can be easily acquired by a front camera of a portable terminal (or a mobile terminal) such as a smart phone or a smart device is created, In view of the fact that it can be acquired and processed irrespective of acoustic noise, the present invention intends to utilize the image information in the voice recognition process of automatic interpretation which is vulnerable to acoustic noise.

First, the face image of the user is utilized for authentication by utilizing the function of the front camera provided in the portable terminal, and since the portable terminal can be used by an individual or a small number of family members, The use of face image can be useful.

Second, it is possible to realize the improved speech recognition performance by using the user face image in the speech recognition process. If the user is identified as the authentication result of the user face image, the speech recognition model based on the speaker adaptation of the identified speaker can be utilized. That is, the present invention can customize a speech recognition model for a specified small number of users and provide enhanced speech recognition performance for each speaker.

Third, the result of tracking the lip motion image is used for voice segment setting. In other words, since the portable terminal is mainly used in a mobile situation, it is difficult to avoid various acoustic noise inflows around, so that it is difficult to set an accurate voice interval under a noisy environment, so that a lip motion image that can be acquired and processed irrespective of acoustic noise .

Fourth, it is possible to replace the existing push-to-talk (PTT) that artificially activates and terminates the start and end of voice utterance.

Fifth, by introducing multiple feature vectors and multiple templates of facial images, facial images in various environments can be utilized in the automatic interpretation process.

The method includes the steps of: acquiring a user's face image, authenticating a user's face image through comparison with a multi-facial feature vector and a face image template associated with a predetermined face image, and extracting a voice section from the input voice signal A step of converting the extracted speech features into a text sentence using an acoustic model constructed by speaker adaptation, and a step of converting the converted text sentence into an target language sentence using an automatic translation engine And converting the target language sentence converted by the speech synthesis engine into a target language speech and outputting the translated target language speech.

According to the present invention, the following effects can be obtained by effectively utilizing the face image in the voice recognition process of the automatic interpretation, and the calculation amount according to the image information processing can be reduced, and the system can be effectively applied to various illumination environments.

First, a user can be identified through user face image authentication of the mobile terminal. To this end, a face image window is set on a user interface screen for automatic interpretation, and multiple feature vectors of the acquired face image and predetermined face image, To compare the results.

Second, improved speech recognition performance can be expected by the speaker adaptation of the identified user, and performance close to the speaker dependent speech recognition performance which can show the maximum speech recognition performance using a small amount of user data can be expected.

Third, the result of tracking the lip motion image can be used for setting the voice interval. That is, since the lips are inevitably moved in the process of voicing for automatic interpretation, the setting of the voice interval can be effectively realized by confirming whether or not the lips are moving during the processing of the voice signal.

Fourth, it can be used to replace push-to-talk (PTT) that artificially activates the start and end of voice utterance. Here, the PTT is an operation mode for preventing processing of a voice signal by acoustic noise and artificially setting a voice section.

1 is a block diagram of an automatic interpretation apparatus according to an embodiment of the present invention;
FIG. 2 is a detailed block diagram of the lip detection block shown in FIG. 1;
3 is a flowchart illustrating a main process of providing an automatic interpretation service through facial image authentication and speaker adaptation of a user according to an embodiment of the present invention and confirmation of a voice interval by tracking a lip motion image,
Figure 4 is a flow chart illustrating the main process of detecting lips movement according to the present invention,
FIG. 5 is a conceptual diagram illustrating a concept of providing an automatic interpretation service based on a face image, a lip motion image, and an acoustic signal according to the present invention.

First, the advantages and features of the present invention, and how to accomplish them, will be clarified with reference to the embodiments to be described in detail with reference to the accompanying drawings. While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. It is to be understood that the following terms are defined in consideration of the functions of the present invention, and may be changed according to intentions or customs of a user, an operator, and the like. Therefore, the definition should be based on the technical idea described throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of an automatic interpretation apparatus according to an embodiment of the present invention, which includes an image acquisition block 102, an outline and element extraction block 104, an image authentication block 106, a speaker adaptation DB 108, A speech recognition block 116, a speech recognition and sentence generation block 118, a target language sentence generation block 120, and a target language Voice generation block 122, and the like.

Here, the automatic interpretation device of the present invention may be installed as an automatic interpreting application and erasably installed (loaded) on a portable terminal (or mobile terminal). Examples of the portable terminal include a mobile phone, a smart phone, Pad, note pad, tablet PC, and the like.

1, the image acquisition block 102 acquires and recognizes a face image of a user (a speaker to receive automatic interpretation service) from a camera (not shown) (for example, a front camera of a portable terminal) And displaying (displaying) the recognized face image frame on a face image window (for example, reference numeral 502 in FIG. 5) on the user interface screen (display panel of the portable terminal).

At this time, the face image of the user displayed on the face image window of the user interface screen can induce adjustment of the face position so as to be adjusted to the face approach inducing line 504, as shown by a dotted line in Fig. 5 . Here, the setting of the facial approach guide line 504 is performed in order to reduce the calculation amount in the contour extraction.

The contour and element extraction block 104 then extracts facial contours and facial elements from the facial image window, which may be, for example, eyebrows, eyes, nose, mouth, etc., , Width, relative position, and the like. The facial contour and facial information extracted here are transmitted to the image authentication block 106 of the next stage, and the information including the lip information in the facial element (i.e., the face lower image frame) is transmitted to the lip detection block 112 do.

The image authentication block 106 also receives face contours of the speaker provided from the contour and element extraction block 104 and the face characteristics of the subjects stored in the speaker adaptation DB 108 (e.g., the owner and family of the mobile terminal) (Multiple face feature vectors), identifies (selects) a speaker larger than a predetermined threshold value (matching threshold value), and calculates a maximum matching rate through matching between the face element of the speaker and the predefined face image template . Here, the template matching means matching in which the images are compared pixel by pixel.

At this time, the face feature vector may be set in various forms considering various use environments in the portable terminal, and the calculated maximum matching rate, that is, the template matching rate between the facial images is calculated as a percentage (%) and displayed on the user interface screen .

That is, in the image authentication block 106, if the maximum matching rate with the predefined face image template is larger than the predetermined matching threshold, it is determined that the user's face image authentication is completed. And the language DB 110, as shown in FIG.

Here, the face image template of the user can be optionally added in accordance with the need for precise user authentication. In consideration of the amount of calculation, the face image template can be specified by minimizing the deformed portion , It is possible to reduce the calculation amount according to the template matching rate calculation. For example, both sides of the eyebrow can be designated as the width, and up to the lip contour portion below can be designated as the face image template.

The voice and language DB 110 is constructed of a sound model, a pronunciation dictionary, a language model, and the like, and a recognition network for combining these models is formed. These models are used to identify (recognize) (I.e., generation of a speaker identification signal), the speech characteristics extracted from the input speech signals are converted into text sentences.

Next, the lip detection block 112 extracts the lip motion image, that is, extracts the lip motion of the speaker, through comparison between temporally adjacent image frames (for example, comparison of pixel values at the same position, etc.) And may include a configuration as shown in FIG.

FIG. 2 is a detailed block diagram of the lip detection block shown in FIG. 1, including a motion extraction block 202, a frame memory 20, a motion separation block 206, a feature value extraction block 208, (210), and the like.

Referring to FIG. 2, the motion extraction block 202 performs a motion comparison between a current image frame (i.e., a face lower image frame) and a temporally adjacent previous image frame (for example, a difference between pixel values at the same position) And extracts a lip motion image. For this purpose, the frame memory 204 may store at least one previous image frame temporally adjacent to the current image frame.

Next, the motion separation block 206 removes noise components from the lip motion image transmitted from the motion extraction block 202 through noise filtering using a general filtering technique well-known in the art, And separating only the moving image (the image showing the lip part and the upper part thereof) from the moving image.

In addition, the feature value extraction block 208 extracts multiple feature values from the lip motion image of the separated motion image using the image feature extraction algorithm widely known in the art, and transmits the extracted feature values to the motion determination block 210 Can be provided.

Meanwhile, the motion decision block 210 compares the multiple feature values provided from the feature value extraction block 208 with the predefined lip motion feature vectors (multiple lip motion feature vectors) stored in the internal memory (not shown) (E.g., nose, eye, eye, etc.) of the upper lip of the lip when it is judged to be lip movement, and comparing the lip top template (Lip motion detection signal or lip motion non-detection signal) and transmits the result to the voice feature extraction block 116 of FIG. 1, for example.

That is, in the motion determination block 210, a maximum matching rate is calculated by matching a partial image of the upper lip of the motion having motion with a predetermined upper lip template, and when the calculated maximum matching rate is equal to or greater than a predetermined threshold value, And generates a lip motion detection signal corresponding thereto. Here, the maximum matching rate between the partial image of the lip upper part and the predefined lip upper template can be displayed on the user interface screen as a percentage (%). At this time, utilizing the lip top template for determining the lip movement reflects various user lighting environments in which the mobile terminal is used.

1, the sound acquisition block 114 acquires an acoustic signal (for example, an acoustic signal including a voice signal or the like) input through a microphone of a portable terminal (not shown) And the like can be provided.

Next, the voice feature extraction block 116 separates the voice section from the voice signal delivered from the sound acquisition block 114 when a lip motion detection signal is provided from the lip detection block 112, It is possible to provide functions such as extracting speech features from a speech signal of a separated speech section using a widely known speech feature extraction algorithm, and then transmitting the extracted speech features to a speech recognition and sentence generation block 118.

At this time, the voice feature extraction block 116, if a lip motion non-detection signal is supplied from the lip detection block 112, even if a voice signal (acoustic signal) is transmitted from the sound acquisition block 114, It is possible to provide a function of not performing the process of extracting speech features by considering it as noise.

The speech recognition and sentence generation block 118 also receives speech characteristic extraction block 116 using the speech recognition model built for the identified speaker stored in the speech and language DB 110 Converting voice features into text sentences, and so on. Here, the converted text sentence can be displayed (output) on the user interface screen (display panel of the portable terminal).

Next, the target language sentence generation block 120 includes, for example, an automatic translation engine and the like. Using the automatic translation engine, the target language sentence generation block 120 converts a text sentence sent from the speech recognition and sentence generation block 118 into a target language sentence And so on. Here, the converted target language sentence can be displayed (output) on the user interface screen.

Finally, the target language speech generation block 122 includes, for example, a speech synthesis engine and the like. The target language speech generation block 122 converts the target language sentence, which is transmitted from the target language speech generation block 120, And outputting it to a speaker (not shown) of the portable terminal.

Next, a series of processes for providing an automatic interpretation service using the automatic interpretation apparatus of the present invention having the above-described configuration will be described in detail.

FIG. 3 is a flowchart illustrating a main process of providing an automatic interpretation service through facial image authentication and speaker adaptation of a user according to an embodiment of the present invention, and confirmation of a voice interval by tracking a lip motion image.

Referring to FIG. 3, when a user's image signal is input from a front camera of a portable terminal (not shown) (step 302), the image acquisition block 102 recognizes the face image, .

At this time, the recognized face image frame is displayed on the face image window on the user interface screen (step 304). The face image displayed on the face image window in this way is, for example, It is possible to induce adjustment of the face position so as to be adjusted to the approach guide line 504.

In response, the contour and element extraction block 104 extracts facial contours and facial elements (e.g., eyebrows, eyes, nose, mouth, length, width, relative position, etc.) from the facial image window Step 306).

Next, in the image authentication block 106, the facial contour of the extracted speaker and the facial feature vectors (multi-facial feature vector) of the subjects (for example, the owner and family of the portable terminal) stored in the speaker adaptation DB 108, The matching rate is calculated, and the maximum matching rate is calculated through matching between the face element of the speaker and the predefined face image template (step 308). If the maximum matching ratio is larger than the predetermined threshold value (matching threshold value) The speaker is selected (identified) (step 310). Here, the template matching means matching in which the images are compared pixel by pixel. Here, the template matching rate between face images is calculated on a percentage (%) basis and displayed on the user interface screen.

Then, the sound acquisition block 114 acquires an acoustic signal (e.g., an acoustic signal including a voice signal) input through a microphone of a portable terminal (not shown) and transmits the acquired acoustic signal to the voice feature extraction block 116 ).

In response to this, the voice feature extraction block 116 extracts the voice feature from the voice section separated from the voice signal based on the lip motion extraction result provided from the lip detection block 112, and checks whether the lip motion is detected or not (Step 314), the input acoustic signal is regarded as acoustic noise (no speech feature extraction process) (step 316) when lip motion is not detected, or a subsequent process is performed when the lip motion is detected. That is, according to the present invention, even if a voice signal (acoustic signal) is inputted, if the lip motion is not detected, the process of extracting the voice feature by considering the input voice signal as noise is not performed.

For this, the lip detection block 112 extracts the lip motion image, that is, extracts the lip motion of the speaker, by comparing (for example, comparing differences of pixel values at the same position) between temporally adjacent image frames, A lips movement detection signal or a lip movement undetection signal is generated and provided to the voice feature extraction block 116, which will be described in detail with reference to Fig. 4 showing the detailed procedure thereof.

Figure 4 is a flow chart illustrating the main process of detecting lips movement in accordance with the present invention.

Referring to FIG. 4, in the motion extraction block 202, a comparison (e.g., comparison of pixel values at the same position) between an input current image frame (i.e., a face lower image frame) and a temporally adjacent previous image frame A lip motion image is extracted (step 402). To this end, the frame memory 204 stores at least one previous image frame temporally adjacent to the current image frame.

Next, the motion separation block 206 removes noise components from the extracted lip motion image through noise filtering using a general filtering technique well known in the art, and then extracts motion images (i.e., (Step 404). ≪ / RTI >

The feature value extraction block 208 extracts multiple feature values from the lip motion image of the separated motion image using the image feature extraction algorithm widely known in the art (step 406).

Next, in the motion decision block 210, whether or not lip motion is detected through comparison between the extracted multiple feature values and predefined lip motion feature vectors (multiple lip motion feature vectors) stored in the internal memory (not shown) (Step 408). When it is determined that the lips move, it is determined whether or not the speaker lips are moved through comparison between some images (e.g., nose, eye, eye, etc.) The maximum matching rate is calculated by matching a partial image of the upper lip of the lip having the motion and the upper lip of the lip that has been defined, and it is checked whether or not the calculated maximum matching rate is equal to or greater than a preset predetermined threshold value Step 410).

If it is determined in step 410 that the maximum matching ratio is less than or equal to a predetermined threshold value, the process proceeds to step 412, and as a result, the motion determination block 210 outputs a lip motion non- And transmits it to the voice feature extraction block 116.

If it is determined in step 410 that the maximum matching rate is equal to or greater than a predetermined threshold value, the motion determination block 210 generates a corresponding lip motion detection signal and outputs the generated lip motion detection signal to the voice feature extraction block 116 (Step 414).

Referring again to FIG. 3, if it is determined in step 314 that the lip motion has been detected (that is, the lip motion detection signal has been generated), the voice feature extraction block 116 extracts The speech features are extracted from the speech signal of the separated speech period using the speech feature extraction algorithm widely known in the art (step 318).

In response, the speech recognition and sentence generation block 118 extracts the speech features extracted using the speech recognition model built for the identified speaker (stored in the speech and language DB 110) into a text sentence (Step 320), the converted text sentence is displayed (output) on the user interface screen (display panel of the portable terminal).

Next, the target language sentence generation block 120 converts the speech text sentence into the target language sentence using the automatic translation engine (step 322), where the translated target language sentence is displayed (output) on the user interface screen .

Thereafter, the target language speech generation block 122 converts the target language sentence transmitted from the target language sentence generation block 120 using the speech synthesis engine to the target language speech, and outputs the target language speech to the speaker of the portable terminal (not shown) 324).

5 is a conceptual diagram illustrating a concept of providing an automatic interpretation service based on a face image, a lip motion image, and an acoustic signal according to the present invention.

Referring to FIG. 5, a face image window is added as a user interface screen of a portable terminal (mobile terminal) in which automatic interpretation is implemented. On the left side of the figure, there is a user (speaker). On the right side, the flow of the element technology for driving the automatic interpretation is shown.

Reference numeral 506 denotes a front face camera of the portable terminal, which acquires the face image of the user and displays the face image in the face image window 502. In the face image window 502, a face approach inducing line 504 is indicated by a dotted line. Here, the facial approach guide line 504 is for reducing the amount of calculation according to the facial contour extraction. The facial contour line is extracted from the facial image window 502, and compared with the predefined facial contour feature, The maximum matching rate is calculated to perform facial image authentication, and the authentication result is recorded in the voice / image sharing memory.

In fact, when you speak the source language for automatic interpreting, you get closer to the screen, with the movement of facial elements including blinking of the eye. The lips movement is detected from various motion images, and the maximum matching rate is calculated by matching a part of the image (e.g., nose) at the upper end with a previously stored template, and the result is recorded in the audio / video sharing memory. Then, the template matching rate according to the facial image authentication and the template matching ratio at the upper part of the lip motion are displayed on the user interface screen.

The source language utterance sound is inputted through the microphone, and at the same time, the lip motion picture signal is confirmed. If there is no lip motion video signal, it is regarded as an acoustic noise and the process is no longer proceeded. Through the lips movement video signal and the speech signal processing process, the speech segment is separated and the speech segment is extracted from the speech segment. Here, an acoustic model, a pronunciation dictionary, and a language model are constructed based on the voice DB and the language DB, and a recognition network in which these are combined is formed.

On the other hand, when the user is identified (identified) through the face image authentication, the voice features extracted in the voice signal processing are converted into text sentences through the acoustic model and the recognition network constructed by speaker adaptation, do. As an example, it shows the result of speaking in the source language "How long does it take to get to Seoul Station?" Here, for speaker adaptation, we utilize a small amount of data obtained from the user, extract common characteristics among a large number of speakers, and transform the recognition model by grouping models having similar characteristics.

Next, the speech recognition sentence is converted into a sentence of the target language by the automatic translation engine and outputted to the user interface screen. As an example, the phrase " How long does it take to the Seoul station? &Quot; The automatic translation sentence is converted into the target language speech by the speech synthesis engine and then outputted through the speaker so that the user can hear the target language.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. It is easy to see that this is possible. In other words, the embodiments disclosed in the present invention are not intended to limit the scope of the present invention but to limit the scope of the technical idea of the present invention.

Therefore, the scope of protection of the present invention should be construed in accordance with the following claims, and all technical ideas within the scope of equivalents should be interpreted as being included in the scope of the present invention.

102: aspect acquisition block 104: contour and element extraction block
106: video authentication block 108: speaker adaptation DB
110: voice and language DB 112: lip detection block
114: Acquisition Acquisition Block 116: Voice Feature Extraction Block
118: speech recognition and sentence generation block 120: target language sentence generation block
122: target language speech generation block 202: motion extraction block
204: frame memory 206: motion separation block
208: Feature value extraction block 210: Motion decision block

Claims (1)

A step of authenticating a user's face image through comparison with a multi-facial feature vector and a face image template related to a predetermined face image after acquiring a user's face image;
Extracting voice features from an input voice signal,
A step of converting the extracted voice features into a text sentence using an acoustic model constructed by speaker adaptation,
Converting the converted text sentence into a target language sentence using an automatic translation engine;
A process of converting the converted target language sentence into a target language speech by using a speech synthesis engine and outputting
.

Images