KR20150050854A - Method and apparatus for recognizing speech based on image - Google Patents

Abstract

The present invention relates to a voice recognition method which can detect a voice section by using an image. According to an embodiment of the present invention, the voice recognition method based on image may include the steps of: receiving an input video including lips; extracting at least two consecutive static images from the input video; detecting lips section from the extracted static images; extracting a characteristic value of the detected lips section by using a chaos pattern, wherein the chaos pattern means an attractor pattern shown according to the changes in pixels between two images; and detecting a voice section by using the characteristic value of the extracted lips section.

Description

METHOD AND APPARATUS FOR RECOGNIZING SPEECH BASED ON IMAGE [0002]

The present invention relates to a speech recognition method, and more particularly, to a speech recognition method for detecting a speech segment using an image.

Speech recognition generally refers to a technique in which a computer interprets and processes a speech language that a person speaks. Such a speech recognition technology feels excellent when it is used in a laboratory, a home, or an office with a low ambient noise. However, when the speech recognition technology is used in a road, hallway, exhibition hall, This is because it does not effectively separate ambient noise and human voice.

On the other hand, speech segment detection in speech recognition has a great influence on speech recognition performance. It is possible to shorten the time required for speech recognition by taking a signal of only the speech section through the speech section detection and reduce the possibility that the noise existing in the non-speech section can lower the speech recognition performance. However, in a noisy environment, detection using only acoustic information is limited. In particular, there is a variety of noise in the vehicle driving situation, so there is a need to compensate for this noise.

Korean Patent Registration No. 10-0883652 discloses a speech segment detection apparatus that detects an accurate speech segment in an input speech including a sudden noise or the like by using dynamic programming so that a short-length disturbance noise is not recognized as speech, Method is disclosed.

However, the above-mentioned Japanese Patent No. 10-0883652 does not disclose a technique for detecting a correct speech interval in various external environments in which illumination is present.

Therefore, it is necessary to study the technology for detecting the accurate speech interval in various external environments in which the illumination is present.

An object of the present invention is to provide an image-based speech recognition method capable of detecting an accurate speech interval in various external environments in which illumination and the like exist.

According to an aspect of the present invention, there is provided a method of processing an input moving image, the method comprising: receiving an input moving image including a lip; Extracting at least two consecutive still images from the input moving image; Detecting a lip region in the extracted at least two still images; Extracting a feature value for the detected lip region using a chaos pattern, the chaos pattern representing a pattern of an attractor represented by a pixel change between two images; And detecting the speech interval using the feature value of the extracted lip region.

The image-based speech recognition method according to an embodiment of the present invention can accurately detect a speech section even in various environments where illumination changes or the like exist.

1 is a block diagram of an image-based speech recognition apparatus in accordance with an embodiment of the present invention.
2 is a flowchart of a video-based speech recognition method in accordance with an embodiment of the present invention.
FIG. 3 shows an example in which a chaos pattern is formed on a phase space plot in an image-based speech recognition method according to an embodiment of the present invention.
FIG. 4 is a diagram for comparing the euclidean dimension and the fractal dimension according to an embodiment of the present invention.
5 is a diagram illustrating an example of a box counting dimension associated with an embodiment of the present invention.
6 is a view for explaining a state transition model for voice interval detection according to an embodiment of the present invention.
FIG. 7 is a graph for comparing the speech segment detection result obtained through the speech recognition method according to an embodiment of the present invention with the speech segment detection result obtained by the conventional method.

Hereinafter, an image-based speech recognition method and apparatus according to an embodiment of the present invention will be described with reference to the drawings.

When an element is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements as well, without departing from the spirit or scope of the present invention. Also, the terms "part," " module, "and the like described in the specification mean units for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software .

1 is a block diagram of an image-based speech recognition apparatus in accordance with an embodiment of the present invention.

As shown, the image-based speech recognition apparatus 100 may include a receiver 100, a preprocessor 120, a feature value extractor 130, a speech interval detector 140, and a controller 150 have. However, not all illustrated components are required. The speech recognition apparatus 100 may be constituted by a larger number of components than the illustrated components, and the speech recognition apparatus 100 may be constituted by fewer components. The speech recognition apparatus 100 may be applied to various electronic products such as a personal portable terminal, a voice recognition TV, an interactive robot, a voice-based automobile interface (e.g., car navigation), and a computer.

The receiving unit 110 may receive the input moving image including the lip. The input moving image may include an audio signal including a voice interval.

The preprocessing unit 120 may extract at least two consecutive still images from the input moving image. The continuous still image means a continuous still image according to the passage of time. The preprocessing unit 120 may detect a face from the extracted still image and detect the lip region in the detected face region.

The feature value extracting unit 130 may extract a feature value for the lip region based on motion information on the lip region. The feature value extracting unit 130 may use a chaotic pattern for detecting motion information on the lip region. The chaos pattern means a pattern of an attractor represented by a pixel change between two images. A method of detecting the feature value using the chaotic pattern will be described later.

The voice section detector 140 can detect the voice section using the extracted feature values of the face area. That is, the voice section detector 140 can determine the voice section and the non-voice section using the feature values extracted from the input moving image. A specific threshold value (T) may be used to distinguish between speech and non-speech.

In addition, the audio interval detection unit 140 may extract the audio signal from the input moving image and detect the start point of the audio interval and the end point of the audio interval using the set upper and lower limit values of the extracted audio signal.

The control unit 150 may control the receiving unit 110, the preprocessing unit 120, the feature value extracting unit 130, and the voice interval detecting unit 140 as a whole.

2 is a flowchart of a video-based speech recognition method in accordance with an embodiment of the present invention.

The receiving unit 110 may receive the input moving image including the lip (S210). The input moving image may include an audio signal including a voice interval.

The preprocessing unit 120 may extract at least two still images that are consecutive from the input moving image according to the passage of time (S220).

The preprocessing unit 120 may detect the face region in the extracted still image and detect the lip region in the detected face region (S230). The lip region can be detected with respect to the lower end portion of the face region detected through the geometric feature that the lip is located at the lower end portion of the face. In the YCbCr color space, the lips are contrasted with the Cr and Cb components more clearly than the facial skin color. This feature can be used to detect the mouth based on the mouth map using Cr and Cb components.

The feature value extracting unit 130 may extract the feature value of the detected lip region using the chaos pattern (S240). The feature value may be used for motion detection of the lip region.

In the present specification, a phase space plot can be defined as a result of obtaining a combined distribution for a pixel value after applying a blur effect through a Gaussian mask operation to an image, respectively.

If two consecutive images at a time extracted from the input video have a fixed background, a chaotic pattern is formed in the phase space plot when there are moving objects in the two images. On the other hand, if there is little difference between the two images or if only the overall brightness changes, a pattern close to the straight line is formed.

FIG. 3 shows an example in which a chaos pattern is formed on a phase space plot in an image-based speech recognition method according to an embodiment of the present invention.

3 (a) shows an example in which a chaos pattern is formed in a non-voice section, and Fig. 3 (b) shows an example in which a chaos pattern in a voice section is formed. In other words, a chaotic pattern appears on the phase space plot for two images with little motion of the lips in the image, and a chaotic pattern appears on the phase space plot due to the movement of the lips in the voice region.

According to an embodiment of the present invention, a feature value for the lip region can be extracted using a fractal dimension of a chaotic pattern. The fractal dimension of the chaotic pattern on the phase space plot calculated from the two images of the speech region can be quantified and the motion of the lips can be detected.

FIG. 4 is a diagram for comparing the euclidean dimension and the fractal dimension according to an embodiment of the present invention. Fig. 4 (a) shows the euclidean dimension, and Fig. 4 (b) shows the fractal dimension.

Fractal represents a structure in which a small structure repeats endlessly in a shape similar to the whole structure, and the fractal dimension represents how perfect the fillet fills the space. If the definition of an euclidean dimension is the number of coordinate axes (two dimensions are x, y, and three dimensions are x, y, and z approaches), then the dimension referred to in the fractal is defined as the number of figures needed to self-replicate. For example, in the case of a square, if the length of each side is tripled, the area becomes 9 times. Therefore, the change of the area with the change of the length is expressed as the square of the length change. Thus, the fractal dimension becomes two-dimensional.

As an example of a method of using the fractal dimension, a method of quantizing a chaotic pattern on a phase space plot using an ox-counting dimension may be used.

5 is a diagram illustrating an example of a box counting dimension associated with an embodiment of the present invention. Fig. 5 (a) shows the case where the box counting dimension is 1, and Fig. 5 (b) shows the case where the box counting dimension is 2.

If the box with size δ is continuously divided in the image, the size of the box becomes smaller in inverse proportion to the number of divisions. If the divisions are repeated, a Richardson plot of the number of divisions and the number of grids including data is approximated to a straight line . The slope of this graph is defined as the box counting dimension.

The box counting dimension may be expressed by Equation (1).

Chaos patterns such as the phase space plot of the speech section have a high fractal dimension, and non-speech sections have a low fractal dimension. According to one embodiment of the present invention, this characteristic can be used to measure the amount of change in the fractal dimension according to the movement of the lips.

The voice section detector 140 may determine the voice section and the non-voice section using the feature values extracted from the input moving image (S250). The threshold value (T) that distinguishes between speech and non-speech is determined by using the distribution of feature values, assuming that the first interval is the non-speech interval in the input image. The threshold value is compared with a feature value between frames to determine a speech / non-speech interval. The determination result may be expressed by Equation (2).

는 시간 t에서의 영상 정보를 나타낸다. 판단 결과 1인 경우 음성 구간임을, 0인 경우는 비음성 구간임을 나타낸다. Where P represents a phase space plot,

Represents image information at time t. As a result of the judgment, it indicates that the voice section is 1, and when it is 0, it is the non-voice section.

In the result of the determination of Equation (2), it may occur that the vocabulary is erroneously detected as a non-speech section in the speech section due to the silence section. To solve this problem, a state transition model can be used. That is, the audio signal included in the input moving picture can be extracted, and the start point of the audio section and the end point of the audio section can be detected using the upper and lower limit values set in the extracted audio signal.

6 is a view for explaining a state transition model for voice interval detection according to an embodiment of the present invention.

In Fig. 6, Silence represents a non-speech section (speech absence section), and In speech represents a speech section. Leaving speech means a section that can be changed into a speech section but a non-speech section. Lower limit value TL (upper threshold), upper limit value TU (upper threshold), and Gap are empirically determined constants to determine the ending point. However, it is assumed that the upper limit value is larger than the lower limit value.

이 TU 보다 작으면 음성이 없는 비음성 구간(Silence)으로 판단하고,

이 TU 보다 커지면 음성 구간이 시작된 것으로 판단할 수 있다. 그리고 음성 구간 검출부(140)는 그 부분을 음성 구간(In speech)의 시작점으로 검출할 수 있다.Using the state transition model shown in Fig. 6, the voice section detection section 140

If it is smaller than the TU, it is determined as a non-speech section (silence)

It can be determined that the voice interval has started. Then, the voice section detector 140 can detect the part as the start point of the voice section (In speech).

이 TL 보다 작아지면 아직 음성 구간이긴 하지만 비음성 구간으로 바뀔 가능성이 있는 구간(Leaving speech)으로 판단하고, Count를 0으로 간주한다. Count는

이 TL과 TU 사이에 연속적으로 존재하는 회수를 의미한다. 음성 구간 검출부(140)는 Count가 Gap 보다 작으면 Leaving speech로 판단하고, Count가 Gap 보다 크면 Silence로 판단한다. Silence로 판단되는 해당 프레임이 음성 구간의 종료 지점이 된다. 또한, 음성 구간 검출부(140)는

이 TL 보다 작아지면 Count를 0으로 잡고 해당 프레임을 Leaving speech 단계로 유지할 수 있다. 그리고

이 TU 보다 커지면 다시 In speech 구간으로 간주할 수 있다.On the other hand, the voice interval detection unit 140

Is smaller than TL, it is determined that the speech is still a speech segment, but it is a segment (Leaving speech) likely to be changed to a non-speech segment, and Count is regarded as 0. Count is

Means the number of consecutive times between TL and TU. If the Count is less than the Gap, the speech interval detector 140 determines Leaving speech. If the Count is greater than the Gap, the speech interval detector 140 determines that the speech is silence. The frame determined as silence is the end point of the voice section. Further, the voice section detection section 140

Is smaller than TL, it is possible to hold the count at 0 and hold the frame at the Leaving speech stage. And

If it is larger than TU, it can be regarded as In speech section again.

Using the state transition model shown in FIG. 6, accurate voice interval detection is possible by adjusting the upper and lower limits according to the size of the noise.

In order to evaluate the performance of the image-based speech segment detection algorithm, an example of speech segment detection using an image taken from a vehicle is described below. The video used in the experiment consists of 913 frames of video and has a speech interval of 197 frames.

Detection rate (PD) and false positive rate (FA) were used as evaluation index for performance measurement. The detection rate is represented by the ratio of the number of frames of the entire speech interval and the number of frames detected correctly as the speech interval. The false detection rate is the ratio of the number of non-speech interval frames to the number of false detection frames. Table 1 shows the results of the performance comparison with the existing voice interval detection algorithm using the optical flow.

Way Detection rate False detection rate Experimental data Conventional 95.98 12.84 Proposed 96.45 0.13

Conventional is a conventional voice segment detection method using an optical path, and Proposed is a method according to an embodiment of the present invention using a chaotic pattern.

FIG. 7 is a graph for comparing the speech segment detection result obtained through the speech recognition method according to an embodiment of the present invention with the speech segment detection result obtained by the conventional method. That is, it is the result graph for Table 1.

Here, Motion Energy represents the motion of the lips through the optical flow, and Chaos Measure represents the movement of the lip through the chaos pattern. In addition, VVAD Result means image-based voice detection, and Ground Truth means actually measured voice interval.

7, it can be seen that the detection of a speech interval according to an embodiment of the present invention is much less false as compared with a conventional speech interval detection method.

As described above, an image-based speech recognition method according to an embodiment of the present invention can detect an accurate speech interval in various external environments such as illumination change by using a chaotic pattern.

The image-based speech recognition method described above may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable recording medium. At this time, the computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. On the other hand, the program instructions recorded on the recording medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software.

The computer-readable recording medium includes a magnetic recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magnetic disk such as a floppy disk, A magneto-optical media, and a hardware device specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

The recording medium may be a transmission medium, such as a light or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, and the like.

The program instructions also include machine language code, such as those generated by the compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above-described image-based speech recognition method and apparatus are not limited to the configuration and method of the above-described embodiments, but the embodiments may be modified so that all or part of the embodiments may be modified Or may be selectively combined.

100: Speech recognition device
110:
120:
130: Feature value extraction unit
140: Voice section detector
150:

Claims (5)

The method comprising: receiving an input moving image including a lip;
Extracting at least two consecutive still images from the input moving image;
Detecting a lip region in the extracted at least two still images;
Extracting a feature value for the detected lip region using a chaos pattern, the chaos pattern representing a pattern of an attractor represented by a pixel change between two images; And
And detecting a speech interval using a feature value of the extracted lip region. The method according to claim 1, wherein the used chaotic pattern
phase space plot of the input image. 3. The method of claim 2, wherein the step of extracting feature values for the lip region comprises:
And using the fractal dimension of the chaotic pattern. 4. The method of claim 3, wherein the use of the fractal dimension
And using a box counting dimension. 4. The method of claim 3, wherein the voice interval detection step
Extracting an audio signal included in the input moving picture;
And detecting an end point of the speech section and an end point of the speech section using the set upper and lower limit values of the extracted audio signal.

Images