KR101134450B1 - Method for speech recognition - Google Patents

Abstract

The present invention relates to a speech recognition method, and according to the disclosed embodiments, the recognition level is reflected by reflecting the reliability for each section of the speech signal, so that the speech recognition is performed when noise is concentrated in some sections or the noise is variable with time. It is possible to prevent the degradation of the performance, and even in the absence of noise, there is an advantage that the intermediate result of the finer speech recognition can be seen for some sections in which the specific words or phrases of the speech recognition exist.

Speech Recognition, Lattice, Segment, Noise, Variable n-best, Reliability

Description

Speech Recognition Method {METHOD FOR SPEECH RECOGNITION}

The present invention relates to a speech recognition method, and more particularly, to a speech recognition method for obtaining a speech recognition result by varying the level-of-detail (LOD) of an input speech signal according to a section.

The present invention is derived from a study conducted as part of the IT growth engine technology development project of the Ministry of Knowledge Economy [Task Management Number: 2008-S-019-02, Task name: Portable Korean / English automatic interpretation technology development].

Automatic Speech Recognition (ASR) is a technology that converts an input speech signal into a string by a speech signal processing technique, and is one of the most important technical fields of speech signal processing along with speech synthesis. Speech recognition technology consists of isolated word recognition that recognizes independently spoken words of tens of words, continuous word recognition, key word spotting that detects and recognizes only key words in consecutive sentences, and continuous speech recognition of sentence-level speech. The speech recognition technology has been developed, and recently, a conversational recognition technology for recognizing a conversational voice naturally spoken between people has been actively studied.

Speech recognition technology is approaching the commercialization level, hundreds of thousands of word-level word recognition application system is applied to a vehicle navigator, etc., but the performance is still degraded by various noise present in the actual application environment. Therefore, robust ASR techniques have been studied for a long time to directly remove noise present in a signal or not to be affected by noise even if the noise is not removed. Nevertheless, many types of noise present in the application environment have become a major factor in the performance of speech recognition.

Noise signals include stationary noise that is constant with time, non-stationary noise that changes with time, such as music, and impulse noise that generates a very short time and distorts only part of the voice. Even if the noise is distorted according to characteristics such as addition noise, room reverberation, and convolution noise, which changes channel characteristics such as echo and microphone variation, the noise varies widely.

1 is a diagram for comparing a normal voice signal with a noise mixed voice signal, (a) is a waveform example of a normal voice signal, (b) is a waveform example of a noise mixed voice signal in which noise is mixed in some section, ( c) is the reliability trajectory for the noise mixed speech signal shown in (b).

In the voice recognition system for voice recognition, a normal voice signal as shown in FIG. 1 (a) may be input, or a noise mixed voice signal having a voice damage section 10 due to noise may be input as shown in FIG. 1 (b). After dividing the speech signal damaged by noise into small sections of several tens of milliseconds as shown in FIG. 1 (b) and measuring the reliability of each section (hereinafter referred to as a voice frame), the reliability value is set to 0. When expressed as a value between 1 shows a curve of the form as shown in Fig. 1 (c). That is, since the reliability is lowered in the section corresponding to the speech damage section 10 of FIG. 1 (b), it has a reliability lowering section 20 showing a low reliability value at the position corresponding to the speech damage section 10.

1 illustrates an example in which noise exists only in a part of a voice signal, but a noise signal may exist in an entire voice signal section, and in this case, the reliability is variable in the entire voice signal section.

2 is a view for comparing the speech recognition method according to the prior art and the speech recognition method according to an embodiment of the present invention, (a) shows the trajectory of the reliability of the noise mixed speech signal as in Figure 1 (c) (B) shows a lattice output result corresponding to the 1-best recognition result for the noise mixed speech signal of FIG. 1 (b) by the speech recognition method according to the prior art. 2 (c) will be described later.

In (b) of FIG. 2, phoneme candidates considered to be closest to each voice section of the entire input signal are indicated in a node represented by a circle. As shown in FIG. 2 (b), according to the speech recognition method according to the related art, the grid information of the same degree is generated regardless of whether it is a normal section having high reliability or a section having low reliability 20. That is, speech recognition is performed with the same granularity for all speech sections.

Therefore, according to the prior art, when performing speech recognition on a noise mixed speech signal, the speech recognition result of the speech damage section including the noise even though the speech recognition result for the section containing no noise is satisfactory. The recognition result does not represent a satisfactory level. That is, the speech recognition result for the speech damage section has a problem that the speech recognition performance is not stabilized because it has a low reliability.

The present invention has been proposed to solve the problems of the prior art as described above. The human brain assumes a plurality of candidates based on speech recognition clues that recognize even a speech section damaged by noise, and then, based on all available information such as context and speech situation of the section, It is known to maximize the speech recognition results. The present invention is a method of partially modeling the human brain function, and since noise often contaminates the sequential sections in the time domain of the voice signal to different levels, the fineness of the input voice signal varies depending on the section. Acquire voice recognition result. In other words, more phonetic or word candidate information is used for the low confidence speech section, that is, richer grid information is generated for the low reliability speech section, and minimum grid information is generated for the high confidence speech section. And by using it to stabilize the speech recognition performance.

To this end, according to an embodiment of the present invention, the speech recognition is robust against noise by estimating the reliability in every section of the speech signal and varying the generation degree of the comparison candidates for each section of the signal when decoding the speech recognition according to the reliability information. Have a characteristic.

According to another exemplary embodiment of the present invention, more detailed grid information is generated for a specific section of interest among all sections of the input voice signal, and the information is used in post-processing of voice recognition. This makes the speech recognition robust to noise.

According to an embodiment of the present invention, a voice recognition method includes dividing an input voice signal into a plurality of sections of a specific unit, and differently determining the granularity of the grid generation for at least one section and another section among the plurality of sections. Generating a lattice for speech recognition according to the granularity of the plurality of sections; and performing decoding to limit the generated lattice for speech recognition to a search space to obtain speech recognition results. It may include a step.

The dividing may be performed by dividing the voice signal into segments.

The determining may include estimating a reliability value for each segment of the voice signal, normalizing the estimated reliability value, and differently n-best values for each segment based on the normalized reliability value. Determining may include.

In the normalizing step, regardless of the type of the reliability estimation method for each segment, the obtained reliability information may be normalized to a real value between 0 corresponding to the lowest value of confidence and 1 corresponding to the highest value.

In the normalizing step, the reliability value expressed as a value between 0 and 1 for each segment may be converted into a value from 1 to a natural number V using a linear or nonlinear function.

The dividing may distinguish one or more specified interest intervals.

In the determining, the n-best value may be specified differently from the other sections for the interest section.

The generating may include performing a forward probability calculation for hidden recognition based on Hidden Markov model (HMM) using the n-best value, and performing a backtracking process after completion of the forward probability calculation. It may include extracting the information of the grid through.

The performing of the step may be performed by decreasing the value obtained by multiplying the n-best forward probability value and the state transition probability from the large value in the HMM states of all previous frames capable of transitioning to the HMM state that is the target of the forward probability calculation of the current frame. After sorting, values corresponding to the n-best number of the current HMM state can be stored.

The performing may include storing information corresponding to which state each n-best value has transitioned in which HMM model of the previous frame in the HMM state that is the target of the forward probability calculation of the current frame.

The extracting may be performed by applying the extracted grid information to a speech recognition search network to perform secondary speech recognition on the input speech signal to obtain a final recognition result.

According to the exemplary embodiment of the present invention, the speech recognition result may be obtained by varying the details of the speech signal according to the section.

For example, by varying the recognition level by reflecting the reliability of each section of the speech signal, it is possible to prevent performance degradation of speech recognition in the case where the noise is concentrated in some section or the noise is variable over time. In other words, even if the voice signal is damaged by noise, packet loss, etc., the voice signal can operate more stably without a sudden deterioration in voice recognition.

In addition, even in the absence of noise, it is possible to see intermediate results of more detailed speech recognition for some sections in which a specific word or phrase of speech recognition exists, which can be used for various purposes such as post-processing of speech recognition. have. In other words, even in the absence of distortion, a more precise level of speech recognition is possible for a specific section of interest of the input speech signal.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the embodiments make the disclosure of the present invention complete, and the general knowledge in the art to which the present invention belongs. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like numbers refer to like elements throughout.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the embodiments of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowchart may be performed by computer program instructions. These computer program instructions may be embedded in the processor of a general purpose computer, special purpose computer, or other programmable data processing equipment such that the instructions executed by the processor of the computer or other programmable data processing equipment may be executed in each block or flowchart of the block diagram. It will create means for performing the functions described in each step of. These computer program instructions may be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, and thus the computer usable or computer readable memory. It is also possible for the instructions stored in to produce an article of manufacture containing instruction means for performing the functions described in each block or flowchart of each step of the block diagram. Computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operating steps may be performed on the computer or other programmable data processing equipment to create a computer-implemented process to create a computer or other programmable data. Instructions that perform processing equipment may also provide steps for performing the functions described in each block of the block diagram and in each step of the flowchart.

In addition, each block or step may represent a portion of a module, segment or code that includes one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative embodiments, the functions noted in the blocks or steps may occur out of order. For example, the two blocks or steps shown in succession may in fact be executed substantially concurrently or the blocks or steps may sometimes be performed in the reverse order, depending on the functionality involved.

3 is a block diagram illustrating a voice recognition device capable of performing a voice recognition method according to an embodiment of the present invention.

As shown in the drawing, the speech recognition apparatus 100 includes a section divider 110, a fineness determiner 120, a grid generator 130, a decoder 140, and the like.

The section divider 110 may divide the input voice signal into a plurality of sections of a specific unit and provide the same to the detail determiner 120. For example, the voice signal may be divided in units of segments or at least one interest section designated by the user may be distinguished from other sections.

The fineness determining unit 120 may differently determine the fineness of the grid generation for at least one section and the other section among the plurality of sections divided by the section separator 110. That is, the variable n-best can be applied. For example, after estimating and normalizing a reliability value for each segment of a voice signal, the n-best value may be differently determined for each segment based on the reliability value. Alternatively, the n-best value may be specified differently from other sections in the interest section designated by the user.

The grid generating unit 130 may generate a grid for speech recognition according to the fineness determined by the fineness determining unit 120 for a plurality of sections. For example, the fineness determining unit 120 performs forward probability calculation for hidden recognition based on Hidden Markov Model (HMM) using the determined or designated n-best value, and after completion of the forward probability calculation, The grid information can be extracted through a back-tracking process.

The decoder 140 may obtain a speech recognition result by performing decoding to limit the grid for speech recognition generated by the grid generator 130 to a search space. For example, after multiplying the n-best forward probability value by the state transition probability in the HMM states of all previous frames capable of transitioning to the HMM state that is the target of the forward probability calculation of the current frame, they are arranged in decreasing order from the large value. Among them, values corresponding to the n-best number of the current HMM state are stored, and each n-best value corresponding to which state of which HMM model of the previous frame has transitioned in the HMM state that is a target for calculating the forward probability of the current frame. Information can be stored. The extracted grid information is applied to a speech recognition search network to perform secondary speech recognition on the input speech signal to obtain a final recognition result.

The voice recognition method by the voice recognition device according to the embodiment of the present invention configured as described above will be described in more detail with reference to FIGS. 4 to 6 as follows.

4 is a flowchart illustrating a voice recognition method according to an embodiment of the present invention. First, the embodiment from step S211 to step S215 of FIG. 4 will be described.

The section divider 110 divides the input voice signal into segments and provides the fineness determination unit 120, and the fineness determiner 120 estimates the reliability of each input segment for each segment (S211). ). The segment of the speech signal may be a speech frame of several tens of milliseconds in length or may be a segment for longer intervals. In the present invention, a voice frame will be described as an example. The reliability of an arbitrary speech frame can be obtained by using a statistical model or by calculating a signal-to-noise ratio (SNR) for a corresponding signal section.

에 의해서 SNR 0dB 이하값은 0에 가깝고, SNR 25dB 이상은 1에 가깝도록 정규화할 수 있다(S213).If the SNR estimated in the t-th frame of the speech signal is snr (t), for example, a nonlinear function such as

By SNR, the value below 0dB is close to 0, and SNR 25dB or more can be normalized to 1 (S213).

의 값이 1에 가까울수록 왜곡이 심하지 않은 프레임이며, 0에 가까울수록 잡음에 의해 손실이 크게 발생하여 신뢰도가 떨어지는 프레임을 의미한다. 도 1 (c)에서 실선으로 표시된 그래프는 도 1 (b)의 잡음 혼입 음성신호에 대한

값의 일례를 보인다.Where the normalized confidence value of the t th frame

The closer the value is to 1, the less distortion the frame is, and the closer to 0, the less loss of noise is caused by noise. The graph indicated by the solid line in FIG. 1 (c) shows the noise mixed speech signal of FIG. 1 (b).

An example of a value is shown.

를 기준으로 프레임 t에서의 n-best 값 v(t)를 결정한다(S215). 여기서는 신뢰도가 낮을수록 n-best 값을 크게 하여 보다 많은 후보들을 생성하므로,

와 v(t)는 반비례 관계이며, 다음과 같이 1개에서 V개까지의 n-best 후보값으로 사상(mapping)된다. 이 때, V는 1보다 큰 정수이다.In addition, the fineness determination unit 120 has a reliability value for each segment.

The n-best value v (t) in the frame t is determined based on (S215). In this case, the lower the reliability, the larger the n-best value, which generates more candidates.

And v (t) are inversely proportional and are mapped to one to V n-best candidate values as follows. At this time, V is an integer greater than one.

를 1에서 V 사이의 값으로 수학식 2와 다른 방법으로 사상할 수 있다. 즉 다른 선형 또는 비선형 함수를 사용하여 1부터 자연수 V까지의 값으로 변환할 수 있다.v (t) is reliability information normalized to a value between 0 and 1 depending on the purpose

Can be mapped to a value between 1 and V in a different way from Equation 2. In other words, other linear or nonlinear functions can be used to convert values from 1 to the natural number V.

According to the exemplary embodiment of step S221 and step S223 of FIG. 4, if the user specifies an interval of interest (IOI), the section divider 110 is different from at least one interest section specified by the user. Are provided to the detail determining unit 120, and the detail determining unit 120 may designate the number of n-best candidates for the corresponding sections, that is, the section of interest and the other section. Also in this case, for the frame t (t = 1, 2, ..., T), the value of v (t) is an integer value from 1 to V, which specifies the number of n-best candidates. The larger the V value, the more various n-best candidates can be generated for the interval.

Next, the grid generating unit 130 obtains v (t) for the frames of the input voice signal, and then generates a decoding and grid for real speech recognition (S230). This process will be described in more detail with reference to FIG. 5 below.

Finally, the decoding unit 140 obtains the final speech recognition result by limiting the second decoding, that is, the generated grid by using the grid information generated by the grid generator 130, to the search space (S240).

5 is an example of forward probability calculation and biter ratio decoding for explaining the operation principle of the Viterbi decoding algorithm in the speech recognition based on the hidden Markov model.

의 HMM의 상태

에서의 출력 확률값

은 수학식 3과 같이 표현할 수 있다.In FIG. 5, the X-axis denotes T feature vectors extracted for each unit section (voice segment or frame) at a constant interval of the voice signal, and the Y-axis represents HMMs for N phonemes, each phoneme being three It consists of states. Each state models a Gaussian mixture density function, for example a feature vector.

State of HMM

Output Probability at

Can be expressed as in Equation 3.

In Equation 3, M denotes the number of Gaussian distributions, and N denotes a general Gaussian distribution function.

는 다음 수학식 4와 같이 계산된다.In FIG. 5, the Viterbi decoding algorithm proceeds one frame (or one feature vector) from left to right on the X axis, and calculates necessary information such as a forward probability value for all states of all HMMs represented on the Y axis. Search for the best recognition path. Forward probability value of the j th HMM state for the phoneme p in the t th frame

Is calculated as in Equation 4 below.

In this case, the first state of the HMM (j = 1) is calculated as shown in Equation 5 including a probability of transition between models from phoneme r to phoneme p, that is, a language model probability value.

값을 계산 및 저장한다(p=2, j=3, t=4). 노드(301)는 이전 프레임에서 음소 2의 상태 2번 노드(302) 또는 3번 노드(303)에서 천이하였으며, 탐색 경로의 역추적을 위해서 이전 프레임의 노드들 중에서 현재 노드까지의 최적 경로 정보를 수학식 6에서처럼 계산하여 저장한다.For example, the node 301 of FIG. 5 is in the last state of phoneme 2 in the fourth frame.

Compute and store the value (p = 2, j = 3, t = 4). Node 301 has transitioned from node 2 302 or node 3 303 of phoneme 2 in the previous frame. Calculate and store as in Equation 6.

와

를 계산한 후에,

값이 최고인 p를 선택하고, 여기에서부터

를 이용하여 최적의 인식결과를 역추적 과정을 통해 얻게 된다. The Viterbi decoding algorithm uses the last frame T of the input speech.

Wow

After calculating

Select p with the highest value, and from here

The optimal recognition result is obtained through the backtracking process.

의 개념을 보다 확장하여, 매 노드마다 N개의 n-best 값들을 저장한다. 이 때, n-best의 N 값은 프레임의 신뢰도 정보에 관련된 값으로 프레임 별로 다른 값을 가질 수 있다.

의 확장된 값은

로 표기하며 수학식 7과 같다.In the present invention, at any frame index t

To extend the notion of, we store N n-best values in every node. In this case, N value of n-best is a value related to reliability information of a frame and may have a different value for each frame.

The expanded value of

It is expressed as Equation (7).

는 t번째 프레임의 신뢰도를 이용하여 계산한 n-best 후보들의 개수로서 1보다 큰 정수값이다.

함수는 괄호 안의 값들을 큰 값으로부터 작은 값의 순서로 정렬하는 함수이며, 이 함수의 괄호 안의 값은 특정 p 값에 대해 변수 i와 w가 정해진 범위 내에서 변함에 따라 각 경우에 대한 값들을 갖게 된다.

함수는 u번째 큰 값을 출력하는 함수이다. 도 6은 이 과정을 개념적으로 도식화한 것으로 도 6에서 t번째 프레임에서 노드(401)의 n-best 값들은 이전 프레임 t-1에서 노드(401)로 천이될 수 있는 각 상태들(402, 403)에 저장된 n-best 값들을 대상으로

값들을 계산한 후, 이로부터 n-best를 선택하여 저장하게 된다.here

Is an integer value greater than 1 as the number of n-best candidates calculated using the reliability of the t-th frame.

A function is a function that sorts the values in parentheses in order from large to small, and the values in parentheses have values for each case as the variables i and w vary within a specified range for a particular p value. do.

The function prints the uth largest value. FIG. 6 conceptually illustrates this process, where n-best values of node 401 in the t-th frame of FIG. 6 are transitioned to states 402 and 403 that may transition to node 401 in previous frame t-1. For n-best values stored in

After calculating the values, we choose n-best from it and store it.

계산 과정에서 j가 HMM의 첫 번째 상태(j=1)인 경우는 음소 r에서 음소 p로의 모델 간 천이를 고려하여, 다음 수학식 8과 같이 계산한다.In addition,

In the calculation process, when j is the first state of the HMM (j = 1), the equation is calculated as shown in Equation 8 in consideration of the transition between the models from the phoneme r to the phoneme p.

들에 대해서 모델 p의 첫 번째 상태로 천이하는 경우에 대해서 v(t)개 만큼의 n-best를 계산하여 저장한다. 이 방식의 역추적 과정은 수학식 6에서 설명한 바와 같은 원리로 이루어지며, 프레임 t에서의 n-best 각각에 대해

를 대신하여

를 계산 및 저장한다.That is, the above Equation 8 is based on the current frame t, n-best values of v (t-1) stored in the end-state of the entire R phoneme models in the previous frame t-1,

For the case of transitioning to the first state of the model p, we calculate and store as many as v (t) n-bests. The traceback process of this method is based on the same principle as described in Equation 6, for each n-best in frame t.

On behalf of

Calculate and store

On the other hand, the description of the HMM-based speech recognition technology that has not been described above according to well-known or tolerant technology, the detailed description thereof is omitted.

As described above, Figure 2 is a view for comparing the speech recognition method according to an embodiment of the present invention and the speech recognition method according to the prior art, (c) is a view of Figure 1 by the speech recognition method according to an embodiment of the present invention The lattice output result corresponding to the n-best recognition result for the noise mixed speech signal of (b) is shown. For example, the speech recognition performance is stabilized by utilizing the grid information of more phonemes or word candidates for speech sections with low reliability. As shown in FIG. 2 (c), it generates richer grid information at a position corresponding to a voice section with low reliability and has a high density grid section 30, and generates minimum grid information for a voice section with high reliability. It becomes possible to use.

1 is a diagram for comparing a normal speech signal and a noise mixed speech signal;

2 is a view for comparing the speech recognition method according to an embodiment of the present invention with the speech recognition method according to the prior art,

3 is a block diagram illustrating a voice recognition device capable of performing a voice recognition method according to an embodiment of the present invention;

4 is a flowchart illustrating a voice recognition method according to an embodiment of the present invention;

5 is an example of a forward probability calculation and a biter ratio decoding for explaining the operation principle of the biter ratio decoding algorithm in speech recognition based on hidden Markov model;

6 is an example of forward probability calculation and bitter ratio decoding when the variable n-best is applied according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: voice recognition device 110: section separator

120: fineness determination unit 130: grid generation unit

140: decoding unit

Claims (11)

Dividing the input voice signal into a plurality of sections of a specific unit; Determining the granularity of the grid generation differently for at least one of the plurality of sections from another section; Generating a lattice for speech recognition according to the fineness of the plurality of sections; Acquiring a speech recognition result by performing decoding to limit the generated grid for speech recognition to a search space Speech recognition method comprising a. The method of claim 1, The dividing may include dividing the voice signal into segments. Voice recognition method. The method of claim 2, The determining step, Estimating a confidence value for each segment of the speech signal; Normalizing the estimated reliability value; Determining the n-best value differently for each segment based on the normalized confidence value Speech recognition method comprising a. The method of claim 3, wherein The normalizing may include normalizing the obtained reliability information to a real value between 0 corresponding to the lowest value of confidence and 1 corresponding to the highest value regardless of the type of reliability estimation method for each segment. Voice recognition method. The method of claim 4, wherein The normalizing may include converting a reliability value expressed as a value between 0 and 1 for each segment into a value from 1 to a natural number V using a linear or nonlinear function. Voice recognition method. The method of claim 1, The dividing may include dividing at least one specified interest region. Voice recognition method. The method of claim 6, The determining may include specifying an n-best value differently from another interval for the interval of interest. Voice recognition method. The method according to claim 3 or 7, The generating step, Performing a forward probability calculation for speech recognition based on Hidden Markov Model (HMM) using the n-best value; Extracting information of the grid through a backtracking process after completion of the forward probability calculation; Speech recognition method comprising a. The method of claim 8, The performing of the step may be performed by decreasing the value obtained by multiplying the n-best forward probability value and the state transition probability from the large value in the HMM states of all previous frames capable of transitioning to the HMM state that is the target of the forward probability calculation of the current frame. After sorting, the values corresponding to the n-best number of the current HMM state are stored. Voice recognition method. The method of claim 9, The performing of the step may include storing information corresponding to which state of each H-model of the previous frame has transitioned to each n-best value in the HMM state that is the target of the forward probability calculation of the current frame. Voice recognition method. The method of claim 8, The extracting may include applying the extracted grid information to a speech recognition search network to perform secondary speech recognition on the input speech signal to obtain a final recognition result. Voice recognition method.

Images