RU2297676C2 - Method for recognizing words in continuous speech - Google Patents

Abstract

FIELD: automatics and computer science, possible usage in systems for controlling technological, home appliance and other equipment, in automatic reference systems, automatic translation systems, speech understanding systems.

SUBSTANCE: in accordance to method, during pronouncing of speech phrase, taken periodically are selections of acoustic signal of this phrase, digitized with given quantization frequency, in fixed time interval or based on combination of selection, functional is calculated, determining current acoustic condition, while received series of current acoustic conditions is used for restoration of series of words (working hypothesis), pronounced in original speech phrase, for that purpose lexical decoding network is used, which sets rules of order of standard acoustic conditions in a language. Working hypothesis is found, being optimal in terms of maximal match thereof with original speech signal, which is ensured by usage of movable marker algorithm, while working hypothesis is restored from marker, which at this time moment is at end vertex of lexical decoding network.

EFFECT: increased precision of words recognition in continuous speech.

4 cl, 12 dwg

Description

The invention relates to automation and computer technology and can be used in control systems for technological, household and other equipment, in automatic reference systems, automatic translation systems, speech understanding systems, etc.

A known method of recognizing words in continuous speech, implemented in the system of automatic speech understanding company LDC [1].

The essence of the method is that with the utterance of a speech utterance, samples of the acoustic signal of this utterance digitized with a given quantization frequency are periodically taken, at fixed time intervals and from the totality of these samples, a functional is determined that determines the current acoustic state, and the obtained sequence of current acoustic states is used to restore the sequence of words (working hypothesis) spoken in the original speech utterance.

A feature of the known method is that the sequence of current acoustic states is converted into an A-matrix, which is used in further analysis instead of the original speech signal. The A matrix contains various parameters of the speech signal, including its phonetic transcription. Based on the A-matrix, a working hypothesis is obtained, i.e. the sequence of words that is allegedly spoken in the original speech utterance. The A-matrix is processed from left to right - the word pre-selection unit processes the words from the list of the most probable words and generates information about the degree of coincidence of the given word and the initial portion of the speech signal, then the control unit analyzes the received information and determines the list of words using the available syntactic and semantic information that can follow the given word and transfers them to the block of preliminary selection of words, then the procedure is repeated until the end of speech in sayings.

The disadvantage of this method is the low accuracy of recognition, due to the fact that: 1) the search hypothesis search strategy is not optimal in terms of the maximum degree of coincidence with the original speech signal, since the decision on the hypothesis optimality is made with each comparison of the word with the speech signal and the optimal hypotheses; 2) the search for a working hypothesis is a two-stage process - the A-matrix is calculated at the first stage, the hypothesis is found at the second stage, so the search for a working hypothesis is based on phonetic transcription, which may contain errors.

A known method of recognizing words in continuous speech, implemented in the system of automatic speech understanding HWIM company BBN [2].

The essence of the method is that with the utterance of a speech utterance, samples of the acoustic signal of this utterance digitized with a given quantization frequency are periodically taken, at fixed time intervals and from the totality of these samples, a functional is determined that determines the current acoustic state, and the obtained sequence of current acoustic states is used to restore the sequence of words (working hypothesis) spoken in the original speech utterance.

A feature of this method is that the sequence of current acoustic states is converted into a series of phonetic transcriptions, which are recorded as a segmented grid. The obtained segmented grid is used in the search for a working hypothesis. The constructions that most correspond to the words, regardless of their location, are sent to the word verification unit, where the degree of coincidence is evaluated. The results of the check are combined with the results of lexical selection, and if the indicator of this combined result is high enough, the words are sent to the syntactical prediction block. Words formed in the syntactical prediction block in accordance with the rules of the grammar used are added to the right and left to the main word and sent to the word verification block. The process then repeats until the entire speech signal is recognized. The selection of words is carried out using lexical decoding networks, which represent all kinds of phonetic representations of the word in all possible phonetic contexts.

The disadvantage of this method is the low recognition accuracy due to the following factors: 1) the search hypothesis search strategy is not optimal in terms of the maximum degree of coincidence with the original speech signal, since the decision on the hypothesis optimality is made with each comparison of the word with the speech signal and the loss of the optimal hypothesis is possible ; 2) the search for a working hypothesis is a two-stage process - at the first stage a segmented grid is computed, at the second stage a working hypothesis is found, thus, the search for a working hypothesis is based on a segmented grid that may contain errors, however, unlike the first analogue, this method is more stable to phonetic transcription errors.

Closest to the proposed one is a method for recognizing words in continuous speech [3], adopted as a prototype, consisting in the fact that with the utterance of a speech utterance, samples of the acoustic signal of this utterance digitized at a given quantization frequency are periodically taken at fixed time intervals and in combination of these the samples calculate the functional that determines the current acoustic state, while the resulting sequence of current acoustic states is used to restore the sequence spacing words (working hypothesis) spoken in the original speech utterance.

A feature of this method is that the working hypothesis is found directly from the sequence of current acoustic states using a lexical decoding network, the vertices of which are reference acoustic states, and the transitions between them define the following possible reference acoustic states. The lexical decoding network sets the patterns for following reference acoustic states in accordance with the grammatical and phonetic rules of the language.

The decoding process begins with the selection of the initial vertices of the lexical decoding network and finding the vertex that corresponds to the reference acoustic state closest to the current acoustic state. The number of the closest vertex is fixed in the storage block of the working path, which is a sequence of vertex numbers. The next current acoustic state is compared with the states associated with the following possible network nodes. The closest peak is fixed in the storage block of the working path. The process is repeated until the arrival of the current acoustic states corresponding to the speech signal. After the process is completed, the found working path is converted into a working hypothesis.

The disadvantage of the prototype is the low recognition accuracy associated with the following factors: 1) the search hypothesis search strategy is not optimal in the sense of maximum degree of its coincidence with the original speech signal, since the decision on the hypothesis optimality is made at each moment of time; 2) when constructing a network of lexical decoding, a language model is not used [5, p.539]; 3) when calculating the degree of coincidence, information on the average duration of acoustic states is not taken into account [4, p. 259].

The technical result obtained from the implementation of the invention is to increase the accuracy of word recognition in continuous speech.

This technical result is achieved due to the fact that in the known method of recognizing words in continuous speech, namely, with the utterance of a speech utterance, samples of the acoustic signal of this utterance digitized with a given quantization frequency are periodically taken at fixed time intervals and from the totality of these samples calculate the functional that determines the current acoustic state, while the resulting sequence of current acoustic states is used to restore the last the sequence of words (working hypothesis) spoken in the original speech utilization by applying a lexical decoding network that defines the patterns for following reference acoustic states in the language, and in order to improve recognition accuracy, a search is made for a working hypothesis that is optimal in terms of the maximum degree of its coincidence with the original speech signal, using the algorithm of the moving marker, and the working hypothesis is restored from the marker, which is currently in the final top of the network of lexical decoding.

A feature of this method is that the search for a working hypothesis is optimal in the sense of the maximum degree of its coincidence with the original speech signal, since the algorithm of the moving marker, which is used to search for a working hypothesis, is based on the dynamic programming method [6, p. 74].

Also, when constructing a network of lexical decoding, a language model can be used [5, p.539] and / or the probability of transition between states by which the average phoneme durations are taken into account.

The invention is illustrated by drawings, where in Fig.1-7 which presents the steps of building a network of lexical decoding; Figs. 8, 9 show a moving marker algorithm; figure 10-12 is a device that implements the method.

A lexical decoding network is created by performing the following operations: presenting a speech with a finite set of words, building possible phonetic representations of words, building a language model, creating a database of context-sensitive phonemes (Trifonov), creating a word graph based on a language model, creating for each word from the set Y is the phonetic network of the word, which defines the possible phonetic realizations of the word, the expansion of the graph of words with the help of phonetic networks and the replacement of phonemes by trifons.

The essence of these operations is as follows.

1) Represent speech with a finite set of words:

where w _i is the i-th word, i = 1, ..., n _Y.

2) Represent words as sequences of phonemes:

where n _p is the number of possible phonetic representations of the word w _i , n _fj is the number of phonemes in the jth phonetic representation of the word w _i , f _kj is the kth phoneme in the jth phonetic representation of the word w _i .

3) Create a language model, for this the probabilities are calculated:

p (w _i / W _n ),

where W _n is the sequence of words preceding the word w _i , length n.

4) Create a database of models of context-sensitive phonemes (Trifonov), which are a network (Fig. 6), the vertices of which are reference acoustic states, and the transitions between them define possible transitions between reference acoustic states in the language. Transitions are associated with transition probabilities between states, which implicitly specify the duration of acoustic states.

5) Design a network of lexical decoding taking into account claim 1, claim 2, claim 3, claim 4.

The stages of constructing a lexical decoding network are presented using an example of a network using a two-gram language model.

The stages of constructing a network of lexical decoding are illustrated by drawings: in Fig. 1, spelling and phonetic representation of words is shown; figure 2 is a graph of words based on the two-gram language model, which is used to specify possible sequences of words in the language; figure 3 - subgraph UNK used to model words that are not included in the set Y; figure 4 - phonetic network of the word "LAWYER"; figure 5 - trifon network of the word "LAWYER"; 6 is a network of the word lawyer, the vertices of which are acoustic states; 7 is a lexical decoding network.

In a first step, a vocabulary Y of verbal communication is determined. The set Y contains the UNK class word, which is responsible for words that do not fall into this set. Next, the lexical and phonetic representation of each word is determined (Fig. 1), a language model for a given dictionary Y and a database of language Trifons are built. At the second stage, a decoding network template is created (Fig. 2) in accordance with the existing language model. The decoding network template contains the start node in which the system is located before the recognition process begins, and the end node in which the system is located after the recognition process is completed. The node corresponding to the word UNK is a subgraph (figure 3) and is intended for modeling words that are not included in the dictionary. The structure of the UNK subgraph is chosen so that it can simulate any sequence of phonemes. At the third stage, a graph is constructed for each word, which models all the expected phonetic representations of words (Fig. 4). The vertices of the graph are phonemes, and the arcs are pointers to the following possible phonemes. The final state of the word graph is indicated by a rectangle. It is intended to mark the end of a word. Next, context-dependent phonemes (trifons) are substituted for phonemes (Fig. 5) and then the acoustic states of which they consist (Fig. 6) are taken from the Trifonov database. At the fourth stage, the constructed word graphs are substituted into the decoding network template, as a result of which the lexical decoding network is obtained (Fig. 7).

The operation of the moveable marker algorithm is illustrated in FIGS. 8, 9; Fig. 8 represents a data structure called a marker; Fig. 9 represents a moveable marker algorithm.

, где

- степень совпадения частичного пути, прошедшего через вершину j в момент времени t,

- запись о состоянии, соответствующем вершине j в момент времени t.To implement the moveable marker algorithm (Fig. 9), each vertex of the lexical decoding network contains a pointer to a data structure called a marker (Fig. 8), which stores information about the partial path ending at a given vertex of the lexical decoding network. The marker structure can be represented as

where

- the degree of coincidence of the partial path that passed through the vertex j at time t,

- record of the state corresponding to the vertex j at time t.

When the algorithm is initialized, a marker with a zero degree of coincidence is placed in the initial network node. Next, the algorithm begins. When the next current acoustic state arrives, copies of the markers from each vertex of the lexical decoding network move to all the vertices that are possible after this, while the degree of coincidence of the marker is recalculated in accordance with the formula

- предыдущее значение степени совпадения, р_ij - вероятность перехода между эталонными состояниями, связанными с вершинами сети лексического декодирования i и j, b_j(V_t) - степень совпадения между текущим акустическим состоянием V_t и эталонным состоянием, связанным с вершиной сети лексического декодирования j, w_i - слово, которому принадлежит состояние i, w_j. - слово, которому принадлежит состояние, связанное с вершиной сети лексического декодирования j. Далее исходные маркеры удаляются. В каждой вершине сети лексического декодирования удаляются все маркеры, кроме маркера, имеющего максимальную степень совпадения, эта процедура называется нормализацией множества маркеров. После того, как обработаны все текущие акустические состояния, соответствующие речевому сигналу, работа алгоритма завершается. Из маркера, находящегося в конечной вершине сети лексического декодирования, извлекается рабочий путь, из которого находится рабочая гипотеза.Where

is the previous value of the degree of coincidence, p _ij is the probability of transition between the reference states associated with the vertices of the lexical decoding network i and j, b _j (V _t ) is the degree of coincidence between the current acoustic state V _t and the reference state associated with the top of the lexical decoding network j, w _i is the word to which state i, w _j belongs. is the word to which the state associated with the top of the lexical decoding network j belongs. Next, the original markers are deleted. At each vertex of the lexical decoding network, all markers are deleted, except for the marker with the maximum degree of coincidence, this procedure is called normalization of the set of markers. After all current acoustic states corresponding to the speech signal are processed, the operation of the algorithm ends. From the marker located at the final vertex of the lexical decoding network, the working path is extracted from which the working hypothesis is found.

A device for implementing the method of recognizing words in continuous speech can be performed in the form of a computer program. In this case, the device is a data structure in the main memory of a computer.

A device for implementing the method of recognizing words in continuous speech is shown in FIGS. 10, 11 and 12. FIG. 10 shows a structural diagram of a system; figure 11 is a structural diagram of a block acoustic analyzer; in Fig.12 is a structural diagram of a lexical analyzer.

A device for recognizing words in continuous speech using a lexical decoding network is shown in FIG. 10. It consists of an acoustic analyzer represented by block 1 and a lexical analyzer represented by block 2.

Block 1 is designed to determine the current acoustic state V _t .

Block 2 is designed to search for a working hypothesis. The input of block 1 is connected to a microphone. The output of block 1 is connected to the input of block 2. From the output of block 2, the desired result is obtained.

Block 1, the structural diagram of which is shown in Fig. 11, contains: block 3 - a frequency spectrum analyzer, block 4 - a calculator of the current acoustic state V _t .

Block 2, the structural diagram of which is shown in Fig. 12, contains: block 5 is a repository of multiple states; block 6 - network storage lexical decoding; block 7 is a calculator of the degree of coincidence b _j (V _t ) between the current acoustic state V _t and the reference acoustic state associated with the current top of the lexical decoding network j; block 8 - marker storage; block 9 - driver of the recognition result; block 10 - storage of recalculated markers; block 11 is a block normalization of many markers; block 12 - block output results recognition; block 13 is a control unit.

Block 3 is designed to calculate the spectrum of the current section of the speech utterance and convert it to digital form.

Block 4 is designed to calculate the current acoustic state V _t corresponding to the current speech portion of speech.

Block 5 is a device that stores a database of acoustic states.

Block 6 is a device in which a lexical decoding network is stored.

Block 7 is designed to calculate an estimate of the degree of coincidence b _j (V _t ) between the current acoustic state V _t and the reference acoustic state corresponding to a given vertex of the lexical decoding network j.

Block 8 is a device that stores a marker database.

Block 9 is designed to generate recognition results using the information stored in the marker from the end vertex at the end time.

Block 10 is a device that stores a database of recalculated tokens.

Block 11 is designed to normalize many markers.

Block 12 is intended for outputting recognition results.

Block 13 is designed to control the recognition system.

The operation of the word recognition system in continuous speech is as follows (see Fig. 10, 11 and 12). The input statement from the microphone is fed to the input of block 3 of the acoustic analyzer 1.

Block 3 using band-pass filters selects the frequency spectrum and converts it into digital form in accordance with the prototype. These digitized signals are fed to the input of block 4.

Block 4 calculates the current acoustic state V _t and simultaneously determines whether the current section of the signal contains speech in accordance with the prototype. After that, the current acoustic state is transmitted to the input of block 13 of the lexical analyzer.

, связанный с начальной вершиной сети лексического декодирования 0.Block 13, unlike the prototype, controls the operation of the lexical analyzer. Having received from the acoustic analyzer the current acoustic state, the control unit instructs block 7 to create a marker

associated with the initial vertex of the lexical decoding network 0.

, связанный с начальной вершиной сети лексического декодирования 0, и через блоки 10 и 11 передает его в блок 8. Далее блок 13 дает команду блоку 6 начать обход вершин сети лексического декодирования.Block 7, unlike the prototype, creates a marker

associated with the initial vertex of the lexical decoding network 0, and through blocks 10 and 11 transfers it to block 8. Next, block 13 instructs block 6 to begin bypassing the vertices of the lexical decoding network.

, связанного с данной вершиной. Также на вход блока 5 передается последовательность номеров состояний, связанных с вершинами, в которые система может перейти из текущей вершины.Block 6 goes on to consider the next vertex i of the lexical decoding network and passes the marker index to the input of block 8

associated with this vertex. Also, the sequence of state numbers associated with the vertices to which the system can go from the current vertex is transmitted to the input of block 5.

.Block 8 to the input of block 7 transmits the current marker

.

Block 5, having received a command from the control unit, transfers to the input of block 7 the next state associated with the vertex j of the lexical decoding network.

, полученного из блока 8, для состояния, связанного с вершиной j, полученного из блока 5, пересчитывая степень совпадения по формуле (1). Вновь созданный маркер

передается в блок 10, который является хранилищем созданных маркеров.Block 7 creates a copy of the marker

obtained from block 8, for the state associated with the vertex j obtained from block 5, recounting the degree of coincidence by the formula (1). Newly created marker

passed to block 10, which is the repository of the created tokens.

After the bypass of all the vertices of the lexical decoding network is completed, the control unit sends to the input of block 10 a signal to normalize the markers accumulated in block 10. For normalization, the markers are transmitted to block 11.

. Нормализованное множество маркеров передается на вход блока 8.Block 11 normalizes the set of markers, removing all the markers associated with one vertex of the lexical decoding network, except for the marker with the maximum degree of coincidence

. The normalized set of markers is transmitted to the input of block 8.

, связанного с конечной вершиной N сети лексического декодирования. Блок 8 передает в блок 9 данный маркер

, по которому блок 9 формирует результат распознавания в виде последовательности распознанных слов. Эта последовательность передается в блок 12, который отображает результат распознавания в удобном для оператора виде.This completes the processing of the current acoustic state V ₁ . The system moves to the next current acoustic state, V ₂ . The recognition procedure ends when the current speech section does not contain speech. In this case, the control unit sends a signal to the input of block 6 to complete the recognition procedure. Block 6 passes the marker index to block 8

associated with the final vertex N of the lexical decoding network. Block 8 transmits this marker to block 9

, according to which block 9 generates a recognition result in the form of a sequence of recognized words. This sequence is transmitted to block 12, which displays the recognition result in a convenient form for the operator.

Thus, the task of increasing recognition accuracy is achieved due to the fact that the search for a working hypothesis is optimal in the sense of the maximum degree of its coincidence with the original speech signal, since the algorithm used for moving the marker is based on the dynamic programming method, and also when constructing the network of lexical decoding, a model is used language and the probability of transition between states by which the average duration of phonemes is taken into account.

Information sources

1. Klatt D.H. Review of the ARPA Speech Understanding Project, J. Acoust. Soc. America, 62, No. 4, pp. 1366, 1977.

2. Woods W.A., Bates M., Brown G., et al. Speech Understanding Systems: Final Tech. Progress Report, Bolt, Beranek, Newman, Inc. Rep. No. 3438, Cambridge, 1976.

3. RF patent No. 2101782 C1, cl. 7 G10L 15/00.

4. Rabiner L. R. A. Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, proceedings of the IEEE, vol. 77, No. 2, February 1989.

5. Huang X., Acero A., Hon H.-W. Spoken Language Processing: a guide to theory, algorithms, and system development. - Prentice-Hall, Inc., 2001.

6. Mottl VV, Muchnik IB Hidden Markov models in the structural analysis of signals. - M .: FIZMATLIT, 1999.

Claims (4)

1. A method for recognizing words in continuous speech, consisting in the fact that with the utterance of a speech utterance, samples of the acoustic signal of this utterance digitized with a given quantization frequency are periodically taken at fixed time intervals and from the totality of these samples, the functional determining the current acoustic state is calculated at In this case, the obtained sequence of current acoustic states is used to restore the sequence of words (working hypothesis) spoken in the original speech showing what the lexical decoding network is used for, which sets the patterns for following reference acoustic states in a language, characterized in that a search for a working hypothesis is carried out, which is optimal in terms of the maximum degree of its coincidence with the original speech signal, which is ensured by the use of a moving marker algorithm, the working hypothesis is restored from the marker, which at this point in time is located at the final vertex of the lexical decoding network. 2. The method for recognizing words in continuous speech according to claim 1, in which a lexical decoding network is created based on a language model. 3. The method of recognizing words in continuous speech according to claim 1, in which when creating a network of lexical decoding, the probabilities of transition between states that specify the average duration of acoustic states are used. 4. The method for recognizing words in continuous speech according to claim 2, in which when creating a network of lexical decoding, the probabilities of transition between states that specify the average duration of acoustic states are used.

Images