JPH08202384A - Speech recognizing method and apparatus therefor - Google Patents

Abstract

PURPOSE: To form a directed graph at a-high speed with high accuracy and eventually to enable real time processing as well by consolidatively executing processing to recognize the system of recognition candidates and processing to form the directed graph in synchronization in time. CONSTITUTION: The arithmetic processing in an arithmetic means 13 is interrupted in mid-way for the recognition candidate system of low cumulative scores. A trellis forming means 12 executes commonage processing to successively integrate chains to the form of a graph in a manner as to prevent the formation of overlapped part trees in synchronization in time as well while developing the succeeding trellis to a tree form. The processing to express the intermediate results of recognition (series of the recognition candidates) as the graph of the trellis at each time executed in the trellis forming means 12, the processing to develop the trellis of the recognition candidates at each time to the trees, the processing to make the freshly added trellis common with the already developed trellis, etc., are progressed in synchronization in time with the arithmetic processing of the cumulative scores and start time at each point on the trellis executed by the arithmetic means 13.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a speech recognition method and apparatus for expressing recognition candidate sequences of phonemes, syllables, words, etc. obtained during or as a final result of recognition using a directed graph type data structure.

[0002]

2. Description of the Related Art In a voice recognition technology for automatically recognizing continuously uttered voices by a device, a process as a voice signal (voice process) and a process as a symbol (language process)
How to connect and is an important issue. Conventionally, a plurality of likely recognition candidates such as phonemes, syllables, and words are extracted from a continuous speech signal using a spotting technique, and a lattice of these recognition candidates is created, or a relatively low-order recognition candidate is created. A method of performing a process such as obtaining a sequence of a plurality of word candidates (N-best solution) using a linguistic constraint as a voice process and performing a higher-order linguistic process based on the result. It was mainly used.

However, in such a conventional method, in order to avoid the uncertainty of voice processing as much as possible, it is necessary to make the "N" of the lattice or N-best considerably large. However, there is a problem in that an excessive amount of overhead occurs when data is passed to the language processing. On the other hand, in recent years, a method of connecting speech processing and language processing using a directed graph type data structure has been proposed. Since the directed graph type data structure can represent a huge number of recognition candidate sequences such as phonemes, syllables, and words in a compact manner, it is possible to solve the above-mentioned problems in the conventional method.

FIG. 3 shows a directed graph representing a syllable recognition candidate sequence as an example of the directed graph. In general, a graph is defined as a set of nodes (nodes) and a set of arcs (branches) that connect two nodes, and a graph in which arcs are oriented is called a directed graph. When a directed graph type data structure is used as an interface between speech processing and language processing, a node usually has a time or a set of time and state as an attribute, and an arc has phonemes, syllables, words, etc. as recognition candidates. Labels (o, m
o, si, ro, i, # indicating silence, etc.) and its score (probability). The node that represents the beginning of each arc is called the in node of that arc, and the node that represents the end of each arc is called the out node. The node at the utterance start time is the start node, and the node at the utterance end (or utterance break) time. Is called an end node.

In a directed graph, by connecting adjacent arcs according to the direction of the arcs, it is possible to consider a path connecting the nodes. When the graph represents the result of speech processing, from the attributes of arcs and nodes, the sequence of recognition candidates in one path, the cumulative score for this series (sum of the scores of arcs on that path) and when The data of how long it exists exists uniquely. Particularly, the path connecting the start node and the end node is important as one speech recognition result, and when there are multiple such paths, by ordering each path by the cumulative score, the corresponding phoneme A valid recognition result can be obtained from a series of recognition candidates such as syllables and words.

Since the directed graph may have a tree shape as a special form, such a tree-shaped directed graph has a structure in which different arcs do not have a common out node. Also, as a special case of a directed graph, there may be only one path connecting the start node and the end node.

Heretofore, there have been proposed some methods for expressing the result of voice processing as a directed graph as described above. The first method is to recognize speech in word units and create a directed graph, and separate the process of generating a word hypothesis and the process of optimizing a directed graph into two steps. Then, the process of connecting the word hypotheses into a word string hypothesis, the process of finding the optimal boundary position between words, the process of grouping the word boundaries corresponding to the same time as one node, and the partial process including the same word candidate sequence. The process of combining the graphs into one is all performed in the optimization process. (M.Oerder and H.Ney, "Word graphs
: An efficient interface between continuous-speec
h recognitionand language understanding ", Proc. IC
ASSP-93, vol.II, pp.119-122)

The second method is to create a mora graph based on the mora (unit of language rhythm) extracted by the spotting technique, and each mora (half mora in the embodiment) is at a certain time. If it is assumed that the utterance ends, the cumulative score up to that time and the start time of the mora are calculated in time synchronization, and after the end of the utterance, the obtained data table is moved back toward the utterance start time. ,
A plurality of mora sequences that are likely to be recognized as recognition candidates are summarized as a graph-type data structure. (JP-A-5-26
No. 5483)

The third method is a three-step process for creating a directed graph in word units. First, after the end of the utterance, the cumulative score and its existence section are calculated when each phoneme starts at a certain time under the condition that a certain phoneme continues next in the direction of the start of the utterance.
Then, this time, from the beginning to the end of the utterance, using the knowledge of how phonemes are arranged in each word and the knowledge of connecting the words, asynchronously with time (synchronized with the words). Create a word-wise directed graph.
Finally, the directed graph is optimized in synchronization with the words from the end of the utterance toward the start. (P.Kenny, et a
l. "New graph search techniques for speech recognit
ion ", Proc. ICASSP-94, vol.I, pp.553-556)

[0010]

Since the directed graph type data structure can represent a huge number of recognition candidate sequences of phonemes, syllables, words, etc. in a compact form, it efficiently links speech processing and language processing. be able to. However, in the conventional method of expressing the result of voice processing as a directed graph, there is a problem that a large amount of processing is required to create a directed graph and rapid processing cannot be performed, and the accuracy of the created directed graph is insufficient. There was a problem of becoming.

That is, the first method is a process of connecting word hypotheses to form a word string hypothesis, a process of finding an optimum boundary position between words, and a process of combining word boundaries corresponding to the same time as one node. , The process of combining partial graphs containing the same word candidate sequence into one is all performed in the optimization process, so if the number of generated hypotheses increases, the optimization process load will increase. Was very heavy. Further, since the second method described above extracts the recognition candidate moras by the spotting technique, there is a risk that the detection accuracy of the boundary position between the respective moras becomes insufficient, and based on this, There is a possibility that the accuracy of the created directed graph may be insufficient.
Further, in the above third method, in order to improve the detection accuracy of the boundary position between the recognition candidates, the amount of processing after the utterance has to be considerably increased.

The present invention has been made in view of the above-mentioned conventional circumstances, and provides a voice recognition method that enables a directed graph as a voice processing result to be created at high speed and with high accuracy, and in turn enables real-time processing. The purpose is to do. Further, the present invention has been made difficult in the past by adding the score of arc, the score of the optimum (highest score) path from each node to the start node and the end node to the data structure. An object of the present invention is to provide a speech recognition method that enables high-speed and high-accuracy recognition even for linguistically diverse utterances including a large amount of vocabulary and unknown words. Another object of the present invention is to provide a voice recognition device for implementing such a voice recognition method.

[0013]

In order to achieve the above object, according to the present invention, a process of generating a recognition candidate of a phoneme, a syllable, a word, etc., and a recognition candidate among the processes of creating a directed graph type data structure are performed. The process of connecting sequences of phonemes, syllables, words, etc. and their scores, the process of optimizing the boundary position between recognition candidates, the boundary position corresponding to the same time (or a set of time and state) as one node The processing to be summarized in 1) is performed in an integrated manner by synchronizing in time. That is, in the present invention, most of the processing for creating the directed graph type data structure is performed by time-synchronized processing.

Further, in the present invention, after the directed graph is created, the backward processing is performed to fix the directed graph.
That is, after detecting the utterance break (or the end of the utterance), the backtracking process is performed, and the most probable one is selected from those obtained by the process of synchronizing the boundary position between the recognition candidates with the time. Determine. Then, the score of the arc corresponding to each recognition candidate of the directed graph is obtained by a simple calculation from the boundary position between the confirmed recognition candidates and the cumulative score of the recognition candidate series obtained when the directed graph is created.

That is, the voice recognition method according to claim 1 recognizes a voice signal as a sequence of recognition candidates such as phonemes, syllables, and words, and expresses the middle or final result thereof using a directed graph type data structure. In the method, the process of recognizing the series of recognition candidates and the process of generating the digraph are performed in a synchronized manner in time.

The speech recognition method according to claim 2 is the speech recognition method according to claim 1, wherein a trellis for each recognition candidate is concatenated using a hidden Markov model in the process of generating the directed graph, and the recognition is performed. The process of recognizing a sequence of candidates is characterized in that the start time of the recognition candidate indicated by the trellis and the cumulative score along the chain of the trellis are obtained.

A speech recognition method according to a third aspect is the speech recognition method according to the first or second aspect, in which the recognition processing of a sequence of recognition candidates and the generation of a digraph are performed in an integrated manner in synchronization with time. After processing and process reach the boundary of utterance,
The data structure of the directed graph is characterized by sequentially determining nodes between recognition candidates of the directed graph and performing score calculation for each arc in a backtracking process from the punctuation side of the directed graph to the utterance start side.

The speech recognition method according to claim 4 is the speech recognition method according to claim 3, which is a recognition candidate treated as a single recognition candidate in the process of the processing which is performed in a synchronized manner temporally. Even if there is a plurality of candidates having different existence intervals in the process of the backtracking, the data structure of the directed graph is determined by using these candidates as different recognition candidates.

The speech recognition method according to claim 5 is the speech recognition method according to claim 3 or 4, wherein, in the process of the backtracking, the optimum path from the node corresponding to the utterance start to each node in the directed graph. And the score of the optimum path from each node in the digraph to the node corresponding to the utterance break, and these scores are held in a digraph-type data structure.

A speech recognition method according to claim 6 is the speech recognition method according to claim 5, wherein the score of the optimum route is
It is characterized in that it is obtained by the acoustic likelihood of a sequence of recognition candidates, or a combination of the acoustic likelihood and a linguistic likelihood concerning a chain of recognition candidates.

A speech recognition apparatus according to claim 7 recognizes a speech signal as a series of recognition candidates such as a phoneme, a syllable, and a word, and configures the middle or final result thereof as directed graph type data. Acoustic analysis means for analyzing the input voice signal to obtain a characteristic parameter series,
A model holding unit that holds a model related to an acoustic model and a chain of acoustic models, a trellis creating unit that generates a trellis corresponding to a recognition candidate by using the model for a feature parameter sequence, and a cumulative score and a start time for the trellis. Calculating means for calculating in synchronization with time, graph data storing means for storing the cumulative score and start time corresponding to the trellis, and recognition based on the starting time and cumulative score stored in the graph data storing means Backtracking processing means for determining a node between candidates and performing score calculation for each arc corresponding to a recognition candidate, wherein the trellis creating means further sequentially synchronizes subsequent trellis based on the calculation result of the calculating means. The backtracking processing means creates the cumulative score and the start time until the break of the utterance. Performing the process at which is stored in the storage means, characterized in that stores the processing result in the graph data storage means.

[0022]

Operation: The process of creating a directed graph data structure is
Phonemes, syllables, processing to generate recognition candidates such as words and their scores, processing to connect recognition candidates and their scores, processing to find the optimal boundary position between recognition candidates, same time (or time The process can be roughly divided into a process of collecting boundaries corresponding to a set of states) as one node and a process of collecting partial graphs representing the same series into one node. Processes are integrated in synchronization with time.

From the acoustic characteristics of a voice signal, phonemes, syllables,
A method based on an HMM (Hidden Markov Model) is generally used in order to recognize a word or the like as a recognition candidate and obtain the likelihood (score) of the recognition candidate and its sequence. The HMM method uses a two-dimensional work space called a trellis, which has a model state and time as two axes for each recognition candidate such as a phoneme, a syllable, and a word. In particular, in continuous speech recognition in which a speech signal is recognized as a series of recognition candidates, these trellis are connected to each other and chained together to set a large work space expressing the series of recognition candidates.

According to the second aspect of the present invention, the cumulative score from the start of utterance at each point (time, trellis number, state) on the trellis, and the recognition candidates (labels such as phonemes, syllables, and words indicated by the trellis). ) Is performed along the trellis chain to create a directed graph. The calculation process of the cumulative score and the start time is advanced in time synchronization between the trellis sequences. If the trellis are connected to each other,
In the process of this processing, the boundary position between the recognition candidates indicated by each trellis is optimized. Further, by comparing the cumulative score values in the final state of each trellis chain at the utterance end time, a series of probable recognition candidates can be obtained as a recognition result.

The trellis graph obtained by the above-mentioned processing is still insufficient to make it the result of the voice processing. For example, the generated graph may include a trellis whose cumulative score at the utterance end time is considerably low, and the score of each arc corresponding to the trellis is undetermined. Therefore, in the invention of claim 3, after detecting the utterance delimiter (end of utterance), the backward processing is performed to determine the data structure of the directed graph type. In the backtracking process, the boundary position (node) between recognition candidates
Is determined and the score of each arc is calculated and summarized as a directed graph type data structure.

Even in the case of a single trellis (recognition candidate) in the process synchronized with time, a plurality of recognition candidate sequences having different existence intervals may appear in the process of backtracking. According to the invention of claim 4, in such a case,
In order to correctly evaluate the scores of these recognition candidate series,
Determine a directed graph with these trellises as separate arcs.

According to the invention of claim 5, in the above-mentioned backtracking process, the score of the optimum route from the node corresponding to the utterance start of the directed graph to each node, and the node from each node to the node corresponding to the utterance break. The scores of the optimum routes are obtained, and these scores are held in a directed graph type data structure for use in later language processing. Also,
In the invention of claim 6, the scores of these optimum routes are obtained by reflecting the linguistic knowledge about the chain between the recognition candidates. Since the amount of data calculated by the above-described backtracking process is usually much smaller than the amount of data for the process of creating a directed graph in time synchronization, it is possible to end the process in time synchronization during utterance. If possible, it is possible to obtain the fixed directed graph data as the recognition result almost at the same time as the utterance break.

In the speech recognition apparatus according to the seventh aspect, a trellis corresponding to the recognition candidate is created, and the cumulative score and start time for this trellis are calculated in time synchronization. In addition, a plurality of trellis corresponding to subsequent recognition candidates are connected to this trellis in synchronization with time, and a trellis sequence chained in a graph or tree shape is sequentially formed corresponding to the recognition candidates. With respect to these trellis sequences, the calculation of the cumulative score and the start time is similarly performed in time synchronization. Therefore, the process of setting the trellis sequence corresponding to the recognition candidate sequence and the process of generating directed graph type data corresponding to the trellis sequence are generally time-synchronized. Then, each data thus obtained is stored in the graph data storage means,
When the above processing reaches the utterance break corresponding to the end of utterance, etc., based on the stored start time and cumulative score, the node between recognition candidates is determined, and the score calculation for each arc corresponding to the recognition candidate is returned. It is obtained by processing and the directed graph type data structure is determined.

[0029]

An embodiment of the present invention will be described with reference to the drawings.
In this embodiment, the HMM method is used, and the acoustic model and the recognition unit are syllables. First, a voice recognition apparatus according to this embodiment will be described with reference to FIGS. 1 and 2.
The speech recognition apparatus of this embodiment is roughly divided into an input unit 1 for inputting a voice signal, an acoustic analysis unit 2 for analyzing a voice signal from the input unit 1 to obtain a characteristic parameter sequence, and an acoustic analysis. A recognition graphing means 3 for recognizing the feature parameter series from the means 2 as a syllable recognition candidate series and collecting the recognition result in a directed graph type data structure, and a graph data storage means for storing the recognition result by the recognition graphing means 3. 4 and a backtracking processing means 5 for performing processing for detecting the utterance delimiter and determining the data structure of the recognition result stored in the graph data storage means 4. The obtained directed graph type recognition result is subjected to language processing by the language processing means 6 based on grammatical constraints and the like, and is output.

The recognition graphing means 3 includes a model holding means 11 holding a model relating to an HMM and a chain of acoustic models as an acoustic model of a syllable, and an HMM of the model holding means 11 for the characteristic parameter series from the acoustic analysis means 2.
Is provided with a trellis creating means 12 for setting a trellis corresponding to a recognition candidate by using, and a calculating means 13 for calculating a cumulative score and a start time for the set trellis in time synchronization. The recognition results such as the cumulative score and the start time corresponding to each obtained trellis are sequentially stored in the graph data storage means 4.

The trellis creating means 12 sets a trellis indicating silence at the utterance start time and then sequentially sets a trellis indicating subsequent recognition candidates in synchronization with the input of the voice signal (feature parameter series). At this time, the trellis creating unit 12 uses the calculation result of the calculating unit 13 to cause a new trellis to follow the trellis corresponding to the recognition candidate sequence having a high cumulative score in a tree shape or a graph shape, while having a low cumulative score. No further trellis chain is developed for the recognition candidate sequence. That is, for the recognition candidate series having a low cumulative score, the calculation processing by the calculation means 13 is terminated halfway. Further, the trellis creating means 12 expands the subsequent trellis into a tree shape, and at the same time, performs a sharing process of collecting the chains in the form of a graph so that overlapping partial trees are not generated. To do.

These trellis creating means 12 carry out:
The process of expressing the intermediate result of recognition (series of recognition candidates) as a graph of the trellis at each time, the process of expanding the trellis of the recognition candidates at each time into a tree, and sharing the newly added trellis with already expanded ones. The conversion processing and the like are carried out in time synchronization with the calculation processing of the cumulative score and the start time at each point on the trellis performed by the calculation means 13.

The backtracking processing means 5 determines the nodes between the recognition candidates based on the directed graph type data stored in the graph data storage means 4 and calculates the score for each arc corresponding to the recognition candidates. A trellis searching means 15 for searching a trellis (arc), a node determining means 16 for determining a predetermined node, and a calculating means 17 for performing score calculation for each arc are provided. That is, in the backtracking processing means 5, when the utterance break is reached, the trellis search means 15 searches for an arc corresponding to the utterance break, and the node deciding means 16 decides the in-node,
The trellis search means 15 searches for an arc having the in-node as an out-node in the direction of utterance start, and the node deciding means 16 decides the in-node of the searched arc until the utterance start node is repeated. . Then, in synchronization with this processing, the calculation means 17 calculates the score or the like of each arc, and the calculation result is stored in the graph data storage means 4.

Here, H used as an acoustic model of a syllable
The MM (Hidden Markov Model) will be explained. FIG. 4 shows an example of the syllable HMM, and states 1 to 3 of the HMM.
Represents the transition as indicated by the solid arrow. The HMM operates so as to repeat transitions between states and acceptance of characteristic parameters of a voice signal. The state transition and the acceptance of the characteristic parameter are probabilistic, and the state transition probability a ^m _{i, j}
And the probability of accepting the feature parameter b ^m _j (O _k ). The HMM is set for each syllable, and the value of the probability of each syllable HMM is previously obtained from the learning voice data so as to best accept the characteristic parameter series of the corresponding syllable. The unknown syllable can be recognized by finding the syllable HMM that best accepts the characteristic parameter sequence.

In the above symbols and symbols in FIG. 4, O
_i is the characteristic parameter sequence of the audio signal (where i =
1, 2, ..., I), a ^m _{i, j} are logarithmic values of the probability of transition from the state i to the state j in the HMM of the syllable m (here, i =
1, 2, ..., S, j = 1, 2, ..., S + 1, m
, 1, 2, ..., M, J = S + 1 corresponds to the transition to the next syllable), b ^m _j (O _k ) is H of syllable m.
The logarithmic value of the probability of accepting the feature parameter O _k in the state j in MM (where j = 1, 2, ..., S, m =
1, 2, ..., M, k = 1, 2, ..., I), Sy
lLabel (j) represents the syllable indicated by trellis j, respectively.

Next, the recognition and graphing processing of the syllable string performed by the recognition graphing means 3 will be described. First, the recognition and graphing process starts by setting a trellis for a syllable that can occur at the start time of utterance, but in the present embodiment, the utterance is treated as a section sandwiched by silence (#), and first, Set the trellis to indicate silence. And
Variables used in the following arithmetic processing are initialized, and trellis number j = 0 ... (Equation 1), SylLabel (j) = “#” ... (Equation 2), AccumScore (i, j, k) = {0.0: (i, j, k) = (0, 0, 0), -∞: otherwise} (Equation 3), InitFrame (i, j, k) = { 1: (i, j, k) = (0,0,0), −1: other than that} (Equation 4).

Note that AccumScore (i, j,
k) is a cumulative score (Viterbi score) from the matching start point (frame 0, trellis number 0, state 0) to (frame i, trellis number j, state k), In
itFrame (i, j, k) is the start frame of the syllable SylLabel (j) indicated by trellis j when viewed from a certain point (frame i, trellis number j, state k),
Respectively. Here, the frame is the time when the characteristic parameter is extracted.

That is, in the initial state, the cumulative score (probability of logarithmic display) is set to "0" in the expression 3, otherwise it is set to error (-∞), and the start frame (start time) is set to "1" in the expression 4. , Otherwise, set to error (-1).

Next, the syllable for which the trellis is set and the characteristic parameter series are collated by the Viterbi search. Generally, frame i = 1, 2, ...
., I, trellis number j = 1, 2, ..., M, state k
= 1, 2, ..., S, the following equations 5 and 6 are calculated. Here, since only the trellis of silence is set by the above, j = 0 and the silence of the silence is set. The calculations of Equations 5 and 6 are performed on the trellis.

AccumScore (i, j, k) = max {AccumScore (i-1, j, k-1) + a ^{SylLabel (j)} _{k-1, k} , AccumScore (i-1, j, k) + a ^{SylLabel ( j)} _{k, k} } + b ^{SylLabel (j)} _k (O _i ) ... (Formula 5), InitFrame (i, j, k) = InitFrame (i−1, j, k−1) or InitFrame (i−) 1, j, k) (Equation 6), where AccumScore (i, j, k) = Accu
mScore (i-1, j, k-1) + a ^{SylLabel (j)}
_{In the case of k-1, k} + b ^{SylLabel (j)} _k (O _i ), InitFr
ame (i, j, k) = InitFrame (i-1,
j, k-1), AccumScore (i, j, k) = AccumSc
ore (i-1, j, k) + a ^{SylLabel (j)} _{k, k} + b
^{In the case of SylLabel (j)} _k (O _i ), InitFrame
(I, j, k) = InitFrame (i-1, j,
k).

For example, in trellis j shown in FIG.
AccumScore (3, j, 2) and InitFr
When determining ame (3, j, 2), the point (3, j,
In 2) AccumScore (2, j, 1) + a
^{SylLabel (j)} _{1, 2} > AccumScore (2, j, 2)
+ A ^{SylLabel (j)} _2,2 , AccumScore (3, j, 2) = AccumSc
ore (2, j, 1) + a ^{SylLabel (j)} _{1, 2} + b
^{SylLabel (j)} ₂ (O ₃ ), InitFrame (3, j, 2) = InitFrame
e (2, j, 1). That is, AccumSc is obtained by sequentially accumulating scores along a path having a high state transition probability and characteristic parameter acceptance probability (thick arrow line in FIG. 5).
ore is calculated, and this processing is performed up to the final point (6, j, 3) of the trellis j.

The syllable matching process as described above is performed by setting the value (AccumScor) at each point of the set trellis.
e (i, j, k) and InitFrame (i, j,
It is possible to proceed by obtaining k)) in synchronization with time. Therefore, after a certain amount of time, the score A of the route from the matching start point to the final point (i, j, S) of the silent HMM
ccumScore (i, j, S) becomes high.

When the score at the final point of the trellis showing silence is thus high, as shown in FIG. 6, the trellis showing silence (j = 0) has the next syllable (“X” and “X”) as recognition candidates. Connect the trellis (j = 1, 2) indicating Y ″). In general, many recognition candidates are considered,
The number of connected trellis (M) is set accordingly, but in FIG. 6 there are two syllables (“X” and “Y”) for simplicity.
Only is shown.

Then, the score and the start frame of the path to the final point on all the increased trellises are calculated in the same manner as in the equations (5) and (6). However, when performing syllable matching on the newly connected trellis, AccumSc
ore (i, j, k) and InitFrame (i, j,
The value of k) should reflect the value in the previous trellis. In the case shown in FIG. 6, the value in the initial state (k = 1) of the newly connected trellis is set to the frame i =
1, 2, ..., I, trellis number j = 1, 2, ...
., M may be calculated based on Equations 7 and 8. It should be noted that the states k = 2, ..., S are calculated in the same manner as Equations 5 and 6.

AccumScore (i, j, k) = max {AccumScore (i−1,0, S) + a ^{SylLabel (j)} _{S, S + 1} , AccumScore (i−1, j, k) + a ^{SylLabel (j) _{k, k} + b SylLabel (}} j) k (O i) ··· ( wherein 7), InitFrame (i, j , k) = i or InitFrame (i-1, j, k) ··· ( equation 8) , Where AccumScore (i, j, k) = Accu
mScore (i-1,0, S) + a ^{SylLabel (j)} _{S, S + 1}
+ B ^{SylLabel (j)} _k (O _i ) then InitFrame
e (i, j, k) = i, AccumScore (i,
j, k) = AccumScore (i-1, j, k) +
In the case of a ^{SylLabel (j)} _{k, k} + b ^{SylLabel (j)} _k (O _i ), InitFrame (i, j, k) = InitFrame
e (i-1, j, k).

When a linguistic score (transition probability between syllables) is added to the acoustic score, equations 7 and 8 are used.
In addition to the above, the calculations of Equations 9 and 10 may be performed for trellis numbers j = 1, 2, ..., M. LangScore (0) = 0 ... (Equation 9), LangScore (j) = LangScore (0) + SylBigram (SylLabel (0), SylLabel (j)) ... (Equation 10) As a result, the cumulative score is AccumScore. (I,
j, k) + wLangScore (j). Here, LangScore (j) is the linguistic score from the first syllable to the syllable SylLabel (j) indicated by trellis j, SylBigram (i, j) is the probability of transition from syllable i to syllable j, and w is the language. Each of the weights assigned to the dynamic scores.

When the trellis is set and chained for each recognition candidate as described above and the process of calculating the score and the start frame is continued, a tree-shaped trellis corresponding to the syllable chain is formed as shown in FIG. It Incidentally, in the process of extending the branches of the tree, the trellis on the tree corresponding to the syllable string having a low recognition score may be pruned to remove it from the target of the subsequent matching process.

In the process of successively chaining subsequent trellis in a tree-like manner in this way, even when the trellis has a sufficiently high score in the final state and no subsequent trellis is connected to this trellis, a new trellis is created. May not be set after it. For example, in a tree of trellis that has already been generated, the syllable represented by the trellis that has already connected the following trellis is the same as the syllable represented by the trellis that has not yet connected the following trellis, and these syllables are If it can be assumed that they start at the same time, the latter trellis is not followed by a new trellis. Then, the trellis succeeding the latter trellis is commonly connected to the trellis succeeding the former trellis, and the trellis to be subjected to the arithmetic processing is reduced to speed up the processing.

For example, consider a case in which after the chain structure shown in FIG. 7, the score in the final state of the trellis numbered 6 at a certain frame i is sufficiently high. Normally, as shown in FIG. 8, the trellis (number 9 '
~ 11 ') are newly set. However, the syllable (SylLabe) indicated by the trellis following the trellis with the number 6
The trellis with the number 3 indicating the same syllable "X" as that of l (6)) has already been set, and the trellis with the number 3 has the following trellis (numbers 9 to 11). In addition, the start frame Ini of the syllable indicated by the trellis number 3
tFrame (i, 3, S) and the start frame of the syllable indicated by the trellis number 6 InitFrame (i, 6,6)
S) is equal to. In such a case, the trellis with the number 6 is the trellis following the trellis with the trellis (numbers 9 to 1) following the trellis with the number 3 as shown in FIG.
As shown in 1), the trellis of number 3 and the trellis of number 6 share the succeeding trellis.

Such common processing is carried out at any time when a predetermined condition is satisfied, whereby the form of the trellis chain dynamically changes during the recognition processing. It should be noted that as a common condition, it is possible to use only the same time property of the syllable without requiring both the same time property of the syllable and the same time property of the syllable.

Here, when the subsequent trellis is shared as described above, the syllable matching process must be slightly modified. In the usual case where the trellis is expanded into a tree, there is only one trellis preceding each trellis,
When the subsequent trellis is shared, there will be a plurality (n) of trellis preceding the shared trellis. Therefore, specifically, when the number of preceding trellis is n (= 1, 2, ..., N), the value of the commonized trellis j in the initial state is set to frame i = 1, 1.
2, ..., I, state k = 1, equation 11 and equation 1
The calculation may be performed based on 2, and the maximum score of the preceding trellis may be inherited.

AccumScore (i, j, k) = max {max {AccumScor e (i-1, n, S)} | ^N _{n = 1} + a ^{SylLabel (j)} _{S, S + 1} , AccumScore (i-1, j, k) + a ^{SylLabel (j)} _{k, k} } + b ^{SylLabel (j)} _k (O _i ) ... (Equation 11), InitFrame (i, j, k) = i or InitFrame (i-1, j, k) (Equation 12), where AccumScore (i, j, k) = max
{AccumScore (i-1, n, S)} | ^N _{n = 1} +
In the case of a ^{SylLabel (j)} _{S, S + 1} + b ^{SylLabel (j)} _k (O _i ), InitFrame (i, j, k) = i, Accum
Score (i, j, k) = AccumScore (i
-1, j, k) + a ^{SylLabel (j)} _{k, k} + b
^{In the case of SylLab el (j)} _k (O _i ), InitFrame
(I, j, k) = InitFrame (i-1, j,
k). Note that max {AccumScore} |
^N _{n = 1} is the maximum AccumScore between n = 1 and N
Indicates.

The syllable matching process and the graphing process for chaining the trellis as described above proceed in synchronization with the frame until the utterance break is detected, and the directed graph formed by the last frame of the utterance is, for example, as shown in FIG. As shown in FIG. Then, in the graph data storage means 4, the number of each trellis, the number of the trellis preceding the trellis, the recognition candidate name (label) SylLabel, and the cumulative score AccumScor obtained by the series of processes described above.
e, start frame InitFrame, and the like are stored in association with each other. However, even the graph of the trellis formed in this way has the result of the audio processing such as including a part where the cumulative score is considerably small, the score of each arc corresponding to each trellis is not found, and the like. There are some points that are not enough.

Therefore, based on the data of the formed directed graph, the backward processing means 5 deletes an extra part of the graph, determines the boundary position (node) between the recognition candidates and calculates the arc score, and further, The score of the optimum route between each node and the start node and the end node is calculated, and the data structure as a directed graph is fixed.

The symbols used in the backtracking process described below will be explained. TrellisID (p) is the trellis number corresponding to arc p, ArcLabel (p).
Is the syllable name indicated by arc p, ArcInNode (p)
Is the in-node of arc p, ArcoOutNode
(P) is an out node of Arc p, ArcScore
(P) is the score of arc p, NodeTime (q)
Is the frame indicated by the node q, FwScore (q)
Is the score of the optimal path from the start node to node q,
BwScore (q) represents the score of the optimum route from the node q to the end node.

The backtracking process begins by setting the end node (number = 0) of the directed graph as shown in FIG. In this node, BwScore
(0) = 0, which is stored in the graph data storage means 4. Next, a trellis having this end node as an out node is selected from the formed directed graph of the trellis. In this selection process, for all the trellis j set in the final frame I, the one with a high cumulative score AccumScore (I, j, S) in the final state S may be selected. When the utterance is divided by silence, a constraint that the syllable indicated by the trellis is silence may be further added.

Then, the syllable label indicated by the selected trellis is inherited as the attribute of the arc, the in-node and the out-node of the arc are set, and the value of the frame which is the attribute of the in-node is set. Specifically, if the number of the selected N trellis is j, the number of the corresponding N arcs is p = 0, 1, ... N-1, and the in-node of each arc is q (= p + 1). , For example: TrellisID (p) = j, ArcLabel (p) = SylLabel (j)
(= “#”), ArcInNode (p) = q, ArcOutNod
e (p) = 0, NodeTime (q) = InitFrame (i,
j, S),

As a result, as shown in the example of FIG. 11, the arcs are connected to the end nodes, and the Trel corresponding to each arc is
lisID, ArcLabel, ArcInNode,
ArcOutNode is stored in the graph data storage means 4. That is, when the arc is determined in this manner, the in-node and the like corresponding to the arc are also determined. In the example shown in FIG. 11, only the trellis numbered 13 and 16 in the trellis chain shown in FIG. 10 are selected.

Since the arc connecting to the end node is fixed, the maximum value of the score of the path from the start node to the end node can be fixed, and this score FwScore (0) is given by Calculate based on FwScore (0) = max {AccumScore
(I, TrellisID (p), S)} | ^N-1 _{p = 0} , that is, the one having the largest cumulative score of the N arcs is FwScore (0). In addition, FwSc
Since the value of ore (0) has already been obtained in the process of creating the directed graph described above, it may be set when the utterance break is detected.

When a linguistic score is added to the acoustic score, the value of FwScore (0) can be calculated by the following equation, for example. FwScore (0) = max {AccumScore
(I, TrellisID (p), S) + wLangS
core (TrellisID (p))} | ^N-1 _{p = 0} ,

Next, from the in-nodes already determined by the arc determination, one that has not become an out-node is selected, and a trellis having this as an out-node is obtained. In the example shown in FIG. 11, for example, the node with number 1 is selected, and the trellis with numbers 11 and 14 are obtained from the corresponding trellis graph shown in FIG.

Then, the obtained trellis is subjected to the same processing as above to determine the arc and the node. Specifically, the number of the selected node is q,
The number of arcs and nodes confirmed so far is L
And K, an arc having a node q as an out node is r = L,
L + 1, ..., L + N-1, the in node of this arc is s = K, K + 1, ..., K + N-1, and the corresponding trellis is j. TrellisID (r) = j, ArcLabel (r) = SylLabel (j), ArclnNode (r) = s, ArcOutNod
e (r) = q, NodeTime (s) = InitFrame (Nod
eTime (q) -1, TrellisID (r),
S),

As described above, since the arc having the node q as the out node is obtained, the attribute of the arc p having the node q as the in node and the attribute other than the frame of the node q can be set. . That is, the score ArcScore (p) of the arc p and the maximum value FwSco of the score of the route (path) from the start node to the node q.
Re (q), the maximum value BwScore (q) of the score of the path (path) from the node q to the end node, is calculated by an operation based on the following equation and stored in the graph data storage means 4. FwScore (q) = max {AccumScore
(NodeTime (q) -1, TrellisID
(R), S)} | ^N _{r = 1} , ArcScore (p) = AccumScore (No
deTime (ArcOutNode (p))-1, T
rellisID (p), S) -FwScore
(Q), BwScore (q) = ArcScore (p) + Bw
Score (ArcOutNode (p)) If these attributes are already set (described later, when different arcs have the same in-node), it is not necessary to set them again as described above.

When the linguistic score is added to the acoustic score, FwScore (q) and BwScor
The values of e (q) are as follows. FwScore (q) = max {AccumScore
(NodeTime (q) -1, TrellisID
(R), S) + wLangScore (Trellis
ID (r))} | ^N _{r = 1} , BwScore (q) = ArcScore (p) + wS
ylBigram (SylLabel (Trellis
ID (r)), SylLabel (TrellisID
(P))) + BwScore (ArcOutNode
(P)),

Here, in the above-described backtracking process, the trellis corresponding to the arc for which the confirmation process is to be performed and the arc corresponding to the trellis developed at the same time in the process of creating the directed graph are already confirmed as the arc. Sometimes. In such a case, under the condition that the frame corresponding to the arc inode already determined and the frame corresponding to the arc innode to be determined are the same, the arc innode to be newly determined is already determined. It must be the same as the arc's innode.

That is, when the condition is satisfied, FIG.
As shown in, the in node of the arc of number 3 (corresponding trellis number is 14) connected to the node of number 1 is made equal to the in node of arc number 1 (corresponding trellis number is 13). The above-described arc and node determination processing proceeds while unifying such in-nodes, and as shown in FIG. 13, the arcs and nodes are sequentially determined toward the utterance start direction.

On the other hand, when the above conditions are not satisfied even with trellises developed simultaneously (that is, when the syllable starts in a different frame), different innodes are set. For example, the arc connected to the node of number 4 shown in FIG. 13 corresponds to the trellis of number 2 from the trellis graph of FIG. 10, and the trellis of number 2 is at the same time as the trellis of number 0 to number 1 in the graph creation process. It has been deployed. Therefore, in the normal confirmation process, an arc as shown by the broken line in FIG. 14 is generated. However, the start time of the syllable indicated by these trellises (number 1 and number 2) InitFrame (No
deTime (5) -1,1, S) and InitFrame
When e (NodeTime (4) -1, 2, S) is not equal, the innodes (number 6 and number 7) of each arc (number 7 and number 8) are divided into two as shown in FIG. Perform confirmation processing.

When the nodes are divided into two in this way, the arc having these nodes as the out nodes is also determined by dividing the corresponding trellis. That is,
Even in the case of treating as a single trellis (recognition candidate) in the process of graph creation processing that is performed synchronously in time, arcs (recognition candidates) whose existence sections are different in the process of backtracking are recognized. When there are a plurality of arcs, as shown in FIG. 16, these arcs are defined as separate arcs and the data structure of the directed graph is determined. By dividing the arc into a plurality of arcs according to the possible recognition candidates in this way, the scores of syllable strings having different syllable boundaries can be correctly evaluated.

Since the arc and node determination processing by the above-mentioned backtracking processing is performed by selecting the route with a high score, the directed graph determined as a result is only the portion with a high score corresponding to the trellis shown by the bold line in FIG. ,
Furthermore, since the score of the arc and the score of the optimum route are also obtained in this confirmation processing, the subsequent language processing etc. can be quickly performed even for linguistically diverse utterances including a large amount of vocabulary and unknown words. be able to.

In the above embodiment, an example in which a syllable is used as a recognition candidate unit is shown, but the present invention can be applied to a case where a phoneme or a word is used as a recognition candidate unit. In addition, although the example using the HMM method is shown in the above embodiment,
In the present invention, other recognition methods such as the DP matching method and the neural network method can be used. In this case, instead of the trellis of the HMM method, a standard pattern or a work space having two axes of unit element (neuron) and unknown voice is set, and the probability (maximum value) calculation is related to the distance (minimum value). It is sufficient to make changes such as replacement with (value) calculation or (maximum value) calculation regarding element output.

[0071]

As described above, according to the present invention,
High-speed and high-accuracy speech recognition can be performed even for linguistically diverse utterances including a large amount of vocabulary, unknown words, etc., which have been difficult problems in the past. In particular,
According to the invention of claim 1 or 2, since the process of recognizing the sequence of recognition candidates and the process of generating the directed graph are performed in a timely synchronized manner, the directed graph can be processed at high speed and with high accuracy. It can be created for real-time processing.

According to the third aspect of the present invention, in addition to the above effect, the node between the recognition candidates of the digraph is determined and the score calculation for each arc is performed by the backtracking process to determine the data structure of the digraph. Therefore, it is possible to obtain a highly useful directed graph in which the extra part is deleted. According to the invention of claim 4, in addition to the above effect, even if a recognition candidate treated as a single recognition candidate in the process of creating a directed graph can have different candidates in the process of the backtracking process. Uses the candidates as separate recognition candidates to determine the data structure of the directed graph, so that the scores of the recognition candidate sequences having different boundaries can be correctly evaluated.

Further, according to the invention of claim 5, in addition to the above effect, the score of the optimum route is obtained in the process of the backtracking process, and these scores are held in the data structure of the directed graph type. It is possible to realize the dynamic processing and the like with high speed and high accuracy. According to the invention of claim 6, in addition to the above effects, the score of the optimum route is the acoustic likelihood of the sequence of recognition candidates, or a combination of the acoustic likelihood and the linguistic likelihood of the chain of recognition candidates. Therefore, the subsequent linguistic processing and the like can be realized at high speed and with high accuracy according to various conditions. According to the invention of claim 7, the voice recognition method can be implemented to obtain the above-mentioned useful effects.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a voice recognition device according to an embodiment of the present invention.

FIG. 2 is a configuration diagram of a voice recognition device according to an embodiment of the present invention.

FIG. 3 is a conceptual diagram showing an example of a syllable directed graph.

FIG. 4 is a conceptual diagram showing an example of a syllable HMM.

FIG. 5 is a conceptual diagram illustrating the calculation of trellis and HMM score calculation.

FIG. 6 is a conceptual diagram showing how a trellis chain evolves with time.

FIG. 7 is a conceptual diagram showing how a trellis chain evolves with time.

FIG. 8 is a conceptual diagram showing how a trellis chain evolves with time.

FIG. 9 is a conceptual diagram showing how a trellis chain evolves with time.

FIG. 10 is a conceptual diagram showing how a trellis chain evolves with time.

FIG. 11 is a conceptual diagram illustrating the progress of confirmation of a directed graph.

FIG. 12 is a conceptual diagram illustrating a process of establishing a directed graph.

FIG. 13 is a conceptual diagram illustrating a process of finalizing a directed graph.

FIG. 14 is a conceptual diagram illustrating a process of finalizing a directed graph.

FIG. 15 is a conceptual diagram illustrating a process of finalizing a directed graph.

FIG. 16 is a conceptual diagram illustrating the progress of confirmation of a directed graph.

FIG. 17 is a conceptual diagram illustrating a trellis chain determined as a directed graph.

[Explanation of symbols]

2 acoustic analysis means, 3 recognition graphing means, 4 graph data storage means, 5 backtracking processing means, 11 model holding means, 12 trellis creating means, 13 computing means, 15 trellis searching means, 16 node determining means, 17 computing means,

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location G10L 9/10 301 C

Claims (7)

[Claims] 1. A speech recognition method for recognizing a speech signal as a sequence of recognition candidates such as phonemes, syllables, words, etc., and expressing the middle or final result thereof by using a directed graph type data structure. A voice recognition method, characterized in that the recognition process and the process for generating the directed graph are integrated in a timely synchronized manner. 2. In the process of generating the directed graph, a hidden Markov model is used to create a trellis for each recognition candidate in a chained manner, and in the process of recognizing a sequence of the recognition candidates, the start time and trellis of the recognition candidate indicated by the trellis. The speech recognition method according to claim 1, wherein a cumulative score along the chain is obtained. 3. After the recognition processing of the recognition candidate sequence and the generation processing of the directed graph, which are integrated in synchronization with each other in time, reach the boundary of utterances, a node between the recognition candidates of the directed graph is determined and an arc is generated. The speech recognition method according to claim 1 or 2, wherein the score calculation for each is performed sequentially in a backtracking process from the utterance delimiter side of the digraph to the utterance start side to determine the data structure of the digraph. . 4. Even if the recognition candidates are treated as a single recognition candidate in the process of the processing which is performed synchronously in time, there are a plurality of candidates having different existence sections in the process of the backtracking process. 4. The voice recognition method according to claim 3, wherein if they can exist, the data structure of the directed graph is determined by using these candidates as separate recognition candidates. 5. The score of the optimum route from the node corresponding to the utterance start to each node in the digraph and the optimum route from each node in the digraph to the node corresponding to the utterance break in the process of the backtracking. Find the score and
The voice recognition method according to claim 3 or 4, wherein these scores are held in a directed graph data structure. 6. The score of the optimum route is obtained by acoustic likelihood of a sequence of recognition candidates, or a combination of the acoustic likelihood and a linguistic likelihood of a chain of recognition candidates. The voice recognition method according to claim 5. 7. A speech recognition apparatus that recognizes a speech signal as a sequence of recognition candidates such as phonemes, syllables, words, etc., and configures the middle or final result thereof as directed graph data to analyze the inputted speech signal. Acoustic analysis means for obtaining a characteristic parameter sequence, model holding means for holding a model relating to an acoustic model and a chain of acoustic models, and trellis creating means for generating a trellis corresponding to a recognition candidate using the model for the characteristic parameter series. A calculation means for calculating a cumulative score and a start time for the trellis in time synchronization; a graph data storage means for storing the cumulative score and the start time corresponding to the trellis; and a graph data storage means for storing the graph data storage means. Based on the start time and cumulative score, the nodes between recognition candidates are determined and each arc corresponding to the recognition candidate The trellis creating means further creates sequentially subsequent trellis on the basis of the calculation result of the calculating means, and the backtracking processing means makes the cumulative score until the break of the utterance. And a start time is stored in the graph data storage means, the processing is performed and the processing result is stored in the graph data storage means.