JPH0375879B2 - - Google Patents

Description

[Detailed description of the invention] The present invention will be explained in the following order.

A. Field of industrial application B. Overview of the disclosure C. Prior art C1. Phonetic model of words (Figure 2) C2. Statistical models and recognition processes C3. Word combinations, text influences and released word graphs (Figure 3) , Fig. 4) C4 Recognition process D using terminal release graph Problem E to be solved by the invention Means for solving the problem F Example F1 Division into subgraphs (clinks) (Fig. 2) F2 Boundary link ( Pre-calculation of F3 (blink) (Fig. 5) Decrease of F3 clink and storage of results (Fig. 6A)
(Fig. 6B) F4 Recognition procedure using pre-calculated blink (Fig. 1) F5 Specific numerical example F6 Environment of the speech recognition system of the present invention F6a General explanation (Fig. 7, Fig. 10 to Fig. 12)
Figure) F6b Auditory model and its realization in the acoustic processor of the speech recognition system (Figures 13 to 19)
Figure) F6c Precision matching (Figures 9 and 20) F6d Basic high-speed matching (Figures 21 to 23)
Figure) F6e alternative high-speed matching (Figures 24 and 25)
Figure) F6f Matching based on initial J level (Figure 25) F6g Phoneme tree structure and fast matching example (Figure 26) F6h Language model (Figure 7) F6i Training by approximation (Figure 27) F6j Acoustic matching Extension of the word pass by the word selected by (Fig. 7, Fig. 10 to Fig. 12, Fig. 28) , relates to a device for recognizing continuous speech using a phonetic graph as a word model.

B. SUMMARY OF THE DISCLOSURE A continuous speech recognition system is disclosed having a speech processor and a word recognition computer as subsystems. In relation to the audio processor, there is a merging link (hereinafter referred to as a link) between the merging nodes.
A means of expanding a graph of confluent links (also called CLINKs) and boundary links (hereinafter also called BLINKs) between adjacent words.
means for expanding a graph of links); and means for storing an inventory of merging links and boundary links as a coded inventory; and means for converting the utterance into a coded sequence of merging links and boundary links. The present invention also includes a method for performing continuous speech recognition by characterizing speech matching candidate words as a sequence of merging links. Furthermore, the invention can be applied to speech recognition of separated words in the same way as continuous speech recognition, except that in that case there are no boundary links.

C. PRIOR ART In systems that use very small vocabularies of several hundred words and are designed to recognize only non-sequential words that are uttered in isolation, memory requirements are low and access speed is not critical.

More powerful speech recognition systems use vocabularies ranging from thousands to tens of thousands of words. Furthermore, in order to recognize consecutive words in continuous speech, the context in which the words occur may be different, so the recognition method needs to be adapted to the many pronunciation changes that can occur in the first and last parts of a word. The traditional model, the phonetic graph, must be extended.

Techniques for dealing with these issues include D.R.
P. S. Cohen and R. L. Mercer's paper “Phoneme Components of Automatic Speech Recognition Systems,” published on pages 275-320 of “Speech Recognition,” 1975 edition, edited by Reddy, published by Academic Press, New York. R.L.Mercer, “The
Phonological Component of an Automatic
Speech Recognition System, “Speech
Recognition, edited by DRReddy,
Academic Press, New York 1975, pp 275−
320) and IBM Technology Disclosure Bulletin No. 320 by the same author.
24, No. 8, January 1982, pp. 4084-4086, "Method for Rapidly Applying Phonemic Laws for Context Discrimination"
Context−Sensitive Phonological Rules”,
IBM Technical Disclosure Bulletin, Vol.24,
No. 8, January 1982, pp. 4084-4086).

C1 Word phonetic model (Figure 2) Speech recognition systems use several methods to represent words in a given vocabulary. One way to represent words is to use strings of phonemes, each of which is a basic phonetic model that roughly corresponds to a vowel or consonant in the alphabet, for example. Because a word can have so many different pronunciations, it is best to have as many models as there are pronunciations, or one
One model must accommodate all pronunciations. An intermediate method is to represent each word in at least one basic form and provide some rules for possible changes. The aforementioned Cohen and Mercer paper provides a good explanation of this technique.

FIG. 2 shows one possible phonetic graph for a hypothetical word with the base form "ABCDEF". Several pronunciations are possible for the same part of a word (phone, letter, or sequence thereof). One pass through the word model or phonetic graph corresponds to one pronunciation of the word.

C2 Statistical models and recognition processes In order to provide a means to recognize speech, word
Each model is shaped, ie, the word is uttered several times and statistics are taken to obtain a value representing the probability that the word, when spoken, follows any path or branch of the word graph. During actual speech recognition, each utterance is converted into a string of acoustic labels by a so-called acoustic processor. next,
This label sequence specifies a specific procedure (e.g.
The Viterbi algorithm compares all word models and generates a probability value for each word to obtain the probability that the label sequence to be recognized actually represents that word.
The word with the highest probability is then generated as the output, ie, the recognized word.

C3 Word Combinations, Text Influences and Open Word Graphs (Figures 3 and 4) The above procedure is suitable for recognizing single or isolated words. However, an important feature of continuous speech is that the words are interconnected when uttered, and that the pronunciation of each particular word depends on the context, that is, which words come before a particular word, and which It depends on whether the word is spoken after a particular word. Therefore, in addition to the task of determining word terminations or boundaries, certain problems arise, as discussed below. Third
The diagram shows two separate words (BRAND and
A phonetic graph of ``NEW'' and a combination graph of these words are shown. (In the figure, vertical lines represent word boundaries.) Combining words can change the end of the first word and the first pronunciation of the next word. In other contexts, the beginning and ending of each of the two words may be varied differently than shown in FIG.

In general, each word can be concatenated with another word during its utterance, resulting in a certain number of different pronunciations at its beginning and end. Such possibilities can be incorporated into each word's phonetic graph or model, as shown in the example of FIG. Figure 4 is a phonetic graph of the word THOSE. Therefore, instead of having a graph for each word that starts and ends at a single node (as is true for many singly spoken words), each word model has a number of graphs at its left and right ends. It has several possible ramifications. This is also described in the Cohen and Mercer paper mentioned above.

C4 Recognition process using open-ended graphs While the goal of speech recognition is to identify sequences of individual words (elements of a given vocabulary), the process of recognizing continuous speech requires processing a stream of consecutive phonemes. No. Therefore, during the recognition process, all possible interconnections of the initially recognized word and each subsequent word must be considered. Therefore, given the rules, the possible choices of phonemes connecting the last known word and each next possible word must be calculated and evaluated quickly, ie during the actual recognition. (This is also described in the above-mentioned Cohen and Mercer paper.) Typically, in an unknown (next) utterance, the probability of several words being spoken is established, and the word with the highest probability is selected as the next Select the word to be recognized.

A large amount of storage is required to store different graphs or models for each word of the desired vocabulary, which requires a large number of open branches at the left and right ends of each word, as well as associated linkage rules. This is because it accompanies

D Problems to be Solved by the Invention In order to increase the vocabulary, expand the word graph, and memorize the connection rules of the expanded graph,
The required storage capacity and access time increases significantly. Therefore, it is desirable to use the available storage space efficiently and to facilitate access of the model during recognition.

A primary objective of the present invention is to store word models or phonetic graphs in continuous or discrete word speech recognition systems, reducing the storage space required. The word model is divided into elementary graphs, each of which corresponds to a "clink". A clink represents at least one pronunciation of a note or a sequence thereof extending between two confluence nodes. In the case of continuous speech, the internal parts of each word and the boundary parts where two words meet are composed of clinks represented by elementary graphs. The invention can also be used in the case of separate word speech without boundaries as in the case of continuous speech.

Furthermore, it is an object of the invention to provide a storage device for phonetic models that allows fast construction of word graphs during the recognition process.

E. Means for Solving the Problems According to the continuous speech recognition embodiment of the present invention, a graph of phonetic words initially created is formed between merging nodes called merging links or clinks, and left and right boundary parts called hooks. is divided into subgraphs. The preprocessing operation performs all possible concatenations of the right and left hook pairs, and the blinks (representing the concatenated links of two words, respectively)
Generate a subgraph of boundary links called .
Although merging links (clinks) and boundary links (blinks) may appear several times in each word or word link, it is desirable to store each only once in a separate inventory. Each hook and link is indicated by a corresponding identifier. Thus, the complete model or graph for each word is not stored, but instead only a string of numbers or identifiers identifying the hooks and links that make up each word graph.

If a word model is to be used during recognition, a string of identifiers can be retrieved to provide fast access to distributed word graph elements. The required boundary link (which depends not only on the word currently being examined but also on the preceding word) is accessed by the identifier of the right hook of this last word and the identifier of the left hook of the current word.

The advantages of the invention are obvious. All words
Instead of memorizing graph details, many words and
The basic graph that is reproduced in the graph is memorized only once. Since the context-dependent boundary graph is pre-assembled, it can be accessed quickly and therefore no evaluation is required to connect hooks during recognition. When adding a new word to your vocabulary, there is a good chance that you already have a klink graph for that word and a blink graph based on the word's hooks in memory, so it is usually a string of numbers that identifies a subgraph of the word. It is sufficient to remember. The need to add new clinks or blinks is extremely rare.

In the case of discrete word sounds, the word is characterized by a string of numbers, or identifiers, corresponding to the clinks forming the word. As in the case of continuous speech, the elementary graph is stored only once, and each word is represented by a string of identifiers, each corresponding to Klink's elementary graph. Therefore, in this case as well, memory requirements are significantly reduced.

F. EXAMPLE The present invention discloses an improved phonetic representation that reduces memory requirements and simplifies computational tasks during the speech recognition process.

F1 Division into subgraphs (klinks) (Figure 2) As is clear from Figure 2, the phonetic graph of a word is divided into several groups (indicated by arrows in the diagram) through which all possible pronunciation paths must pass. It may have points of A representation of the word graph of FIG. 2, with these common points indicated by stars, is shown below.

*A*XYCD XCD BCD BZ*E*F* These common points are called confluence nodes and are represented by arrows in FIG. Each word is divided into sections by a merging node. Each section between two merging nodes is an elementary subgraph called a merging link or clink for short.

Each word graph in the context of continuous speech has at least one clink, as well as a left hook (an open branch that starts at the first confluence node) and a right hook (an open branch that starts at the last confluence node). branches). Thus, each word in the speech sequence can be represented as a sequence of clinks, preceded by a left hook and followed by a right hook.

Since any vocabulary has only a certain number of clicks, left hooks, and right hooks, each of which can be used in several different word models,
Each clink, left hook and right hook are
It is stored only once as a phonetic graph (including statistical values). Thus, each word of the vocabulary is stored as a string of code numbers that identify the strings of clinks etc. that make up each word. This is the first improvement that reduces the storage requirements of the desired vocabulary.

F2 Precomputation of boundary links (blinks) (Figure 5) As explained in the Cohen and Mercer paper, a combination of graphs of two words creates links between these words that are themselves phonetic subgraphs. occurs. Using the model representation described above, which includes a confluence link, a left hook, and a right hook, each connection of two words is actually a connection between the last confluence node of the left word and the first confluence node of the right word. yields a combination of right and left hooks that constitute a boundary subgraph of . This subgraph is called a boundary link or blink.

FIG. 5 provides an overview of the process described above for concatenating two words. This example uses the word “THOSE”
(graph in Figure 4) is selected to be connected to itself, and shows the case of saying "THOSE THOSE". The upper left of the drawing shows the right hook of the word, and the upper right shows the left hook. A specific set of rules is applied to each branch, determining how and when each pronunciation occurs. In any particular concatenation, only a subset of the terminating and initiating branches, ie, those that are compatible with each other, can be concatenated. An example of the resulting boundary link is shown at the bottom of FIG.

In Figure 5, the asterisk indicates the matching terminal node,
IRS stands for a set of interconnection rules.

The second improvement is the precomputation of boundary links or blinks that arise as subgraphs from the combination of right-hook branches and left-hook branches. As previously mentioned, each vocabulary word can be stored as a string of identifiers for a left hook, some clinks, and a right hook, with the phonetic graph of each possible clink and hook being stored separately.

In the case of the blink precomputation described above, all possible connections of right and left hooks are evaluated. For example, suppose we are given 50 different right hooks and 30 different left hooks.
The precomputation results in a set of 1500 blinks. In normal vocabulary, many of these blinks are equal, so in reality only a few hundred different blinks may occur.

Appendix 1 generates a blink from a pair of hooks,
The procedure for determining the links constituting each blink is shown.

F3 Clink reduction and memory of results (6th A)
(Fig. 6B) The next step is to convert the blink into a clink (merging link). Each blink containing an internal merge node (which may occur for a small number of blinks) is decomposed into a sequence of clinks, each spanning from one merge node to another. Each blink that does not have a merging node inside it (which may occur for the majority of blinks) is considered a blink itself.

The set of clinks resulting from the set of blinks is then compared to the initial set of clinks collected to represent the internal portions of all word models. As a result of the comparison, some clinks based on the set of blinks can be deleted because equivalent clinks were already available in the basic link inventory of the internal word part. New clinks (ie, clinks not present in the base link inventory) are added to the currently stored inventory of clinks, and each added clink is assigned a respective identification letter.

As a result of Brink's precomputation step, another data table is stored. This table shows, for each combination of right and left hooks, the identification letter of the resulting blink and the identification letters of the clinks that make up each blink (which can often be a single clink). including.

Figures 6A and 6B provide an overview of the tables or lists that are actually stored for recognition vocabularies when using the technique of the present invention. In accordance with a particular embodiment of the present invention, a commercially available IBM System 370 computer is used, and the various inventories described below are stored in corresponding locations in the IBM System 370's memory.

(A) Word Inventory: Each word of the vocabulary is stored in the form of a left hook, a sequence of clinks, and a string of right hook identifiers, as described above. Even if it is stored, it may be an empty hook that is identified as empty (numbered 00).

(B) Clink inventory: Each clink is stored as a phonetic graph whose branches represent phonemes.
Graphs can be stored in the form of structures usable in certain programming languages. In addition to the structure, a table of probability values representing the probability of occurrence of each branch of the structure is also stored. These values are retrieved during the training phase. The link inventory also includes links resulting from the blink calculation procedure.

(C) Initial hook inventory: Initially, there is an inventory of all left hooks and right hooks identified in the word vocabulary. Also, these hooks have a clink-like structure (phonetic graph), but are open on one side. Additionally, there is a table giving the concatenation rules that apply to each of the branches opened for each hook.
This hook inventory is not used after the blink is precomputed.

(D) Hook vs. inventory: After the blink is precomputed,
For each combination of a right hook and a left hook, the identification characters of each blink and the identification characters of the links constituting each blink are stored. Therefore, given a pair of right and left hooks, this table gives the identifier of the corresponding link that can be retrieved from the link inventory,
represents the resulting blink.

Of the three catalogs (A), (B) and (D) used for identification, only the Clink catalog (B) requires a large amount of storage space, while the other word catalogs (A), (D) and The hook pair inventory stores only strings of identifiers, each requiring only two bytes of storage.

Appendix 2 shows a sample portion of the Clink inventory.

F4 Recognition procedure using pre-computed blink (Figure 1) To briefly describe continuous speech recognition, it is desirable to operate as follows.

(a) Convert the audio signal to a string of acoustic labels.

(b) Represent each word in word elementary form by a sequence of Markov models. Each word base form can be matched to a string of labels.

(c) For matching procedures, it is recommended to use the following two steps:

First, a rough or fast match is performed to obtain a subset of vocabulary words that roughly correspond to the beginning of the remaining label sequence.

A precision matching procedure is then performed on the model or graph of the candidate words selected in the fast match. The resulting probability value is used to ultimately select the output word. (The language model is also used in the matching procedure.) In a continuous speech embodiment using the phonetic graph storage technique of the present invention, the recognition procedure is modified as follows.
Although recognition is performed word by word, the procedure actually proceeds from the last confluence node of one word to the last confluence node of the next word. This is done by right-footing the last recognized word (in continuous speech, it is affected by the pronunciation of the next word).
must be available to recognize the next word.

To continue the recognition procedure, the identifier string for each candidate word is retrieved from storage. Using the left hook identifier of the candidate word and the right hook identifier of the last identified word as a pair, find the identifiers of the links that make up each blink in the hook pair inventory. These clink identifiers, as well as the identifiers from the strings representing the internal parts of each candidate word, are then used to form a sequence of clink models (graphs) selected from the clink inventory. The resulting clink sequence corresponds to a label string that matches the input caravelle string using the ending distribution of the previous step as the starting distribution to obtain the probability value of the candidate word currently being investigated.

FIG. 1 shows this recognition procedure. In the diagram, word N
-1 has a right hook (RH36) that forms a boundary with the left hook of word N (LH25). Word N includes links CL117, CL085, along the interior portion extending between each merging link.
and CL187. CL117, CL08
5, and CL187 are obtained from a pre-formed Clink inventory. As mentioned above, the word N
is a model (ie, a graph) from the last confluence node of word N-1 to the last confluence node of word N. The first clink in this model is the identifier RH3 of word N-1.
6 and the identifier LH25 of word N, and finds a corresponding pre-formed blink (blink 118). blink 118
corresponds to Clink 267 in the Clink catalog. Clinks 267 can be associated only with blinks, or they can form an elementary graph that may exist between confluence nodes of internal parts of words.

When inspecting the blink formed by the right hook identified by RH64 of word N and the left hook identified by LH18 of word N+1,
To form Blink 169, two Klinks (Clink 023 and Klink 125) are required from the link inventory. When blink 169 appears, it can be replaced by a series of blinks 025 and 125 following it.

In the figure, small circles represent merging nodes, and stars represent the last merging node. The symbols CL, RH and LH represent the identifiers of the link, right hook and left hook, respectively. The dashed frame represents a confluent link graph.

A matching process is performed to determine the sequence of words from the string of labels generated by the acoustic processor. The matching process treats each word (formed by its left foot and subsequent sequence of clinks up to the last confluence node) as if it followed the right foot of the last recognized word. . In FIG. 1, each word is treated separately as if following RH 36 of word N-1. That is, the left hand of each word in the vocabulary is
It is connected (one-to-one) to the right hook of word N-1 to form a blink, followed by the appropriate sequence of links. A blink is converted into a link or a set of links. Each of the links is 1
It can be considered as one phoneme or multiple phonemes arranged in series and parallel arrays. Each phoneme has a corresponding Markov model phoneme machine specified. For each of the clinks, the phoneme machine or array of phoneme machines is used to determine the probability of a given clink or sequence of clinks, and is used to determine the feneme (feneme) of the string produced by the acoustic processor. This is how we call microphonemes generated by a processor, etc.). The details of the matching process will be explained below. Please note that U.S. Patent Application No.
No. 06/672974 (filed November 19, 1984) also describes the matching process.

The matching technique ensures that when a particular label string is fed to a given phoneme machine, the given phoneme machine (from the data stored in it)
Determine the likelihood that a given phoneme machine will generate a label in a particular incoming string.

As explained below, there are different types of phoneme machines depending on the data they store and use. first,
A precision matching phoneme machine that incorporates predetermined probabilities into a phoneme model will be described. This machine has the following features:

(a) Has multiple states and transitions between states.

(b) have a probability associated with each transition, i.e. the probability that a particular transition will occur;

(c) At a given transition, a particular label has a probability of being produced by a phoneme.

Precision matching phoneme machines can be used to determine very precisely how closely a phoneme matches a label in an incoming string, but doing so requires an enormous amount of computation. .

To avoid the enormous computational demands associated with precision matching phoneme machines, the matching technique considers another phoneme machine. Second
The phoneme machine (referred to as the basic fast matching phoneme machine) performs approximations and simplifies the calculations for the precise matching phoneme machine. In particular, whenever a given label has a probability of occurring at a transition in a given phoneme, that probability is assigned a particular value. (preferably at least the magnitude of the highest probability of a label occurring at any transition in that phoneme) In another embodiment (referred to as alternative fast matching), an arbitrary label string of some length is generated. The probabilities of phonemes are considered uniform by the corresponding phoneme machine. An alternative fast matching phoneme machine specifies a minimum and maximum length between which a uniform length distribution is formed.
In an alternative fast matching phoneme machine, the probability of any length within the length distribution is replaced by one prescribed value, and the probability of any length outside the distribution is zero. Additionally, an alternative fast matching phoneme machine can be used in place of the basic fast matching phoneme machine to further reduce the amount of computation required in determining whether a word formed by a string of phoneme machines is suitable as a candidate word. can do.

Processing the word with a basic fast matching phoneme machine or an alternative fast matching phoneme machine, preferably including both label probability replacement values and length distribution replacement values, according to the phoneme graph of the invention,
It is desirable to derive a list of words from a vocabulary of words. The words in the retrieved word list are then processed by a precision matching phoneme machine and one word is selected.

Alternatively, each word represented by the Clink phoneme machine could be matched using only the precision matching phoneme machine. In this case, the amount of processing and the time required for it will increase.

Each of the aforementioned phoneme machines receives as input a starting distribution and a label string and determines a matching value. This matching value indicates the likelihood that a given phoneme will generate a sequence of labels in the string.

The stack algorithm can be used for the decoding operation, ie, matching the input label string to the word model.

F5 Specific numerical example The storage requirements for a system with a vocabulary of 5000 words are as follows.

(a) For the old configuration without precomputing blinks: Total vocabulary: 634000 bytes Each additional word: 2500 bytes (b) For the new configuration according to the invention: Total vocabulary: 72500 bytes Each additional word: 18-20 bytes Number of links and hooks: For a vocabulary of 1000 words, the basic graph below was used.

237 clinks 72 right hooks 40 left hooks The combination of right and left hooks resulted in 2880 blinks, many of which were equal. To construct each blink, 122 new links were required in addition to the 237 existing links. Therefore,
After precomputing the blinks, the total blink inventory is
It was 359 Klink.

F6 Environment of the speech recognition system of the present invention F6a General description (Figs. 7, 10 to 12)
FIG. 7 shows a schematic block diagram of a speech recognition system 1000 that provides an environment for the present invention. This system includes a stack decoder 1002 and an audio processor (AP) 1004 connected to it.
an array processor 1006 that performs fast approximate acoustic matching; an array processor 1008 that performs precise acoustic matching; and a language model 10.
10, as well as a workstation 1012.

Acoustic processor 1004 is designed to convert audio waveform input into a string of labels or finemes, each of which identifies a corresponding acoustic type. In the present system, audio processor 1004 is based on a unique model of human hearing, as described in US patent application Ser. No. 06/665,401 (filed October 26, 1984).

Labels or finemes from audio processor 1004 are sent to stack decoder 1002. FIG. 8 shows a stack decoder 100.
2 shows a logic element. That is, stack decoder 1002 includes searcher 1020 and workstations 1012, interfaces 1022, 1024, 1026, and 1028 connected thereto. Each of these interfaces includes an acoustic processor 1004, an array processor 1006, 1008, and a language model 101.
0 respectively.

In operation, the finemes from the acoustic processor 1004 are sent by the seeker 1020 to the array processor 1006 (fast match). The fast matching procedure described below is also described in the aforementioned US patent application Ser. No. 06/672,974, filed November 19, 1984. The purpose of matching is, simply, to determine at least one most likely word of a given label string.

Fast matching is designed to examine words in a vocabulary of words and reduce the number of candidate words for a given incoming label string. Fast matching is based on stochastically bounded state machines (also referred to herein as Markov models).

Precision matching is preferably based on language model calculations that examine these words from a list of fast matching candidates that have a reasonable likelihood of being spoken words. Precision matching is also described in the aforementioned US patent application Ser. No. 06/672,974 (filed November 19, 1984). Precision matching is the 9th
It is executed by a Markov model phoneme machine as shown in the figure.

After precision matching, it is desirable to invoke the language model again to determine the word likelihood.

The purpose of stack decoder 1002 (FIG. 7) is to determine the word string that gives the highest probability for the string of labels y ₁ , y ₂ , y ₃ .

This can be expressed mathematically as follows.

Max(Pr(W|Y) (1) This is the maximum probability of W giving Y over all word strings W. As is well known, Pr
(W|Y) can be written as follows.

Pr(W｜Y)=Pr(W)・Pr(Y｜W)/Pr(Y)
(2) However, Pr(Y) is unrelated to W.

One way to determine the most likely path (i.e. sequence) of consecutive words W ^* is to examine each possible path and determine the probability of each path resulting in the label string that we are trying to decode. . Then select the path with the highest associated probability. For a vocabulary of 5000 words, this method becomes unwieldy and impractical, especially when the strings of words are long.

Two other known methods of finding the maximum likelihood word sequence W ^* are Viterbi decoding and stack decoding. Each of these techniques is described in the IEEE Proceedings on Pattern Analysis and Machine Information
PAMI Vol. 5 No. 2, March 1983 issue
LRBahl et al, “A Maximum Likelihood Approach to Continuous Speech Recognition”
Likelihood Approach to Continuous Speech
Recognition”, IEEE Transactions on Pattern
Analysis and Machine Intelligence, Vol.
PAMI-5, No. 2, March 1983).

The stack decoding method in this paper is related to a single stack decoding. That is, paths of different lengths are listed in a single stack by likelihood, and decoding is performed based on this single stack. Single stack decoding is due to the fact that the likelihood depends somewhat on the path length and therefore normalization is commonly performed. However, normalization may result in excessive searching and search errors due to improper searching if the normalization factor is not estimated correctly.

The Viterbi method does not require normalization, but is generally only practical for small tasks.
With large vocabularies, the essentially time-synchronous Viterbi algorithm may have to interface with asynchronous acoustic matching components. In this case, the result is that the interface is not suitable.

An alternative novel apparatus and method (described below) by LRBahl et al. is a method that can decode the most likely word sequence W ^* with lower computational demands and higher accuracy than other techniques. related to. In particular, a technique is provided which is characterized by multiple stack decoding and a unique decision method to determine which word sequence to expand at a given time. According to this determination method, paths of relatively short length are not penalized because of their shortness, but are instead judged by their relative likelihood.

Figures 10, 11, and 12, which will be explained later.
The figures illustrate this apparatus and method.

Stack decoder 1002 actually acts to control other elements, but does not perform much computation. Therefore, the stack decoder 1002 runs on the IBM VM/370 Operating System (Model 155, VS2, Release 1.7).
It is desirable to include a 4341 processor running under the control of the 4341 processor. Arrays that perform a significant amount of computation
The processor is realized by a commercially available 190L manufactured by Floating Point Systems (FPS).

F6b Auditory model and its realization in the acoustic processor of the speech recognition system (Figures 13 to 19)
Figure) Figure 13 shows the sound processor 11 as described above.
A specific example of 00 is shown. Acoustic wave input (eg, natural speech) enters an A/D converter 1102 that samples at a predetermined rate. A typical sampling rate is one sample every 50 microseconds. A time window generator 1104 is provided to shape the edges of the digital signal. The output of the time window generator 1104 enters an FFT (Fast Fourier Transform) device 1106 which provides a frequency spectral output for each time window.

Then, the output of the FFT device 1106 is labeled
L ₁ , L ₂ ... are processed to generate L _f . Feature selector 1108, clusterer 1110, prototyper 1112, and encoder 1114 jointly generate labels. When generating labels, prototypes are formed as points (or vectors) in space based on selected features. The acoustic input is characterized by the same selected features to provide corresponding points (or vectors) in space that can be compared to the prototype.

Specifically, when defining a prototype, clustering device 1110 collects a set of points and groups them into clusters. The method of forming clusters is based on probability distributions (such as Gaussian distributions) applied to speech. The prototype of each cluster (with respect to the center locus or other characteristics of the cluster) is stored in the prototype device 11
12. The generated prototype and the acoustic input (both with the same features selected) enter the encoder 1114. Symbolization device 1114
performs a comparison procedure and, as a result, assigns a label to a particular acoustic input.

Selection of appropriate features is an important factor in retrieving labels representing acoustic (speech) wave inputs. The acoustic processor is associated with an improved feature selector 1108. A unique auditory model is derived and used according to the acoustic processor. The auditory model will be explained with reference to FIG.

FIG. 14 shows a portion of the human inner ear. In particular, the inner hair cells 1200 and the distal end 1202 extending into a fluid-containing groove 1204 are shown in detail. Furthermore, upstream from the inner hair cells 1200,
Outer hair cells 1206 and distal ends 1208 extending into grooves 1204 are shown. Nerves that transmit information to the brain are connected to the inner hair cells 1200 and the outer hair cells 1206. Specifically, the neuron undergoes electrochemical changes, causing electrical pulses to be carried along nerves to the brain for processing. Electrochemical changes are
Stimulated by mechanical movement of the basement membrane 1210.

It is known in the art that basilar membrane 1210 acts as a frequency analyzer of acoustic wave input, with sections along basilar membrane 1210 responding to respective critical frequency bands. Each portion of basilar membrane 1210 that responds to a corresponding frequency band influences the perceived loudness of the acoustic waveform input. That is,
The loudness of a tone is greater than if two tones of similar power intensity occupied the same frequency band.
It is perceived as louder when the two tones are in separate critical frequency bands. It has been found that there are 22 orders of critical frequency bands defined by the basilar membrane 1210.

Tailored to the frequency response of the basilar membrane 1210, the present invention advantageously defines the input acoustic waveform in some or all of the critical frequency bands and then separately determines the signal components for each defined critical frequency band. inspect. This function is available in FFT device 1
106 (FIG. 13) by appropriately filtering the signal from
This is done by providing a separate signal to the feature selector 1108 for each critical frequency band tested.

The separate inputs are also blocked into time frames (preferably 25.6 milliseconds) by time window generator 1104. Therefore, the feature selector 1108 has 22
It is desirable to include the following signals. Each of these signals represents the sound intensity of a given frequency band for each time frame.

The signal is preferably filtered by a conventional critical band filter 1300 of FIG. The signal is then processed separately by a volume equalization transformer 1302 that perceives changes in volume as a function of frequency. Incidentally, the perceived loudness of a first tone at a given dB level at one frequency may be different from the loudness of a second tone at the same dB level at another frequency. Volume equalization converter 1302
is based on empirical data and transforms the signals in each frequency band so that each is measured on the same loudness scale. For example, volume equalization converter 1
302, with some modifications to the 1933 work of Fletcher and Munson, can map acoustic energy to equivalent loudness. Figure 16 shows the results of a modification to the previous study. According to Figure 16, at 40dB
It can be seen that a 1KHz tone corresponds to the volume level of a 100Hz tone at 60dB.

The volume equalization converter 1302 adjusts the volume according to the curve shown in FIG. 16, producing the same volume regardless of frequency.

In addition to the dependence on frequency, changes in power do not correspond to changes in volume, as is clear from examining specific frequencies in Figure 16. That is, the intensity of the sound,
That is, variations in amplitude are not reflected in equivalent changes in perceived loudness at all points. for example,
At a frequency of 100Hz, 10dB around 110dB
The perceived volume change is around 20dB.
Much larger than the 10dB perceived volume change.
This difference is processed by a volume compression device 1304, which compresses the volume in a predetermined manner. Volume compression device 13
04 converts the power P to its cube root by replacing the volume amplitude measurement value in units of phons with units of sones.
It can be compressed to P ^1/3 .

FIG. 17 shows the known empirically determined Hong-to-Thorn relationship. With the use of sone units,
Our model remains nearly accurate even with large audio signal amplitudes. One sone is defined as a 1KHz tone with a volume of 40dB.

FIG. 15 shows a new time-varying response device 13.
06 is shown. The device operates with volume equalization and volume compression signals associated with each critical frequency band. Specifically, for each frequency band examined, the neural firing rate f is determined for each time frame. The firing rate f is defined according to the acoustic processor of the present invention as follows.

f=(So+DL)n (1) where n is the amount of neurotransmitter; So is the spontaneous firing constant involved in neural firing independent of acoustic waveform input; L is the measured sound volume; D is the displacement constant.
So·n corresponds to the spontaneous neural firing rate that occurs regardless of the presence or absence of acoustic wave input, and DLn corresponds to the firing rate due to acoustic wave input.

An important point is that the present invention has the characteristic that the value of n changes with time according to the following equation.

dn/dt=Ao−(So+Sh+DL)n (2) where Ao is the recruitment constant; Sh is the spontaneous neurotransmitter decay constant. The new relationship shown in equation (2) shows that while neurotransmitters are generated at a constant rate Ao, (a) decay (Sh・n), (b) spontaneous firing (So・
n), and (c) neural firing due to acoustic wave input (DL・
n) is taken into account. It is assumed that these modeled phenomena occur at the locations shown in FIG.

As is clear from equation (2), the next amount of neurotransmitter and the next firing rate are proportional to at least the square of the current amount of neurotransmitter, indicating the fact that the acoustic processor of the present invention is nonlinear. There is. In other words, the amount of neurotransmitter in state (t+△t) is
It is equal to the amount of neurotransmitter in the state (t+dn/dt・Δt). Therefore, n(t+Δt)=n(t)+(dn/dt)·Δt (3) holds true.

Equations (1), (2) and (3) represent the operation of the time-varying signal analyzer. Time-varying signal analyzers point to the fact that the auditory system is time-adaptive, and the signals of the auditory nerve are non-linearly related to the acoustic wave input.
Incidentally, the acoustic processor of the present invention can better track obvious temporal changes in the nervous system.
It provides the first model for implementing nonlinear signal processing in speech recognition systems.

In order to reduce the number of unknown terms in equations (1) and (2), the present invention uses the following equation that is applied to a constant volume L.

So+Sh+DL=1/T (4) where T is the measurement of the time after the audio wave input is generated until the auditory response drops to 37% of its maximum value. T is a function of loudness and is taken from a known graph that displays the attenuation of the response for various loudness levels by the sound processor of the present invention. That is, when a tone of constant volume is generated, a high level response initially occurs, after which the response decays with a time constant T toward a steady state level. When there is no acoustic wave input, T=T ₀ . This is 50
It is about milliseconds. If the volume is L _nax , T=
T _nax . This is about 30 milliseconds. Ao=
By setting it to 1, 1/(So+Sh) becomes
If L=0, it is determined to be 5 centiseconds. L is
When L _nax and L _nax = 20 sones, the following equation holds true.

So+Sh+D(20)=1/30 (5) Based on the above data and formula, So and Sh are determined by formulas (6) and (7) shown below.

So=DL _nax / [R+(DL _nax T ₀ R)-1] (6) Sh=1/T ₀ -So (7) However, R=f stable state/f stable state L _nax /L=0 (8 ) f steady state represents the firing rate at a given volume when dn/dt is 0.

R is the only variable left in the audio processor. Therefore, the performance of this processor is changed simply by changing R. That is, R is one parameter that can be adjusted to change performance, usually meant to minimize steady-state effects versus transient-state effects. Inconsistent output patterns for similar speech inputs are generally caused by differences in frequency response, speaker differences, background noise, and other factors (affecting the steady-state but not the transient portions of the speech signal). ) It is desirable to minimize steady-state effects since they are caused by distortion. The value of R is preferably set to optimize the error rate of the complete speech recognition system. The optimal value thus found is R=1.5. In that case, the values of So and Sh are 0.0888 and 0.11111, respectively, and the value of D is 0.00666.

FIG. 18 is a flow diagram of the operation of the audio processor according to the present invention. Preferably, the digitized audio, sampled at 20 KHz, during a 25.6 millisecond time frame is passed through a Hanning window 1320 and the output is double Fourier transformed in a DFT 1322 at 10 millisecond intervals. The conversion output is filtered in block 1324 to provide a power density output in each of at least one frequency band (preferably all critical frequency bands, or at least 20 bands). The power density is then converted from the recorded loudness to a volume level at block 1326. This operation is the first
By changing the graph in Figure 6 or later
It is carried out on the basis of the limit values derived by the process outlined in the figure.

In FIG. 19, first the perceptual limit T _f and the audible limit T _h of each filtered frequency band m
are set to be 120 dB and 0 dB, respectively (block 1340). The audio counter, total frame register, and histogram register are then reset (block 134).
2).

Each of the histograms includes bins, with each bin representing the number of samples or counts during which the power or similar measurement (for a given frequency band) is within a respective range. In the present invention, the histogram preferably represents the number of centiseconds during which the volume is within each of a plurality of volume ranges (for a given frequency band). For example, in the third frequency band, there may be 20 centiseconds between 10 dB and 20 dB power. Similarly, the
For 20 frequency bands, 50dB and 60dB may have 150 centiseconds of the total 1000 centiseconds between them. Percentiles are taken from the total number of samples (ie, centiseconds) and the binned counts.

At block 1344, the frame of filter output for each frequency band is examined, and at block 1346, the bins in the appropriate histogram (one per filter) are incremented. block 1348
Then, the total number of bins with amplitudes greater than 55 dB is tallied for each filter (i.e., frequency band) to determine the number of filters that indicate the presence of speech. At block 1350, a minimum number (e.g. 20
If there is no filter in 6), block 1
At 344, the next frame is inspected. If there are enough filters to indicate the presence of audio, block 135
2, increment the voice counter. The audio counter is incremented at block 1354 until audio appears for 10 seconds and at block 1356 new T _f and T _h values are determined for each filter.

The new T _f and T _h values for a given filter are determined as follows. For T _f , dB of the bin holding the 35th sample from the top of 1000 bins
The value (i.e. the 96.5th percentile of volume) is
BIN _H is defined, and T _f is set to T _f =BIN _H +40dB. For T _h , from the lowest bin (0.01)
The dB value of the bin holding the (total number of bins - voice count)-th value is defined as BIN _L. That is,
BIN _L is a bin of 1% of the number of samples excluding those classified as voices in the histogram. T _h is defined as T _h =BIN _L -30dB.

Blocks 1330 and 1332 of FIG.
Then, the amplitude of the sound is updated with the limit value as described above, and based on the updated limit value, the amplitude of the sound is converted into units of sones and compressed. An alternative method of introducing and compressing Sohn units is to take the filter amplitude "a" (after the bins have been incremented) and convert it to dB by:

a ^dB = 20log ₁₀ (a)-10 (9) Each of the filter amplitudes is then compressed to a range between 0 dB and 120 dB to give equivalent loudness by:

a ^eql = 120 (a ^dB − T _h ) / (T _f − T _h ) (10) Next, a ^eql is converted from the volume level (in units of phons) to an approximate value of the volume in units of sones (40 dB It is desirable to map a 1KHz signal to 1).

L ^dB = (a ^eql −30)/4 (11) Next, the approximate value L _s of the sound volume per son is given by the following equation.

L _s =10(L ^dB )/20 (12) At step 1334, L _s is used as input to equations (1) and (2) to determine the output firing rate f for each frequency band. For 22 frequency bands, a 22-dimensional vector characterizes the acoustic wave input over consecutive time frames. However, in general, 20
The frequency bands are examined using a conventional filter bank scaled by Mel.

Block 1 before processing the next time frame.
At 337, the "next state" of n is determined according to equation (3).

The acoustic processors described above require improvement for use with DC pedestals where the firing rate f and neurotransmitter content n are large. That is, if the dynamic range of terms in the f and n equations is important, the following equation is derived to lower the height of the pedestal.

In a stable state and when there is no acoustic wave input signal (L=0), equation (2) can be solved for the internal state n' in the stable state as follows.

n'=A/(So+Sh) (13) The internal state of the neurotransmitter amount n(t) is expressed as a stable state part and a fluctuating part as follows.

n(t)=n′+n″(t) (14) Combining equations (1) and (14), the firing rate is obtained as follows.

f(t)=(So+D・L)(n′+n″(t)) (15) The So・n′ term is a constant, but all other terms are the varying parts of n or (D・Contains an input signal represented by It will be done.

f″(t)=(So+D・L)・[{n″(t)+D・L
・
A} /(So+Sh)] (16) Considering equation (3), the “next state” is as follows.

n(t+Δt)=n′(t+Δt)+n″(t+Δt)(17
) n(t+Δt)=n″(t)+A−(So+Sh+D・
L)・(n′+n″(t)) (18) n(t+Δt)=n″(t)−(Sh・n″(t) −(So+Ao・L ^A )・n″(t) −(Ao・L ^A・D)/(So+Sh) +Ao−(So・Ao)+(Sh・Ao))/(So+Sh) (19) Equation (19) becomes as follows if all constant terms are ignored.

n″(t+Δt)=n″(t)(1−So・Δt)−f″(
t)
(20) Equations (15) and (20) constitute the output and state update equations applied to each filter during each 10 ms time frame. The result of using these equations is a vector of 20 elements every 10 milliseconds, where each element of this vector corresponds to the firing rate of each frequency band in the filter bank scaled by Mel.

With respect to the embodiment described above, the flowchart of FIG. and n(t+Δt) except that the expressions are replaced.

A unique value for each equation term (i.e., t ₀ =
5csec, t _L = 3csec, Ao = 1, R = 1.5 and L _nax
= 20) can be set to other values, So, Sh
and D terms have their desired values of 0.0888, 0.11111, respectively, when the other terms are set to different values.
and will be a different value from 0.00666.

The invention can be implemented with a variety of software or hardware.

F6c Precision Matching (FIGS. 9 and 20) FIG. 9 shows a precision matching phoneme machine 2000 as an example. Each of these machines is a stochastically limited state machine, with (a) multiple states S _i ; (b) multiple transitions tr (Sj → Si): a transition is a transition between different states; transitions between the same states, and each transition has a corresponding probability; (c) is characterized in that it has a corresponding actual label probability for each label that can be generated at a particular transition.

In FIG. 9, seven states S ₁ to S ₇ and 13 transitions tr1 to tr13 are shown in the precision matching phoneme machine 200.
0, of which the paths of the three transitions tr11, tr12 and tr13 are indicated by dashed lines. In each of these three transitions, a phoneme may change from one state to another without generating a label. Therefore, such a transition is called a null transition. transition
Labels can be generated along tr1 to tr10. In particular, at least one label along each of transitions tr1-tr10 may have a unique probability of being generated there. For each transition, there is a probability associated with each label that can be generated by the system. That is, if there are 200 labels that can be selectively generated by an acoustic channel, each (non-null) transition has an "actual level probability" associated with it of 200, each of which means that the corresponding label is Corresponds to the probability of being produced by a phoneme in a particular transition. The actual label probability of transition tr1 is represented by the symbol P followed by a column from 1 to 200 in brackets, as shown. Each of these numbers represents a given label. In the case of label 1, there is a probability P[1] that the precision matching phoneme machine 2000 generates label 1 at transition tr1. Various actual label probabilities are stored in association with labels and corresponding transitions.

When a string with labels y ₁ y ₂ y ₃ . . . is presented to the precision matching phoneme machine 2000 corresponding to a given phoneme, the matching procedure is performed. The procedure related to the precision matching phoneme machine will be explained with reference to FIG.

FIG. 20 is a trellis diagram of the phoneme machine of FIG. 9. As in the case of the phoneme machine, this trellis diagram also has a null transition from state S ₁ to state S ₇ , state S ₁
shows the transition from to state S ₂ and from state S ₁ to state S ₄ . Transitions between other states are also shown. Additionally, the trellis diagram shows the measured times in the horizontal direction. The start probabilities q ₀ and q ₁ represent the probability that a phoneme has a start time at time t=t ₀ or t=t ₁ of the phoneme, respectively. Each start time
The respective transitions at t ₀ and t ₁ are also shown. Incidentally, it is desirable that the interval between successive start (and end) times be equal to the time interval between labels.

When using the Precision Matching Phoneme Machine 2000 to determine how closely a given phoneme is matched to the label of an incoming string, the end time distribution of that phoneme is searched to determine the matching value for that phoneme. used for. The method of performing a precise matching depending on the end time distribution is common to all phoneme machine embodiments described herein with respect to the matching procedure. When generating end time distribution to perform precise matching, the Precision Matching Phoneme Machine 2000
requires precise and complex calculations.

First, according to the trellis diagram of FIG. 20, time t
Examine the calculations required to obtain the start and end times at = t ₀ . In the case of the example phoneme machine structure shown in FIG. 9, the following probability equation applies.

Pr(S ₇ , t=t ₀ )=q ₀・T(1→7) +Pr(S ₂ , t=t ₀ )・T(2→7) +Pr(S ₃ , t=t ₀ )・T( 3→7)
(21) However, Pr represents the probability, and T is the 2 in parentheses.
represents the transition probability between two states. This formula is t
We show that the probabilities of each of the three states that can have an end time of = t ₀ are limited to end time occurrences in state S ₇ in this example.

Next, when we look at the end time t=t ₁ , we have to perform calculations for every state other than state S ₁ . State S ₁ starts at the end time of the previous phoneme. For convenience of explanation, only the calculations for state S ₄ are shown.

For S ₄ , the calculation becomes:

Pr(S ₄ , t=t ₁ )=Pr(S ₁ , t=t ₀ )・T(1 → 4)・Pr(y ₁ , 1→4) +Pr(S ₄ , t=t ₀ )・T (4→4)・Pr(y ₁ , 4→4) (22) Equation (22) shows that at time t=t ₁ , the phoneme machine is in state S _{4
The probability that _} The value obtained by multiplying the probability that a given label y ₁ transitions from state S ₁ to state S ₄ and (b) the probability of being in state S ₄ at time t = t ₀ multiplied by the probability of transitioning from state S ₄ to itself. Multiply the transition probability and further state S ₄
We show that it is determined by the sum of the values obtained by multiplying the probability of generating a given label y ₁ as a transition from to itself.

Similarly, calculations for other states (except state S ₁ ) are also performed to generate the corresponding probabilities that the phoneme is in the particular state at time t=t ₁ . In general, in determining the probability of being in a target state at a given time, precise matching involves (a) recognizing the respective probability of each state and each previous state before the transition that leads to the target state; (b) for each said previous state, recognize a value representing the probability of a label that must be generated on a transition between each said previous state and the current state to match its label string; c) Combine the respective values representing the probability of each previous state and the label probability to give the probability of the target state due to the corresponding transition.

The overall probability of being a target state is determined from the target state probabilities due to all transitions leading to it.
The calculation for state S ₇ includes terms for three null transitions, allowing the phoneme to start and end at time t=t ₁ with the phoneme ending in state S ₇ .

As in the case of determining the probabilities for times t=t ₀ and t=t ₁ , the determination of probabilities for other sets of end times is preferably done in such a way as to form an end time distribution. The end time distribution value for a given phoneme indicates how well the given phoneme is matched to the incoming label.

In determining how well a word is matched to an incoming label, the phonemes representing the word are processed sequentially. Each phoneme generates an end time distribution of probability values. The matching value of a phoneme is obtained by summing the end time probabilities and taking the logarithm of the sum. The start time distribution of the next phoneme is derived by normalizing the end time distribution. This normalization involves, for example, scaling each of the values by dividing them by their sum, such that the scaled values sum to one.

There are at least two ways to determine the number h of phonemes to test for a given word or word string. In the depth-first method, calculations follow a basic format (calculating subtotals for each successive phoneme in succession). The calculation ends if this subtotal is found to be less than or equal to a predetermined limit for a given phoneme position along it. Another method, the breadth-first method, involves calculating similar phoneme positions in each word. The calculations are the calculation of the first phoneme of each word, followed by the calculation of the second phoneme of each word, and so on.
Do it sequentially. In the breadth-first method, calculations along the same number of phonemes in each word are compared at the same relative phoneme position. In either method, the word with the largest sum of matching values is the desired target word.

Precise matching is achieved using APAL (Array Processor Assembly Language). This is an assembler 190L manufactured by Floating Point Systems, Inc. By the way, precise matching is
The actual label probability (i.e., the probability that a given phoneme produces a given label y at a given transition), the transition probability for each phoneme machine, and the probability that a given phoneme produces a given label y after a given start time It requires considerable memory to store each probability of being in a given state at a time. The aforementioned 190L is an end time, preferably a matching value based on the logarithmic sum of the end time probabilities, a start time based on the previously generated end time probability,
and the respective calculations of word matching scores based on the matching values of sequential phonemes in the word. Furthermore, for precise matching, it is desirable to calculate the tail probabilities of the matching procedure. Tail probability measures the likelihood of consecutive labels independent of words. In a simple example, a given tail probability corresponds to the likelihood of a label following another label. This likelihood is easily determined, for example, from a string of labels generated by some sample speech.

Therefore, precise matching provides sufficient storage to include the base form, the Alcoff model statistics, and the tail probabilities. For a vocabulary of 5000 words, each word containing about 10 phonemes, the basic form is 5000
Requires ×10 memory capacity. If there are 70 distinct phonemes (with a Markov model for each phoneme), 200 distinct labels, and 10 transitions with a probability of generating any label, the statistic is 70 × 10
x200 storage locations would be required. However, the phoneme machine is divided into three parts (starting part, middle part and ending part),
The statistical table should correspond accordingly (preferably one of the three self-loops is included in each part). Therefore, the storage requirement is reduced to 60x2x200. For tail probabilities, 200x200 storage locations are required. 50K integer and 82K floating point storage will work satisfactorily with this array.

F6d Basic high-speed matching (Figures 21 to 23)
Figure) Because precise matching calculations are expensive, basic fast matching and alternative fast matching are provided that reduce the required calculations without sacrificing too much accuracy. It is desirable to use high-speed matching in conjunction with precision matching. High-speed matching lists promising candidate words from the vocabulary, and precise matching uses
In most cases, this list of candidate words is used.

A fast approximate acoustic matching technique is described in the aforementioned US patent application Ser. No. 06/672,974, filed November 19, 1984. In fast approximate acoustic matching, each phoneme machine is preferably simplified by replacing the actual label probability for each label with a specified replacement value at every transition in a given phoneme machine. A particular replacement value is such that when using that replacement value, the matching value for a given phoneme overestimates the matching value obtained by precise matching if the replacement value does not replace the actual label probability. It is desirable to select. One way to ensure this condition is to choose each replacement value such that the probability of corresponding to a given label in a given phoneme machine is no greater than that replacement value. By replacing the actual label probabilities in the phoneme machine with corresponding replacement values, the number of calculations required in determining word match scores can be significantly reduced. Furthermore, it is desirable to overestimate the replacement value so that the resulting matching score is less than what would have previously been determined without replacement.

In a particular embodiment of performing acoustic matching in a language decoder with a Markov model, each phoneme machine is shaped to include (a) a plurality of states and transition paths between states, and (b) each of the current state S Given _i has a transition probability T(i→j) to state S _j , and S _i and S _j can represent the same state or different states, tr(S _i →S _j ), ( c) characterized to obtain the actual label probability p(y _k −i→j) of each transition from one state to the next. In this case, k is a label identification symbol.

Each phoneme machine has: (a) one particular value for each y _k in each of said phoneme machines;
(b) means for assigning each actual output probability p(y _k −i _→ j) at each transition in a given phoneme machine to one assigned to the corresponding y _k means for substituting a specific value p′(y _k ). Preferably, the replacement value is at least the magnitude of the actual maximum label probability of the corresponding y _k label at any transition in a particular phoneme machine.
The fast matching example uses the 10 to 10 words selected as the most likely words in the vocabulary corresponding to the incoming label.
It is used to form a list of candidate words on the order of 100. Preferably, candidate words are subject to a language model and precise matching. By cutting down the number of words considered in precise matching to the order of 1% of the words in the vocabulary, computational cost is significantly reduced while maintaining accuracy.

Basic fast matching can be simplified by replacing the actual label probability of a given label at every transition with a single value, allowing the given label to be generated by a given phoneme machine. That is, regardless of the transition in a given phoneme machine that a label has a probability of occurring, we replace that probability with one particular value. This value is overestimated and is preferably at least the magnitude of the maximum probability of a label occurring at any transition in a given phoneme machine.

By setting the probability replacement value of a label as the maximum of the actual label probabilities for a given label in a given phoneme machine, the matching value produced by the basic fast matching is at least It is guaranteed to be as large as the matching value that occurs. Thus, more words are generally selected as candidate words because basic fast matching generally overestimates the matching value of each phoneme. Words that are considered candidates through precise matching also pass according to basic high-speed matching.

Figure 21 shows the basic high-speed matching phoneme machine 30.
Indicates 00. The labels (also referred to as symbols and finemes) enter the basic fast matching phoneme machine 3000 along with the start time distribution. The start time distribution and label string inputs are similar to the precision matching phoneme machine inputs described above. The start time is sometimes not distributed over multiple times, but instead represents a precise (phoneme onset) time following a silence interval, for example. However, if the audio is continuous, the end time distribution is used to form the start time distribution (as explained in more detail below). The basic high-speed matching phoneme machine 3000 generates an end time distribution and also generates a matching value for a specific phoneme from the generated end time distribution. The matching score of a word is defined as the sum of the matching values of the constituent phonemes (at least the first h phoneme of the word).

FIG. 22 shows the basic high-speed matching calculation. The basic fast matching calculation is concerned only with the start time distribution, the number or length of labels generated by the phoneme, and the replacement value p'(y _k ) associated with each label y _k .

By replacing all the actual label probabilities for a given label in a given phoneme machine with the corresponding replacement values, the basic fast matching replaces transition probabilities with length distribution probabilities (for a given phoneme machine It becomes unnecessary to include the actual label probabilities (which may be different for each transition) as well as the probability of being in a given state at a given time.

Incidentally, the length distribution is determined from a precise matching model. Specifically, for each length of the length distribution, the matching procedure preferably examines each state individually and, for each state, determines the respective transition path. Thereby, the currently examined state can occur (a) given a particular label length, and (b) independently of the output along the transition. The probabilities of all transition paths of a particular length to each destination state are summed, and then the sums of all destination states are summed to represent the probability of a given length in the distribution. The above procedure is repeated for each length. In accordance with a preferred form of the invention, these calculations are performed on a trellis diagram as is known in the art of Markov modeling. For transition paths that share branches along a trellis structure, the computation for each common branch only needs to be performed once, and the results are added to each path that contains the common branch.

In FIG. 22, two restrictions are included as an example. Initially, the length of labels generated by phonemes may be 0, 1, 2, or 3 with probabilities 1 ₀ , 1 ₁ , 1 ₂ and 1 ₃ respectively. The start times are also restricted, allowing only four start times, each with probabilities q ₀ , q ₁ , q ₂ and q ₃ .
That is, L (1 ₀ , 1 ₁ , 1 ₂ , 1 ₃ ) and Q (q ₀ , q ₁ ,
q ₂ , q ₃ ) are assumed. Due to these restrictions, the end distribution of the target phoneme is defined as shown in the following equation.

Φ ₀ =q ₀ 1 ₀ Φ ₁ =q ₁ 1 ₀ +q ₀ 1 ₁ p ₁ Φ ₂ =q ₂ 1 ₀ +q ₁ 1 ₁ p ₂ +q ₀ 1 ₂ p ₁ p ₂ Φ ₃ =q ₃ 1 ₀ +q ₂ 1 ₁ p ₃ +q ₁ 1 ₂ p ₂ p ₃ +q ₀ 1 ₃ p ₁ p ₂ p ₃ Φ ₄ =q ₃ 1 ₁ p ₄ +q ₂ 1 ₂ p ₃ p ₄ +q ₁ 1 ₃ p ₂ p ₃ p ₄ Φ ₅ = q ₃ 1 ₂ p ₄ p ₅ + q ₂ 1 ₃ p ₃ p ₄ p ₅ Φ ₆ = q ₃ 1 ₃ p ₄ p ₅ p ₆ Examining these equations, we find that Φ ₃ is equal to It can be seen that the corresponding terms are included.
The first term represents the probability that a phoneme starts at time t= _t3 and generates a zero-length label (the phoneme ends at the same time as it starts). The second term represents the probability that a phoneme starts at time t= _t2 , the label length is 1, and label 3 is generated by that phoneme. The third term represents the probability that the phoneme starts at time t= _t1 , the label length is 2 (ie, labels 2 and 3), and that labels 2 and 3 are generated by that phoneme. Similarly, the fourth
The term is such that the phoneme starts at time t = t ₀ , the label length is 3, and there are three labels 1, 2, and 3.
represents the probability of being generated by that phoneme.

Comparing the calculations required for basic high-speed matching and the calculations required for precise matching, it is found that the former is relatively simpler than the latter. By the way,
The value of p′(y) is
The label length remains the same value as in the case of probability. Furthermore, the length and start time limitations make later end time calculations easier. For example, in Φ ₆ , the phoneme starts at time t=t ₃ and all three labels 4, 5 and 6 must be generated and used by the phoneme at that ending time.

When generating a matching value for a target phoneme, end time probabilities along the formed end time distribution are summed. Then, the matching value is obtained by taking the logarithm.

Matching value=log ₁₀ (Φ ₀ +...+Φ ₆ ) As mentioned above, the matching score of a word is easily determined by summing the matching values of consecutive phonemes in a particular word.

Next, generation of the start time distribution will be explained with reference to FIG. In Figure 23(a), the word
THE ₁ is broken down into its constituent phonemes and repeated.
In FIG. 23(b), a string of labels is shown along the time axis. FIG. 23(c) shows the initial start time distribution. The initial start time distribution is derived from the end time distribution of the most recent preceding phoneme (in the preceding word, which may include a silent word). Based on the label input and the start time distribution in FIG. 23(c), the end time distribution Φ _DH of the phoneme DH is generated (FIG. 23(d)). The start time distribution of the next phoneme UH1 is determined by recognizing the time at which the previous phoneme end distribution exceeds the limit value A shown in FIG. 23(d). A is determined individually for each end time distribution.
A is a function of the sum of the end time distribution values of the target phoneme. Therefore, the interval between time a and time b is the phoneme
Represents the time at which the UH1 start time distribution is set.
In FIG. 23(e), the interval between time c and time d is
This corresponds to the time when the end time distribution of the phoneme DH exceeds the limit value A and the start time distribution of the next phoneme is set. The value of the start time distribution is obtained, for example, by dividing each end time value by the sum of end times exceeding the limit value A to normalize the end time distribution.

The basic high-speed matching phoneme machine 3000 is realized using the above-mentioned Floating Point Systems Assembler 190L using the APAL program. Additionally, certain forms of the invention may be deployed using other hardware and software in accordance with the description herein.

F6e. Alternative high-speed matching (Fig. 24, 25)
Figure) Basic fast matching, used alone or preferably in conjunction with precise matching and/or language models, significantly reduces computational requirements. In order to further reduce the computational requirements, the present invention further provides two lengths (minimum length L _nio and maximum length L nio
Precise matching is simplified by forming a uniform label length distribution between L _nax ). In basic fast matching, the probabilities of generating labels of a given length (ie, 1 ₀ , 1 ₁ , 1 _{2 ,} etc.) generally obtain different values. Alternative fast matching replaces each length probability of a label with one uniform value.

The minimum length is preferably equal to the minimum length that has a non-zero probability in the initial distribution of lengths, but other lengths can be chosen if desired. The choice of maximum length is more arbitrary than the choice of minimum length, but the probability of a length less than the minimum and greater than the maximum is 0.
is set to By setting the length probability to exist only between the minimum length and the maximum length, a uniform pseudo-distribution can be shown. As one method, the uniform probability can be set as the average probability due to a pseudo-distribution. As an alternative, the uniform probability can be set as the maximum value of the length probabilities and replaced by the uniform value.

The effect of making all label length probabilities equal can be easily recognized from the above-mentioned formula for end time distribution in basic high-speed matching. Specifically, the length probability can be extracted as a constant.

By setting L _nio to 0 and replacing all length probabilities with one constant value, the end time distribution can be expressed as:

Φ _n =m/1=q _n +Φ _n-1 p _n (23) However, “1” is one uniform replacement value,
Preferably, the value of p _n corresponds to a replacement value for a given label generated at time m for a given phoneme.

In the case of the above formula for Φ _n , the matching value is defined as follows.

Matching value = log ₁₀ (Φ ₀ +Φ ₁ +…+Φ _n ) +log ₁₀ (1) (24) Comparing the basic fast matching and the alternative fast matching, the number of additions and multiplications required is lower than that using the alternative fast matching phoneme machine. This will significantly reduce the amount. When L _nio = 0, the basic fast matching requires 40 multiplications and 20 additions because length probabilities must be taken into account.
In the case of alternative fast matching, it can be seen that since Φ _n is determined iteratively, one multiplication and one addition are required for each successive Φ _n .

Figures 24 and 25 detail the computational simplification by alternative fast matching. Figure 24a
is the phoneme machine 310 corresponding to the minimum length L _nio = 0
An example of 0 is shown below. The maximum length is assumed to be infinite so that the length distribution is uniform. FIG. 24b shows the trellis diagram resulting from the phoneme machine 3100.
Assuming that the start times after q _o are outside the start time distribution, if m<n, the determination of each successive Φ _n requires only one addition and one multiplication. When determining the end time after that, only one multiplication is required and no addition is necessary. Figure 25a is
Specific phoneme machine 320 for minimum length L _nio = 4
FIG. 25b shows the corresponding trellis diagram. Since L _nio = 4, the second
The trellis diagram in Figure 5b is marked by the symbols U, V, W and Z.
yields zero probability along the path of . For an end time between Φ ₄ and Φ _o , four multiplications and one addition are required. For end times greater than n+4,
Only one multiplication is required; no addition is necessary. This embodiment is implemented with the APAL code on the FPS 190L.

In accordance with the present invention, any desired additional states may be added to the embodiment of FIG. 24 or 25.
For example, any number of states with null transitions can be included without changing the value of L _nio .

F6f Matching based on first J level (25th
Figure) The present invention contemplates a further improvement of the basic fast matching and the alternative fast matching by considering only the matching of the first J label of the string entering the phoneme machine. A reasonable value for J is 100, assuming that labels are generated by the acoustic channel's acoustic processor at the rate of one label every sensor second. In other words, labels corresponding to audio on the order of one second are provided to establish a match between phonemes and labels entering the phoneme machine. By limiting the number of labels to be tested, two advantages are obtained.
The first is a reduction in decoding delay, and the second is that the problem of comparing short and long word scores can be largely avoided. Of course, the length of J can be changed as desired.

The effect of limiting the number of labels to be inspected can be observed from the trellis diagram of Figure 25b. Without the improvements of the present invention, the fast matching score is the sum of the probabilities of Φ _n along the bottom row of this drawing. That is, t=t ₀ (if L _nio = 0) or t=t ₄ (L _nio =
The probability of being in state S ₄ at each time starting at (case 4) is determined as a _n and then all Φ _n are summed. When L _nio =4, the probability of being in state S ₄ at any time before t ₄ is 0. With the improvement, the summing of Φ _n ends at time J.
In FIG. 25b, time J corresponds to time t _o+2 .

By completing the inspection of the J label that exceeds the interval up to time J, the sum of the following two probabilities occurs when determining the matching score. First, as mentioned above, there is a row calculation along the bottom row of this trellis diagram. However, this calculation is performed at time J−
Up to 1. State S ₄ at each time up to time J-1
The probabilities that are summed to form a raw score. Second, there is a column score corresponding to the sum of the probabilities that the phoneme is in each of the states S ₀ to S ₄ at time J.
This column score is calculated as follows.

Column score = ₄ 〓 ^f=0 Pr (S _f , J) (25) The phoneme matching score is obtained by summing the row score and column score and taking the logarithm of the sum. To continue the fast matching of the next phoneme, values along the bottom row (preferably including time J) are used to derive the start time distribution of the next phoneme.

After determining the matching score for each of the J consecutive phonemes, the total phoneme total is the sum of all matching scores for that phoneme, as described above.

Examining the method of generating end time probabilities in the Basic Fast Matching and Alternative Fast Matching embodiments described above, we find that determining column scores is not easily compatible with fast matching calculations. In order to better adapt the improvements that limit the number of labels to be tested to the fast matching and alternative matching,
The present invention allows column scores to be replaced with additional row scores. That is, (in Figure 25a)
An additional low score for the phoneme in state S ₄ between times J and J+K is determined. However, K is the maximum number of states in any phoneme machine. Therefore, if any phoneme machine has 10 states, our refinement adds 10 end times along the bottom row of its trellis diagram and determines the probability for each of them. All probabilities along the lowest row up to time J+K (including the probability at time J+K) are added to generate a matching score for a given phoneme. As described above, the matching values of consecutive phonemes are summed to obtain a word matching score.

This example is an APAL on the FPS 190L mentioned above.
Although implemented in code, it can also be implemented in other hardware and in other code, as is the case with other parts of the invention.

F6g Phoneme tree structure and high-speed matching example (Second
(Figure 6) By using basic fast matching or alternative fast matching (with or without a maximum label limit), less computational time is required in determining phoneme matching values. Furthermore,
Even when performing precise matching on the words in the list obtained by fast matching, the amount of computation is significantly saved.

Once the phoneme matching value is determined,
As shown in FIG. 26, comparisons are made along the branches of tree structure 4100 to determine which path of phonemes is most likely. In Figure 26, (point 410
The sum of the phoneme matching values for the phonemes DH and UH1 of the spoken word “the” (from branch 4104 from phoneme MX) must be a higher value than for each sequence of phonemes branching from phoneme MX. . Incidentally, the phoneme matching value for the first phoneme MX is calculated only once and then used for each base form that extends. (Branch 4104 and 4
See 106. ) Furthermore, if we find that the total score computed along the first sequence of branches is lower than the critical value or lower than the total score of the other sequences of branches, then All base forms extending from the sequence may be removed from the candidate words at the same time. For example, the base forms associated with branches 4108-4118 are discarded as soon as it is determined that MX is not a likely path. The fast matching implementation and tree structure generates an ordered list of candidate words with significant savings in associated computation.

For storage requests, the phoneme tree structure, phoneme statistics, and tail probabilities are to be stored. For the tree structure, there are 25,000 arcs and four data words characterizing each arc. The first data word represents an index of a subsequent arc or phoneme. The second data word represents the number of subsequent phonemes along the branch. The third data word represents at which node of the tree structure the arc is located. Fourth
The data word represents the current phoneme. Therefore,
This tree structure requires a storage space of 25000×4. For fast matching, there are 100 different phonemes and 200 different finemes. Since a finem has one probability of being generated anywhere in the phoneme, a storage space of 100 x 200 statistical probabilities is required. For the tail structure, 200x200 storage space is required. In the case of high-speed matching,
100K integer and 60K floating point storage space is sufficient.

F6h Language Model (Figure 7) As mentioned above, the probability of correctly selecting a word can be increased by including a language model that stores information about the word in context (such as triple letters). The language model is described in the above paper.

Language model 1010 (Figure 7) preferably has unique characters. Specifically, a modified trigraph method is used. According to the invention, the sample
The text is examined to determine the likelihood of each ordered triple word and word pair and single word in the vocabulary. A list of most likely triple words and word pairs is then formed. Additionally, the likelihoods of triple words not in the list of triple words and word pairs not in the list of word pairs are determined, respectively.

According to the language model, if the target word follows two words, a determination is made as to whether the target word and the two preceding words are in the list of triple words. If in a list of triple words, the stored probability assigned to that triple word is specified. If the target word and the preceding word are not in the list of triple words, a determination is made as to whether the target word and its adjacent preceding word are in the list of word pairs. If it is in a list of word pairs, multiply the probability of that word pair by the probability that there is no triple word in the list of triple words mentioned above, and assign the product by the target word. If said triple word and word pair containing the target word are not in the list of triple words and the list of word pairs, respectively, then the probability of the target word alone is added to the probability that said triple word is not in the list of triple words, and the word pair. Multiply by the probability that is not in the list of word pairs and assign the product to the target word.

Training by F6i Approximation (Figure 27) The flowchart 5000 in Figure 27 shows training of a phoneme machine used in acoustic matching. At block 5002, a vocabulary of words (typically on the order of 5000 words) is defined. block 5004
, each word is displayed by a sequence of phoneme machines. The phoneme machine is shown, for example, as a phonetic phoneme machine, but may alternatively include a sequence of finemone phonemes. The display of words by a phonetic phoneme machine sequence or a finemone phoneme machine sequence will be described below. The phoneme machine sequence of a word is called the word base form.

Block 5006 arranges the word base forms into the tree structure described above. Statistics for each phoneme machine in the basic form of each word can be found in IEEE Bulletin Volume 64 (1976)
F. Jelinek's paper “Continuous speech recognition using statistical methods” (pages 532-556)
“Continuous Speech Recognition by
Statististal Methods”Proceedings of the
IEEE, Vol. 64, 1976, pp. 532-556) (block 5008).

Block 5009 determines the actual parameter values used in the fine matching, ie, the values to be used in place of the statistical values. For example, a value to be used in place of the actual label output probability is determined. At block 5010, the determined values replace the stored actual probabilities so that the phonemes in the base form of each word contain approximate replacement values. All calculations related to basic high-speed matching are in block 501.
Executes at 0.

Next, at block 5011, a determination is made as to whether the acoustic matching should be further approximated or improved. If no improvement is required, block 5012 sets the value determined for base approximation matching to be used and does not configure to use any other approximation. If improvement is desired, block 5018 creates a uniform string length distribution. block 50
At 20, a decision is made as to whether further enhancement is desired. If you do not wish to improve, block 50.
At 12, label output probability values and string length probability values are estimated and set for use in acoustic matching. If further improvement is desired, at block 5022 the acoustic matching is limited to the first J label of the generated string. Regardless of whether one of the enhanced embodiments is selected, determined parameter values are set at block 5012. At this point, each phoneme machine in the basic form of each word has been trained with the desired approximation.

F6j Expansion of word paths by words selected by acoustic matching (Figures 7, 10 to 1)
(FIGS. 2 and 28) Next, a good stack decoding method used in the speech recognition shown in FIG. 7 will be explained.

10 and 11, a plurality of continuous labels y ₀ y _{1 .} . . generated at continuous “label intervals” or “label positions” are shown.

FIG. 11 also shows a plurality of generated word paths, namely path A, path B, and path C.
It is shown. In the context of Figure 10, path A is the entry "to be or" and path B is the entry "two".
b”, path C would correspond to entry “too”. For a target word path, there is a label (i.e., equivalently, a label interval) for which the target word path has the highest probability of being terminated. .Such a label is called a "boundary label".

For a word path W representing a sequence of words, the most likely ending time (shown in the label string as a "boundary label" between two words) is given in IBM Technical Disclosure Bulletin, No. 23.
Volume No. 4, September 1980, paper “Fast Acoustic Matching Calculations” by LRBahl et al.
et al, “Faster Acoustic Match
IBM Technical Disclosure
Bulletin, Vol. 23, No. 4, September 1980). Briefly, this paper focuses on two important questions: (a) how many label strings Y are words (or word sequences); and (b) at what label spacing. It describes how to deal with the termination of partial sentences (corresponding to parts of label strings).

For any given word path, there is a "likelihood value" associated with each label or label interval in the label string, including the first label and the boundary label. All of the likelihood values for a given word path collectively represent the "likelihood vector" for the given word path. Therefore, for each word pass there is a corresponding likelihood vector. The likelihood value L _t is shown in FIG.

The "likelihood envelope" Λ _t of a collection of word paths W ¹ , W ² , . . . , W ^s at label interval t is defined mathematically as follows.

Λ _t = max(L _t (W ¹ ), ..., L _t (W ^s )) In other words, for each label interval, the likelihood envelope is
Contains the highest likelihood value associated with any word path in the collection. Figure 11 shows the likelihood envelope 804.
0 is shown.

A word pass is considered "complete" if it corresponds to a complete sentence. Preferably, the complete path is identified when the typing speaker reaches the end of the sentence, for example by pressing a button.
The entered input is synchronized with the label interval that marks the end of the sentence. A complete word pass cannot be extended by appending words to it. Partial word paths accommodate incomplete sentences and can be extended.

Partial paths are classified as "alive" or "dead" paths. A Word Pass is "dead" when it has already been extended, but
It is “alive” when it has not yet been extended. By this classification, a path that has already been extended to form at least one longer extended word path will not be considered for extension again at the next time.

Each word path can be characterized as "good" or "bad" with respect to the likelihood envelope. A word path is a good word path if it has a likelihood value that is within the maximum likelihood envelope with the label corresponding to its boundary label. Otherwise, word
Pass is a bad word pass. Although it is desirable, it is not necessary to reduce each value of the maximum likelihood envelope by a fixed value to act as a good (bad) limit level.

There is a stack element for each label interval. Each live word path is assigned to a stack element corresponding to the label interval that corresponds to the boundary label of such live path. A stack element may have zero, one, or more word path entries (listed in order of likelihood value).

Next, the steps performed by stack decoder 1002 of FIG. 7 will be described.

Forming the likelihood envelope and determining which word passes are good are interrelated as shown in the stack decoding technique flow diagram of FIG.

In the flowchart of FIG. 12, block 5050
So, first, the null path is the first stack (0)
to go into. At block 5052, the (complete) stack element containing the previously determined complete path, if any, is provided. Each complete path in a (complete) stack element has a likelihood vector associated with it. The likelihood vector of the complete path with the highest likelihood at its boundary label first determines the maximum likelihood envelope. If there is no complete path in the (intelligent) stack element, the maximum likelihood envelope is initialized to -∞ at each label interval. Additionally, the maximum likelihood envelope may be initialized to −∞ even if the complete path is not specified. Initialization of the envelope occurs in blocks 5054 and 5056.

The maximum likelihood envelope is initialized and then adjusted by a predetermined amount Δ
is reduced by Δ, forming a well-defined region where Δ exceeds the reduced likelihood and Δ below the reduced likelihood.
Form a poorly defined area. The larger Δ, the larger the number of word passes that are considered possible to extend. When using log ₁₀ to determine L _t , a value of Δ of 2 gives satisfactory results. Although it is desirable, it is not necessary that the value of Δ be uniform along the length of the label spacing.

As shown in Figure 12, the likelihood envelope is updated,
A “good” (extendable) password,
or the loop you want to mark as a “bad” path.
The process begins at block 5058, which searches for the longest unmarked word path. If more than one unmarked word path is in the stack corresponding to the longest word path length, then the word path with the highest likelihood for its boundary label is selected. If the password is found, block 5060
Then, check whether the likelihood at that boundary label is within the region with good Δ regulation. If it is not in the good region, block 5062 marks the path as being in the bad region of the Δ prescription, and block 5058 searches for the next unmarked live path. If it is within the good region, block 5064 marks the path as being within the good region of the Δ prescription, and block 50
At 66, the likelihood envelope is updated to include the likelihood values of the paths marked "good." That is, for each label interval, the updated likelihood value is the sum of (a) the current likelihood value within its likelihood envelope, and (b) the likelihood associated with the word path marked “good”. determined as the larger likelihood value between the values. This operation occurs in blocks 5064 and 5066. After the envelope has been updated, block 5058
Go back and look again for the longest, best unmarked living word path.

This loop is repeated until there are no more unmarked word paths. When there are no more unmarked word passes, block 5070
, the shortest word path marked "good" is selected. If there are two or more "good" word paths with the shortest length, block 50
At 72, the word path with the highest likelihood for its boundary label is selected and the selected shortest path is extended. That is, at least one possible successor word is determined by successfully performing the fast matching, language model, precision matching, and language model procedures, as described above. For each potential successor word, an extended word path is formed. Specifically, an extended word path is formed by appending a likely successor word to the end of the selected shortest word path.

After the selected shortest word path forms an extended word path, the selected word path
The path is removed from the stack of which it was an entry and is replaced by each extended word.
The path is inserted into the appropriate stack. In particular, the extended word path becomes an entry into the stack corresponding to its boundary label (block 507).
2).

The act of extending selected pulses in block 5072 will be described in conjunction with the flowchart of FIG. After finding the path in block 5070,
The procedure shown in Figure 28 is executed to extend the word path based on the appropriate approximate matching.

At block 6000 of FIG. 28, (of FIG. 7)
Audio processor 1004 generates a string of labels. Label string is block 600
2, and at block 6002,
One of the basic or improved approximate matching procedures is performed to obtain an ordered list of candidate words. Thereafter, at block 6004, the language model described above is used as described above. After using the language model, the remaining target words along with the generated labels are sent to the precision matching processor at block 6006. Block 6
At 008, the fine matching yields a list of remaining candidate words that are better presented to the language model. Probable words (as determined by approximate matching, precise matching, and language model) are used to extend the path found in block 5070 of FIG. Block 6008 (28th
Each potential word determined in Figure) is appended separately to the discovered word path, and
Multiple extended word passes may be formed (block 6010).

In FIG. 12, after the extension path is formed and the stack is re-formed, the process returns to block 5052 to repeat the process.

Therefore, at each iteration, the shortest, best "good" word path is selected and extended. A word path marked as a "bad" path in one iteration may become a "good" path in a later iteration. Then,
The characteristics of whether a living word path is a "good" or "bad" path are uniquely assigned to each iteration. In practice, the likelihood envelope does not change significantly from one iteration to the next, so the calculations to determine whether a word pass is good or bad are performed efficiently. Furthermore, normalization becomes unnecessary.

When identifying complete sentences, it is desirable to include block 5074. That is, if there are no living word paths left unmarked and no "good" word paths to extend, decoding terminates. The complete word path with the highest likelihood for each of its boundary labels is identified as the most likely word sequence of the input label string.

In the case of continuous speech where sentence ends are not identified, path extension is performed continuously, ie, for a predetermined number of words as desired by the user of the system.

F7 Appendix Appendix 1: Procedure for generating a blink from a pair of hooks and changing the blink to a clink (a) Determine a node from the right hook that is eligible to be connected to the left hook.

(b) Determine a node with the left hook that can be connected to a qualified node with the right hook.

(c) Perform concatenation and remove unused branches.

(d) Determine the confluence node of the resulting graph (blink graph) and insert the blink into the sequence of clinks.

Appendix 2: Sample Portion of the Klink Inventory G Effects of the Invention As mentioned above, the invention allows each word in the vocabulary to be identified by a sequence of Klink identifiers and a corresponding phoneme machine. That is, the basic form of each word is stored simply as a sequence of identifiers, rather than as a sequence of phonemes filled with statistical tables for each phoneme, due to the precise matching as well as the fast matching used. As mentioned above, the statistics table for each phoneme machine is stored only once.

[Brief explanation of drawings]

Figure 1 shows the recognition procedure, showing the assembly of a word graph using separately drawn elementary graphs, or clinks, Figure 2 is a schematic diagram of a single word phonetic graph, and Figure 3 shows the construction of two word graphs. Figure 4 shows a simple phonetic string of separate words and a phonetic graph connecting both words. Figure 5 shows the phonetic graph of a word with branches; Figure 5 shows the interconnection of the respective branches of the two hooks of the phonetic graph resulting in a boundary subgraph or blink; Figures 6A and 6B illustrate the invention. FIG. 7 is a schematic block diagram of a system environment in which the present invention may be implemented; FIG. A detailed block diagram of a stack decoder in the system environment of FIG. 9 is identified in storage by statistics obtained during a training session,
Figure 10 is a diagram showing the displayed precise matching phoneme machine, Figure 10 is a diagram showing the steps of continuous stack decoding, Figure 11 is a diagram showing the stack decoding method, Figure 12 is a flowchart of the stack decoding method, and Figure 1 is a diagram showing the stack decoding method.
Fig. 3 is a diagram showing the elements of the acoustic processor, Fig. 14 is a diagram showing the parts of a typical human ear representing the locations forming the components of the acoustic model, and Fig. 15 is a block diagram showing the parts of the acoustic processor. , Fig. 16 is a diagram showing the relationship between sound intensity and frequency, which is used in the design of an audio processor, Fig. 17 is a diagram showing the relationship between the sound and the horn, and Fig. 18 is a diagram showing the characteristics of the sound using the audio processor shown in Fig. 13. 19 is a flowchart showing how to update the limits in FIG. 18; FIG. 20 is a flowchart showing the trellis or grid of the precision matching procedure;
Figure 21 is a diagram showing the phoneme machine used to perform matching, Figure 22 is a time distribution diagram used in a matching procedure with specific conditions, and Figure 23 (a).
~(e) is a diagram showing the interrelationship between phonemes, label strings, and start and end times determined by the matching procedure; Figures 24(a) and (b) are diagrams showing the relationship between phonemes, label strings, and start and end times determined by the matching procedure; 25(a) and (b) are diagrams showing a specific phoneme machine with a minimum length of 4 and its corresponding trellis. FIG. 26 is a diagram showing multiple phoneme machines at the same time. Diagram showing the phoneme tree structure that allows processing of words,
Figure 27 is a flowchart showing the shaping of the phoneme machine, the second
FIG. 8 is a flow diagram illustrating how the word path is lengthened in the stack decoding procedure. 1000...Voice recognition system, 1002...
Stack decoder, 1004...Acoustic processor, 1006, 1008...Array processor, 1010...Language model, 1012...Workstation, 1020...Search device, 102
2,1024,1026,1028...interface.

Claims (1)

[Claims] 1. A sound system in which partial Markov models corresponding to phonemes are connected to form a chain of partial Markov models representing a plurality of continuously uttered words, each of which can be assigned to a minute time interval. A device that recognizes continuous speech by selecting a string of labels according to the input speech from a set of labels representing types, and matching this string of labels to a chain of partial Markov models, performs the following (a) to ( A continuous speech recognition device characterized by having the means of g). (a) means for storing first data representing a plurality of possible utterances of a first audio portion; The possible utterances of this first audio portion do not vary with the audio portions that precede or follow the first audio portion. The first data is specified by a first identifier. (b) means for storing second data representing a plurality of possible utterances of a second audio portion different from said first audio portion; The possible utterances of this second audio portion do not vary with the audio portion that precedes it, but vary with the audio portion that follows it. The second data is specified by a second identifier. (c) means for storing third data representing a plurality of possible utterances of a third audio portion different from said first audio portion and said second audio portion; The possible utterances of this third audio portion vary with the audio portions that precede it and do not vary with the audio portions that follow it. The third data is specified by a third identifier. (d) means for storing fourth data representing a first word comprising said second audio portion; This fourth data includes a second identifier representing a portion of the first word. (e) means for storing fifth data representing a second word different from the first word, including the third audio portion; This fifth data includes a third identifier representing a part of the second word. (f) Means for storing sixth data including a first identifier representing the second audio portion and a third audio portion following the second audio portion. (g) Converting a plurality of consecutive words into a chain of the first, second and third identifiers above based on the contents of the storage means in (d) and (e) above, and
and the third identifier into first identifiers according to the contents of (f) above, and match them to the label string of the input voice based on the chain of the converted first identifiers. A means of constructing chains of partial Markov models.