KR20220154422A - Method and apparatus for speech recognition - Google Patents

Abstract

Disclosed are a method for speech recognition and a device for speech recognition including a decoding process based on a beam search algorithm. According to an embodiment, the speech recognition method comprises the steps of: obtaining candidate word sequences included in a beam corresponding to a target frame of a speech signal and the probabilities of the candidate word sequences; expanding the candidate word sequences based on the target frame and the probabilities of the candidate word sequences; determining whether the expanded candidate word sequences are included in a predetermined number of first upper word sequences in an order with a high probability in a preliminary candidate group including the candidate word sequences and the expanded candidate word sequences; updating the preliminary candidate group by additionally expanding at least one expanded candidate word sequence included in the first upper word sequences based on the determination result; and determining a predetermined number of second upper word sequences in an order with high probability in the preliminary candidate group as beams corresponding to a frame next to the target frame. Therefore, provided is a beam search method for reducing calculation time and a calculation amount required for expanded search.

Description

Voice recognition method and apparatus {METHOD AND APPARATUS FOR SPEECH RECOGNITION}

The following embodiments relate to a voice recognition method and apparatus, and specifically, to a voice recognition method and apparatus including a decoding process based on a beam search algorithm.

Speech recognition technology converts a voice signal generated by speech into text data for processing, and is also referred to as STT (speech-to-text). Recent advances in deep learning technology have resulted in the development of end-to-end frameworks that directly map speech signals to character sequences. The proposed end-to-end framework includes an encoder-decoder network, a recurrent neural aligner, and a recurrent neural network transducer (RNN-T).

In particular, speech recognition based on RNN-T shows considerable accuracy, and research is being actively conducted. RNN-T uses best-first search or breadth-first search as a decoding method, and a technique for improving the speed of a decoding method based on beam search is required.

The embodiment may provide a beam search method for reducing computation time and computation amount required for extended search.

The embodiment decodes efficient speech recognition based on beam search by using the feature that most of the frames of the speech signal correspond to blank characters in the decoding process because the length of the word string, which is the result of speech recognition, is shorter than the length of the data in which the speech signal is encoded. method can be provided.

However, the technical challenges are not limited to the above-described technical challenges, and other technical challenges may exist.

A speech recognition method performed by a processor according to one aspect may include obtaining candidate word sequences included in a beam corresponding to a target frame of a speech signal and probabilities of the candidate word sequences corresponding to the speech signal; expanding the candidate word sequences based on the target frame and the probabilities of the candidate word sequences; Preliminary candidate group-The preliminary candidate group includes the candidate word sequences and extended candidate word sequences of the candidate word sequences-A predetermined number of first upper word sequences according to an order of high probability within the extended candidate word sequences determining whether at least one of them is included; updating the preliminary candidate group by additionally extending the at least one extended candidate word string included in the first upper word strings based on a result of the determination; and determining, as beams corresponding to a frame next to the target frame, a predetermined number of second higher order word sequences in an order with a higher probability in the preliminary candidate group.

In the step of updating the preliminary candidate group, when it is determined that the at least one extended candidate word sequence is included in the first higher-order word sequences, at least one of the target frame and the first higher-order word sequences included in the first higher-order word sequences is determined. further extending the at least one extended candidate word string based on a probability of the extended candidate word string; and updating the preliminary candidate group by adding the additionally extended candidate word sequence to the preliminary candidate group.

The step of determining the beam corresponding to the next frame may include selecting a predetermined number of second higher order word sequences in an order with a higher probability in the updated preliminary candidate group; and determining the second upper word sequences as beams corresponding to the next frame.

The updating of the preliminary candidate group may include not updating the preliminary candidate group when it is determined that the at least one extended candidate word sequence is not included in the first higher order word sequences.

The first high-order word strings and the second high-order word strings may match each other.

The extending of the candidate word sequences may include: obtaining an extended candidate word sequence of the corresponding candidate word sequence by adding a candidate word to the corresponding candidate word sequence, corresponding to each of the candidate word sequences; obtaining a probability that the target frame corresponds to the candidate word based on the corresponding candidate word string; and obtaining a probability of the extended candidate word sequence based on the probability of the corresponding candidate word sequence and the probability that the target frame corresponds to the candidate word.

The voice recognition method may further include obtaining a plurality of frames including features extracted from the voice signal by encoding the voice signal.

The extending of the candidate word sequences may include obtaining the expanded candidate word sequences and the probabilities of the extended candidate word sequences based on an end-to-end speech recognition model.

The end-to-end speech recognition model may include a neural network trained to receive voice data and output text labels based on training data including voice data and text labels corresponding to the voice data.

When the target frame is the last frame of the voice signal, determining the beam corresponding to the next frame includes outputting at least some of the second upper word strings as a voice recognition result corresponding to the voice signal. can include more.

The extended candidate word sequences of the candidate word sequences may include word sequences having the candidate word sequences as prefixes.

The word sequence includes one word and a sequence of a plurality of words, and the word may include a combination of a single character corresponding to a specific language and a plurality of characters corresponding to the specific language.

A beam search method performed by a processor according to one aspect includes adding candidate word sequences included in a beam corresponding to a target frame of encoded data to a preliminary candidate group; performing an expansion operation on the candidate word sequences; determining whether or not to repeat the expansion operation based on whether word sequences selected based on the expansion operation include an extended word sequence of the candidate word sequences obtained based on the expansion operation; performing the expansion operation on the selected word sequences when it is determined to repeat the expansion operation; When it is determined not to repeat the expansion operation, determining the selected word strings as beams corresponding to the next frame of the encoded data.

The expansion operation may include adding extended word sequences of word sequences subject to the expansion operation to the preliminary candidate group; and selecting a certain number of word strings according to a predetermined criterion from the preliminary candidate group.

The step of determining whether to repeat the expansion operation may include determining to repeat the expansion operation when the selected word sequences include the extended word sequence; and determining not to repeat the expansion operation when the extended word sequence is not included in the selected word sequences.

The determining whether to repeat the extension operation may include determining whether to repeat the extension operation based on whether a predetermined repetition termination condition is satisfied.

The speech recognition apparatus according to one side obtains candidate word sequences included in a beam corresponding to a target frame of a speech signal and probabilities corresponding to the speech signal of the candidate word sequences, and determines the target frame and the candidate word sequences Based on the probabilities, the candidate word sequences are expanded, and a preliminary candidate group-the preliminary candidate group includes the candidate word sequences and extended candidate word sequences of the candidate word sequences-a schedule in order of high probability It is determined whether at least one of the extended candidate word strings is included in the number of first upper word strings, and the at least one extended candidate included in the first upper word strings is determined based on a result of the determination. At least one method for updating the preliminary candidate group by additionally extending a word sequence, and determining a predetermined number of second upper word sequences according to an order with a high probability in the preliminary candidate group as a beam corresponding to a frame next to the target frame. contains the processor of

In updating the preliminary candidate group, when it is determined that the at least one extended candidate word sequence is included in the first higher-order word sequences, the processor includes the target frame and the first higher-order word sequences. Based on the probability of the at least one extended candidate word string, the preliminary candidate group may be updated by additionally extending the at least one extended candidate word sequence and adding the additionally extended candidate word sequence to the preliminary candidate group. have.

In determining the beam corresponding to the next frame, the processor selects a certain number of second higher order word sequences in an order with a higher probability in the updated preliminary candidate group, and selects the second higher order word sequences in the next frame. It can be determined as a beam corresponding to .

In updating the preliminary candidate group, the processor may not update the preliminary candidate group when it is determined that the at least one extended candidate word sequence is not included in the first higher-order word sequences.

The first high-order word strings and the second high-order word strings may match each other.

The processor, in extending the candidate word sequences, obtains an extended candidate word sequence of the corresponding candidate word sequence by adding a candidate word to the corresponding candidate word sequence, corresponding to each of the candidate word sequences, and A probability that the target frame corresponds to the candidate word is obtained based on the word string, and a probability of the extended candidate word string is determined based on the probability of the corresponding candidate word string and the probability that the target frame corresponds to the candidate word. can be obtained

A beam search apparatus according to one side adds candidate word sequences included in a beam corresponding to a target frame of encoded data to a preliminary candidate group, performs an expansion operation on the candidate word sequences, and selects based on the expansion operation. Based on whether or not the extended word sequences of the candidate word sequences obtained based on the expansion operation are included in the upper word sequences obtained by the expansion operation, whether or not to repeat the expansion operation is determined, and as it is determined to repeat the expansion operation, At least one of performing the expansion operation on the selected word sequences and determining that the expansion operation is not repeated, determining the selected word sequences as beams corresponding to the next frame of the encoded data contains the processor.

The expansion operation includes an operation of adding extended word sequences of word sequences to be subjected to the expansion operation to the preliminary candidate group and an operation of selecting a predetermined number of word sequences from the preliminary candidate group according to a predetermined criterion.

The embodiment may provide a beam search method with improved calculation speed by parallelly processing an extended search process in the beam search method.

In the embodiment, by adopting a process for determining whether to continue the extended search during the beam search process, the occurrence of unnecessary extended search processes can be prevented while maintaining the accuracy of speech recognition, and computational overhead due to unnecessary extended search processes can be reduced. can reduce

1 is an operation flowchart of a voice recognition method according to an embodiment.
2A and 2B are diagrams for explaining a detailed operation of expanding candidate word strings according to an embodiment.
3 is a detailed operation flowchart of a voice recognition method according to an embodiment.
4 is a diagram illustrating the structure of an RNN-T, which is an example of an end-to-end speech recognition model according to an embodiment.
5 is an operational flowchart of a decoding process for speech recognition performed based on RNN-T according to an embodiment.
6 is an operation flowchart of a beam search method according to an embodiment.
7 is an exemplary diagram of a configuration of a device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is an operation flowchart of a voice recognition method according to an embodiment.

The voice recognition method according to one side may proceed with a process of outputting a character string corresponding to the voice signal by decoding the encoded information of length T obtained by encoding the input voice signal based on a beam search algorithm. . Encoding information of length T may be composed of T frames, and may be decoded in time order from the first frame to the last T frame. A decoding process of the t-th frame may correspond to a t-step of a decoding process of all encoded information. In the decoding process, a candidate group of speech recognition results of each frame may be determined, and the determined candidate group may be referred to as a beam. A beam corresponding to each frame may include a predetermined number (eg, N) of candidate word strings. A beam corresponding to a certain frame t+1 of encoded information may be determined based on a beam corresponding to frame t and a result of voice recognition of frame t in a decoding process for frame t, which is a previous frame in time order. In the case of the last frame T of the encoded information, a candidate word sequence included in a beam generated as a result of decoding the frame T may be output as a final recognition result of the input speech signal.

A voice recognition method according to one aspect may be performed by a server or device including an end-to-end voice recognition model. More specifically, steps included in the voice recognition method according to an embodiment may be performed by at least one processor included in a server or device. A specific configuration of a device or server that performs a voice recognition method according to one side will be described in detail with reference to FIG. 7 below.

Referring to FIG. 1, the speech recognition method includes obtaining candidate word sequences included in a beam corresponding to a target frame of a speech signal and probabilities of candidate word sequences (110), and determining the target frame and candidate word sequences of a speech signal. Expanding candidate word sequences based on the probabilities (120), determining whether the extended candidate word sequences are included in a predetermined number of first upper word sequences in the preliminary candidate group (130), based on the determination result Updating the preliminary candidate group by additionally extending the extended candidate word sequences included in the first upper word sequences (140), and determining a predetermined number of second upper word sequences in the preliminary candidate group as beams corresponding to the next frame Step 150 may be included. According to an embodiment, steps 110 to 150 may correspond to a decoding process of a voice recognition method.

A target frame of a voice signal according to one aspect may be included in a plurality of frames constituting encoding information obtained by encoding the voice signal. In other words, the voice recognition method may further include obtaining a plurality of frames including features extracted from the voice signal by encoding the voice signal. According to an embodiment, encoding of a voice signal may include an encoding operation performed by an encoder of an end-to-end voice recognition model.

According to an embodiment, the target frame means a frame currently being processed by a processor, that is, a frame on which a decoding process is performed, and may be referred to as a current frame hereinafter. The next frame may refer to a frame immediately after the current frame in time order. For example, if the current frame corresponds to the tth frame, the next frame may correspond to the t+1th frame.

A word sequence according to one aspect may include one or more words, and may include a sequence of words in which a plurality of words are arranged. A word according to an embodiment may mean a string composed of at least one character, not a word in a dictionary meaning. In other words, a word according to an embodiment may include one letter corresponding to a specific language and a combination of a plurality of letters corresponding to the specific language.

According to an embodiment, a beam corresponding to the current frame may be obtained by decoding a previous frame of the voice signal. The previous frame may correspond to a frame (t−1) immediately preceding the current frame (t) according to time order. More specifically, the candidate word sequences corresponding to the current frame of the speech signal obtained in step 110 and the probabilities of the candidate word sequences are the candidate word sequences and candidate words obtained by decoding the previous frame of the speech signal. It may correspond to probabilities corresponding to the speech signals of the columns. In other words, it may correspond to candidate word strings obtained by performing steps 110 to 150 using the previous frame as the current frame and probabilities corresponding to speech signals of the candidate word strings.

According to an embodiment, a beam corresponding to the next frame obtained by performing steps 110 to 150 corresponding to the current frame of the voice signal is used in step 110 performed for the next frame of the current frame. It can be. According to an embodiment, when the current frame of the voice signal is the last frame of the voice signal, a beam corresponding to the next frame obtained by performing steps 110 to 150 may be output as a voice recognition result. have.

According to one embodiment, when the current frame is the first frame of the voice signal, since the previous frame does not exist, step 110 may be omitted. Alternatively, when the current frame is the first frame according to an embodiment, step 110 may include obtaining a predetermined candidate word sequence(s) and probability(s). For example, when the current frame is the first frame, step 110 obtains a candidate word sequence including a start token, and the probability of the start token that is the candidate word sequence may be predetermined to be 1.

According to an embodiment, a probability corresponding to a specific candidate word string may be obtained based on an end-to-end speech recognition model. An end-to-end speech recognition model according to one aspect may include a neural network trained to receive voice data and output text labels based on training data including voice data and text labels corresponding to the voice data. In other words, the end-to-end speech recognition model may include a deep learning-based speech recognition model based on a neural network, and may include, for example, an RNN transducer (RNN-T) based on a recurrent neural network (RNN). A process of obtaining a probability corresponding to a candidate word string based on an end-to-end speech recognition model will be described in detail with reference to FIGS. 4 and 5 below.

Candidate word sequences according to one side may be matched with corresponding probabilities. For example, candidate word sequence A may be matched with a first probability. According to one embodiment, the probability of matching the candidate word string may be stored in a memory accessible to the processor performing step 110 .

According to an embodiment, the probabilities of candidate word sequences included in the beam corresponding to the current frame are probabilities obtained in a decoding process performed corresponding to the previous frame, and may be matched with the candidate word sequences and stored in memory. . According to an embodiment, step 110 may include obtaining probabilities of candidate word sequences by loading probabilities matched and stored in memory with candidate word sequences.

Step 120 according to one side corresponds to a step of obtaining extended candidate word sequences of candidate word sequences and probabilities of the extended candidate word sequences, based on the current frame of the speech signal and probabilities of the candidate word sequences. can The extended candidate word string(s) of a specific candidate word string may include word string(s) in which one or more candidate words are added to the corresponding candidate word string. Candidate words may be determined based on training data of an end-to-end speech recognition model. For example, when an end-to-end speech recognition model is trained to output an alphabetic label corresponding to a speech signal based on training data including a speech signal and an alphabetic label corresponding to the speech signal, the candidate word corresponds to the alphabetic label. can do. According to an embodiment, extended candidate word sequences of candidate word sequences may include word sequences having candidate word sequences as prefixes. For example, when candidate word strings include 'a' and 'bc', word strings prefixed with candidate word strings are 'aa', 'ab', 'ac', 'bca', 'bcb' and May contain 'bcc'.

According to an embodiment, there may be cases where an extended candidate word sequence obtained by expanding a specific candidate word sequence overlaps another candidate word sequence. For example, when 'a' and 'ab' are included in the candidate word sequences, the extended candidate word sequence 'ab' may be generated as a result of extending the candidate word sequence 'a'. 'ab' is duplicated with the candidate word string 'ab'. In this way, when an extended candidate word string of a specific candidate word string overlaps another candidate word string, the corresponding word string may be treated as a candidate word string rather than an extended candidate word string. For example, when 'ab' obtained by extending the candidate word sequence 'a' is already included in the candidate word sequence, 'ab' may be treated as a candidate word sequence other than the extended candidate word sequence.

According to an embodiment, the step 120 of expanding the candidate word strings may be performed corresponding to each of the candidate word strings. For example, in step 120 of extending a specific candidate word string, the step of adding candidate words to the corresponding candidate word string to obtain an extended candidate word string of the corresponding candidate word string, the current frame is a candidate word based on the candidate word string Obtaining a probability corresponding to , and acquiring a probability of an extended candidate word sequence of the corresponding candidate word sequence based on a probability of the corresponding candidate word sequence and a probability that the current frame corresponds to the candidate word.

According to an embodiment, the step 120 of expanding the candidate word sequence may be performed in parallel in correspondence with each of the candidate word sequences. For example, referring to FIG. 2A , when candidate word sequences 201 of 'a', 'ab', and 'c' are included in a beam corresponding to a current frame, candidate words are included in each of the candidate word sequences. An operation of obtaining each of the extended candidate word strings 202 of candidate word strings by adding 'a', 'b', and 'c' may be performed in parallel by a plurality of processors. In other words, the process of acquiring the extended candidate word string 'aa' by adding the candidate word 'a' to the candidate word string 'a' or the extended candidate word by adding the candidate word 'c' to the candidate word string 'c' The process of obtaining the column 'cc' may be performed by a plurality of processors. For example, a process of obtaining each extended candidate word sequence may be simultaneously performed by different processors.

According to an embodiment, the step of expanding candidate word sequences 120 may include obtaining expanded candidate word sequences and probabilities of the extended candidate word sequences based on an end-to-end speech recognition model. A process of acquiring extended candidate word strings and probabilities of the extended candidate word strings in the end-to-end speech recognition model according to one side will be described in detail with reference to FIGS. 4 and 5 below.

Step 130 according to one side may correspond to a step of determining whether an extended candidate word string is included in a predetermined number of first upper word strings in an order with a high probability in the preliminary candidate group. Here, the preliminary candidate group may include candidate word sequences and extended candidate word sequences of the candidate word sequences. For example, referring to FIG. 2A , the preliminary candidate group is a set including candidate word sequences 201 and extended candidate word sequences 202 [a, ab, c, aa, ab, ac, aba, abb , abc, ca, cb, cc].

The first upper word strings according to one side may include a predetermined number of upper rank word strings having a high probability in the preliminary candidate group. According to an embodiment, step 130 may further include selecting a predetermined number of first upper word strings in an order of high probability from the preliminary candidate group.

According to an embodiment, when the processor performing step 130 includes one or more of the candidate word strings expanded in step 120 in the first higher order word strings, the first higher order word strings are extended. It can be determined that the candidate word string is included. Alternatively, when the first upper word strings do not include any one of the candidate word strings expanded in step 120, it may be determined that the extended candidate word strings are not included in the first upper word strings.

For example, referring to FIG. 2A , a preliminary candidate group including a candidate word sequence 201 and an extended candidate word sequence 202 [a, ab, c, aa, ab, ac, aba, abb, abc, ca , cb, cc], when the top three word strings 'a' (211), 'ab' (212), and 'cb' (214) with high probability correspond to the first high word strings, the first high word Since the extended candidate word string 'cb' is included in the columns, it may be determined in step 130 that the extended candidate word string is included in the first upper word strings.

For another example, referring to FIG. 2A , a preliminary candidate group including a candidate word sequence 201 and an extended candidate word sequence 202 [a, ab, c, aa, ab, ac, aba, abb, abc, ca, cb, cc], when the top three word strings 'a' (211), 'ab' (212), and 'c' (213) with high probability correspond to the first high word strings, the first high Since the extended candidate word strings are not included in the word strings, it may be determined in step 130 that the extended candidate word strings are not included in the first upper word strings.

Step 140 according to one side includes updating a preliminary candidate group by additionally extending at least one extended candidate word string included in the first upper word strings based on the determination result according to step 13. may apply.

According to an embodiment, the step of updating the preliminary candidate group (140) is not updating the preliminary candidate group when it is determined that the extended candidate word string is not included in the first upper word strings according to the step (130). can include In this case, the preliminary candidate group may include candidate word sequences and extended candidate word sequences of the candidate word sequences.

According to an embodiment, the step of updating the preliminary candidate group (140) includes the current frame and the first higher word string when it is determined that the extended candidate word string is included in the first upper word string according to step 130. Based on the probability of the at least one extended candidate word sequence included in , further extending at least one extended candidate word sequence included in the first upper word sequences, and adding the additionally extended candidate word sequence to a preliminary candidate group. By adding, updating the preliminary candidate group may be included.

According to an embodiment, the extended candidate word string included in the first upper word string may correspond to a word string in which the candidate word string is expanded once according to step 120 . The step of additionally extending the expanded candidate word sequence included in the first upper word sequences may include extending the once-extended candidate word sequence included in the first upper word sequences one more time. In other words, the additionally expanded candidate word sequence(s) may correspond to word sequence(s) obtained by extending at least some of the candidate word sequences included in a beam corresponding to the current frame twice.

For example, referring to FIG. 2B , the preliminary candidate group before updating is a set [a, ab, c, aa, ab including candidate word sequences 201 and candidate word sequences 202 expanded once. , ac, aba, abb, abc, ca, cb, cc]. When 'a', 'ab', and 'cb' are included in the first upper word strings including the top three word strings with high probability in the preliminary candidate group, the candidate word extended once to the first upper word strings Since the column 'cb' is included, it may be determined in step 130 that the extended candidate word sequence is included in the first upper word sequences. The extended candidate word string 'cb' included in the first upper word strings may be additionally extended by step 140, and the candidate word strings 203 to which the extended candidate word string 'cb' is additionally extended are It can be added to the preliminary candidate group. The preliminary candidate group updated by the execution of step 140 is [a, ab, c including candidate word sequences 201, extended candidate word sequences 202, and additionally extended candidate word sequences 203. , aa, ab, ac, aba, abb, abc, ca, cb, cc, cba, cbb, cbc].

Meanwhile, in FIG. 2B , when the first upper word strings including the top three word strings with high probability in the preliminary candidate group include 'a', 'ab', and 'c', 1 is included in the first upper word strings. Since none of the candidate word strings 202 extended 202 are included, it may be determined in step 130 that the extended candidate word string is not included in the first upper word strings. In this case, the preliminary candidate group is not updated in step 140, and the set [a, ab, c, aa, ab, ac, aba including the candidate word strings 201 and the extended candidate word strings 202 , abb, abc, ca, cb, cc]. In addition, additional extended candidate word strings 203 may not be generated either.

Referring back to FIG. 1 , step 150 according to one side may correspond to a step of determining a predetermined number of second upper word strings in an order with high probability in the preliminary candidate group as beams corresponding to the next frame of the current frame. can The number of word strings included in the first upper word string may be the same as the number of word strings included in the second upper word string.

According to an embodiment, when it is determined in step 130 that the extended candidate word string is included in the first upper word strings, in step 140, the candidate word string is selected according to the order of high probability within the updated preliminary candidate group. A certain number of second upper word strings may be determined as beams corresponding to the next frame. The second upper word strings selected in the order of high probability within the updated preliminary candidate group may remain the same as the first upper word strings or may be different. For example, when it is assumed that the number of first upper word strings and second upper word strings is N, the candidate word strings extended twice added to the preliminary candidate group in step 140 are ranked higher in order of probability. If they do not belong to N number of word strings, the second upper word strings may be determined identically to the first upper word strings.

According to an embodiment, when it is determined in step 130 that the extended candidate word sequence is not included in the first upper word sequences, the preliminary candidate group is not updated, so that the second upper word sequences are the first upper word sequences. can be the same as In other words, the first upper word strings and the second upper word strings may match each other.

According to an embodiment, the first high-order word strings and the second high-order word strings are to indicate some of the word strings included in the preliminary candidate group, and the first high-order word strings are distinguished from the second high-order word strings. It does not correspond to instances or objects. For example, both the first upper word strings and the second upper word strings may mean top N word strings with high probability among word strings included in the preliminary candidate group at a specific time point.

According to an embodiment, whether or not the preliminary candidate group is updated depends on the determination result of step 130, and the candidate sequences determined as the beam corresponding to the next frame are the top N word sequences with high probability in the preliminary candidate group. may apply.

For example, referring to FIG. 3 , the speech recognition method according to an embodiment determines whether the extended candidate word string is included in the top N word strings with high probability in the preliminary candidate group after step 120 shown in FIG. 1 . In step 310, the preliminary candidate group is not updated according to the determination result, and the upper N word sequences in the preliminary candidate group including the candidate word sequences and the candidate word sequences expanded once are determined as beams corresponding to the next frame. Any one of an operation and an operation of updating the preliminary candidate group and determining upper N word sequences from the updated preliminary candidate group as beams corresponding to the next frame may be performed.

More specifically, when it is determined that the extended candidate word string is included in the top N word strings in the preliminary candidate group, the speech recognition method according to one side further extends the extended candidate word string included in the top N word strings. step 320, adding the extended candidate word string to the preliminary candidate group 330, re-selecting the top N word strings from the preliminary candidate group 340, and selecting the top N word strings selected again as follows. A step 350 of determining a beam corresponding to the frame may be included.

Meanwhile, when it is determined that the extended candidate word strings are not included in the first upper word strings, the speech recognition method according to one side determines the upper N word strings in step 310 as beams corresponding to the next frame. It may include step 350 of doing.

Referring back to FIG. 1, if the current frame is the last frame of the voice signal, in step 150 of determining a beam corresponding to the next frame, at least some of the second upper word strings are used for voice recognition corresponding to the voice signal. A step of outputting as a result may be further included.

4 is a diagram illustrating the structure of an RNN-T, which is an example of an end-to-end speech recognition model according to an embodiment.

RNN-T is an end-to-end speech recognition framework that converts input sound frames into character sequences. RNN-T can transcribe x = {x _t } ^T _t=1 to y = {y _u } ^U _u=1 . Here, x _t denotes a sound frame, t denotes a frame index, y _u denotes an output label, and u denotes an output label index.

The length (or number) T of sound frames and the length (or number) U of output labels may be different, for example, T may have a larger value than U. RNN-T can handle the length difference between x and y by introducing a blank symbol Ø. Blank symbols enable the RNN-T to decide whether to generate output labels during the decoding process of T steps. For example, |y| = U, the decoding path of an RNN-T for y = [y ₁ , y ₂ , y ₃ ] is |y*| = T, y* = [y ₁ , Ø, Ø, ..., y ₂ , Ø, y ₃ ].

Referring to FIG. 4 , the RNN-T may include an encoder 410, a prediction network 420, and a joint network 430. The encoder 410 may map sound frames x to hidden representations as shown in Equation 1 below.

The prediction network 420 may estimate a hidden state p _u based on the previous non-blank output label y _u−1 as shown in Equation 2 below.

The joint network 430 may combine latent expressions by the encoder 410 and the prediction network 420 as shown in Equation 3 below.

Here, W represents a weight matrix and b represents a bias vector. The posterior probability of each kth output label can be obtained by applying the softmax layer 440 as shown in Equations 4 and 5 below.

where y = [y ₁ , . . . , y _u ].

5 is an operational flowchart of a decoding process for speech recognition performed based on RNN-T according to an embodiment. More specifically, FIG. 5 illustrates a decoding process of obtaining a beam corresponding to frame t+1 (or t+1-step) based on a beam corresponding to frame t (or t-step).

A decoding process according to an embodiment may be composed of a plurality of expansion blocks. An expansion block may correspond to a set of operations for expansion search on a beam search. The number of expansions possible in a single expansion block may be limited to one time, and expansion search may be performed in parallel by batch processing hypotheses. In other words, all hypotheses can be expanded simultaneously in an extension block. Two expansions can be performed by cascading two expansion blocks. For example, the operation of a single extension block may include an operation of extending the above-described candidate word sequences (eg, step 120 of FIG. 1), and the operation of two extension blocks connected in a cascade manner may include the above-described extended extension block operation. An operation of additionally extending the candidate word string (eg, step 140 of FIG. 1 or step 320 of FIG. 3) may be included.

According to one embodiment, a decoding process may include two or more extension blocks. For example, it may include three extension blocks to improve word error rate (WER).

Batch computation applied to expansion blocks according to one side may accelerate expansion exploration, but expansion exploration may be performed even for hypotheses that do not require expansion, which may increase computational overhead. Accordingly, in order to avoid unnecessary extended search in a decoding process, an extension block according to one side may include a process of adaptively determining whether to continue an extended search.

In the case of using two cascaded extension blocks in a decoding process according to one side, there may be three scenarios of not extending, extending once, or extending twice according to the posterior probability of the hypothesis.

According to an embodiment, an input for an extension block in step 510 is a list of hypotheses, and beam _t may be allocated as a list of a first extension block. Beam _t includes hypotheses in the decoding process for frame t (or the decoding process of t-step). For example, beam _t may correspond to a beam corresponding to the above-described specific frame t. Hypotheses of beam _t may correspond to candidate word sequences included in a beam corresponding to a specific frame t, and step 510 may correspond to step 110 of FIG. 1 described above.

According to an embodiment, in the first expansion block (expansion block 1) 520, _LoadLMStates is a prediction network (e.g., the prediction network 420 of FIG. )) is an operation that loads the hidden state of The hidden state of the batched prediction network as a result of LoadLMStates can be expressed as lmh _batch (521).

According to an embodiment, BatchJointFn related to Equations 3 to 5 performs batch calculation using e _t and lmh _batch output from an encoder (eg, the encoder 410 of FIG. 4 ) to obtain all possible hypotheses in the list. It is an operation that obtains posterior probabilities for extensions. The result of BatchJointFn is batched into prob _batch (522). For example, prob _batch may correspond to a probability that the above-described specific frame t corresponds to a candidate word.

According to an embodiment, all logPr(k|y, t) of the prob _batch are added to the accumulated log probabilities for previous hypotheses in the list using the AccumulateProb operation, and the result is assigned to the acc_prob _batch (523). For example, the accumulated log probabilities of the previous hypotheses may correspond to the probabilities of the above-described candidate word sequence, and acc_prob _batch may include the above-described extended probability of the candidate word sequence. acc_prob _batch may further include the above-described probabilities of the candidate word sequence.

According to one embodiment, FindTopIndexes is an operation for extracting indices of higher (N + β) posterior probabilities in acc_prob _batch . The index of the top (N + β) posterior probabilities by FindTopIndexes is assigned to top_indexes (524). For example, step 524 may correspond to the operation of selecting the above-described first upper word strings. Here, β is the margin factor. This will be described in detail below.

According to an embodiment, UpdateHyps is an operation that updates hypotheses in a list by referring to top_indexes including an index of a specific hypothesis whose output label is k. For example, if k represents a non-blank label, the corresponding hypothesis may be updated by adding the k th output label using the UpdateHyps operation, and in this case, the posterior probability of the updated hypothesis is incomplete probabilities. can be matched. As another example, when k represents a blank label, the corresponding hypothesis may be updated by adding information indicating that the extension to the corresponding hypothesis has ended in frame t using the UpdateHyps operation. Probabilities may correspond to complete probabilities. The posterior probability of the updated hypothesis is the value of acc_prob _batch indexed by top_indexes. Updated hypotheses are assigned to newhyps (525). For example, operation 525 may correspond to the above-described step of obtaining extended candidate word sequences and probabilities of the extended candidate word sequences.

According to an embodiment, duplication between hypotheses may occur during the expansion procedure. For example, if beam _t includes two hypotheses [a] and [a, b], hypothesis [a] can be expanded to [a, b] through the expansion process of UpdateHyps, and in this case, between hypotheses duplication occurs.

According to an embodiment, prior to performing the decoding process shown in FIG. 5, a prefix search may be performed in correspondence to beam _t . The prefix search may correspond to a process of correcting the posterior probability of the searched hypothesis by searching for a hypothesis having one of the hypotheses included in beam _t as a prefix. For example, if beam _t includes [a] and [a, b], since [a, b] corresponds to a hypothesis having [a] as a prefix, The posterior probabilities can be calibrated. The posterior probability of [a, b] is calculated by calculating the posterior probability when [a] expands to [a, b] in frame t, and by adding it to the posterior probability of beam _t hypothesis [a, b] in advance can be calculated.

The posterior probability of [a, b] in which [a] is expanded through the prefix search process according to an embodiment may be aggregated into the posterior probability of [a, b] already included in beam _t . Therefore, in the UpdateHyps operation, the case of extending to a hypothesis that already exists in beam _t is ignored. In other words, in order to prevent duplication of hypotheses in beam _t+1 , if an extended hypothesis already exists in beam _t , the extension of the corresponding hypothesis is ignored. In one embodiment, since extended hypotheses that overlap with hypotheses included in beam _t are ignored, the number of hypotheses assigned to newhyps may be less than the number of elements of higher indexes. Therefore, the number of hypotheses of newhyps can be kept higher than N by including N + β elements in top_indexes.

According to an embodiment, newhyps may be divided into list _b and list _e1 by BlankHyps operation and NonBlankHyps operation (526). For example, among the hypotheses included in newhyps, the hypothesis to which information indicating that expansion has been completed is added to list _b by the BlankHyps operation, and the hypothesis to which information indicating that expansion has been completed is not added to list _e1 by NonBlankHyps operation. can be distinguished. The posterior probability of list _b is a perfect probability, and the posterior probability of list _e1 may correspond to an incomplete probability probability. To obtain perfect probability, list _e1 can be further processed. For example, list _b may correspond to candidate word sequences included in the above-described first upper word sequences, and list _e1 corresponds to extended candidate word sequences included in the above-described first upper word sequences. It can be. Also, for example, the step of checking whether list _e1 exists (527) is a step of determining whether the above-described first higher order word strings include an extended candidate word string (eg, step 130 of FIG. 1 or FIG. It may correspond to step 310 of 3).

According to an embodiment, when list _e1 does not exist, the search procedure may be terminated by selecting 530 top N hypotheses from list _b . This case is referred to as Scenario 1. For example, in scenario 1, when the extended candidate word sequences are not included in the above-described first higher-order word sequences, the step of determining the first higher-order word sequences as beams corresponding to the current frame without updating the preliminary candidate group. can be matched.

Meanwhile, when list _e1 exists, a search procedure for list _e1 may be continued to determine whether list _e1 is further extended.

According to an embodiment, BatchDecoderFn is an operation that performs batch calculation of f ^pre according to Equation 2 above using the original lmh _batch of list _e1 and label _prev corresponding to the last label batch of hypotheses in list _e1 . . BatchDecoderFn operation updates lmh _batch of list _e1 (528). If the search procedure follows scenario 1, since the BatchDecoderFn operation does not need to be performed, BatchDecoderFn may be performed after checking the existence of list _e1 (527), not before the BatchJointFn operation (522). The updated lmh _batch is allocated (529) to the lmh _batch of list _e1 by SaveLMStates.

According to an embodiment, in the second expansion block (expansion block 2) 550, an operation similar to that of the first expansion block 520 is performed on list _e1 . The branching point between scenarios 2 and 3 is whether list _e2 exists (557). For example, list _e2 may correspond to additionally extended candidate word sequences included in the above-described second upper word sequence. If list _e2 does not exist, the top N hypotheses are selected from list _b and list _e1b (560), and the search procedure is terminated. This case is referred to as Scenario 2. Scenario 2 is the second higher order word sequence including the candidate word sequence and the extended candidate word sequence when the additionally extended candidate word sequence is not included in the N second higher word sequences in the order of high probability in the above-described updated candidate word sequence. This may correspond to a step of determining top N word strings in an order with high probability in the word string as beams corresponding to the current frame.

According to an embodiment, when list _e2 exists, the second extension block 550 outputs list _e2 having an incomplete probability by performing operations 558 and 559. Additional blank probabilities can be calculated and aggregated 570 as the posterior probabilities of list _e2 by AccumulateBlankProb, which includes the BatchJointFn operation and the AccumulateProb operation. Display the completed version of list _e2 as list _e2b . Thereafter, the top N hypotheses are selected from list _b , list _e1b , and list _e2b (580), and the search procedure is terminated. This case is referred to as Scenario 3. For example, in scenario 3, when the additionally extended candidate word sequences are included in the N second higher order word sequences with high probability in the above-described updated candidate word sequences, the candidate word sequences, the extended candidate word sequences, and the additional expansion This may correspond to the step of determining, as beams corresponding to the current frame, the top N word strings in the order of high probability in the second upper word string including the candidate word string.

6 is an operation flowchart of a beam search method according to an embodiment.

1 to 5, the speech recognition method including a decoding process based on the beam search algorithm according to one side has been described above, but the beam search algorithm according to one side can be applied to various embodiments using tree search.

Referring to FIG. 6, the beam search method according to an embodiment includes adding candidate word sequences (beam _t ) included in a beam corresponding to a current frame (t) of encoded data to a preliminary candidate group (610), Steps 620 to 640 of performing an expansion operation on word sequences, and extending candidate word sequences obtained based on the expansion operation on word sequences (eg, N upper word sequences) selected based on the expansion operation. Determining whether to repeat the expansion operation based on whether or not the word sequence is included (650), and performing the expansion operation on the selected word sequences according to the decision to repeat the expansion operation (670, 630, and 640) ), and determining that the expansion operation is not repeated, determining upper word sequences as beams _t+1 corresponding to the next frame t+1 of the encoded data (step 660). .

The expansion operation according to one side includes adding extended word sequences of word sequences subject to the expansion operation to a preliminary candidate group (630) and a certain number of word sequences according to a predetermined criterion in the preliminary candidate group (eg, N A step of selecting (640) the upper word string) may be included. Here, the predetermined criterion is a criterion for determining the priority of word strings included in the preliminary candidate group, and may include, for example, an order in which probabilities corresponding to word strings included in the preliminary candidate group are high.

Step 650 of determining whether to repeat the expansion operation according to one side includes determining to repeat the expansion operation when the selected word sequences include the extended word sequence, and When the word string is not included, determining not to repeat the expansion operation may be included.

According to an embodiment, the step of determining whether to repeat the extension operation ( 650 ) may include determining whether to repeat the extension operation based on whether a predetermined repetition end condition is satisfied ( 680 ). Here, the repetition end condition is a condition for terminating repetition of a predetermined extension operation in relation to the beam search. For example, when the number of repetitions of the extension operation exceeds a threshold value (eg, 2 times), the number of repetitions corresponding to the upper word sequence A case where the probability exceeds a predetermined threshold value may be included.

7 is an exemplary diagram of a configuration of a device according to an embodiment.

Referring to FIG. 7 , a device 700 includes a processor 701 , a memory 703 and an input/output device 705 . The apparatus 700 may include, for example, a user device (eg, a smart phone, a personal computer, a tablet PC, etc.) and a server.

The apparatus 700 according to an embodiment may include an apparatus for performing the above-described voice recognition method and/or beam search method. The processor 701 may perform at least one method described above through FIGS. 1 to 5 . For example, the processor 701 may perform speech recognition including the above-described beam search-based decoding process. The processor 701 may obtain encoded data corresponding to the voice signal by encoding the input voice signal, and may obtain text data corresponding to the voice signal by decoding the encoded data. According to an embodiment, the processor 701 may perform a decoding process based on an end-to-end speech recognition model.

The memory 703 may store information related to a voice recognition method and/or a beam search method, and may store data related to an end-to-end voice recognition model. Memory 703 may be volatile memory or non-volatile memory.

Device 700 according to one side may be connected to an external device (eg, a personal computer or network) through an input/output device 705 and exchange data. For example, the device 700 may receive a voice signal through the input/output device 705 and output text data corresponding to the voice signal as a result of voice recognition of the voice signal.

According to an embodiment, the memory 703 may store a program in which the above-described voice recognition method and/or beam search method are implemented. The processor 701 may execute a program stored in the memory 703 and control the device 700 . Program codes executed by the processor 701 may be stored in the memory 703 .

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer readable medium may store program instructions, data files, data structures, etc. alone or in combination, and program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. have. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

In the voice recognition method performed by the processor,
obtaining candidate word sequences included in a beam corresponding to a target frame of a speech signal and probabilities of the candidate word sequences corresponding to the speech signal;
expanding the candidate word sequences based on the target frame and the probabilities of the candidate word sequences;
Preliminary candidate group-The preliminary candidate group includes the candidate word sequences and extended candidate word sequences of the candidate word sequences-A predetermined number of first upper word sequences according to an order of high probability within the extended candidate word sequences determining whether at least one of them is included;
updating the preliminary candidate group by additionally extending the at least one extended candidate word string included in the first upper word strings based on a result of the determination; and
Determining a predetermined number of second higher order word sequences in an order with a higher probability in the preliminary candidate group as a beam corresponding to a frame next to the target frame
including,
voice recognition method.
According to claim 1,
Updating the preliminary candidate group
When it is determined that the at least one extended candidate word sequence is included in the first upper word sequences,
further extending the at least one extended candidate word sequence based on a probability of the at least one extended candidate word sequence included in the target frame and the first upper word sequences; and
Updating the preliminary candidate group by adding the additionally extended candidate word sequence to the preliminary candidate group;
including,
voice recognition method.
According to claim 2,
The step of determining the beam corresponding to the next frame
selecting a predetermined number of second high-ranking word strings in an order of high probability within the updated preliminary candidate group; and
Determining the second upper word sequences as beams corresponding to the next frame
including,
voice recognition method.
According to claim 1,
Updating the preliminary candidate group
not updating the preliminary candidate group when it is determined that the at least one extended candidate word sequence is not included in the first upper word sequences;
including,
The first upper word strings and the second upper word strings match each other
voice recognition method.
According to claim 1,
Expanding the candidate word strings
Corresponding to each of the candidate word sequences,
obtaining an extended candidate word sequence of the corresponding candidate word sequence by adding a candidate word to the corresponding candidate word sequence;
obtaining a probability that the target frame corresponds to the candidate word based on the corresponding candidate word string; and
obtaining a probability of the extended candidate word sequence based on the probability of the corresponding candidate word sequence and the probability that the target frame corresponds to the candidate word;
including,
voice recognition method.
According to claim 1,
Encoding the voice signal to obtain a plurality of frames including features extracted from the voice signal
Including more,
voice recognition method.
According to claim 1,
Expanding the candidate word strings
obtaining the extended candidate word sequences and the probabilities of the extended candidate word sequences based on an end-to-end speech recognition model;
including,
The end-to-end speech recognition model includes a neural network trained to receive the speech data and output the text label based on training data including speech data and text labels corresponding to the speech data,
voice recognition method.
According to claim 1,
When the target frame is the last frame of the voice signal,
The step of determining the beam corresponding to the next frame
outputting at least some of the second upper word strings as a voice recognition result corresponding to the voice signal;
Including more,
voice recognition method.
According to claim 1,
The extended candidate word sequences of the candidate word sequences are
Including word sequences having the candidate word sequences as prefixes,
voice recognition method.
According to claim 1,
The word sequence is
contains a word and a sequence of plural words;
said word
Including one character corresponding to a specific language and a combination of a plurality of characters corresponding to the specific language,
voice recognition method.
In the beam search method performed by the processor,
adding candidate word sequences included in a beam corresponding to a target frame of encoded data to a preliminary candidate group;
performing an expansion operation on the candidate word sequences;
determining whether or not to repeat the expansion operation based on whether word sequences selected based on the expansion operation include an extended word sequence of the candidate word sequences obtained based on the expansion operation;
performing the expansion operation on the selected word sequences when it is determined to repeat the expansion operation;
Determining the selected word strings as beams corresponding to the next frame of the encoded data when it is determined not to repeat the extension operation
including,
The expansion operation is
adding extended word strings of word strings subject to the expansion operation to the preliminary candidate group; and
Selecting a certain number of word strings according to a predetermined criterion from the preliminary candidate group
including,
Beam search method.
According to claim 11,
The step of determining whether to repeat the expansion operation is
determining to repeat the expansion operation when the selected word sequences include the extended word sequence; and
determining not to repeat the expansion operation when the extended word sequence is not included in the selected word sequences;
including,
Beam search method.
According to claim 11,
The step of determining whether to repeat the expansion operation is
determining whether to repeat the expansion operation based on whether a predetermined repetition end condition is satisfied;
including,
Beam search method.
A computer program stored in a medium to execute the method of any one of claims 1 to 13 in combination with hardware.
Obtaining candidate word sequences included in a beam corresponding to a target frame of a speech signal and probabilities of the candidate word sequences corresponding to the speech signal;
Based on the target frame and the probabilities of the candidate word strings, expand the candidate word strings;
Preliminary candidate group-The preliminary candidate group includes the candidate word sequences and extended candidate word sequences of the candidate word sequences-A predetermined number of first upper word sequences according to an order of high probability within the extended candidate word sequences determine whether at least one of them is included,
Based on the determination result, updating the preliminary candidate group by additionally extending the at least one extended candidate word sequence included in the first upper word sequences;
Determining the predetermined number of second high-order word strings in order of high probability in the preliminary candidate group as a beam corresponding to a frame next to the target frame,
at least one processor
including,
voice recognition device.
According to claim 15,
the processor,
In updating the preliminary candidate group,
When it is determined that the at least one extended candidate word sequence is included in the first upper word sequences,
Further extending the at least one extended candidate word sequence based on a probability of the at least one extended candidate word sequence included in the target frame and the first upper word sequences;
Updating the preliminary candidate group by adding the additional extended candidate word sequence to the preliminary candidate group.
voice recognition device.
According to claim 16,
the processor,
In determining the beam corresponding to the next frame,
Selecting a certain number of second upper word strings in an order with a higher probability in the updated preliminary candidate group;
Determining the second upper word strings as beams corresponding to the next frame,
voice recognition device.
According to claim 15,
the processor,
In updating the preliminary candidate group,
When it is determined that the at least one extended candidate word string is not included in the first upper word strings, the preliminary candidate group is not updated;
The first upper word strings and the second upper word strings match each other,
voice recognition device.
According to claim 15,
the processor,
In expanding the candidate word sequences,
Corresponding to each of the candidate word sequences,
Adding candidate words to the corresponding candidate word sequence to obtain an extended candidate word sequence of the corresponding candidate word sequence;
Obtaining a probability that the target frame corresponds to the candidate word based on the corresponding candidate word string;
obtaining a probability of the extended candidate word sequence based on a probability of the corresponding candidate word sequence and a probability that the target frame corresponds to the candidate word;
voice recognition device.
Adding candidate word sequences included in a beam corresponding to a target frame of encoded data to a preliminary candidate group;
performing an expansion operation on the candidate word sequences;
Determine whether or not to repeat the expansion operation based on whether word sequences selected based on the expansion operation include an extended word sequence of the candidate word sequences obtained based on the expansion operation;
When it is determined to repeat the expansion operation, the expansion operation is performed on the selected word strings,
Determining the selected word sequences as beams corresponding to the next frame of the encoded data according to determining not to repeat the extension operation,
at least one processor
including,
The expansion operation is
Adding extended word strings of word strings subject to the expansion operation to the preliminary candidate group and selecting a certain number of word strings according to a predetermined criterion from the preliminary candidate group,
Beam search device.

Images