JP6358744B2 - Speech recognition error correction device - Google Patents

Description

  The present invention relates to a speech recognition error correction apparatus.

In recent years, research has been conducted on techniques for automatically providing subtitle texts in programs such as television.
Non-Patent Document 1 describes an example of Spanish CARTV as one of the approaches for providing captions to a live broadcast news program. The system described in Non-Patent Document 1 is based on the premise that news items, their order, and a reading manuscript are given. The reading part is specified. The forced alignment is an algorithm for obtaining correspondence between each phoneme and a speech voice section when a phoneme string of speech content is given. In this system, it is possible to specify which part of the original is being read in units of phonemes, so that it is possible to determine a caption sentence at an early stage and output a caption with high consistency with a program video.

  Patent Document 1 discloses a technique for correcting a speech recognition error by comparing a recognized word string and a read original in units of words. The text correction device described in Patent Document 1 is characterized in that a word string in a speech recognition result is compared with a word string in a document in a word chain block of length N.

JP 2012-128188 A

JE Garcia, A. Ortega, E. Lleida, T. Lozano, E. Bernues, D. Sanchez, Aragon Inst. For Eng. Res., Univ. Of Zaragoza, Zaragoza, "Audio and text synchronization for TV news sub-titling based on Automatic Speech Recognition ", BMSB '09., IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, 2009

  The algorithm employed by the system described in Non-Patent Document 1 has a problem that the accuracy is significantly lowered when there is a discrepancy between the uttered speech and the phoneme string.

  On the other hand, in many news programs, it is difficult to specify news items and their order in advance. The order of news items is often changed during the broadcast of a program. In the example of CARTV described in Non-Patent Document 1, since a large-scale system linked with a news editing computer system (NRCS) is constructed, it is used for alignment whenever a news item or item order is updated. The phoneme sequence is updated.

  However, even if Non-Patent Document 1 adopts such a method, if the phoneme string update based on the management by NRCS is not in time, or if the document is skipped or paraphrased, appropriate alignment is performed. become unable. Therefore, in order to correct the alignment position, it is necessary to construct a complicated and large-scale system by acquiring and using a lap time that is a guide for program progression and the playback timing of the insert video. In addition, there are parts of the program where the content of utterances is not managed by the NRCS, such as an interview of an insert video. Therefore, in order to prevent the alignment from progressing in such a portion, complicated control such as managing the above-described insert video and stopping the input to the alignment based on the reproduction information is also required.

  For example, in a local broadcasting station that requires an automatic caption production system, it is difficult to construct and maintain such a complicated system, and a simpler system is required.

  Further, although the technique of Patent Document 1 does not require the complicated system configuration as described above, it does not estimate the sentence boundary of the recognition word hypothesis sequence, so that an inappropriate manuscript section and words of speech recognition result As a result of collating with the column, if there is a recognition error at the sentence boundary or block boundary, skipping or paraphrasing the manuscript, or adding an utterance that is not in the manuscript, the word is missing at the block boundary Or automatic correction errors such as the same word being output twice. Such an automatic correction error is the same even if the unit of the section in which the speech recognition result corresponds to the manuscript is set as a sentence or other unit as long as the boundary (sentence boundary) of the word hypothesis sequence based on the speech recognition result is unknown. Arise.

The present invention has been made in view of the above circumstances, and provides a speech recognition error correction apparatus with higher accuracy with a simple system configuration.
In particular, the present invention provides a speech recognition error correction device that can determine an output sentence (corrected sentence) at an appropriate stage in the middle even before the speaker reads the entire document. It is to provide.

  In order to solve the above problems, a speech recognition error correction apparatus according to an aspect of the present invention is a finite state transducer that sequentially transitions while accepting words input as speech recognition results in response to a document text. Information about the state of the finite state transducer that makes a state transition while also accepting an error in a word included in the speech recognition result, and a transition source state, a transition destination state, and an input symbol with respect to the state transition. A finite state transducer information storage unit that stores information including an output symbol and transition weight (cost required for transition), and a node data storage unit that stores a score representing the likelihood of the state in the finite state transducer And receiving an input of a recognition word that is a speech recognition result corresponding to the document text from outside In addition, a node data update unit that calculates a score at the time for each state in the finite state transducer according to the received recognition word, and updates the node data storage unit using the calculated score And when an activation signal indicating the start of processing is received from the outside, by referring to the node data storage unit, the maximum likelihood node at that time is determined, and the finite state transducer information storage unit, the node data storage unit, The traceback process is performed until the predetermined time when the state transition is confirmed by tracing back the state transition to the maximum likelihood node, and the path of the state transition subjected to the traceback process is set as an output candidate. , For the path that is an output candidate, the ratio of the state transition related to the error to the path A corresponding error score is calculated, and when the degree of error is smaller than a predetermined threshold based on the calculated error score, an output candidate of the path is obtained by the document search unit that determines output and the document search unit An output unit for outputting the determined output, wherein the finite state transducer information storage unit outputs the output transition where the output symbol is a non-empty state transition to the state. The output transition is provided after a path of state transition in which a sentence included in the original text is divided into a plurality of chunks and a word included in the chunk is accepted as an input symbol. The information on the state and the information on the state transition of the finite state transducer are stored.

In the speech recognition error correction apparatus having this feature, a sentence is divided into a plurality of chunks. Further, after accepting a word included in the chunk as an input symbol, an output transition is executed and an output symbol is output. That is, there is an output transition in the middle of the sentence. Therefore, this speech recognition error correction apparatus can output the determined output symbol even in the middle of a sentence.
The above transition weight is numerical data representing the cost related to the state transition. As an example, the transition weight is defined such that the cost is minimum when the transition weight is zero, and the cost increases as the negative transition weight value decreases (that is, the absolute value of the negative value increases). However, for the transition weight, the sign of this definition may be reversed, or another definition may be used. Using this weight by setting its weight for each state transition. The score for obtaining the maximum likelihood node can be calculated.

  According to another aspect of the present invention, in the speech recognition error correction apparatus, the document search unit calculates the error score for each chunk, and the degree of error is based on the error score for each chunk. If the output symbol of the output transition corresponding to the chunk is smaller than a predetermined threshold, the output symbol of the output transition is used as the definite output, and the document search unit calculates the error score for each chunk. An error score corresponding to a ratio of the state transition related to the error with respect to the chunk and a section of the chunk immediately before the chunk is calculated as an error score of the chunk.

  The speech recognition error correction apparatus according to an aspect of the present invention is a finite state transducer that sequentially transitions while accepting words input as speech recognition results in response to a document text. Information on the state of the finite state transducer that makes a state transition while accepting an error in a word included in the recognition result, and a transition source state, a transition destination state, an input symbol, an output symbol, and a transition weight ( A finite state transducer information storage unit that stores information including the cost of transition), a node data storage unit that stores a score representing the likelihood of the state in the finite state transducer, and the original text The input of the recognition word which is the recognition result of the corresponding voice is accepted from the outside, and the received A node data update unit that calculates a score at the corresponding time for each state in the finite state transducer according to a sense word, updates the node data storage unit using the calculated score, and starts processing. When receiving the activation signal shown from the outside, by referring to the node data storage unit, to determine the maximum likelihood node at that time, with reference to the finite state transducer information storage unit and the node data storage unit, By tracing back the state transition to the maximum likelihood node, traceback processing is performed until a predetermined time when the state transition has been confirmed, and the path of the state transition subjected to the traceback processing is set as an output candidate and becomes an output candidate. For the path, an error score is calculated according to the ratio of the state transition related to the error to the path. When the degree of error is smaller than a predetermined threshold based on the calculated error score, a document search unit that determines the output candidate of the path as a definite output and the definite output obtained by the document search unit are output The finite state transducer information storage unit includes an output transition in which the output symbol is a non-empty state transition as a part of the state transition. The document search unit is characterized in that the node having the maximum likelihood among the nodes corresponding to the transition source state of the output transition is determined as the maximum likelihood node at that time.

  The speech recognition error correcting apparatus having this feature determines the maximum likelihood node only from the nodes that are the transition source states of the output transition. As a result, the speech recognition error correction apparatus can perform processing with a smaller amount of calculation, and can determine the output at an early stage when the error level of the corresponding document exceeds the threshold value only with the already input recognition result. That is, the speech recognition error correction apparatus can quickly perform processing that leads to the determination of output.

According to the present invention, an error in a speech recognition result can be corrected without requiring a system having a complicated configuration.
Furthermore, according to the present invention, it is possible to reduce the occurrence of automatic correction errors that have occurred due to block boundaries when using the prior art.
Further, according to the present invention, it is possible to quickly determine and output a corrected recognition result.

It is a block diagram which shows the outline of a structure of the system containing the speech recognition error correction apparatus by embodiment of this invention. It is a state transition diagram showing an example of WFST (weighted finite state transducer). It is a state transition diagram which shows the example of WFST provided with the characteristic for using in this embodiment. It is a functional block diagram which shows schematic function structure of the speech recognition error correction apparatus by the embodiment. It is a flowchart which shows the procedure of the process by the speech recognition error correction apparatus by the embodiment. It is the schematic which shows the example of the determination method of a caption text output by the embodiment.

  Next, an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, a technique is used in which the correspondence between a word string that is inevitably mixed with an error such as a speech recognition result and a document that is an information source of the word string is determined early. Thereby, the error included in the speech recognition result is automatically corrected.

[1. System configuration to which this embodiment is applied]
FIG. 1 is a block diagram showing an outline of a system configuration including a speech recognition error correction apparatus according to the present embodiment. As shown in the figure, the system includes a speech recognition error correction device 100, a document text storage device 200, a speech recognition device 220, and a transducer construction device 240.

  The system shown in FIG. 1 includes a speech recognition error correction apparatus 100 that is applied when subtitles are added to a news program at a broadcasting station using speech recognition. The utterance voice in such a program is characterized in that it is generally based on a manuscript text prepared in advance. By the way, in a large-scale key broadcasting station, there is a case where an operator for manually correcting a recognition error included in a speech recognition result is arranged in order to broadcast subtitles. On the other hand, local broadcasters often find it difficult to locate such operators due to various circumstances. According to this embodiment, even in such a local broadcasting station, it becomes possible to correct the speech recognition apologization without arranging an operator.

The speech recognition error correction apparatus 100 receives a recognition result word string (recognition word string) output from the speech recognition apparatus 220 as an input, and estimates a word string of a corresponding document stored in advance, thereby being included in the recognition word string. It corrects errors.
Here, the information stored in advance for the estimation process by the speech recognition error correction apparatus 100 is a set of corresponding originals constructed by reading the original text stored in the original text storage device 200 in advance. This set of corresponding manuscripts includes a weighted finite state transducer (hereinafter referred to as “WFST”) having nodes representing states and branches (arcs) representing state transitions between the nodes as a network (directed graph). Or simply "finite state transducer").
The speech recognition error correction apparatus 100 sequentially checks the best (maximum likelihood) hypothesis on the WFST network, and based on the edit distance between the word string of the corresponding manuscript on the WFST and the recognized word string, the word strings of all recognition results. Without waiting for input, the final best hypothesis is approximated and the correction results are partially determined sequentially.

  In response to the recognition result word being input from the speech recognition device 220, the speech recognition error correction device 100 uses WFST to obtain a transition that can accept the input word, calculates its score, and accumulates the score. A search (Viterbi search) using a Viterbi algorithm is used while pruning using a threshold for. The Viterbi algorithm is an existing technique, and is effective in performing a search using a trellis diagram when estimating a code sequence closest to a transmission code with respect to a received sequence, that is, a code sequence having the maximum likelihood. It is a simple method.

  In normal Viterbi search, after all the inputs are observed, the path with the best score is traced back and the best hypothesis is output. Therefore, in the normal search method, it is not possible to output the correction results sequentially from the oldest input before all the inputs are observed. For example, assuming that the subtitles are produced from the result of recognizing the broadcast sound of a TV broadcast program and superimposed on the image of the TV screen in real time, the maximum likelihood sequence by the normal Viterbi search must input words until the end of the program It cannot be confirmed. This will end the program, so normal Viterbi search is not suitable for such operations.

  On the other hand, the speech recognition error correction apparatus 100 according to the present embodiment sequentially approximates the maximum likelihood sequence and traces back in a timely manner while using the Viterbi search. That is, every time a predetermined processing start condition is satisfied, a path with the best score at that time is traced back and an output transition that can be determined is determined, so that correction results can be output sequentially. The path traced back here is an approximation of the best hypothesis, but whether or not the path is determined based on the reliability of the edit distance between the input word string corresponding to each output transition and the word string of the manuscript. To improve the approximation accuracy. Details thereof will be described later.

The document text storage device 200 stores document text data. The manuscript text includes a plurality of articles. Each article includes a plurality of sentences.
The manuscript text storage device 200 stores the entire transcript of what the person plans to speak. The manuscript text storage device 200 stores, for example, texts representing a number of subdivided individual contents according to word string delimiter units such as sentences, sentences, paragraphs, and content classifications such as themes and topics. Such individual contents are hereinafter simply referred to as a manuscript. The unit of the word string is a sentence as an example.
The document text storage device 200 is constructed by using a general storage means such as a hard disk device or a semiconductor memory. The document text storage device 200 may be accessed through a communication network or the like as a so-called cloud service.

The speech recognition device 220 recognizes a speech voice read by a person (an announcer or the like) from the document 201 included in the document text storage device 200, and outputs a word string as a recognition result.
When raw speech data is input, the speech recognition device 220 recognizes speech data using an acoustic model based on a hidden Markov model (HMM) and a language model, and recognizes the recognized result as a recognition word string. Is generated as In the present embodiment, the voice recognition device 220 is not particularly limited, and a device using a known technique can be employed.

  As will be described later, in a broadcast news program, a plurality of versions of a manuscript are submitted for each news item, and it is not possible to determine in advance which version will be broadcast in which order. In such a situation, the voice recognition device 220 is required to perform voice recognition and immediately check whether or not a corresponding document exists for the uttered voice. For this reason, the language model used for speech recognition is preliminarily adapted using text data stored in the document text storage device 200 in order to make correspondence between the speech recognition result and the document with high accuracy, and according to the document. It is preferable that the recognition accuracy when reading is increased.

  The transducer construction device 240 constructs a WFST as a collection of corresponding originals (corresponding original collection) used in the speech recognition error correction apparatus 100. The transducer construction device 240 constructs in advance a WFST to be used by the speech recognition error correction device 100 from a read-out original to be speech-recognized, that is, a document sentence included in the document text storage device 200. WFST is a finite state machine having input symbols, output symbols, and transition weights, and can efficiently handle input / output of different granularities such as words and sentences. The construction of this WFST will be described later.

  The transducer construction device 240 includes a word network registration unit 241 and an editing network registration unit 242 as shown in the figure.

The word network registration unit 241 reads the document text included in the document text storage device 200 and performs a series of processes described below for each predetermined unit (for example, sentence unit). That is, the word network registration unit 241 starts from the start point node of the WFST network, and each time a word included in the document text is read from the document text storage device 200, a state transition (input transition) for accepting the word is received. Create a branch and a new node. The word network registration unit 241 sequentially creates the above branches and nodes for each read word until the end of a predetermined unit (for example, sentence unit) in the document.
Then, in the WFST network, an output transition branch of the read word string is added and connected to the end node.

  The editing network registration unit 242 corresponds to a branch representing a state transition that accepts an arbitrary word corresponding to a word replacement and a word insertion between nodes of the WFST network created by the word network registration unit 241. Thus, a branch representing a state transition that accepts an arbitrary word and a branch representing a state transition that transitions to the output side even if there is no input in response to the deletion of the word are added.

In the present embodiment, the system operates under the following conditions (A1) to (A7), for example.
(A1) A plurality of document sentences among the document sentences stored in the document text storage device 200 are read out as speech recognition targets.
(A2) A number of updated versions (versions) are also prepared for a news manuscript for one news item, and it is impossible to determine in advance which version of the manuscript will be read in the news program.
(A3) The order in which a plurality of manuscript sentences are read is not known in advance.
(A4) Some document texts included in the document text storage device 200 are skipped without being read out.
(A5) Some readers (announcers, casters, reporters, etc.) do not read the original manuscript faithfully, but may change the wording in consideration of ease of transmission to the viewer, An error may occur.
(A6) We want to make it a priority to avoid sending unclear subtitles due to recognition errors of the speech recognition device 220 to mislead or make viewers uncomfortable. Therefore, it is not sent in the case of an unknown recognition result, and instead, the subtitle is a manuscript (prior manuscript) that is automatically estimated to be the closest to the utterance content that has been proofread by the editor and confirmed in advance. Send it out.
(A7) If there is no original corresponding to the recognition result in an interview part or the like, automatic estimation is impossible, so no subtitles are transmitted for an interview part without an original.

[2. General WFST example]
FIG. 2 is a schematic diagram showing an example of a general WFST. WFST is expressed as a definition of state transition. The state transition has a node representing the state and a branch representing the state transition. Note that state transition may be simply referred to as transition. In the illustrated WFST, an input symbol is a word and an output symbol is a predetermined word string. The predetermined word string is a sentence. In the figure, the nodes are indicated by ellipses. Further, the branches are represented by lines with arrows. That is, the branch has a direction.

  In the figure, each ellipse node is given a three-digit number for identifying the node. The start point node is node 001, and the end point node is node 008. In this example, nodes 002 to 007 are arranged in a straight line between the start point and the end point. Further, between the start point and the end point, the nodes 010 to 015 are arranged in a straight line in parallel with the above-described series of nodes 002 to 007. Furthermore, between the start point and the end point, the above-described series of nodes 002 to 007 and the series of nodes 010 to 015 are arranged in a straight line with nodes 018 to 023. Further, a branch representing a state transition is given a word or a symbol such as <S>, <I>, <D>, <Emi1>, <Emi2>, <Emi3>, <eps>. Further, the branch representing the state transition extends from the transition source node to the transition destination node.

The state transition shown in the figure is generalized and described. In this WFST, a parameter (S ⁱ / S ^o : ω) is set for each transition. Here, S ⁱ represents a word input accepted by the state transition. S ^o represents a word string output by the state transition. The output word string is a sentence or a part of a sentence. Ω represents a state transition weight. That is, a triple parameter is set for each transition. In FIG. 2, for the sake of convenience, not all the parameters of the state transition are shown, but only one of the three parameters, ^Si or ^So , is shown corresponding to each state transition. ing.

  Here, the word described in the figure is generalized and expressed as a word s. Note that uppercase and lowercase letters are distinguished. In the figure, a word s is a word included in the word string of the document. When a word s is added to a certain state transition, it indicates that the state transition occurs only when the word s is input. That is, in a certain state, when the word s is added to the state transition having the state as a transition source, the state transition occurs only when the word s is input as a word in the recognized word string. That is, the state transition in which the word s is described is a transition that proceeds by accepting the speech-recognized word s. When a state transition occurs, the transition destination state becomes the next state.

  In the state transition shown in the figure, the sequence of the nodes 002 to 007 corresponds to the word string “Last month /// Kanto Koshin / local / ha /. Here, the slash “/” is a word break. The series of nodes 010 to 015 corresponds to the word string “This week / Moon / Matched / Rain / Ha /. The series of nodes 018 to 023 corresponds to the word string “Meteorological Agency / Nii / Yori / Masu / To / ...”. As described above, the WFST constructed here is a network that can freely connect all document texts.

  In the figure, the parameter for the transition in which the word s is described is represented by (s / ε: 0.0). That is, this triplet parameter accepts the word s as input and represents no output. ε is a symbol indicating that there is no word. That is, the output other word strings are empty. The triple parameter represents that the transition weight value is 0.0. There are various methods for setting the weight value. For example, when there is no penalty, 0.0 can be used as the transition weight value, and a negative weight can be used as the penalty. For example, when the current state is estimated based on the definition of the state transition, the total of the transition weight values in a predetermined series can be used as a score. As an example, the state transition to which the word “last month” is attached in the same figure is represented by (last month / ε: 0.0) as a triple parameter.

  In the figure, a state transition with <S> is a transition for accepting a replacement word. That is, when the word input at the position of the recognized word string corresponding to the position of a certain word s included in the document word string is replaced with an arbitrary word different from the word s on the document, the replacement is performed. A transition for accepting a word. Hereinafter, an arbitrary word different from the word s at the position of a certain word s included in the word string of the document is referred to as an arbitrary word *. The asterisk is a notation representing a wild card. This replacement includes, for example, a case where “restart” is recognized as a transliteration of the homonym “reunion”.

In the figure, a state transition with <S> can accept any word *. The parameter for the transition in which <S> is described is represented by (* / ε: ω _s ). In this triplet representation, * represents any word input that the transition can accept, and ε means there is no output at this transition. Further, ω _s is one of transition weights, and means a penalty imposed on the transition when an arbitrary word * different from the word s is input (hereinafter referred to as a replacement penalty). This replacement penalty ω _s is expressed by a numerical value that lowers the node score, and for example, −1.0 is used. For example, a state transition with <S> can be expressed as (* / ε: -1.0) as a triple parameter.

  In the figure, a state transition with <I> is a transition for accepting an insertion word. In other words, if there is a repetition of information addition or segmentation due to utterance due to the speaker, etc., the word inserted at the position following the word string recognized as the original or replaced is accepted. It is a transition to do. In addition, due to the voice recognition device 220, a word inserted in a position following the one word as the manuscript is caused by a recognition error that recognizes a word that should be recognized as the manuscript as a plurality of words if it is as the manuscript. Can also be used for state transitions marked with <I>.

In the figure, the state transition with <I> can accept any word *. The parameter for the transition in which <I> is described is expressed by a triplet (* / ε: ω _i ). Here, * represents an arbitrary word input, and ε represents an empty word. That is, this state transition accepts an arbitrary word and has no output word. Further, ω _i is one of transition weights, and means a penalty imposed when an arbitrary word * is input for this transition (hereinafter referred to as an insertion penalty). This insertion penalty ω _i is represented by a numerical value that lowers the node score, and for example, −1.0 is used. For example, a state transition marked with <I> is (* / ε: −1.0) in terms of a triple parameter.
It should be noted that the transition destination state in the state transition with <I> is the same as the transition source state in the transition.

  In the figure, a state transition with <D> is a transition for accepting a dropped word. That is, this is a transition for specifying the position of a word dropped from the original in the recognized word string when a phrase or the like is dropped in a part of the utterance content due to the speaker. In addition, due to the voice recognition device 220, a word that should be recognized as a plurality of words according to the original is caused by a recognition error in which the word is deleted and recognized as one word, and is dropped from the original in the recognized word string. It is a transition for specifying the position of a word.

In the figure, a state transition with <D> is a transition that can occur even if no word is input. The triple parameter for the transition in which <D> is described is represented by (ε / ε: ω _d ). Here, ε before the slash indicates that there is no input accepted in this transition. Also, ε after the slash means that there is no output at this transition. Further, ω _d is one of transition weights, and means a penalty imposed when a word is dropped in this transition (hereinafter referred to as a drop penalty). This dropout penalty ω _d is represented by a numerical value that lowers the node score, and for example, −1.0 is used. For example, the transition in which <D> is described in FIG. 2 is (ε / ε: −1.0) in terms of a triple parameter.
It should be noted that the transition source and transition destination states in the state transitions marked with <D> are the same as the state transitions relating to the dropped words. For example, the state transition with <D> from the node 002 to the node 003 enables the transition from the node 002 to the node 003 even when the parallel word “no” is omitted (however, the above-described dropping) With a penalty).

  In the figure, the state transitions to which <Emi1>, <Emi2>, and <Emi3> are attached are transitions for outputting a sentence as a predetermined word string. The sentence output here is the correction result of the word string recognized by the speech recognition process. The parameter for the transition to which <Emi1>, <Emi2>, and <Emi3> are attached is represented by (ε / L: 0.0). Here, ε represents that no word is input in this transition. L indicates that the word string (sentence) output in this transition is L.

  For example, a state transition to which <Emi1> is attached is represented by three parameters (ε / Last Kanto Koshin region ...: 0.0). In other words, in this case, L is in order (in series) all the word strings “last month /// Kanto Koshin / local / ha / ...” arranged in each transition from the start node 001 to the node 007 via the node 002. N) Connected word strings. Note that 0.0 in the parameter is one of transition weights, and means that no penalty is imposed when a sentence is output for this transition. Similarly, the parameter of the state transition to which <Emi2> is attached is (ε / rains gathered in this week ...: 0.0). In addition, the parameter of the state transition to which <Emi3> is attached is similarly (ε / 0.0 by the JMA).

In the figure, a transition in which <eps> is described is a transition connecting an end point node and a start point node, and is called an epsilon transition (ε transition). The state transition with <eps> is a transition that gives a constraint that a predetermined word string (sentence) included in the document text set is continuously spoken. The parameter for the state transition with <eps> is expressed by (ε / ε: ω _u ). Here, ε before the slash indicates that no word is input in this transition. In addition, ε after the slash indicates that there is no output in this transition. Also, ω _u is one of transition weights, and by giving an appropriate weight (numerical value), the WFST can increase the score of sentences that match longer.

[3. WFST in this embodiment]
FIG. 3 is a schematic diagram showing an example of WFST used in the present embodiment. The following description will focus on the difference between the general WFST described with reference to FIG. 2 and the WFST used in the present embodiment.

  The WFST shown in FIG. 3 is also represented by a state transition diagram composed of nodes (states) and branches (state transitions). In the figure, nodes are represented by black circles or black square symbols, and branches are represented by lines with arrows. Each node is given a three-digit integer for identifying the node. Also in the state transition diagram shown in the figure, there are a start node 101 and an end node 108. In the state transition diagram shown as an example, there are two sequences in parallel from the start node 101 to the end node 108. The first sequence extends from the node 101 at the start point to the node 108 at the end point via the nodes 102 to 107. Further, the second series reaches from the node 101 at the start point to the node 108 at the end point via the nodes 112 to 117. Each of these series corresponds to a sentence in the manuscript text. Here, only two sentences are shown for the sake of simplicity, but in reality there is no restriction on the number of sentences. There is a branch (the above-described epsilon transition) for returning from the end point node 108 to the start point node 101.

Each state transition is defined by a transition source state, a transition destination state, an input symbol (word), an output symbol (word string), and a transition weight value. As an example, in the figure, the state transition from node 102 to node 103, an input word to accept a is W _2, the symbol to be output is epsilon (i.e., the output word is not). As another example, in the figure, the state transition from node 114 to node 115 is the symbol for receiving the epsilon (i.e. no word to accept), symbols output from a word string of C _3.

  In the figure, the display of the weight of state transition is omitted. Further, in the figure, a transition for accepting a speech recognition error is omitted. That is, the state transition for accepting a replacement word, the state transition for accepting an insertion word, and the state transition for allowing omission are omitted.

A feature of the WFST shown in the figure is that a document sentence is divided into chunks, and output transitions are arranged as state transitions at chunk breaks. The output transition is a state transition in which the output symbol is not empty. That is, the word string may be output not only at the end of the sentence but also at a state transition in the middle of the sentence. As a specific example in the figure, an original sentence (W ₁ , W ₂ , W ₃ , W ₄ , W ₅ ) is divided into a plurality of chunks of o ₁ and o ₂ . Chunk o ₁ is a word string (W ₁ , W ₂ , W ₃ ), and chunk o ₂ is a word string (W ₄ , W ₅ ). An output transition in which the input symbol is ε (that is, there is no input word) and the output symbol is C ₁ is provided between the chunks o ₁ and o ₂ . The state transition from node 107 to node 108 is the output transitions at the end of the sentence, the output symbol is C _2. Similarly, the original text (W ₆ , W ₇ , W ₈ , W ₉ , W ₁₀ ) is also divided into a plurality of chunks o ₃ (not shown) and o ₄ (not shown). Chunk o ₃ is a word string (W ₆ , W ₇ , W ₈ ), and chunk o ₄ is a word string (W ₉ , W ₁₀ ). Then, after the chunk o ₃ , an output transition for outputting the word string C ₃ is provided. Further, after the chunk o ₄ , an output transition for outputting the word string C ₄ is provided.

  As described above, in this embodiment, a sentence is divided into chunks and an output transition is provided at the end of the chunk. The method of dividing the sentence into chunks is arbitrary. Divide into chunks at grammatically or semantically breakable points in the sentence (for example, syntactic phrases can be used), or mechanically divide into multiple chunks with a predetermined number of words May be. Moreover, you may divide | segment in the location where the reporter or the announcer has started a new line for readability. Further, it may be divided into chunks of an appropriate length according to the degree of redundancy of sentences in the original and the required recognition error correction capability. Further, it may be divided into chunks of an appropriate length according to the allowable subtitle display delay. When a long word string is set as a chunk, the correction accuracy is improved, but the determination of the subtitle word string to be transmitted is delayed. If a short word string is set as a chunk, the subtitle word string to be sent can be determined quickly, but the correction accuracy is reduced. Further, division into chunks of an appropriate length may be performed according to the expected recognition accuracy of voice recognition and the degree of matching between the original and the reading voice.

[4. WFST construction method]
Next, a WFST construction method by the transducer construction device 240 will be described. When the transducer construction device 240 constructs a WFST, the manuscript text included in the manuscript text storage device 200 is divided into sentences in advance. If the end of a sentence is delimited by a punctuation mark, the punctuation mark can be divided into landmarks. Further, by analyzing the text of the manuscript, sentence breaks may be detected from the syntax rules and divided into sentences. Further, it may be possible to divide into sentences at places marked manually.

  In the present embodiment, the text stored in the manuscript text storage device 200 is a reading manuscript that may be picked up by a news program. The manuscript prepared here includes manuscripts for a plurality of news items. Each news item includes a plurality of sentences. In this embodiment, it is not necessary to specify in advance the order in which news items are read or the order in which sentences are read. All documents that may be read may be prepared and stored in the document text storage device 200. In addition, among these manuscripts, there may be news items and sentences that cannot be read as a result.

  Prior to the construction of the WFST, the manuscript is previously shaped manually. By this shaping, word strings (sentences) that are always read continuously are collected as one sentence unit. Also, by this shaping, a portion that may be skipped in the document is separated as another sentence.

The sentence is divided into chunks in advance. If the chunk (that is, the unit of the word string until the output transition is performed) is set long, the correction accuracy increases, but the determination of the word string to be output is delayed. On the other hand, if the chunk is set to be short, the determination of the subtitle word string to be sent out becomes fast, but the correction accuracy decreases. Therefore, what unit should be used may be appropriately designed according to the expected recognition accuracy of voice recognition and the degree of matching between the original and the reading voice. Whatever method is used to divide a sentence into chunks, a chunk is a shorter unit than a sentence.
The transducer construction device 240 then configures the WFST to place an output transition immediately after receiving the chunk.

A specific WFST construction procedure by the transducer construction device 240 is as follows. The construction of the WFST is started from the start point of the WFST.
Each time the transducer construction device 240 reads each word from the document text storage device 200, the transducer construction device 240 sequentially creates a transition with a weight of 0 and a new node for accepting the word. Here, the transition of weight 0 means (s / ε: 0.0) when expressed by a triple parameter. Note that s is a word. Then, after adding an output transition after the chunk as a unit, the same processing is repeated for each word included in the next chunk. When the end of the chunk is the end of the sentence, the output transition related to the last chunk is connected to the end point node of WFST. That is, the transition destination state of the output transition related to the last chunk is set as the end node.
If there are more sentences left, start again from the starting point and repeat the above process.
Thereafter, the same processing is repeated until the processing for all sentences included in the document text storage device 200 is completed.

After reading all the original texts from the original text storage device 200, the transducer construction device 240 then connects the end point node and the start point node with an epsilon transition. As already described, the epsilon transition is a state transition of (ε / ε: ω _u ) in terms of a triple parameter. The transducer construction device 240 gives an appropriate value as the weight ω _u for this epsilon transition. As a result, the WFST can increase the score of sentences that match longer. This also allows the WFST to operate properly even when a sentence that matches the prefix of another sentence exists in the document. Finally, the transducer construction device 240 adds a transition that accepts substitution, omission, and insertion to the transition of each word. In FIG. 3, state transitions for replacement, omission, and insertion are omitted.

[5. Adaptation of speech recognition]
When the speech recognition device 220 actually recognizes the speech read out based on the document, it is desirable to adapt the language model for speech recognition based on the word string included in the document in advance. As a result, it is possible to increase the accuracy of voice recognition when the text is read as it is.

[6. Traceback from output transition]
In the present embodiment, the node that starts the traceback is limited to the node immediately before the output transition. That is, in the WFST state transition diagram illustrated in FIG. 3, only the nodes indicated by black square symbols are the target of traceback. The nodes indicated by the black circle symbols are not subject to traceback. In other words, the node that is the target of traceback is only the node corresponding to the end of each chunk.

Compared with the case where the traceback is performed for all nodes of WFST from the maximum likelihood node (the node having the highest score among all the nodes at time t), in the above method of the present embodiment, the output delay is increased. Can be shortened. That is, in the present embodiment, it is possible to reduce the delay in displaying the corrected subtitles according to the result of the voice recognition process.
In the WFST construction method already described, a sentence is divided into chunks in advance, and an output transition is provided following the end of each chunk. However, it is also possible not to divide the sentence into chunks (in other words, one sentence as one chunk) and to limit the node that starts traceback to only the node immediately before the output transition. In such a case, the output delay can be shortened as compared with the case where the traceback is performed from the maximum likelihood node for all nodes.

[7. How to confirm subtitle text output]
A method for the speech recognition error correction apparatus 100 to determine an output word string will be described.
N (tilde) (t), which is the maximum likelihood node immediately before the output transition at time t (in other words, the maximum likelihood node in the node corresponding to the end of the chunk), is expressed by the following equation (1).

In Expression (1), Q is a set of nodes that are subject to traceback. That is, Q is a set of nodes immediately before the output transition. n is a node belonging to the set Q. L _n (t) is the score of node n at time t. The score L _n (t) is expressed by the following formula (2).

In Expression (2), e is a state transition, and ^ef and ^et are the transition source and transition destination nodes of the state transition e, respectively. E ^w is the value of the state transition weight of the state transition e. E ⁱ is an input symbol of the state transition e. E ⁱ is one of the words W, ε (empty input), and * (wildcard that matches any input). That is, as shown in Expression (2), the score (likelihood) of the node n at the time t is the transition of the state transition e at the time t−1 among all the state transitions with the node n as the transition destination node. the score of the source node e ^f, is obtained by adding the weight value e ^w a state transition e.

In order to obtain the maximum likelihood node according to the above equation (1), the speech recognition error correction apparatus 100 performs an efficient Viterbi search for the score L _n (t) while performing pruning with a threshold.

Then, the speech recognition error correction apparatus 100 sequentially traces back the state transitions e passed until reaching the maximum likelihood node obtained by the equation (1) to obtain a state transition sequence. Already it results traced back to the time t-t'of already output word h _t-t'is expressed by the following equation (3).

In Equation (3), {e _{t−t ′} ,...} Is a set of state transitions that do not include the start of WFST. Also, {..., E _t } is a set of state transitions that do not include the end of WFST. O (tilde) _t is a sequence of a section l ^t _k (that is, l ^t _k is a sentence) sandwiched between the start and end, as expressed by the following expression (4) (k = 1, 2, ...).

In Expression (4), r ^t is the number of sections l ^t _k (sentences) included in O (tilde) _t . Furthermore, this l ^t _k is a sequence of chunks, as shown in equation (5) below.

In equation (5), m _{t, k} is the number of chunks included in the interval l ^t _k .
The speech recognition error correction apparatus 100 outputs a sequence of chunks represented by the following equation (6) for each section l ^t _k . That is, what is represented by the following formula (6) is a caption sentence output by the speech recognition error correction apparatus 100.

In Equation (6), E (tilde) (o ^{t, k} _u ) is an error score based on the error rate. T is a threshold value regarding the error score, and 0 <T <1. The error score E (tilde) (o ^{t, k} _u ) is obtained by the following equation (7).

In the equation (7), the two-input function E (tilde) (o ₁ , o ₂ ) is expressed by E (tilde) (o ₁ ) and E (tilde) (o ₂ ) as shown in the following equation (8). ) Of the word weighted average.

In Expression (8), N ₁ and N ₂ are the numbers of words included in chunks o ₁ and o ₂ , respectively. E (tilde) (o) is an error rate (edit distance) obtained from the number of accepted words N _r in chunk o and the number of transitions N _e due to acceptance of an error. 9).

That is, as shown in the equation (7), E (tilde) (o ^{t, k} _u ) ^, which is an error score of the chunks o ^{t, k} _u included in the interval l ^t _k , is defined as follows. . That is, the error E ^{(tilde) (o t,} _{k u),} the chunk ^{o t,} and the error rate calculated by Equation (9) in the _{k u,} the previous chunk ^{o _t,} is calculated in the _{k u-1} Based on the score and the weighted average value by number of words. However, when the weighted average value based on the number of words is smaller than a predetermined threshold value T, the value of E (tilde) (o ^{t, k} _u ) is zero (the upper case in the right side of Expression (7)). ). On the other hand, when the weighted average value based on the number of words is equal to or greater than the threshold value T, the weighted average value is directly used as the value of E (tilde) (o ^{t, k} _u ) (lower right side of Expression (7)). Case). In this way, the value of E ^{(tilde) (o t,} _{k ^u)} is dependent on the value of the previous chunk ^{o _t,} E for _{k u-1 ^{(tilde) (o t, k u-}} 1) The output can be determined by calculating recursively using equation (7).

In addition, when the error score in the previous chunk is small by cutting off the error score when the value is lower than the threshold T to 0 by dividing into cases in Expression (7), The error score is not spread to the chunk.
Note that the specific value of the threshold T shown in the equations (6) and (7) may be determined as appropriate. For example, T = 0.5, or 0.4 ≦ T ≦ 0.6. It is preferable to set the value within the range. If the value of T is too large (approaching 1), a high error rate is allowed. On the other hand, if the value of T is too small, it becomes too strict against errors, and there is a disadvantage that the density of subtitles that can be output for a document that is originally read out becomes too low. Therefore, the range of 0.4 ≦ T ≦ 0.6 is preferable, and T = 0.5 is particularly preferable.

[8. Configuration of voice recognition error correction device]
Next, the configuration of the speech recognition error correction apparatus will be described.
FIG. 4 is a functional block diagram showing a schematic functional configuration of the speech recognition error correction apparatus 100. As illustrated, the speech recognition error correction apparatus 100 includes a WFST storage unit 110, a node data update unit 120, a node data storage unit 130, a document search unit 140, and an output unit 150.

  The WFST storage unit 110 (finite state transducer information storage unit) is a finite state transducer that performs state transition while sequentially accepting words input as speech recognition results corresponding to the original text. Information on the state of a finite state transducer that makes a state transition while accepting an error in a word included in the result, and includes a transition source state, a transition destination state, an input symbol, an output symbol, and a transition weight regarding the state transition. Information is stored. The WFST storage unit 110 stores an output transition where the output symbol is a non-empty state transition as a part of the state transition. Also, information on the state and information on the state transition of the WFST in which a sentence included in the manuscript text is divided into a plurality of chunks and an output transition is provided after a state transition path that accepts a word included in the chunk as an input symbol. Remember.

  Specifically, the WFST storage unit 110 stores the WFST constructed in advance by the transducer construction device 240 based on the document text storage device 200 (FIG. 1). Specifically, the WFST storage unit 110 stores information equivalent to the state transition diagram (network). The information includes state (node) identification information and state attribute information, state transition (branch) identification information and state transition attribute information. The state transition attribute information includes at least transition source state identification information, transition destination state identification information, and a weight value related to the transition.

The node data updating unit 120 receives an input of a recognized word that is a speech recognition result corresponding to the document text from the outside, and according to the received recognized word, a score at the time (that time) for each state in the WFST. And the node data storage unit 130 is updated using the calculated score.
Specifically, the node data update unit 120 calculates, as node data, a score of a state that can be transited on the WFST network at each time when an input of a word included in the recognized word string output by the speech recognition device 220 is received. And to update. The node data update unit 120 performs, for example, a Viterbi search sequentially with reference to the WFST stored in the WFST storage unit 110 every time a recognition word is received from the voice recognition device 220 and stores it in the node data storage unit 130. Update the node data.

  The node data updating unit 120 calculates a score when the state transition is performed according to the word input as a part of the recognized word string. The node data update unit 120 calculates a score based on the weight value of the state transition. As described in the example of WFST, when words matching the corresponding original are input in the order of the original, “0” is added to the score as a weight value. When a word different from the corresponding manuscript is input, “−1” is added to the score as a weight value, and “−1” has a penalty meaning.

  For example, in the example shown in FIG. 2, when the word string input as the recognition word string is exactly the same word string as the corresponding manuscript, the word “Last Month” is received from the start node 001 and the word of the corresponding manuscript is handled. Go to node 002 through the transition to. Then, in response to this state transition, the node data update unit 120 adds “0” to the score. Next, for example, the word “no” is received from the node 002 and the process proceeds to the node 003. Then, in response to this state transition, the node data update unit 120 adds “0” to the score. Similarly, when “Kanto Koshin”,... Are sequentially received, “0” that is the value of the weight of state transition corresponding to each word is added to the score.

  On the other hand, for example, in the example shown in FIG. 2, when the word string input as the recognition word string is a word string different from the corresponding document, the state transition is different. In other words, when the word “Last week” is received from the start point node 001, for example, the word “Last month” of the corresponding document is replaced. In this case, the node data updating unit 120 adds “−1” (penalty), which is the value of the weight of the state transition, to the score. Similarly, when a transition corresponding to an insertion error or a drop error is passed, the node data updating unit 120 similarly adds a penalty “−1” to the score.

  Thus, when the input recognition word is the same as the word s accepted in the state transition in WFST, the score in the transition is the best. On the other hand, state transitions corresponding to editing of replacement, insertion, and deletion are factors that deteriorate the score. Here, “edit” refers to a result obtained by replacing or deleting a word that should be originally inserted, or by inserting a word that is not a word that should originally be. For example, a state transition with <D> can be transitioned even if there is no matching input word. However, in the case of a path that passes only a transition with <D>, the penalty increases as the number of transitions increases. And the score goes down. WFST produces a result that if the recognition word string includes an error or paraphrase, the score is deteriorated accordingly.

  The node data storage unit 130 stores a score representing the likelihood of a state in WFST. That is, the node data storage unit 130 stores the node data calculated by the node data update unit 120. The node data update unit 120 updates the node data storage unit 130 in a timely manner based on the calculation result described above. The node data storage unit 130 is configured using a general storage unit such as a semiconductor memory or a hard disk device, for example.

  When the document search unit 140 receives an activation signal indicating the start of processing from the outside, the document search unit 140 refers to the node data storage unit 130 to determine the maximum likelihood node at that time, and at the same time, the WFST storage unit 110 and the node data storage unit 130. And tracing back the state transition from the maximum likelihood node to perform a traceback process until a predetermined time when the state transition has been confirmed, and setting the path of the state transition subjected to the traceback process as an output candidate, For an output candidate path, an error score is calculated according to the ratio of the state transition related to the error in the path, and when the error degree is smaller than a predetermined threshold based on the calculated error score, The output candidate is determined output. Further, the manuscript search unit 140 calculates an error score for each chunk and, when the error level is smaller than a predetermined threshold based on the error score for each chunk, outputs an output transition symbol corresponding to the chunk, When the document search unit 140 calculates an error score for each chunk, the manuscript search unit 140 corresponds to the ratio of the state transition related to the error with respect to the chunk and the section of the chunk immediately before the chunk. The error score is calculated as the error score of the chunk.

  In addition, the document search unit 140 determines the node having the maximum likelihood among the nodes corresponding to the transition source state of the output transition as the maximum likelihood node at that time. In other words, the document search unit 140 does not consider whether or not it is a maximum likelihood node for a state that is not the transition source state of the output transition.

  Specifically, the manuscript search unit 140 does not wait for input of recognition results of all recognition word strings for all manuscripts for determining the final best hypothesis, and each time the predetermined processing start condition is satisfied, The correction result is obtained. When the processing start condition is satisfied, the document search unit 140 traces the final best hypothesis partially while tracing back on the WFST network based on the node data stored in the node data storage unit 130 at that time. Approximate hypotheses are sequentially determined as error correction results.

  The manuscript search unit 140 approximates the final best hypothesis based on the edit distance between the word string of the corresponding manuscript included in the WFST and the input recognition word string. The document search unit 140 determines that the path section is reliable if the edit distance in the path section from the beginning to the end is small to some extent between the paths divided for each predetermined range on the WFST network. And output. Here, the short edit distance means that the path through which the recognized word string and the original word string are almost matched has been passed. On the other hand, since the reliability of a path section with a long editing distance is low, it is not determined at that time and is used for the next traceback. It is presumed that a path section having a low reliability forever is a section talking about a difference that is not originally described in the manuscript. Therefore, a path section with low reliability is not output.

  The processing start condition in the manuscript search unit 140 is, for example, a case where a silence period in which there is no uttered speech reaches a predetermined period, or the number of input words as a recognition word string output by the speech recognition device 220 is a predetermined word This is the case when the number is reached. The predetermined period is not particularly limited, but is 3 seconds as an example. The predetermined number of words is not particularly limited, but is 20 words as an example. The activation signal input to the document search unit 140 is a signal indicating that the above process start condition is satisfied. The activation signal at this time may be automatically turned on, for example, by the voice recognition device 220 or may be turned on by a manual operation by the operator. This operator performs an operation of turning on the activation signal when recognizing a pause (silence of a predetermined length) or when recognizing a predetermined number of words output by the speech recognition device 220. When the speech recognition device 220 automatically turns on the activation signal, the activation signal is detected by detecting a pause using the result of the speech recognition or counting the number of words output as the speech recognition result. As a trigger to switch. With this configuration, the processing load can be reduced compared to the case where the search process is started each time a recognized word is input. Further, if the silent period continues for a predetermined period, the recognition result word is not passed from the voice recognition device 220 during that time, so that the processing load of the device is not concentrated at that time, and the node score Can be easily compared.

  In order to realize the above function, the document search unit 140 further has the following configuration. That is, the document search unit 140 includes a maximum likelihood node detection unit 141, a traceback unit 142, a document division unit 143, an output candidate storage unit 144, an error score calculation unit 145, an error score determination unit 146, and a confirmation. An output storage unit 147 and a fixed time storage unit 148 are included.

  The maximum likelihood node detection unit 141 operates using an activation signal received from the outside as a trigger, and detects a node having the maximum score in the node data stored at that time. As described above, this activation signal is supplied from the outside when a predetermined processing start condition is satisfied.

  The traceback unit 142 follows the WFST network from downstream to upstream for the path from the node specified by the maximum likelihood node detection unit 141 to the node. At this time, the traceback unit 142 goes back to the network to the place determined by the previous traceback. That is, the traceback unit 142 traces back to the time corresponding to the last input word of the word series that has been finalized and output.

  The document dividing unit 143 cuts out a word string for each section corresponding to the output transition in the traceback path. Here, the output transition is a state transition accompanied by the output of a symbol as described above. When the transducer construction device 240 constructs a WFST, an output transition is provided immediately after the end of the chunk. That is, the section obtained as a result of the division by the document dividing unit 143 is the above-described chunk unit. The document dividing unit 143 divides the entire path obtained as a result of the traceback, and outputs each word string obtained as a result of the division.

  The output candidate storage unit 144 stores an output transition output symbol (cut out document) corresponding to the path section obtained as a result of the division by the document dividing unit 143 as an output candidate. The output candidate storage unit 144 is configured using, for example, a general storage unit such as a semiconductor memory or a hard disk device.

  The error score calculation unit 145 calculates an error score for each section (chunk) divided and cut out by the document dividing unit 143. The error score calculation unit 145 calculates the error score using the equation (7) already described.

  The error score determination unit 146 determines whether or not the error score calculated by the error score calculation unit 145 is smaller than a predetermined threshold T. That is, the error score determination unit 146 performs determination based on the condition included in the already described formula (6).

  The confirmed output storage unit 147 stores an output that is confirmed based on the determination result by the error score determination unit 146 (referred to as a confirmed output). Specifically, when the error score determination unit 146 determines that the error score is smaller than the threshold T, the confirmed output storage unit 147 stores the output symbol of the output transition in the predetermined path section as an error correction result. In other cases, the definite output storage unit 147 does not store the output symbol of the output transition in the predetermined path section as an error correction result. The confirmed output storage unit 147 is configured using a general storage unit such as a semiconductor memory or a hard disk device. The storage structure of the definite output storage unit 147 is a stack, and the definite output storage unit 147 holds data in a last-in first-out (LIFO) structure.

  The fixed time storage unit 148 stores the fixed time determined by the latest traceback process. The fixed time storage unit 148 is a general storage unit such as a semiconductor memory or a hard disk device. The fixed time storage unit 148 stores the latest path symbol corresponding to the output symbol loaded on the stack at the time when the determination process by the error score determination unit 146 is completed for all the path sections (all cut out documents) traced back this time. The time of the confirmed word is stored as the confirmed time.

  The output unit 150 sequentially outputs the corresponding document (that is, the confirmed output) determined as the error correction result by the document search unit 140. The output unit 150 is determined by the time when the determination process for the error score calculated for each path section of all the cut out corresponding documents in the path traced back through the WFST network is completed. Outputs output symbol data. Specifically, the output unit outputs the output symbol data stacked on the stack of the definite output storage unit 147 until the stack becomes empty.

  The speech recognition error correcting apparatus 100 performs an operation of correcting and outputting a word that has been erroneously recognized, and not outputting a significantly incorrect recognition result. In other words, if the correction result by the speech recognition error correction device 100 can be viewed in advance by a person, an error is enough to feel that “this is not a sentence” or “the meaning is different”. The operation in which the speech recognition error correction apparatus 100 detects the part in the process and does not output the detected part is included as error correction in a broad sense. This is because an error score is calculated and discrimination is performed based on the error score.

[9. Operation procedure of speech recognition error correction device]
FIG. 5 is a flowchart showing a processing procedure performed by the speech recognition error correction apparatus 100. The premise of the processing of this flowchart is the following four points.
(Assuming 1) the recognition result of a word input _{{ω 0, ω 1, ...} , ω k, ..., ω j, ...} and.
(Premise 2) Let ω _k be the last input word of the portion determined by the previous traceback, and let the output transition at that time be a _P (P-th output transition along the time axis).
(Premise 3) Consider a case in which, after a word ω _{j as a} recognition result is input, a predetermined silence is used as a trigger for successive determination.
(Premise 4) The node data updating unit 120 accepts the last input word ω _j before silence and calculates all nodes that can make a transition.

  Hereinafter, the flow of processing by the speech recognition error correction apparatus 100 will be described with reference to this flowchart.

  First, in step S1, using the input of an activation signal from the outside as a trigger, the maximum likelihood node detection unit 141 detects the node having the highest score in the node data stored at that time as the maximum likelihood node. The state represented by this detection node is the maximum likelihood state at the start of traceback.

Next, in step S2, the traceback unit 142 traces back from the maximum likelihood node detected in step S1 to the time determined in the previous traceback process. Specifically, the traceback unit 142 traces the word history on the WFST in the reverse direction (in the direction of going back in time) for the path that has reached the detected maximum likelihood node, and confirms and outputs it in the previous traceback. Find the last input word ω _k of the word sequence. Further, the traceback unit 142 obtains a corresponding state transition (a state transition in which the accepted word is ω _k ). Then, trace back is performed until a fixed time corresponding to the word and the state transition. The traceback unit 142 refers to the fixed time storage unit 148 and sets the fixed time stored in the fixed time storage unit 148 as the fixed time corresponding to the last input word ω _k described above.
Note that instead of the state transition in which the accepted word is ω _k , the traceback may be performed until the output transition a _P is reached.

In step S3, the original dividing unit 143 divides the original, and sets each word string obtained as a result of the division as an output candidate. Specifically, the document dividing unit 143 divides the document for each path section sandwiched between two output transitions in the path traced back this time. Then, the original dividing unit 143 stores the obtained output candidates in the output candidate storage unit 144.
Note that, as a processing method here, the output transition a _L (Lth (but L) along the time axis) that can be output while proceeding in the reverse direction (that is, in the direction of going back in time) until the output transition a _P is reached. The document may be divided every time it passes (> P) output transition). It is also possible to divide the original each time passing through the output transition a _L can be output from the side of the output transition a _P.
Note that the output transition a _L that can be output is an output transition symbol that is an output candidate, but is an output transition that is later rejected by the error score determination unit 146 (that is, it is not output from the output unit 150 after all). Output transition).

Next, in step S4, the error score calculation unit 145 calculates the error score of the output candidate (step S4). Specifically, the error score calculation unit 145 calculates the equation (7) by the chunk ^{o t,} _{k u} of the error score E ^{(tilde) (o t,} _{k u).}

  In step S5, the error score determination unit 146 selects one of the output candidates, and determines whether or not the error score calculated for the output candidate is less than the threshold value T. If the error score is less than the threshold T (step S5: YES), the process proceeds to the next step S6. If the error score is greater than or equal to the threshold T (step S5: NO), the process jumps to step S8.

When the process proceeds to step S6, the error score determination unit 146 determines the output transition of the path section (chunk) on the WFST network, and determines the output symbol as an error correction result.
In step S <b> 7, the error score determination unit 146 writes the output symbol data determined this time in the determined output storage unit 147. As described above, the definite output storage unit 147 is a stack having a last-in first-out structure.
The processes in steps S6 and S7 are executed only when the error score of the current chunk is less than the threshold T by the branch control in step S5.

  Next, in step S8, the error score determination unit 146 determines whether or not all output candidates have been determined. If discrimination of all output candidates has been completed (step S8: YES), the process proceeds to the next step S9. If discrimination of all output candidates has not been completed, that is, if there are still output candidates to be selected (step S8: NO), the process returns to step S5 to select the next output candidate.

That is, when the determination process for the error score calculated for each path section corresponding to all the cut out documents is completed, the process proceeds to the next step S9.
In step S9, the error score determination unit 146 updates the confirmed time storage unit 148. That is, the error score determination unit 146 stores, in the determination time storage unit 148, the determination time determined by the current traceback process as the determination time determined by the current traceback process, with the latest determination word corresponding to the output symbol stacked on the stack.

  Next, in step S10, the output unit 150 sequentially reads and outputs the output symbol data written in the finalized output storage unit 147 at that time until the finalized output storage unit 147 becomes empty. As a result, all the output data loaded on the stack is output. Note that the output unit 150 outputs the document sequentially from the document placed on the front side of the WFST network (that is, sequentially from the earlier time side).

  That is, the output unit 150 sequentially outputs the data accumulated on the stack in each traceback process as a confirmed document. At this time, if the error score of a predetermined path section is equal to or greater than the threshold T among the speech recognition results, the output symbol of the output transition of the path section is not adopted as the error correction result because the path has low reliability. Accordingly, the output unit 150 does not output an output symbol in such a path section with low reliability.

  When the process of step S10 ends, the speech recognition error correction apparatus 100 ends the process of the entire flowchart.

[10. Specific examples and evaluation of subtitle text output]
As described above, the features of the processing of the speech recognition error correction apparatus 100 according to the present embodiment are (1) dividing a sentence into chunks, (2) providing an output transition at the end of the chunk, (3) Trace back from the location of the output transition. The effect | action by performing such a characteristic process is demonstrated below with a specific process example.

  FIG. 6 is a schematic diagram illustrating an example of a subtitle sentence output determination method according to the present embodiment. (A) and (B) in the figure respectively show the relationship between the input recognition result word string, the WFST state transition corresponding to those words, and the resulting output symbols, and the calculation. The error score to be added is added.

  First, in the example of FIG. 5A, the input word string is “today / no / kanto / region / ha”. When these words are accepted, the WFST transitions as follows. That is, starting from the beginning, the input word “today” matches “today” in the corresponding document, and a state transition without penalty occurs. The next word “NO” matches “NO” in the corresponding manuscript, and a state transition without a penalty occurs. The next word “Kanto” matches “Kanto” in the corresponding manuscript and a state transition without penalty occurs. The next word “region” is a word that is input due to a replacement by the speaker or a recognition error by the speech recognition device 220. Therefore, the “region” in the corresponding manuscript does not match and a replacement state transition occurs. The next word “ha” matches “ha” in the corresponding document, and a state transition without penalty occurs. “Sunny” and “Is” are not input, but when a dropout state transition occurs, the end state of the chunk indicated by the black square is reached. The output symbol “Today's Kanto region is clear”, which is an output symbol in the output transition, is an output candidate. Here, the error rate for this chunk is 3/7. That is, the error score is 3/7. When the threshold value T is 0.5, this error score is smaller than the threshold value T. Then, the speech recognition error correction apparatus 100 can determine the output of “Today's Kanto region is sunny”, which is an output candidate.

  Next, in the WFST network shown in the example of FIG. 5B, a series of sections is divided into abcdef chunks and ghijk chunks. An output transition is provided at the end of each chunk. The input word string is abcxeyegh. When these words are accepted, the WFST transitions as follows. That is, starting from the beginning, each of the input words a, b, and c matches a, b, and c in the corresponding manuscript, and a state transition without penalty for each of these words occurs. For the next input word x, a state transition of replacement of the word d in the document occurs. The next input word e matches e in the manuscript, and a state transition without penalty occurs. Then, for the next input word y, the state transition of replacement of the word f in the document occurs. The transition destination is the end state of the chunk, and the output symbol in the next output transition is abcdef. The chunk error score so far is 2/6 (because there are 4 normal state transitions and 2 replacement state transitions, 2 / (4 + 2)).

  Subsequently, in FIG. 5B, each of the input words g and h matches g and h in the corresponding manuscript, and a state transition without penalty for each of these words occurs. Here, the input word string ends, but o, j, k in the corresponding manuscript may be dropped. Here, the apology rate is only 3/5 for the section of the ghijk column in the document, which exceeds the threshold T (= 0.5). This is because g, h match the input and i, j, k are missing. However, the error score calculated by the equation (9) is 5/11 by taking a weighted average (weighted average based on the number of words in the chunk) with the error score in the previous chunk. This error score 5/11 is smaller than the threshold value T. Therefore, the speech recognition error correction apparatus 100 outputs the output candidate abcdef and also outputs the output candidate ghijk corresponding to the next chunk.

As shown in the example of FIG. 5A, the speech recognition error correction apparatus 100 can detect the input word after the word “ha” all wrong (eg, recognition error by the speech recognition apparatus 220) at an early stage. The corresponding subtitle sentence can be output. This is because the error score is smaller than the threshold T even if all input words after the word “ha” are incorrect.
Further, as shown in the example of FIG. 5B, when the chunk is divided in advance, the error score of the chunk that has been confirmed ahead of the currently focused chunk is obtained by the action of the equation (9). By using this, it is possible to determine the output of the currently focused chunk even earlier.

[11. How to determine threshold T for error score]
If the recognition accuracy of the speech recognition apparatus is about 90%, the error rate of the word that is the basis of the error score may be about 10%. In this embodiment, the threshold T used for determination by the error score determination unit 146 is preferably set with a margin corresponding to the reliability of the word matching rate according to the recognition accuracy of speech recognition. Here, the reliability of the word match rate depends on the number of words between two output transitions of the WFST network.

As another factor for setting the threshold value T, there is a ratio of the overlapping of the candidate document sentences stored in the document text storage device 200 as a ratio. For example, in the case of sentences shown in (E1) to (E3) below, overlapping as sentences is included at a rate of about 80%.
(E1) Today's weather is sunny (E2) Today's weather is rainy (E3) Today's weather is cloudy In this case, if the threshold T is set to about 80%, the desired movement will be realized Can not.

  It was confirmed that when the test was performed under the condition that the sentence of the news manuscript was appropriately divided into chunks and the output transitions were arranged and the threshold T was 50%, it was confirmed that the operation was good.

  As described above, the speech recognition error correction apparatus 100 according to the present embodiment uses the constraint that the sentences included in the manuscript text are continuously uttered in principle although the order of the sentences can be changed. Thus, the recognition error can be corrected by associating the recognition result with the original text. That is, the automatic correction error of the block matching method according to the prior art is solved.

  In addition, the speech recognition error correction apparatus 100 according to the present embodiment collates the word string of the recognized word most closely with the word string included in the manuscript text in a longer range than the conventional technique. ing. That is, while the conventional block collation method collates only the section corresponding to the word chain block, the speech recognition error correction apparatus 100 collates the entire longer sentence while tracing the sentence of the document. Therefore, it is clearly known where to make the match, and automatic correction errors can be reduced more than before.

  The speech recognition error correction apparatus 100 according to the present embodiment divides a sentence into a plurality of chunks and provides an output transition at the end of the chunk. As a result, the confirmed word string can be output even in the middle of the sentence.

  Further, the speech recognition error correction apparatus 100 according to the present embodiment is limited to trace back only from a place where an output transition exists. Thereby, the amount of calculation can be suppressed.

  Note that the functions of the speech recognition error correction device, the document text storage device, the speech recognition device, and the transducer construction device in the above-described embodiment may be realized by a computer. In that case, the program for realizing these functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included, and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  Although the embodiment has been described above, the present invention can also be implemented in the following modified example. Moreover, you may implement combining a some modification.

[Modification 1] Configuration of Speech Recognition Error Correction Device The speech recognition error correction device 100 may include a transducer construction device 240 therein.

[Modification 2] Construction of WFST that accepts paraphrasing An announcement manuscript that is an information source of WFST includes phrases that are skipped when read, paraphrased phrases, and supplemented phrases There is a case. Some of these are regularly omitted and paraphrased / supplemented with high frequency. For example, in a news program manuscript, phrases such as “according to the Metropolitan Police Department” representing the interview source are often skipped. Even if such a phrase is skipped, there is no change in the meaning of the main text of the news, in other words, there is no change regarding so-called 5W1H, and there is no practical problem as a news program.

  In the present modification, such a standard phrase variation is added to the WFST so that the correction result can be output with high accuracy. The addition of the above-mentioned wording variations is performed efficiently by constructing another WFST for adding wording variations separately from the WFST constructed from the manuscript and combining it with the WFST constructed from the manuscript. Can do. A known algorithm for performing WFST synthesis, minimization, determinization, or the like can be applied.

  In order to construct a WFST including the above paraphrasing example, the difference between the original text of the same type of program in the past and the word string actually read out is analyzed, and the frequency is high and the meaning of the text is determined by paraphrasing. Select and collect items that do not change in advance. Then, for each selected paraphrase example, a WFST for synthesizing the paraphrase is constructed, and an operation (computation based on the existing technology) for synthesizing the WFST constructed from the original and the WFST of the paraphrase example is performed In other words, a WFST that can be used in other words can be constructed.

[Modification 3] Addition to Recognition Result Symbols that cannot be obtained as a result of speech recognition (punctuation marks, reading marks, other symbols, etc.) may be included in the output symbols in the output state transition. For example, those symbols and the like are previously included in the news manuscript, and when the WFST is constructed based on the news manuscript, the symbols and the like remain in the output symbol. By operating the speech recognition error correction apparatus 100 using such WFST, it is possible to output easy-to-read subtitles including those symbols and the like.

[Modification 4] Other output corresponding to the recognition result (multilingual subtitles)
In addition to the third modification described above, a word string obtained as a result of translating the text of a document into another language or the like may be used as an output symbol in output transition. Thereby, it is possible to output subtitles in a language different from that of the original document read out. Also, subtitles in a plurality of languages can be output by operating a plurality of WFSTs in parallel. Further, a symbol corresponding to a queue for controlling the progress of a program may be included as an output symbol in the output transition. With the output of this queue as a trigger, it is possible to give instruction information such as start-up and scene change to a program-linked service such as hybrid cast. As a result, more various broadcasting services can be realized.

[Modification 5] Minimization of WFST In this modification, when creating a WFST, the WFST is minimized if possible. The minimization of WFST is to aggregate a plurality of states (nodes) that can be aggregated or to aggregate a plurality of state transitions (branches) that can be aggregated in a given state transition diagram. Aggregating WFST itself can be performed by existing technology. An example of WFST minimization is as follows. That is, in the WFST network, a plurality of common partial networks are reconfigured as the same state transition sequence. By using such WFST minimization, state transitions for common word strings can be reduced. For example, word strings (sentences) having the same prefix can be shared by the same transition. Thereby, the amount of calculation can be reduced.

[Modification 6] Determining the WFST When creating the WFST, the WFST is determinized if necessary. Specifically, in the WFST network, when an output symbol is determined during the state transition, the position of the output sentence is changed to the front in order to output the estimation result as soon as possible. By determinizing the WFST, for example, the output sentence is moved to a transition with a unique prefix. This has the advantage that the output sentence can be finalized in the long term. However, if the WFST is determinized when creating the WFST, it is necessary to change the setting so that the search process of the maximum likelihood hypothesis by the manuscript search unit 140 can also cope. In other words, it is necessary to shift the path interval for calculating the error score before and after the output transition as compared with the case where WFST is not determinized. In addition, it is necessary to set the threshold T to a stricter value (a value with a smaller error rate) so that the expansion and contraction of the preceding and following path sections can be absorbed.

[Modification 7] How to Find Error Score In this embodiment, an error score is calculated based on the error rate (edit distance) shown in Equation (9). However, the present invention is not limited to this, and the error score may be calculated by using the matching rate, matching accuracy, dropout rate, and insertion rate of the document and the recognition result, or using them together.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention can be used to create content such as subtitles using a speech recognition result. The present invention can be used in, for example, a broadcasting business and other content providing businesses.

100 Speech recognition error correction device 110 WFST storage unit (finite state transducer information storage unit)
120 Node data update unit 130 Node data storage unit 140 Document search unit 141 Maximum likelihood node detection unit 142 Traceback unit 143 Document division unit 144 Output candidate storage unit 145 Error score calculation unit 146 Error score determination unit 147 Final output storage unit 148 Determination Time storage unit 150 Output unit 200 Document text storage device 220 Speech recognition device 240 Transducer construction device 241 Word network registration unit 242 Editing network registration unit

Claims (3)

Corresponding to the manuscript text, it is a finite state transducer that sequentially transitions while accepting words input as speech recognition results, and transitions while accepting errors in words contained in the speech recognition results A finite state transducer information storage unit for storing information about the state of the finite state transducer to be performed and information including a transition source state, a transition destination state, an input symbol, an output symbol, and a transition weight regarding the state transition When,
A node data storage unit for storing a score representing the likelihood of the state in the finite state transducer;
While receiving an input of a recognition word that is a speech recognition result corresponding to the document text from outside, according to the received recognition word, a score at the time for each state in the finite state transducer is calculated, A node data update unit that updates the node data storage unit using the calculated score;
When an activation signal indicating the start of processing is received from the outside, the node data storage unit is referred to determine the maximum likelihood node at that time and refer to the finite state transducer information storage unit and the node data storage unit Then, by tracing back the state transition to the maximum likelihood node, a traceback process is performed up to a predetermined time when the state transition has been confirmed, and the path of the state transition subjected to the traceback process is set as an output candidate and output. With respect to the candidate path, an error score is calculated according to a ratio in which the state transition related to the error is included in the path, and the error score is smaller than a predetermined threshold based on the calculated error score. A manuscript search unit that uses a path output candidate as a final output;
An output unit for outputting the determined output obtained by the document search unit;
A speech recognition error correction apparatus comprising:
The finite state transducer information storage unit stores an output transition where the output symbol is a non-empty state transition as a part of the state transition, and stores sentences included in the original text in a plurality of chunks. Storing information related to the state and information related to the state transition of the finite state transducer that is divided and accepts the words included in the chunk as an input symbol after the state transition path.
A speech recognition error correction apparatus characterized by the above. The document search unit calculates the error score for each chunk and outputs the output transition corresponding to the chunk when the error level is smaller than a predetermined threshold based on the error score for each chunk. When the symbol is used as the definite output and the document search unit calculates the error score for each chunk, state transition related to the error with respect to the chunk and a section of the chunk immediately before the chunk An error score corresponding to the percentage of the error is calculated as the error score of the chunk.
The speech recognition error correction apparatus according to claim 1. Corresponding to the manuscript text, it is a finite state transducer that sequentially transitions while accepting words input as speech recognition results, and transitions while accepting errors in words contained in the speech recognition results A finite state transducer information storage unit for storing information about the state of the finite state transducer to be performed and information including a transition source state, a transition destination state, an input symbol, an output symbol, and a transition weight regarding the state transition When,
A node data storage unit for storing a score representing the likelihood of the state in the finite state transducer;
While receiving an input of a recognition word that is a speech recognition result corresponding to the document text from outside, according to the received recognition word, a score at the time for each state in the finite state transducer is calculated, A node data update unit that updates the node data storage unit using the calculated score;
When an activation signal indicating the start of processing is received from the outside, the node data storage unit is referred to determine the maximum likelihood node at that time and refer to the finite state transducer information storage unit and the node data storage unit Then, by tracing back the state transition to the maximum likelihood node, a traceback process is performed up to a predetermined time when the state transition has been confirmed, and the path of the state transition subjected to the traceback process is set as an output candidate and output. With respect to the candidate path, an error score is calculated according to a ratio in which the state transition related to the error is included in the path, and the error score is smaller than a predetermined threshold based on the calculated error score. A manuscript search unit that uses a path output candidate as a final output;
An output unit for outputting the determined output obtained by the document search unit;
A speech recognition error correction apparatus comprising:
The finite state transducer information storage unit stores an output transition where the output symbol is a non-empty state transition as a part of the state transition;
The manuscript search unit determines a node having the maximum likelihood among the nodes corresponding to the transition source state of the output transition as the maximum likelihood node at that time point.
A speech recognition error correction apparatus characterized by the above.

Images