JP6327745B2 - Speech recognition apparatus and program - Google Patents

Description

  The present invention relates to a speech recognition apparatus and a program.

  A technology that uses speech recognition to produce subtitles for live broadcast programs has been put into practical use. Broadcast subtitles are created by manually correcting the result of speech recognition of broadcast program audio (see, for example, Patent Document 1).

JP 2004-226910 A

  Speech recognition of broadcast programs is mainly aimed at information compensation for hearing impaired and elderly people. The target of voice recognition at this time is only the voice of the voice language in the broadcast program. However, the sound of many broadcast programs is not composed solely of sound languages. For example, a non-linguistic sound (for example, laughter), applause, background music, and other acoustic events are added in response to a program production request. The acoustic event is considered to play an important role in information transmission, like the spoken language, such as supplementarily explaining the scene of the broadcast program or notifying the change of the scene. From this point, it can be said that the acoustic event is one of the elements indispensable for the viewer to understand the program.

  However, in current caption production by voice recognition, acoustic events are not taken into account, and information for understanding a program may not be sufficiently conveyed to viewers. If the information of an acoustic event is reflected in subtitles, supplemental information such as color, accent, or nuance will be added to the subtitles to be transmitted, which will greatly contribute to viewer understanding of the program. To that end, it is required to produce captions with information on acoustic events.

  The present invention has been made in consideration of such circumstances, and provides a speech recognition apparatus and program capable of producing subtitles with information on acoustic events added thereto.

According to one aspect of the present invention, a speech recognition unit that recognizes speech data and outputs character string data indicating the utterance content of the speech recognition result; and an acoustic event based on an acoustic feature obtained from the speech data. A sound event recognition unit for calculating a posteriori probability and outputting character string data representing an acoustic event detected based on the calculated posteriori probability; and a character string data of the utterance content output by the voice recognition unit And the character string data representing the acoustic event output by the acoustic event recognition unit is displayed on the correction terminal, and the annotation insertion position in the character string of the utterance content specified from the displayed is displayed. An annotation insertion instruction indicating the character string representing the acoustic event selected from the inside is received from the correction terminal, and the character string indicating the utterance content is received according to the received annotation insertion instruction. A recognition result correction unit for generating an annotated caption data inserting a string of characters representing the acoustic events over data, a speech recognition apparatus comprising: a.
According to this invention, the voice recognition device displays a character string indicating the utterance content obtained by voice recognition of voice data and a character string representing an acoustic event detected for the voice data on the correction terminal. The speech recognition device inserts a character string representing an acoustic event into the utterance content according to the annotation insertion position in the character string of the utterance content designated by the corrector at the correction terminal and the character string representing the acoustic event selected as the annotation to be inserted. To generate subtitles with annotations.
Thus, the voice recognition device allows the corrector to view the acoustic event by a simple operation of selecting the position of the utterance content to which the annotation is to be inserted and the character string representing the acoustic event to be inserted as the annotation while viewing the display on the correction terminal. Subtitles with information added can be generated.

One aspect of the present invention is the speech recognition device described above, wherein the speech data is divided into frames, and the acoustic feature amount of each frame is compared with the acoustic feature amount of each of silence, acoustic event, and speech language. An acoustic event section detection unit that detects a section including an acoustic event, and the acoustic event recognition unit is based on an acoustic feature amount obtained from the audio data of the section detected by the acoustic event section detection unit. A posterior probability of the acoustic event is calculated, and character string data representing the acoustic event detected based on the calculated posterior probability is output.
According to this invention, the voice recognition device detects a section including an acoustic event from voice data, and performs acoustic event recognition on the voice data of the detected section.
Thereby, since the speech recognition apparatus sets only the section including the acoustic event as a target for the acoustic event recognition, the accuracy of the acoustic event recognition can be improved.

One aspect of the present invention is the speech recognition device described above, in which the acoustic event recognition unit arranges acoustic feature amounts of time-ordered frames obtained by dividing the speech data and inputs the acoustic feature amounts to a convolutional neural network to input an acoustic event. The posterior probability is calculated, and the convolutional neural network has an input layer, a hidden layer, a pooling layer, and an output layer, and the input layer receives the acoustic feature values of the frames arranged in time order as inputs, and Each unit of the layer is combined with a predetermined number of frames of the input layer while maintaining a shift by a predetermined number of frames, and shows the result of convolution calculation of the acoustic feature amount of the frame of the input layer combined, Each unit of the pooling layer is coupled to the number of hidden layer units corresponding to the number of units of the pooling layer. A maximum value is propagated among the hidden layer units, each unit of the output layer corresponds to a different type of acoustic event, and all the units of the pooling layer and the corresponding posterior probability of the acoustic event Are combined by respective weights for calculating.
According to this invention, the speech recognition apparatus divides speech data into frames that are processing units of acoustic feature amounts in acoustic event recognition, and arranges the acoustic feature amounts of the divided frames in the order of the times of the corresponding frames to perform a convolutional neural network. By inputting to the network, the posterior probability of each acoustic event is calculated.
Thereby, the speech recognition apparatus can obtain the posterior probability of each acoustic event using the acoustic feature amount of each frame obtained from the speech data.

One aspect of the present invention is the speech recognition device described above, wherein the acoustic feature amount is a feature amount in a time-frequency domain.
According to the present invention, the voice recognition device recognizes an acoustic event using the feature quantity in the time frequency domain of the voice data.
Thereby, since the speech recognition apparatus can recognize the acoustic event by connecting the feature quantities in the frequency domain for a predetermined time or more, it can improve the accuracy of the acoustic event recognition.

  According to one aspect of the present invention, a computer recognizes voice data, outputs voice string data indicating the utterance content of the voice recognition result, and an acoustic feature obtained from the voice data. A sound event recognition means for calculating a posterior probability of an acoustic event and outputting data of a character string representing an acoustic event detected based on the calculated posterior probability; and a character of the utterance content output by the voice recognition means Annotation insertion position in the character string of the utterance content designated from among the data of the column and the character string data representing the acoustic event output by the acoustic event recognition unit is displayed on the correction terminal, An annotation insertion instruction indicating the character string representing the acoustic event selected from among the displayed events is received from the correction terminal, and the previous instruction is received according to the received annotation insertion instruction. A recognition result correction means for generating an annotated caption data inserting a string of characters representing the acoustic event data string indicating the speech content is a program to function as the speech recognition apparatus comprising.

  According to the present invention, it is possible to produce a caption with information on an acoustic event added.

It is a figure which shows the comparison with the closed caption production method by one Embodiment of this invention, and the conventional closed caption production method. It is a functional block diagram which shows the structure of the caption production system by the same embodiment. It is a figure which shows the whole processing flow of the speech recognition apparatus by the embodiment. It is a figure which shows HMM for the acoustic event area detection by the same embodiment. It is a figure which shows the acoustic event area detection process flow of the acoustic event area detection part by the embodiment. It is a figure which shows the neural network for acoustic event recognition by the embodiment. It is a figure which shows the acoustic event recognition process flow of the acoustic event recognition part by the embodiment. It is a figure which shows the correction work screen displayed on the display part of the correction terminal by the embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In speech recognition for the purpose of subtitle production, output of recognition result character strings without delay is regarded as important. Conventionally, only a speech language important for information transmission to a viewer is a subject of subtitle conversion from speech to a character string, and non-language sounds such as acoustic events are not subject to subtitle conversion. This is because, particularly in a live broadcast program, there is not enough time for correcting a speech recognition error, and it is difficult to subtitle information other than the speech language.

  In a program such as news, the speech language occupies a very important weight, and hardly includes sound events such as sound effects. Therefore, it is possible to transmit necessary information to the viewer only by subtitling only the audio language. On the other hand, in sports programs and information programs, acoustic events such as laughter, applause, and cheers, which are non-verbal sounds, play a larger role. While the news focuses on telling the facts, other programs are partly due to the increasing importance of non-verbal sounds due to staging demands such as telling a sense of reality. Sound events, which are important in production, have a background that they have not been regarded as important in conventional caption production for live broadcasting. However, from the viewpoint of hearing impaired people and elderly people more enjoying or understanding broadcast programs, it is natural to enhance non-verbal sound events as subtitles.

FIG. 1A is a diagram showing a conventional caption production method. In the conventional caption production method, only a speech-capable speech language included in the input speech is subject to caption production. Therefore, a speech section including the speech language is detected from the input speech, and the corresponding section is cut out. Next, the extracted speech section is speech-recognized, and text data of a word string as a recognition result is output. Since the recognition result usually includes a recognition error, the error in the recognition result is manually corrected, and the correction result is transmitted as a broadcast subtitle.
This series of procedures is performed sequentially every time a voice section is cut out, and caption production can be performed with low delay.

When inserting an acoustic event in the conventional caption production method based on speech recognition, the corrector interprets the content represented by the non-language sound as appropriate, and then uses a keyboard or other input method to annotate the character string representing the acoustic event. Can be inserted into the speech recognition result. However, since keyboard input takes time, it is actually very difficult for the corrector to perform additional keyboard input work while correcting the speech recognition result.
The speech recognition apparatus according to the present embodiment solves such a problem and performs caption production for performing information transmission regarding an acoustic event to a viewer.

  Therefore, the speech recognition apparatus according to the present embodiment outputs the recognition result of the acoustic event as an annotation together with the speech recognition result similar to the conventional caption production method. Here, the “annotation” is a text (character string) representing an acoustic event, which is additional information for the speech language, as a language expression. In addition, an annotation inserted with respect to a conventional caption based on a speech recognition result of a speech language is referred to as “annotated caption”.

FIG. 1B is a diagram showing a caption production method by the speech recognition apparatus of the present embodiment.
As shown in the figure, in the caption production method by the speech recognition apparatus of the present embodiment, the acoustic event section detection process and the acoustic event recognition process are executed in parallel with the conventional speech section detection process and the speech recognition process. . In the acoustic event section detection process, a voice section including an acoustic event is detected from the input voice, and the corresponding section is cut out. In the acoustic event recognition process, an acoustic event in the extracted acoustic event section is recognized, and text data of a word string representing the recognized acoustic event is output. Through the parallel operation of the speech recognition process and the acoustic event recognition process, the speech recognition apparatus according to the present embodiment can mount an optimal algorithm independently for each recognition process. If the recognition of the acoustic event is unnecessary, the sound recognition device of the present embodiment may be set so that the execution program for the acoustic event recognition process is not operated. This makes it possible to select a caption production method that meets the needs of the caption producer.

  Then, in the caption production method by the speech recognition apparatus of the present embodiment, the annotated broadcast subtitle that is an annotated caption to be broadcast is integrated by integrating the speech recognition result and the acoustic event recognition result at the time of manually correcting the speech recognition result. Produce. As described above, when the speech recognition apparatus of the present embodiment executes speech recognition processing and acoustic event recognition processing in parallel, it is necessary to integrate the final speech recognition result and the acoustic event recognition result given as an annotation. There is. Normally, manual correction is performed on the voice recognition result at the correction terminal. When the corrector inputs a correction instruction to the voice recognition result displayed on the correction terminal, the voice recognition device of the present embodiment also displays the annotation that is the acoustic event recognition result on the correction terminal, and the voice recognition result With an efficient interface for insertion into This interface will save labor for creating acoustic event character strings by keyboard input.

  Due to the parallel execution of the speech recognition process and the acoustic event recognition process as described above, and the integration of the speech recognition result and the acoustic event recognition result at the time of the correction work, the speech recognition apparatus according to the present embodiment has been difficult to achieve. Enables efficient caption production with annotations about events.

  FIG. 2 is a block diagram showing a configuration of a caption production system according to an embodiment of the present invention, and shows only functional blocks related to the present embodiment. As shown in the figure, the caption production system includes a voice recognition device 1 and a correction terminal 5. The speech recognition apparatus 1 and the correction terminal 5 are connected via a network. In the figure, the subtitle production system includes two correction terminals 5, but only one correction terminal 5 or three or more correction terminals 5 may be provided. The two correction terminals 5 are referred to as correction terminals 5-1 and 5-2, respectively.

  The voice recognition device 1 is realized by a computer device. As shown in the figure, the speech recognition apparatus 1 includes a storage unit 10, a speech branching unit 11, a speech segment detection unit 12, a speech recognition unit 13, an acoustic event segment detection unit 14, an acoustic event recognition unit 15, and a recognition result correction. A portion 16 is provided.

  The storage unit 10 stores a statistical acoustic model for speech section detection, a statistical acoustic model for speech recognition, and a statistical language model. Further, the storage unit 10 stores a statistical acoustic model for acoustic event section detection and a neural network for acoustic event recognition. The voice branching unit 11 branches the voice data D1 input to the voice recognition device 1 into two and outputs it to the voice segment detection unit 12 and the acoustic event segment detection unit 14.

  The speech section detection unit 12 uses the statistical acoustic model for speech section detection stored in the storage unit 10 and uses the statistical data of speech language to be converted into text in the speech data D1 input from the speech branching unit 11. A speech language segment that is a speech segment is detected. The speech segment detection unit 12 outputs the speech language segment data D2, which is the speech language segment of the detected speech data D1, to the speech recognition unit 13. The speech recognition unit 13 recognizes speech language section data D2 using the statistical acoustic model and statistical language model for speech recognition stored in the storage unit 10. The voice recognition unit 13 outputs the voice recognition result data D3 set with the voice recognition result of the utterance content to the recognition result correction unit 16.

  The acoustic event section detection unit 14 uses a statistical acoustic model for acoustic event section detection stored in the storage unit 10, and the voice section including the acoustic event in the voice data D <b> 1 input from the voice branching unit 11. An acoustic event section is detected. The acoustic event section detection unit 14 outputs acoustic event section data D4 that is an acoustic event section of the detected audio data D1 to the acoustic event recognition unit 15. The acoustic event recognition unit 15 recognizes the acoustic event of the acoustic event section data D4 using the neural network for acoustic event recognition stored in the storage unit 10. The acoustic event recognition unit 15 outputs the acoustic event recognition result data D5 in which the acoustic event recognition result is set to the recognition result correction unit 16. The acoustic event recognition result is a text expression (character string) representing the detected acoustic event.

  The recognition result correction unit 16 outputs the speech recognition result data D3 output from the speech recognition unit 13 and the acoustic event recognition result data D5 output from the acoustic event recognition unit 15 to the correction terminal 5 for display. The recognition result correction unit 16 corrects the speech recognition result based on the correction instruction received from the correction terminal 5 and inserts an annotation character string into the speech recognition result based on the annotation insertion instruction received from the correction terminal 5. Additional broadcast caption data D6 is generated. The correction instruction indicates a correction location in the speech recognition result and correction contents such as deletion, insertion, and replacement of characters at the correction location. The annotation insertion instruction indicates an annotation insertion location in the speech recognition result and an annotation character string to be inserted at the annotation insertion location. The annotation character string is selected by the corrector from the text representation of the acoustic event of the acoustic event recognition result data D5 displayed on the correction terminal 5. The recognition result correction unit 16 outputs the generated annotated broadcast caption data D6.

  The correction terminal 5 is realized by a computer device such as a personal computer, for example. The correction terminal 5 includes a control unit 51, a display unit 52, an input unit 53, and an audio output unit 54. The display unit 52 is a display and displays a screen. The input unit 53 is a keyboard, a mouse, or the like, and receives an operation by a corrector. In the present embodiment, a case where the correction terminal 5 includes a touch panel and a keyboard will be described as an example. The touch panel serves as both the display unit 52 and the input unit 53. The audio output unit 54 is a headphone or a speaker, and outputs a reproduction sound of the audio data D1. The control unit 51 causes the display unit 52 to display the voice recognition result data D3 and the acoustic event recognition result data D5 received from the voice recognition device 1. In addition, the control unit 51 outputs a speech recognition result correction instruction input by the corrector through the input unit 53 and an annotation insertion instruction to the speech recognition result to the speech recognition apparatus 1. Further, the control unit 51 causes the audio output unit 54 to output the reproduced audio of the audio data D1.

Next, the operation of the voice recognition device 1 will be described.
First, the speech recognition apparatus 1 stores a statistical acoustic model for speech segment detection and acoustic event segment detection, a statistical acoustic model and statistical language model for speech recognition, and a neural network for acoustic event recognition. Stored in the unit 10. A statistical acoustic model for detecting a speech section, a statistical acoustic model for speech recognition, and a statistical language model can be the same as those in the past. In the present embodiment, HMM (Hidden Markov Model) and GMM (Gaussian Mixture Model) are used as statistical acoustic models for detecting acoustic event intervals. The acoustic event section detection HMM and GMM are learned by the same learning method as in the prior art, using speech data labeled with three classes of speech, acoustic event, and silence as learning data. Note that the voice label is attached to the voice data of the voice language. For example, in the case of GMM of acoustic events, each of the mixed Gaussian distributions represents a characteristic of a different type of acoustic event. For learning of a neural network for acoustic event recognition, speech data labeled with each acoustic event is used as learning data, and learning is performed by a learning method similar to that of the prior art. The HMM for detecting an acoustic event section will be described later with reference to FIG. 4, and the neural network for acoustic event recognition will be described later with reference to FIG.

FIG. 3 is a diagram showing an overall processing flow of the speech recognition apparatus 1. The voice recognition device 1 performs the process shown in the figure every time the voice data D1 is input.
When the audio data D1 of the broadcast program is input to the audio recognition device 1, the audio branching unit 11 branches the input audio data D1 into two in order to input the audio recognition and the acoustic event recognition. This is because there is an overlap between the speech language and the acoustic event. By dividing the speech recognition process and the acoustic event recognition process, an optimal recognition algorithm can be applied independently. The audio branching unit 11 outputs one of the audio data D1 branched into two to the audio interval detecting unit 12 that performs preprocessing for audio recognition, and the other audio event interval that performs preprocessing for acoustic event recognition. It outputs to the detection part 14 (step S1).

  The speech section detection unit 12 detects and cuts out a speech language section that needs to be converted into text in the speech data D1 by the conventional technique (step S2). This speech language segment may include overlap with acoustic events such as background sounds. In the present embodiment, a voice section is detected by the techniques described in Japanese Patent Application Laid-Open No. 2007-233148 and Japanese Patent Application Laid-Open No. 2007-233149. The speech segment detection unit 12 outputs the speech language segment data D2, which is the speech language segment of the detected speech data D1, to the speech recognition unit 13.

Specifically, every time voice data D1 is input, the voice section detection unit 12 divides the voice indicated by the voice data D1 into input frames that are frames of one processing unit at a predetermined time interval. The voice section detection unit 12 calculates the acoustic feature amount of each of a predetermined number of input frames selected in order from the earliest time. The state transition network for detecting an utterance section is a left-to-right type HMM that makes a transition between non-speech language, speech language, and silence without skipping from the start of utterance to the end of utterance. Note that a non-speech language state may be used instead of the silent state. The speech section detection unit 12 reads out the acoustic models of the non-speech language and the speech language from the storage unit 10 and calculates the acoustic score (logarithmic likelihood) of each input frame using the read out acoustic models. A non-speech language acoustic model represents an HMM such as silence or an acoustic event. The acoustic model of the speech language is composed of phoneme HMMs for each phoneme. The voice section detection unit 12 stores a record of the state transition of each input frame, traces the state transition record from the current state toward the start state, and inputs a process start (starting end) using the state transition network. The cumulative acoustic score of each state series from the frame is calculated. When the difference between the maximum accumulated acoustic score of each state series and the starting acoustic score is larger than the threshold, the speech section detecting unit 12 is the last in the series in which the maximum accumulated acoustic score is obtained. A time that is a predetermined time later than the time when the voice language was in use is set as the utterance start time.
Further, the speech section detection unit 12 calculates the accumulated acoustic score of each state series from the input frame at the start of processing to the current input frame in the same manner as described above for the input frame after the speech start time is detected. The speech section detection unit 12 determines the difference between the maximum accumulated acoustic score in each state sequence and the maximum accumulated acoustic score in the state sequence from the spoken language to the end of the non-speech language in each state sequence. Determines whether the threshold value has been exceeded. When a predetermined time has elapsed after the threshold value is exceeded, the voice section detection unit 12 sets a time that is a predetermined time later than the elapsed time as the utterance end time.
The speech section detection unit 12 outputs speech language section data D2 in which input frames in a section from the utterance start time to the utterance end time are collected.

  The speech recognition unit 13 recognizes the speech language section data D2 using the statistical acoustic model and the statistical language model for speech recognition stored in the storage unit 10 according to the conventional technique (step S3). In the present embodiment, the speech recognition unit 13 uses HMM and GMM for the statistical acoustic model. In the present embodiment, the speech recognition unit 13 obtains a recognition result by multipath speech recognition using a word n-gram language model as a statistical language model. This recognition result is a segmentation in units of words, and the speech recognition unit 13 assigns each word with time information when the word is uttered. The voice recognition unit 13 outputs the voice recognition result data D3 set with the voice recognition result to the recognition result correction unit 16 (step S4).

  On the other hand, the acoustic event section detection unit 14 detects and cuts out an acoustic event section including an acoustic event such as a background sound in the audio data D1 (step S5). This acoustic event section may include an overlap with a part that needs to be converted into text by speech recognition. The acoustic event section detection unit 14 detects the acoustic event section using the GMM and the HMM for acoustic event section detection stored in the storage unit 10, using the same algorithm as the speech section detection unit 12. However, while the speech segment detection unit 12 is targeted for the speech segment (speech language segment) of the speech language, the acoustic event segment detection unit 14 is different in that the speech segment of the non-language sound is the detection target. . Further, an HMM for detecting an acoustic event section is used in place of the state transition network for detecting an utterance section.

  FIG. 4 is a diagram illustrating an HMM for detecting an acoustic event section stored in the storage unit 10. In the present embodiment, the configuration of the HMM is a so-called ergodic HMM. As shown in the figure, the ergodic HMM is an HMM that expresses three classes of transition of voice, acoustic event, and silence. Each transition is given a transition probability obtained by learning.

  FIG. 5 is a diagram showing an acoustic event section detection processing flow of the acoustic event section detection unit 14, and shows detailed processing in step S5 of FIG. First, every time the audio data D1 is input, the acoustic event section detection unit 14 divides the audio data D1 into input frames D11 that are frames of one processing unit at a predetermined time interval.

  The acoustic event section detection unit 14 acquires a predetermined number of input frames D11 in order from the earliest time among the input frames D11 that are not yet processed (step S51). The acoustic event section detection unit 14 calculates the acoustic feature amount of each acquired input frame D11. The acoustic event section detection unit 14 reads from the storage unit 10 each voice, acoustic event, and silent GMM that is each state of the HMM. The acoustic event section detection unit 14 compares the read GMM and the acoustic feature quantity of each input frame D11 to calculate the acoustic score of each input frame D11, and performs transition between HMM states if necessary. (Step S52). When the predetermined number of input frames necessary for the traceback are not processed (step S53: NO), the acoustic event section detection unit 14 returns to step S51 to acquire a new input frame D11, and acquires the acoustic score. Perform the calculation.

  When the predetermined number of input frames necessary for traceback are processed (step S53: YES), the acoustic event section detection unit 14 obtains a list of state series up to the current state by traceback (step S54). ). That is, the acoustic event section detection unit 14 traces the record of the state transition from the current state toward the start state, and uses the ergodic HMM shown in FIG. 4 from the state (start state) of the input frame D11 at the start of processing. The cumulative acoustic score of each state series up to the current state is calculated. At this time, the acoustic event section detection unit 14 sorts the series in descending order of the accumulated acoustic score.

  The acoustic event section detection unit 14 compares the first rank series and the second rank series from the HMM state series obtained by the traceback (step S55). The acoustic event section detection unit 14 determines that the section is not fixed when the accumulated acoustic score difference is equal to or less than a predetermined threshold (step S56: NO), returns to step S51, and performs a new input frame D11. To calculate the acoustic score. When the acoustic event section detection unit 14 determines that the difference between the accumulated acoustic scores exceeds a predetermined threshold (step S56: YES), the acoustic event section detection unit 14 sets the first rank series as the confirmed section. The acoustic event section detection unit 14 outputs a frame sequence that finally collects the frames of the confirmed section of the acoustic event as acoustic event section data D4 (step S57).

  In FIG. 3, the acoustic event recognition unit 15 recognizes an acoustic event from the acoustic event section data D4 obtained by the acoustic event section detection unit 14 using a neural network for acoustic event recognition stored in the storage unit 10. (Step S6). First, the acoustic event recognizing unit 15 connects the acoustic event frame data constituting the acoustic event section data D4 until the frame reaches a predetermined length N or more by connecting the frame sequences. This is because it is difficult to capture temporal changes in the frequency characteristics of acoustic events from a frame sequence that is too short, and it is difficult to estimate acoustic events with high accuracy. When the acoustic event recognizing unit 15 obtains an input frame sequence including N or more frame sequences by frame connection, the acoustic event recognizing unit 15 performs acoustic event recognition using a neural network stored in the storage unit 10.

  FIG. 6 is a diagram showing a neural network for acoustic event recognition stored in the storage unit 10. As shown in the figure, in this embodiment, the acoustic event recognition unit 15 uses a convolutional neural network, which is a kind of neural network, for acoustic event recognition. An example of a convolutional neural network is described in, for example, a document “Andrew L. Maas et al.,“ Word-level Acoustic Modeling with Convolutional Vector Regression ”, ICML Representation Learning Workshop, 2012”.

The convolutional neural network shown in the figure is composed of four layers: an input layer, a hidden layer, a pooling layer, and an output layer. The input layer corresponds to a plurality of time-ordered frames output from the acoustic event section detection unit 14, and the value of the input layer is an acoustic feature quantity in the time frequency domain such as a mel frequency cepstrum obtained from the corresponding frame. is there. This acoustic feature amount is represented by a vector, for example. In the present embodiment, the total number of frames N _s (≧ N) of the acoustic feature quantity of the input layer is variable.

Each unit (element) of the hidden layer is coupled to only the continuous n _s frames among the frames (elements) of the total number of frames N _s of the input layer. The n _s number of frames of the input layer units of the hidden layer are bonded, but the previous adjacent units corresponds to the time after the n _s frames attached, some Shifting by k frames so as to overlap (k < _ns ). For example, it is assumed that the i to (i + 2) -th frame in the input layer is coupled to the i-th unit in the hidden layer. The value of the i-th unit of the hidden layer is an addition (convolution operation) of the values of the i to (i + 2) -th frames of the input layer. However, the connection weight (weight at the time of addition) of the i-th frame of the input layer, the (i + 1) -th frame, the (i + 2) -th frame, and the i-th unit of the hidden layer may not be equal. . For example, the first to third frames of the input layer are coupled to the first unit of the hidden layer, the second to fourth frames of the input layer are coupled to the second unit of the hidden layer, and 3 The fifth frame binds to the third unit of the hidden layer. At this time, (combination weight of the first unit of the hidden layer from the first frame of the input layer) = (joint weight of the second unit of the hidden layer from the second frame of the input layer) = (3 of the input layer) The coupling weight of the third unit of the hidden layer from the th frame) =. Similarly, (combination weight of the first unit of the hidden layer from the second frame of the input layer) = (joint weight of the second unit of the hidden layer from the third frame of the input layer) = (4 of the input layer) The coupling weight of the third unit of the hidden layer from the th frame) =. In other words, the coupling between the hidden layer unit and the input layer frame connects the elements of the input layer and the hidden layer with the same coupling weight while maintaining a shift of k frames. The number of hidden layer units _Nh is a number corresponding to the number of units in the input layer.

Pooling layer above the hidden layer is constituted by a unit of the number of units N _p of predetermined fixed. Each unit in the pooling layer is coupled to a unit having a variable number of units n _h = N _p / N _h among the units in the hidden layer. The coupling between the pooling layer unit and the hidden layer unit has a characteristic that only the maximum value among the values of the hidden layer units coupled to the same pooling layer unit propagates to the pooling layer.

The pooling layer and the output layer are all connected to each other. The value of the output layer is obtained by applying the weighting coefficient matrix representing the weight between each unit of the pooling layer and each unit of the output layer to the value of the pooling layer, and then using the Softmax function for each unit of the output layer. Is calculated by normalizing the output of. The unit of the output layer represents a text expression (character string) corresponding to an acoustic event, and gives a posterior probability of the text expression when an acoustic feature amount is given.
In this embodiment, the pooling layer and the output layer are connected, but any number of hidden layers and pooling layers can be inserted between them.

FIG. 7 is a diagram showing an acoustic event recognition processing flow of the acoustic event recognition unit 15, and shows detailed processing in step S6 of FIG.
The acoustic event recognition unit 15 concatenates the frame sequence of the acoustic event section data D4, which is the output from the acoustic event section detection unit 14, in order of time so that the input feature amount of the convolutional neural network is sufficiently long, and the input frame A column is generated (step S61). When the length of the input frame sequence has not reached N (step S62: NO), the acoustic event recognition unit 15 returns to step S61, and new acoustic event section data D4 until an input frame sequence of N frames or more is obtained. Frame frames of the frames are connected. When the length of the input frame sequence is greater than or equal to N necessary for acoustic event recognition (step S62: YES), the acoustic event recognition unit 15 performs acoustic event recognition using the convolutional neural network stored in the storage unit 10. (Step S63). The acoustic event recognition unit 15 calculates the acoustic feature amount of each frame constituting the input frame sequence. The acoustic event recognition unit 15 calculates the values of the units of the hidden layer, the pooling layer, and the output layer by using the acoustic feature amount calculated for each frame of the input frame sequence as the input of the input layer of the convolutional neural network shown in FIG. To do.

  Finally, the acoustic event recognition unit 15 selects a unit in the output layer of the convolutional neural network based on the posterior probability indicated by the output of each unit. For example, the acoustic event recognition unit 15 may select a predetermined number of units in order from the largest posterior probability, may select a unit having a posterior probability of a predetermined value or higher, and the posterior probability is higher than a predetermined value. Up to a predetermined number of units may be selected in descending order of the posterior probability. The storage unit 10 stores in advance the unit number of the output layer and the text expression selected by the user for the acoustic event represented by the unit of that number. The acoustic event recognition unit 15 reads the text representation of the acoustic event corresponding to the selected unit from the storage unit 10.

  In this embodiment, the text representation of the acoustic event according to the classification as shown in Table 1 to Table 5 below is used.

  Tables 1 to 5 show examples of the text representation of the corresponding acoustic event, but it is difficult to uniquely determine the text representation corresponding to a certain acoustic event. Therefore, the text of subtitle broadcasting performed in the past is analyzed, and a representative expression with high frequency is selected as the text expression. For example, these expressions are expressed as a subtitle broadcast (footnote explaining the scene). Then, the unit number of the output layer and the text expression selected as the acoustic event represented by the unit of that number are stored in the storage unit 10 in association with each other.

  In FIG. 3, the acoustic event recognition unit 15 assigns ranks to the text representations of the read acoustic events in descending order of posterior probabilities. The acoustic event recognizing unit 15 sets an annotation character string, which is a text representation of the acoustic event to which the ranking is given, in the acoustic event recognition result data D5, and outputs it to the recognition result correcting unit 16 (step S7).

  The recognition result correction unit 16 integrates the voice recognition result indicated by the voice recognition result data D3 and the annotation character string indicated by the acoustic event recognition result data D5 to create a final broadcast subtitle (step S8). The speech recognition apparatus 1 of the present embodiment provides an efficient interface that can efficiently implement both. There are the following two methods for providing this interface.

  The first interface providing method is a method of inserting an annotation when the corrector corrects the recognition result. The correction of the recognition result is performed based on the input of the operator using the correction terminal 5 realized by a computer device having a touch panel.

FIG. 8 shows a correction work screen 8 which is a computer display screen displayed on the display unit 52 of the correction terminal 5. The correction work screen 8 includes a speech recognition result display window 80, an acoustic event recognition result display window 83, an acoustic event recognition result candidate window 86, and a history display window 87.
The voice recognition result display window 80 displays a voice recognition result and a character string obtained by correcting the voice recognition result and inserting an annotation character string. The acoustic event recognition result display window 83 displays an annotation character string. The annotation character string displayed in the acoustic event recognition result display window 83 is the annotation character string having the highest order set in the acoustic event recognition result data D5. The acoustic event recognition result candidate window 86 displays annotation character string candidates. Annotation character string candidates are annotation character strings with the second or lower rank set in the acoustic event recognition result data D5. The history display window 87 displays a corrected character string for the voice recognition result.

  The recognition result correction unit 16 of the speech recognition device 1 outputs the speech recognition result data D3 output from the speech recognition unit 13 and the acoustic event recognition result data D5 output from the acoustic event recognition unit 15 to the correction terminal 5 as needed. To do. At this time, the recognition result correction unit 16 also outputs the voice data D1 corresponding to the voice recognition result data D3 to the correction terminal 5. The recognition result correction unit 16 sets the voice recognition result indicated by the voice recognition result data D3 output to the correction terminal 5 as a working subtitle.

  The control unit 51 of each correction terminal 5 outputs the reproduced sound of the received sound data D1 from the sound output unit 54. The control unit 51 causes the voice recognition result display window 80 to display the voice recognition result read from the received voice recognition result data D3 as the correction target character string on the bottom line of the voice recognition result display window 80. At this time, the control unit 51 displays the speech recognition result as a character string with a vertical bar between words. If the voice recognition result corrected to the bottom line has already been displayed in the voice recognition result display window 80, the control unit 51 corrects the earliest time among the corrected voice recognition results displayed. Erase the completed speech recognition result. After erasure, the control unit 51 moves the remaining corrected speech recognition results to a line above the current level, and displays the speech recognition results read from the received speech recognition result data D3 in the speech recognition result display window 80. Display on the bottom line.

  Further, the control unit 51 of each correction terminal 5 displays the latest annotation character strings in order from the right end of the acoustic event recognition result display window 83. That is, every time new acoustic event recognition result data D5 is received from the speech recognition apparatus 1, the control unit 51 shifts the annotation character string displayed in the acoustic event recognition result display window 83 to the left and displays it. The control unit 51 displays the highest-ranked annotation character string read from the newly received acoustic event recognition result data D5 on the right end of the acoustic event recognition result display window 83. Further, the control unit 51 causes the acoustic event recognition result candidate window 86 to display the annotation character strings ranked in the second or lower rank set in the received acoustic event recognition result data D5 as a menu.

  The speech recognition result correction operation is performed as follows. While listening to the program sound, the corrector touches the display portion of the character including the character string to be corrected from the character string displayed on the display unit 52 by the voice recognition result display window 80 with a finger or the like. The corrector may move a finger and touch a plurality of characters. The input unit 53 outputs information on the screen position at which contact is detected to the control unit 51. The control unit 51 selects a word including the character displayed at the screen position where contact is detected, and transmits indication information for specifying the selected word to the speech recognition apparatus 1. For example, the time at which a word is pronounced can be used as the indication information. It is assumed that the recognition result correction unit 16 of the voice recognition device 1 has received the indication information from the correction terminal 5-1 earliest. The recognition result correction unit 16 instructs each correction terminal 5 to change the display of the character string indicated by the indication information received from the correction terminal 5-1 to a selected color such as red. The control part 51 of each correction terminal 5 changes the display of the character string selected based on the instruction | indication from the speech recognition apparatus 1 to a selection color. Furthermore, the recognition result correction unit 16 instructs the correction terminal 5-2 to change the guard to the selected color. In the correction terminal 5-2 in which the correction guard is instructed, correction work and annotation insertion work cannot be performed.

  The corrector using the correction terminal 5-1 uses the input unit 53 to perform correction operations such as replacement, insertion, and deletion for the character string displayed in the selected color. For example, the corrector inputs characters using a keyboard while a word is selected. When the correction work is completed, the corrector presses a key such as Enter on the keyboard as the correction work end operation. When receiving the input of the correction work end operation, the control unit 51 transmits the content of the correction work to the voice recognition device 1. The recognition result correction unit 16 of the speech recognition apparatus 1 corrects the selected character string in the working subtitle according to the correction work content received from the correction terminal 5-1, and generates a new working subtitle. The recognition result correction unit 16 transmits a new subtitle during work and a character string input from the keyboard by the corrector in the correction work to each correction terminal 5. The control unit 51 of each correction terminal 5 replaces the display of the speech recognition result displayed in the speech recognition result display window 80 with the working subtitle received from the speech recognition device 1. Further, the control unit 51 of each correction terminal 5 causes the history display window 87 to display a character string input from the keyboard by the corrector as a list work history. The correction terminal 5-2 releases the correction guard.

Annotation is inserted as follows. The corrector inserts an arbitrary annotation character string displayed in the acoustic event recognition result display window 83 into an arbitrary portion of the character string displayed in the voice recognition result display window 80 while listening to the program sound. Go.
For example, immediately after the voice recognition result (or the corrected voice recognition result) indicated by the character string 81 “I am good at cooking,” the annotation character string 84 “(laughter) displayed in the acoustic event recognition result display window 83 is displayed. ) ", The corrector performs the following operation. The corrector touches the last character “.” Of the character string 81 into which the annotation character string is to be inserted. The input unit 53 outputs information on the screen position at which contact is detected to the control unit 51. The control unit 51 selects the word “.” That includes the character displayed at the screen position where the contact is detected, and transmits the indication information that identifies the selected word to the speech recognition apparatus 1. That is, the indication information at this time indicates the annotation insertion position. It is assumed that the recognition result correction unit 16 of the voice recognition device 1 has received the indication information from the correction terminal 5-1 earliest. The recognition result correction unit 16 instructs each correction terminal 5 to change the display of the character string indicated by the indication information received from the correction terminal 5-1 to a selected color such as red. The control unit 51 of each correction terminal 5 changes the display of the selected character string to the selected color based on an instruction from the voice recognition device 1. Furthermore, the recognition result correction unit 16 instructs the correction terminal 5-2 to change the guard in accordance with the change to the selected color.

  The corrector using the correction terminal 5-1 presses the “Insert” key on the keyboard, and touches any character of the annotation character string 84 “(laugh)”. The input unit 53 outputs to the control unit 51 information on the screen position where the “Insert” key is pressed and the contact is detected. When the control unit 51 determines the annotation character string including the character displayed at the screen position where the contact is detected, the control unit 51 recognizes the information specifying the annotation character string or the insertion annotation information in which the annotation character string is set. Transmit to device 1. A combination of the previously sent indication information and insertion annotation information corresponds to the annotation insertion instruction. The recognition result correction unit 16 of the speech recognition apparatus 1 inserts the annotation character string specified by the insertion annotation information or indicated by the insertion annotation information immediately after the selected word “.” In the working subtitle, and newly adds the annotation character string. The subtitle "I'm good at cooking. (Laughs)" is generated. The recognition result correction unit 16 transmits a new working subtitle to each correction terminal 5. The control unit 51 of each correction terminal 5 replaces the display of the voice recognition result (or the corrected voice recognition result) displayed in the voice recognition result display window 80 by the working subtitle received from the voice recognition device 1. The correction terminal 5-2 releases the correction guard.

When the corrector wants to insert the annotation character string “(laugh)”, the corrector replaces the annotation character string “(laugh)” displayed in the acoustic event recognition result display window 83 with the annotation character string 85 “( You may touch any of the characters "laughter".
Further, for example, when an annotation character string is inserted immediately after the corrected recognition result “What is Mr. XX's hobby?” Indicated by the character string 82 displayed in the voice recognition result display window 80, the character string Touch the last character “.” Of 82.

  If the acoustic event recognition result is incorrect, a correct annotation character string cannot be selected from the acoustic event recognition result display window 83. In this case, the operator selects an annotation character string to be inserted from a list of annotation character string candidates displayed on the acoustic event recognition result candidate window 86 as a menu.

  The second interface providing method is a method of inserting an annotation character string when decorating a corrected character string. In the production of subtitles for information programs and sports broadcasts, the corresponding subtitle color is classified into white, blue, yellow, etc., depending on the speaker (program performer). Color coding is often performed by another operator for the subtitles after correction. In this case, on the screen shown in FIG. 8, instead of correcting the character string, an appropriate color is designated for each line of the displayed character string, and at the same time, an appropriate acoustic event is displayed from the acoustic event recognition result display window 83. What is necessary is just to insert a recognition result. In the following, a case where the speech recognition result is corrected by the correction terminal 5-1 and decoration is applied to the corrected speech recognition result by the correction terminal 5-2 will be described focusing on differences from the first interface providing method. To do.

  The recognition result correction unit 16 of the speech recognition apparatus 1 uses the speech recognition result data D3 output from the speech recognition unit 13, the corresponding speech data D1, and the acoustic event recognition result data D5 output from the acoustic event recognition unit 15. And output to the correction terminal 5 as needed. The control part 51 of each correction terminal 5 outputs the reproduction | regeneration audio | voice of the received audio | voice data D1 from the audio | voice output part 54, and shows the correction work screen 8 shown in FIG. The correction work of the speech recognition result by the corrector of the correction terminal 5-1 is the same as the method for providing the first interface. However, the recognition result correction unit 16 of the speech recognition apparatus 1 transmits a correction guard when there is another correction terminal 5 that corrects the speech recognition result, but a correction terminal that decorates the corrected speech recognition result. In 5-2, the correction guard may not be transmitted.

  Subsequently, the recognition result correction unit 16 of the speech recognition device 1 outputs the newly generated speech recognition result data D3 and the corresponding speech data D1 to the speech recognition device 1. The control unit 51 of each correction terminal 5 outputs the reproduction sound of the newly received audio data D1 from the audio output unit 54. Further, as in the first interface providing method, the control unit 51 displays the voice recognition result read from the received voice recognition result data D3 as the correction target character string on the bottom line of the voice recognition result display window 80. Let

  The corrector of the correction terminal 5-2, while listening to the program audio, corrects the corrected voice recognition result (for example, the color of the character string displayed on the display unit 52 by the voice recognition result display window 80). A character color including a character string 82) is touched with a finger or the like to input a character color. The character color may be input using a keyboard or the like, or may be input by providing a button for selecting a character color in the voice recognition result display window 80 and touching the button. The input unit 53 outputs information on the screen position at which contact is detected to the control unit 51. The control unit 51 selects a line including the character displayed at the screen position where contact is detected, and the voice recognition device 1 displays the decoration information indicating the information specifying the selected line and the input character color. Send to. The recognition result correction unit 16 of the speech recognition apparatus 1 changes the character string of the line in the working subtitle indicated by the decoration information received from the correction terminal 5-2 to the character color indicated by the decoration information, and performs a new work. Generate subtitles. The recognition result correction unit 16 instructs each correction terminal 5 to display the character string of the selected line with the changed character color. The control unit 51 of each correction terminal 5 displays the character string of the designated line (corrected voice recognition result) in the voice recognition result display window 80 with the changed character color in accordance with the instruction from the voice recognition device 1. .

Further, the corrector of the correction terminal 5-2 changes the arbitrary annotation character string displayed in the acoustic event recognition result display window 83 to any of the corrected voice recognition results displayed in the voice recognition result display window 80. Insert it in the place of.
For example, when the annotation character string 84 “(laughs)” is inserted immediately after the corrected speech recognition result “What is Mr. XX ’s hobby?” Indicated by the character string 82, the corrector selects “ Press the “Insert” key and touch the last character “.” Of the character string 82. The input unit 53 outputs to the control unit 51 information on the screen position where the “Insert” key is pressed and the contact is detected. The control unit 51 selects a word including the character displayed at the screen position where contact is detected, and generates annotation insertion position information that identifies the selected word. Further, the corrector touches any character of the annotation character string 84 “(laugh)”. The input unit 53 outputs information on the screen position at which contact is detected to the control unit 51. When the control unit 51 determines the annotation character string including the character displayed at the screen position where the contact is detected, the control unit 51 generates information for specifying the annotation character string or insertion annotation information in which the annotation character string is set. . The control unit 51 transmits an annotation insertion instruction in which annotation insertion position information and insertion annotation information are set to the speech recognition apparatus 1. The recognition result correction unit 16 of the speech recognition apparatus 1 identifies the word “.” As the annotation insertion target in the working subtitle based on the annotation insertion position information. The recognition result correcting unit 16 inserts the annotation character string specified by the inserted annotation information or indicated by the inserted annotation information immediately after the word “.” As the annotation insertion target in the working subtitle, and creates a new working subtitle. Is generated. The recognition result correction unit 16 transmits a new working subtitle to each correction terminal 5. The control unit 51 of each correction terminal 5 replaces the display of the corrected voice recognition result displayed in the voice recognition result display window 80 with the working subtitle received from the voice recognition device 1.

  In FIG. 2, the recognition result correction unit 16 of the speech recognition apparatus 1 outputs annotated broadcast subtitle data D <b> 6 in which the in-work subtitle is set reflecting the above-described speech recognition result correction operation and annotation insertion operation. (Step S9). The annotated broadcast subtitle data D6 is broadcast by being superimposed on the broadcast wave in the broadcast station.

  As described above, the corrector can create an annotated caption by inserting an annotation, which is a text representation of an acoustic event, into the speech recognition result by a simple operation. Therefore, it is possible to greatly improve the work efficiency as compared with the case where an annotation character string is inserted by keyboard input.

When the caption production system includes only one correction terminal 5, in the first interface providing method, the recognition result correction unit 16 of the speech recognition device 1 transmits the indication information earliest among the above-described processes. The operation with the correction terminal 5 other than the correction terminal 5 is not executed.
Further, the recognition result correction unit 16 may output the acoustic event recognition result data D5 to the correction terminal 5 for display at the timing when the acoustic event recognition result is changed. Thereby, it is possible to prevent the same annotation character string from being continuously displayed in the acoustic event recognition result display window 83.

  According to the present embodiment, the speech recognition apparatus 1 performs acoustic event recognition in parallel with conventional speech recognition and displays the recognition results on the correction terminal 5. Specify the annotation insertion position and the annotation to be inserted (text representation of the acoustic event) from the display. Therefore, it is possible to greatly reduce the burden of manual production of annotated captions. In addition, since the speech recognition apparatus 1 can obtain text representations of various types of acoustic events as recognition results, a richer subtitle can be obtained by inserting the text representation of the obtained acoustic events into the subtitles as annotations. Expression is possible.

  The voice recognition device 1 described above has a computer system inside. The operation process of the speech recognition apparatus 1 is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 ... Voice recognition apparatus, 5 ... Correction terminal, 10 ... Memory | storage part, 11 ... Voice branching part, 12 ... Voice area detection part, 13 ... Voice recognition part, 14 ... Acoustic event area detection part, 15 ... Acoustic event recognition part, 16 ... Recognition result correction unit 51 ... Control unit 52 ... Display unit 53 ... Input unit 54 ... Audio output unit

Claims (4)

A speech recognition unit that recognizes speech data and outputs character string data indicating the utterance content of the speech recognition result;
An acoustic event recognition unit that calculates a posteriori probability of an acoustic event based on an acoustic feature obtained from the voice data and outputs character string data representing the acoustic event detected based on the calculated posteriori probability; ,
The character string data of the utterance content output by the voice recognition unit and the character string data representing the acoustic event output by the acoustic event recognition unit are displayed on the correction terminal, and specified from the display. An annotation insertion instruction indicating the annotation insertion position in the character string of the utterance content and the character string representing the acoustic event selected from the displayed contents is received from the correction terminal, and the utterance is received according to the received annotation insertion instruction. A recognition result correction unit that generates annotated caption data in which character string data representing the acoustic event is inserted into character string data indicating content;
Equipped with a,
The acoustic event recognizing unit calculates the posterior probability of the acoustic event by inputting the acoustic feature amounts of the frames in time order obtained by dividing the audio data and inputting the acoustic feature amounts into a convolutional neural network,
The convolutional neural network has an input layer, a hidden layer, a pooling layer, and an output layer,
The input layer has an acoustic feature amount of each of the frames arranged in time order as an input,
Each unit of the hidden layer is combined with a predetermined number of frames of the input layer while maintaining a shift by a predetermined number of frames, and the result of convolution calculation of the acoustic feature amount of the frame of the input layer that is combined is obtained. Show
Each unit of the pooling layer is coupled to a number of the hidden layer units corresponding to the number of units of the pooling layer, and the maximum value of the coupled hidden layer units is propagated,
Each unit of the output layer corresponds to a different type of acoustic event, and is coupled to all the units of the pooling layer by respective weights for calculating the posterior probability of the corresponding acoustic event.
A speech recognition apparatus characterized by that. An audio event section detection unit that divides the audio data into frames and compares the acoustic feature amount of each frame with the acoustic feature amount of each of silence, acoustic event, and speech language to detect a section including the acoustic event; Prepared,
The acoustic event recognizing unit calculates a posterior probability of the acoustic event based on the acoustic feature amount obtained from the audio data of the section detected by the acoustic event section detecting unit, and based on the calculated posterior probability. Output character string data representing the detected acoustic event,
The speech recognition apparatus according to claim 1. The acoustic feature amount is a feature amount in a time-frequency domain.
The speech recognition apparatus according to claim 1 or claim 2 , wherein Computer
Voice recognition means for voice recognition of voice data and outputting character string data indicating the utterance content of the voice recognition result;
An acoustic event recognition means for calculating a posteriori probability of an acoustic event based on an acoustic feature obtained from the voice data, and outputting character string data representing the acoustic event detected based on the calculated posteriori probability; ,
The character string data of the utterance content output by the voice recognition unit and the character string data representing the acoustic event output by the acoustic event recognition unit are displayed on the correction terminal and designated from among the displayed items. An annotation insertion instruction indicating the annotation insertion position in the character string of the utterance content and the character string representing the acoustic event selected from the displayed contents is received from the correction terminal, and the utterance is received according to the received annotation insertion instruction. Recognition result correcting means for generating annotated subtitle data in which character string data representing the acoustic event is inserted into character string data indicating contents;
Equipped with,
The acoustic event recognizing means calculates the posterior probability of the acoustic event by inputting the acoustic feature amounts of the frames in time order obtained by dividing the audio data and inputting the acoustic feature amounts into a convolutional neural network,
The convolutional neural network has an input layer, a hidden layer, a pooling layer, and an output layer,
The input layer has an acoustic feature amount of each of the frames arranged in time order as an input,
Each unit of the hidden layer is combined with a predetermined number of frames of the input layer while maintaining a shift by a predetermined number of frames, and the result of convolution calculation of the acoustic feature amount of the frame of the input layer that is combined is obtained. Show
Each unit of the pooling layer is coupled to a number of the hidden layer units corresponding to the number of units of the pooling layer, and the maximum value of the coupled hidden layer units is propagated,
Each unit of the output layer corresponds to a different type of acoustic event, and is coupled to all the units of the pooling layer by respective weights for calculating the posterior probability of the corresponding acoustic event.
A program for functioning as a voice recognition device.

Images