JP2015212732A - Sound metaphor recognition device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a text expression representing a sound metaphor of a non-voice included in audio data and display properties of the text expression.SOLUTION: A non-voice section detection part 24 detects a non-voice section of audio data. An acoustic feature quantity extraction part 25 extracts an acoustic feature quantity from audio data in the non-voice section. A language feature quantity extraction part 26 specifies a language feature quantity extraction section including parts before and after the non-voice section, and extracts a language feature quantity from data on utterance contents corresponding to audio data in the specified language feature quantity extraction section. A rhythm feature quantity extraction part 27 specifies a rhythm feature quantity extraction section including the parts before and after the non-voice section, and extracts a rhythm feature quantity from audio data in the specified rhythm feature quantity extraction section. A sound metaphor recognition part 28 calculates a posterior probability of a sound metaphor comprising a text expression of a non-voice and display properties of the text expression from the extracted acoustic feature quantity, language feature quantity, and rhythm feature quantity by using a statistically learnt sound metaphor recognition model, and outputs a sound metaphor selected based upon the posterior probability.

Description

  The present invention relates to a phonetic recognition apparatus and a program.

  The sound of many broadcast programs is not composed only of a speech language, but non-linguistic sound (for example, laughter), applause, background music, and the like are added in response to a request for directing the program. Such non-speech is considered to play an important role in information transmission, as in the case of spoken language, such as notifying the change of scene. Therefore, non-speech can be said to be one of the elements indispensable for viewers to understand a program.

  On the other hand, there is a technique for recognizing environmental sounds out of non-speech and outputting them as onomatopoeia (see, for example, Patent Document 1). In this technique, environmental sounds are simply converted into onomatopoeia, which is one form of text representation, and presented to the disabled. In addition, a technique for automatically generating a hypothesis of onomatopoeia that is close to the user's impression evaluation from the phonological features of onomatopoeia (onomatopoeia / mimetic words) (see, for example, Patent Document 2), and onomatopoeia selected by the user for an image There is a technique for adding an environmental sound linked to the onomatopoeia (see, for example, Patent Document 3). In addition, there is a technique for generating an onomatopoeia from handwriting (for example, see Patent Document 4) and a technique for adding a dynamic attribute of a metaphor from a line drawing in a digital comic (for example, see Non-Patent Document 1). Furthermore, it recognizes the onomatopoeia uttered by the user and realizes an interface for changing line drawing attributes (such as line type) in computer graphics and performing game operations based on the speech recognition result. There is a technique (see, for example, Non-Patent Document 2).

JP 2007-334149 A JP2013-33351A JP-A-11-261947 JP 2012-18544 A

Ayaka Terashima, 2 others, “Applying Dynamic Phrase Focusing on Effect Line Drawing”, 2012 Annual Conference of the Japanese Society for Artificial Intelligence (26th), 2012, lM2-OS-8b-3 , P. 1-4 Keisuke Kanbara and 1 other, “Multimodal Interaction Using Onomatopoeia”, 2011 Annual Conference of the Japanese Society for Artificial Intelligence (25th), 2011, 1C2-OS4b-12, p. 1-2

  When considering a metaphor from the viewpoint of information compensation for hearing impaired and elderly people, what kind of text representation should be selected from non-speech, and the graphic representation that can sufficiently compensate the selected text representation As a result, how to arrange the video content is a big problem. In general, the work of expressing non-speech as a metaphor and placing it in a video is performed manually, and thus the production cost tends to be high. By reducing this manual work, production costs can be greatly reduced.

  However, the technique of Patent Document 1 is intended for broadcast audio, and does not estimate display attributes for displaying not only non-speech text representation but also the text representation graphically. In other words, the use of the text representation obtained by recognizing the environmental sound as the representation in the video is out of scope. Patent Documents 2 to 4 and Non-Patent Document 1 do not generate onomatopoeia from environmental sounds. Non-Patent Document 2 is an interactive application example based on speech recognition of a user's utterance, and does not recognize non-speech in an actual environment.

  The present invention has been made in consideration of such circumstances, and a phonetic recognition apparatus and program capable of obtaining a text expression representing a non-speech metaphor included in audio data and a display attribute of the text expression. I will provide a.

One aspect of the present invention is to collate a statistical acoustic model for detecting a non-speech segment with speech data and detect a non-speech segment in the speech data, and to detect the non-speech segment An acoustic feature quantity extraction unit that extracts an acoustic feature quantity from the voice data in the non-speech section detected by the unit, and the non-speech section detected by the non-speech section detection unit, and more than the non-speech section. A language feature quantity extraction unit that identifies a language feature quantity extraction section that is a predetermined long section, extracts a language feature quantity from speech content data corresponding to the speech data of the identified language feature quantity extraction section, and the non-speech A prosodic feature quantity extraction section that includes the non-speech section detected by the section detection unit and that is longer than the non-speech section by a predetermined length is specified, and the speech data of the specified prosodic feature quantity extraction section is specified. A prosody feature quantity extraction unit that extracts prosody feature quantities from the input, and obtains a metaphor comprising non-speech text expressions and display attributes of the text expressions by inputting acoustic feature quantities, language feature quantities, and prosodic feature quantities Using the statistically learned phonetic recognition model, the acoustic feature amount extracted by the acoustic feature amount extraction unit, the language feature amount extracted by the language feature amount extraction unit, and the prosodic feature amount extraction unit A metaphor recognition unit that calculates a posteriori probability of a metaphor from the extracted prosodic feature value and outputs data of a metaphor selected based on the calculated a posteriori probability; It is a recognition device.
According to this invention, the metaphor recognition device detects a non-speech segment in speech data, and extracts an acoustic feature amount from the speech data of the detected non-speech segment. Further, the phonetic recognition device identifies a language feature amount extraction section that includes a non-speech section and is longer than the non-speech section by a predetermined amount, and utterance content corresponding to the speech data of the identified language feature amount extraction section Extract language features from. Further, the phonetic recognition device identifies a prosodic feature amount extraction section that includes a non-speech section and is longer than the non-speech section by a predetermined length, and obtains a prosodic feature amount from the speech data of the identified prosodic feature quantity extraction section Extract. The phonetic recognition device uses a statistically learned phonetic recognition model to extract the non-speech text expression and the display attributes of the text expression from the extracted acoustic feature value, prosodic feature value, and language feature value. Calculate the posterior probability of the syllable. The posterior probability of this syllable is a conditional probability that represents the certainty of the syllable when a feature value is given. The phonetic recognition device outputs data of a phonetic selected based on the calculated posterior probability.
Thereby, the phonetic recognition apparatus can obtain an appropriate phonetic metaphor included in the audio data.

One aspect of the present invention is the above-described phonetic recognition apparatus, wherein the display attribute is one or more of a character font, a size, a color, and a display operation representing a text expression.
According to the present invention, the metaphor recognition device is a metaphor consisting of a non-speech text expression included in the audio data and one or more of the font, size, color, and display operation of the character representing the text expression. Output data.
Thereby, the phonetic recognition apparatus can obtain data of richly expressed phonetics.

One aspect of the present invention is the above-described phonetic recognition device, in which the phonetic recognition model receives an acoustic feature obtained from each of the time-ordered frames obtained by dividing the speech data of the non-speech section, and inputs A first neural network that outputs an acoustic feature amount obtained by converting the acoustic feature amount into a lower dimension than the input, and the prosodic feature amount of the prosodic feature amount extraction section as an input, and the input prosodic feature amount is input A second neural network that outputs prosodic feature values converted to a lower dimension, and a language feature value obtained by converting the input language feature value into a lower dimension than the input, with the language feature value of the language feature value extraction section as an input A third neural network that outputs a feature value, the acoustic feature value that is an output of the first neural network, and an output of the second neural network. A fourth neural network that outputs the posterior probabilities of text representations and display attributes of the metaphor with the prosodic feature value and the language feature value that is the output of the third neural network as inputs, and The acoustic feature quantity extraction unit extracts an acoustic feature quantity from each of the frames obtained by dividing the voice data of the non-speech section detected by the non-speech section detection unit, and the metaphor recognition unit performs each of the frames in time order. The acoustic feature value extracted by the acoustic feature value extraction unit from the first neural network, the prosody feature value extracted by the prosody feature value extraction unit to the second neural network, and the language feature value extraction unit The extracted linguistic feature quantity is input to the third neural network, and the text of the metaphor which is the output of the fourth neural network is input. Calculating a preparative representation and display attribute each posterior probability, wherein the.
According to this invention, the metaphor recognition device uses the acoustic feature amount of each frame in time order obtained by dividing the speech data of the non-speech section as the input of the first neural network, and uses the acoustic feature amount expressed in a low dimension. calculate. Also, the phonetic recognition device uses the prosodic feature value in the prosodic feature value extraction section as an input to the second neural network, and calculates the prosodic feature value represented in a low dimension. Furthermore, the phonetic recognition device calculates a language feature amount expressed in a low dimension using the language feature amount in the language feature amount extraction section as an input to the third neural network. The phonetic recognition device includes a low-dimensional acoustic feature calculated by the first neural network, a low-dimensional prosodic feature calculated by the second neural network, and a low-dimensional calculated by the third neural network. The dimensional linguistic feature quantity is used as the input of the fourth neural network, and the posterior probabilities of the text representation of the phonetic and the display attributes are calculated. The phonetic recognition device outputs data of a phonetic selected based on the calculated posterior probability.
Thereby, the metaphor recognition apparatus uses the acoustic feature amount of each frame obtained by dividing the speech data of the non-speech interval, the prosodic feature amount of the prosodic feature amount extraction interval, and the linguistic feature amount of the language feature amount extraction interval. The posterior probability of the metaphor can be obtained accurately.

One aspect of the present invention is the above-described phonetic recognition apparatus, which is a result editing unit that generates video data for displaying the phonetic selected by the phonetic recognition unit on video data corresponding to the audio data. Is further provided.
According to the present invention, the syllable recognition device generates video data that displays a non-speech syllable included in the audio data superimposed on the video data corresponding to the audio data.
Thereby, the metaphor recognition device can generate a program video to which the metaphor is added.

One aspect of the present invention is the above-described phonetic recognition apparatus, wherein the result editing unit receives a correction instruction for the phonetic selected by the phonetic recognition unit and corrects the sound based on the correction instruction. Generating video data to display a metaphor superimposed on video data corresponding to the audio data, the acoustic feature amount being the input of the metaphor recognition model when the metaphor is obtained, the prosodic feature amount, and It further comprises a phonetic recognition model learning unit that updates the phonetic recognition model based on the language feature and the corrected phonetic.
According to this invention, the metaphor recognition device corrects the recognized metaphor according to the correction instruction, generates video data that displays the corrected metaphor superimposed on the video data corresponding to the audio data, and the correction Update the phonetic recognition model based on the results.
Thereby, the phonetic recognition device can learn a phonetic recognition model that matches the tendency and preference of the user's performance.

One aspect of the present invention is the above-described phonetic recognition device, wherein a speech section detection unit that detects a speech section in the speech data by collating with a statistical acoustic model for speech section detection, and the speech section detection unit A speech recognition unit that recognizes the speech data in the speech section detected by the speech recognition unit and outputs data of speech content obtained as a result of speech recognition, and the language feature amount extraction unit includes: A linguistic feature amount in the linguistic feature amount extraction section is extracted from the output utterance content data.
According to this invention, the metaphor recognition device extracts the language feature amount in the language feature amount extraction section from the speech recognition result of the speech data.
Thereby, the syllable recognition device can obtain the language feature amount necessary for the recognition of the syllable even when the utterance content data is not added to the voice data.

  In one aspect of the present invention, the computer collates a statistical acoustic model for detecting a non-speech segment with speech data, and detects a non-speech segment detection unit that detects a non-speech segment in the speech data. An acoustic feature quantity extracting means for extracting an acoustic feature quantity from the speech data in the non-speech section detected by the speech section detection means; and the non-speech section including the non-speech section detected by the non-speech section detection means. A linguistic feature quantity extraction unit that identifies a linguistic feature quantity extraction section of a section longer than the section, and extracts a linguistic feature quantity from speech content data corresponding to the speech data of the identified linguistic feature quantity extraction section; The prosodic feature amount extraction section that includes the non-speech section detected by the non-speech section detection unit and that is longer than the non-speech section by a predetermined amount is specified, and the specified prosody A prosody feature quantity extraction means for extracting prosody feature quantity from the speech data in the collection amount extraction section; and a non-speech text expression with the input of acoustic feature quantity, language feature quantity, and prosodic feature quantity, and display attributes of the text expression, Using a statistically learned phonetic recognition model for obtaining a phonetic, comprising: the acoustic feature extracted by the acoustic feature extractor; the language feature extracted by the language feature extractor; And a metaphor recognition means for calculating a posteriori probability of the metaphor from the prosodic feature quantity extracted by the prosodic feature quantity extraction means, and outputting data of the metaphor selected based on the calculated a posteriori probability; It is a program for functioning as a phonetic recognition device.

  According to the present invention, it is possible to obtain a text expression representing a non-speech metaphor included in sound data and a display attribute of the text expression.

It is a figure which shows the outline | summary of the metaphor recognition process in the metaphor recognition apparatus by one Embodiment of this invention. It is a functional block diagram which shows the structure of the phonetic recognition apparatus by the embodiment. It is a figure which shows the metaphor recognition process flow of the metaphor recognition process part by the embodiment. It is a figure which shows HMM (Hidden Markov Model, hidden Markov model) for the non-voice area detection by the embodiment. It is a figure which shows the non-voice area detection process flow of the non-voice area detection part by the embodiment. It is a figure which shows the feature-value extraction area by the embodiment. It is a figure which shows the phonetic recognition model by the embodiment. It is a figure which shows the edit screen by the embodiment. It is a figure which shows the phonetic table contained in the learning data by the embodiment. It is a figure which shows the caption table contained in the learning data by the embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The metaphor recognition apparatus according to the present embodiment recognizes non-speech (sound other than speech and speech) included in the sound data, outputs text representing the non-speech, and a display attribute associated with the text, and provides a graphic representation As a metaphor.

Currently, a technology that uses speech recognition to produce subtitles for broadcast programs has been put into practical use. Since the subtitles of broadcast programs are mainly intended to compensate information for hearing-impaired persons and the elderly, the target of speech recognition is only the speech languages that can be converted into texts that constitute the broadcast programs. However, many broadcast programs are not composed solely of speech languages. For example, non-speech sounds such as non-linguistic sounds (for example, laughter), applause, and background music are added to the broadcast program sound in response to a program production request. Non-speech, which is a sound other than the spoken language, is considered to play an important role in information transmission like the spoken language, such as notifying the change of scene. For this reason, non-speech can be said to be one of the elements indispensable for viewers to understand programs.
Broadcast subtitles can be understood as a visual representation method using text in audio language, but only the audio language is subject to subtitles in subtitle broadcasting in the current broadcast, and non-speech text representation is not often performed. It is a fact.

  On the other hand, there is a technique called metaphor as a visual expression of text in comics. This phonetic metaphor is one of the elements that make up comics, and is defined as a linguistic expression with a visual symbol that expresses the sound symbol such as an onomatopoeia or mimetic word or the emotion of the characters. If conventional terms are used, a metaphor is a kind of typography, and it is characterized in that it includes attributes such as character font (including handwriting), color scheme, and arrangement in a cartoon expression. In addition, the story of a comic is usually progressed by the lines of characters appearing in a speech balloon, but these phonetics are also important information for readers to understand the story and to feel a sense of reality. Yes.

  The metaphor in comics is a technique of expressing still images as frames by arranging them on a medium such as paper or a computer display. On the other hand, even in broadcast programs, expressions similar to these metaphors can be seen prominently as a method of production, especially in variety programs of private broadcast stations. However, in the video of a broadcast program, in addition to text arrangement and color scheme, which are attributes of phonograms in comics, attributes such as time change (motion such as change in size) are given. Therefore, the metaphor in a broadcast program is defined as a dynamic metaphor (usually called character animation or motion typography), not a static metaphor performed in comics. Similar to the phonogram in comics, the phonogram in broadcast programs has the potential to be a video expression that leaves a strong impression on the viewer. Therefore, if non-speech can be visually expressed by phonetic metaphor, for example, industrial application can be greatly expected as a production method for program production that deepens the understanding of hearing impaired people.

Whether it is a manga or a broadcast program, the metaphor is placed on a still image or video depending on the intention of the author or director. If you are an experienced director, you can select a suitable text expression and attributes from non-speech and then arrange the metaphor, but generally convert non-speech to a linguistic expression suitable for the production. That is difficult. This is because the image recalled from non-speech depends on the subjectivity of the individual, and the production means for accurately conveying the message to the viewer cannot be determined uniquely.
Also, when considering metaphors from the viewpoint of information compensation for hearing impaired and elderly people, what text representation should be selected from non-speech and how it should be placed in video content as a graphic that can fully compensate the information It becomes a big problem.

For this problem, if non-speech is automatically recognized and the color scheme and size of characters are properly estimated as a metaphor in video content, the production cost of visual sound presentation methods will be greatly increased. Can be reduced. Therefore, it is only necessary to develop a method for statistically estimating the non-speech text expression and its attributes under the condition that appropriate learning data is given.
A text animation generation method based on such a metaphor can be considered as one of program production methods based on so-called pattern recognition and machine learning, and can be a new method that greatly renews the conventional production method.
The phonetic recognition apparatus according to the present embodiment uses voice recognition, a non-speech acoustic feature in speech data based on a statistically estimated phonetic recognition model, and a linguistic feature near the non-speech. Recognizes non-speech and generates a metaphor containing the text representation and the display attributes of the text representation on the screen.

FIG. 1 is a diagram showing an outline of a phonetic recognition process in the phonetic recognition device of the present embodiment.
The metaphor recognition apparatus according to the present embodiment estimates information about a metaphor from information on sound (speech language) and non-speech (non-language sound) included in sound data of a broadcast program or the like. The phonetic recognition apparatus according to the present embodiment applies a statistical method to estimate a phonetic from speech data. The phonetic is composed of non-speech text and display attributes. The user modifies the phonetic phrase estimated by the phonetic recognition device, for example, using a video editor or the like according to the production intention. Thereby, the metaphor recognition device of the present embodiment realizes production of metaphor expression at low cost.

The metaphor of this embodiment is intended for presentation on a display screen of a broadcast program. Therefore, the phonetic metaphor is composed of a non-speech text representation and a display attribute.
The non-speech text expression is a word string for expressing non-speech. Non-speech text expressions are, for example, onomatopoeia, mimetic words, onomatopoeia, interjections such as “One One” and “Gachagacha”, and are composed of one word or a short phrase.
The display attribute is an attribute associated with the text expression when the non-speech text expression is displayed on the screen. The display attributes include character attributes and display operations. The character attributes are the font (typeface, character design), size, color, etc. of the character when displaying the non-speech text expression. The display operation (motion) is a change in size or movement when displaying the text expression. The movement includes not only the case where the display position moves, but also the case where the display position is not moved.

For details on the components of the phonogram, see, for example, “Mitsumi Mizuguchi, 1 other person,“ Trial of automatic synthesis of character animation ”, Information Processing Society of Japan Research Report 2005, 2005-HI-116 (15), p. 97-104 ".
Note that the above-described text expression and display attributes are merely examples, and any form that can be displayed on the screen can be a constituent element of a metaphor. For example, ASCII art (a kind of graphic expression using characters) used in bulletin boards and e-mails on the Internet and pictograms used in social network services can be constituent elements.

  FIG. 2 is a block diagram showing the configuration of the syllable recognition device 1 according to one embodiment of the present invention, in which only functional blocks related to the present embodiment are extracted and shown. The metaphor recognition device 1 is realized by a computer device. As shown in FIG. 1, the metaphor recognition device 1 includes a metaphor recognition processing unit 2, a display unit 3, an input unit 4, and a metaphor recognition model learning processing unit 5. The phonetic recognition processing unit 2 performs phonetic recognition using a phonetic recognition model. The phonetic recognition model is used to obtain posterior probabilities of metaphors (conditional probabilities representing the probabilities of metaphors when given features) using acoustic features, linguistic features, and prosodic features. It is a statistical model. The display unit 3 is a display that displays an image. The display unit 3 may display an image on a display of a computer terminal connected to the phonetic recognition apparatus 1 via a network. The input unit 4 includes a keyboard, a mouse, a sensor arranged on a touch panel, and the like, and receives an input operation by a user. Note that the input unit 4 may receive information on an input operation performed by a user at a computer terminal connected to the metaphor recognition device 1 via a network. The phonetic recognition model learning processing unit 5 learns a phonetic recognition model that the phonetic recognition processing unit 2 uses for phonetic recognition.

  The syllable recognition processing unit 2 obtains a linguistic feature amount from text data obtained as a result of speech recognition of a speech section included in the input speech data D1 as a feature amount used for the speech recognition, and also from the input speech data D1. Get the prosodic feature value. Therefore, the syllable recognition processing unit 2 recognizes the speech section of the input speech data D1 separately from the syllable recognition. Therefore, the syllable recognition processing unit 2 separates the input voice data D1 into two, which are used as input for speech recognition and input for syllable recognition, respectively.

  The phonetic recognition processing unit 2 includes an acoustic model storage unit 20, a language model storage unit 21, a speech segment detection unit 22, a speech recognition unit 23, a non-speech segment detection unit 24, an acoustic feature quantity extraction unit 25, and a language feature quantity extraction unit. 26, a prosody feature quantity extraction unit 27, a metaphor recognition unit 28, and a result editing unit 29.

  The acoustic model storage unit 20 stores a statistical acoustic model for detecting a speech segment, a statistical acoustic model for detecting a non-speech segment, and a statistical acoustic model for speech recognition. The language model storage unit 21 stores a statistical language model for speech recognition. The speech section detection unit 22 collates a statistical acoustic model for speech section detection stored in the acoustic model storage unit 20 with the input speech data D1 as preprocessing for speech recognition, and performs speech in the input speech data D1. Identify the interval. The speech segment is a segment of speech (speech language). The speech recognition unit 23 stores the input speech data D1 of the speech segment identified by the speech segment detection unit 22 in the statistical acoustic model and language model storage unit 21 for speech recognition stored in the acoustic model storage unit 20. Speech recognition using a statistical language model for speech recognition. The voice recognition unit 23 outputs voice recognition result data D2 in which the voice recognition result of the utterance content is set.

  The non-speech section detection unit 24 collates a statistical acoustic model for non-speech section detection stored in the acoustic model storage unit 20 with the input speech data D1 as preprocessing for metaphor recognition, A non-speech segment in D1 is identified. The acoustic feature quantity extraction unit 25 obtains an acoustic feature quantity from the input voice data D1 of the non-speech section identified by the non-speech section detection unit 24. The language feature quantity extraction unit 26 extracts the language feature quantity in the language feature quantity extraction section from the text data of the speech recognition result indicated by the speech recognition result data D2. The language feature amount extraction section includes a non-speech section detected by the non-speech section detection unit 24 and is a section longer than the non-speech section by a predetermined amount. The prosodic feature quantity extraction unit 27 extracts prosodic feature quantities from the input speech data D1 in the prosodic feature quantity extraction section. The prosodic feature amount extraction section includes a non-speech section detected by the non-speech section detection unit 24 and is a section longer than the non-speech section by a predetermined amount. In this embodiment, the prosodic feature quantity extraction section is the same section as the language feature quantity extraction section, but may be a section different from the language feature quantity extraction section.

  The metaphor recognition unit 28 uses the acoustic feature obtained by the acoustic feature extraction unit 25, the language feature extracted by the language feature extraction unit 26, and the prosodic feature obtained by the prosody feature extraction unit 27. Phrase recognition is performed using the input of a phonogram recognition model, which is a statistical model for phonogram recognition. The phonetic recognition unit 28 outputs the phonetic data D3 in which the phonetic recognition result is set to the result editing unit 29. The phonetic recognition result is a phonetic metaphor composed of non-speech text representation (character string) and its display attributes. The display attribute is one or more of the font, size, color, and display operation of the character representing the text expression.

  The result editing unit 29 generates a screen in which the metaphor indicated by the metaphor data D3 is superimposed on the display screen of the input video data D4 from which the input audio data D1 is extracted, and displays the screen on the display unit 3. The result editing unit 29 corrects the text expression and display attribute of the phonetic phrase indicated by the phonetic data D <b> 3 according to the correction instruction input by the input unit 4. The result editing unit 29 outputs completed video data D5 generated as a result of video editing so that the corrected metaphor is displayed on the display screen of the input video data D4. Further, the result editing unit 29 sets the corrected metaphor data D10 in which the result of correcting the metaphor indicated by the metaphor data D3 according to the correction instruction, and the acoustic feature when the metaphor indicated by the metaphor data D3 is obtained. The quantity, prosodic feature quantity, and linguistic feature quantity are output.

  The phonetic recognition model learning processing unit 5 learns a phonetic recognition model prior to the phonetic recognition in the phonetic recognition processing unit 2. The phonetic recognition model learning processing unit 5 includes a spoken language resource storage unit 50, a learning acoustic feature quantity extraction unit 51, a learning language feature quantity extraction unit 52, a learning prosodic feature quantity extraction unit 53, and a phonetic recognition model learning unit. 54, and a phonetic recognition model storage unit 55.

  The spoken language resource storage unit 50 stores learning data of a phonetic recognition model. The learning data is data in which the learning voice data, the metaphor, and the text data of the utterance content are associated with each other. Non-speech section data indicated by the start time and end time of non-speech in the speech data for learning is added to the metaphor. To the text data of the utterance content, voice section data indicated by the start time and end time of the utterance in the learning voice data is added.

  The learning acoustic feature quantity extraction unit 51 acquires non-speech data D6, which is learning speech data of a non-speech section corresponding to a phonetic phrase, from the learning data stored in the speech language resource storage unit 50. The learning acoustic feature quantity extraction unit 51 extracts an acoustic feature quantity from the acquired non-voice data D6. The learning language feature quantity extraction unit 52 specifies a language feature quantity extraction section corresponding to the non-speech section of the non-speech data D6, and the speech language resource storage section stores the text data D7 of the utterance content of the specified language feature quantity extraction section. 50 is acquired from the learning data stored in 50. The learning language feature extraction unit 52 extracts a language feature from the acquired text data D7. The learning prosody feature quantity extraction unit 53 identifies the prosodic feature quantity extraction section corresponding to the non-speech section of the non-speech data D6, and uses the speech data D8, which is the training speech data of the identified prosodic feature quantity extraction section, as a speech language. Obtained from the learning data stored in the resource storage unit 50. The prosodic feature quantity extraction unit 53 for learning extracts prosodic feature quantities from the acquired speech data D8.

  The phonetic recognition model learning unit 54 includes an acoustic feature extracted by the learning acoustic feature extraction unit 51, a language feature extracted by the learning language feature extraction unit 52, and a learning prosodic feature extraction unit 53. The extracted prosodic feature quantity is received. The phonetic recognition model learning unit 54 reads the phonetic data D9 corresponding to the non-speech section of the non-speech data D6 from the learning data stored in the spoken language resource storage unit 50. The phonetic recognition model learning unit 54 learns a phonetic recognition model by statistical means using the received acoustic feature value, language feature value, prosodic feature value, and the read phonetic data D9. The phonetic recognition model learning unit 54 stores the learned phonetic recognition model in the phonetic recognition model storage unit 55. The phonetic recognition unit 28 performs phonetic recognition using the phonetic recognition model stored in the phonetic recognition model storage unit 55.

  The phonetic recognition model learning unit 54 updates the phonetic recognition model based on the corrected phonetic. The spoken language resource storage unit 50 associates the corrected syllable data D10 output from the result editing unit 29 with the acoustic feature value, prosodic feature value, and language feature value when the uncorrected syllable is obtained. Remember. The phonetic recognition model learning unit 54 reads out these data from the spoken language resource storage unit 50 and updates the phonetic recognition model stored in the phonetic recognition model storage unit 55. Note that the phonetic recognition model learning unit 54 may update the phonetic recognition model for each category. In this case, the result editing unit 29 outputs the user profile data D11 of the user who made the correction in association with the corrected syllable data D10. The phonetic recognition model learning unit 54, for each category obtained from the user profile data D11, the corrected phonetic data D10 and the acoustic feature, prosodic feature, and language feature when the phonetic before correction is obtained. And update the phonetic recognition model. The phonetic recognition model learning unit 54 stores the category in association with the updated speech recognition model. The phonetic recognition unit 28 reads a phonetic recognition model corresponding to the category input by the input unit 4 from the phonetic recognition model storage unit 55 and performs phonetic recognition.

Next, the operation of the phonetic recognition apparatus 1 will be described.
First, the phonetic recognition process in the phonetic recognition device 1 will be described.
The metaphor recognition processing unit 2 of the metaphor recognition device 1 uses the acoustic feature quantity, the language feature quantity, and the prosodic feature quantity as input for the metaphor recognition process. Therefore, the phonetic recognition processing unit 2 performs an acoustic feature extraction process by the acoustic feature extraction unit 25, a language feature extraction process by the language feature extraction unit 26, and a prosodic feature extraction by the prosody feature extraction unit 27. Processes are executed cooperatively.

  For example, it is assumed that the speech segment detection unit 22 detects a total of N non-speech segments. First, the acoustic feature quantity extraction unit 25 extracts acoustic feature quantities for the nth (n = 1,..., N) non-speech section. Next, the language feature quantity extraction unit 26 extracts a language feature quantity from the language feature quantity extraction section specified based on the start time and end time of the nth non-voice section. The language feature amount is based on the frequency of the word string in the speech recognition result in the language feature amount extraction section. Similarly, the prosodic feature quantity extraction unit 27 extracts the prosodic feature quantity of the prosodic feature quantity extraction section specified by the nth non-voice section. Finally, the phonetic recognition unit 28 extracts the acoustic feature amount and the language feature amount extracted by the acoustic feature amount extraction unit 25, the language feature amount extraction unit 26, and the prosodic feature amount extraction unit 27 for the n-th non-speech interval. , And the prosodic feature value are integrated into an input feature value. The phonetic recognition unit 28 performs phonetic recognition for each of the N input feature values.

Hereinafter, the details of the phonetic recognition process in the phonetic recognition device 1 will be described.
FIG. 3 is a diagram showing a metaphor recognition process flow of the metaphor recognition device 1.
First, the phonetic recognition device 1 stores the statistical acoustic models for speech segment detection, non-speech segment detection, and speech recognition in the acoustic model storage unit 20, and the statistical language model for speech recognition as a language model. Stored in the storage unit 21. The phonetic recognition model storage unit 55 stores a phonetic recognition model learned by a phonetic recognition model learning process described later.
A statistical acoustic model for detecting a speech section, a statistical acoustic model for speech recognition, and a statistical language model can be the same as those in the past. In the present embodiment, HMM (Hidden Markov Model) and GMM (Gaussian Mixture Model) are used as statistical acoustic models for detecting non-speech intervals. The HMM and GMM for detecting non-speech intervals are learned by the same learning method as that of the prior art using speech data labeled with three classes of speech, non-speech, and silence as learning data. For example, in the case of a non-speech GMM, each of the mixed Gaussian distributions represents a different type of non-speech feature. The HMM for detecting the non-voice section will be described later with reference to FIG.

  The metaphor recognition processing unit 2 of the metaphor recognition device 1 performs the process shown in FIG. 3 every time the input voice data D1 is input. The metaphor recognition processing unit 2 branches the input voice data D1 into two in order to cut out a voice section and a non-voice section from the input voice data D1 in which both spoken voice and non-voice are mixed. The metaphor recognition processing unit 2 inputs one branched to the speech segment detection unit 22 and the other to the non-speech segment detection unit 24.

  The voice section detection unit 22 detects and cuts out a voice section that needs to be converted into text in the input voice data D1 by the conventional technique (step S105). This voice section may include an overlap with non-voice such as background sound. In the present embodiment, a voice section is detected by the techniques described in Japanese Patent Application Laid-Open No. 2007-233148 and Japanese Patent Application Laid-Open No. 2007-233149. The voice section detection unit 22 outputs voice section data obtained by cutting the detected voice section from the input voice data D1 to the voice recognition unit 23.

Specifically, every time the input voice data D1 is input, the voice section detection unit 22 divides the voice indicated by the input voice data D1 into input frames that are frames of one processing unit at a predetermined time interval. The voice section detection unit 22 calculates the acoustic feature amount of each of a predetermined number of input frames selected in order from the earliest time. The state transition network for detecting a voice section is a left-to-right type HMM that makes a transition between non-voice, voice, and silence without skipping from the start of utterance to the end of utterance. Note that a non-speech state may be used instead of the silent state. The speech section detection unit 22 reads out acoustic models of non-speech and speech from the acoustic model storage unit 20 and calculates acoustic scores (logarithmic likelihood) of each input frame using these read out acoustic models. The non-speech acoustic model represents an HMM such as silence or non-speech. Moreover, the acoustic model of speech is composed of phoneme HMMs of each phoneme. The voice section detection unit 22 stores a record of the state transition of each input frame, traces the state transition record from the current state toward the start state, and inputs the processing start (starting point) using the state transition network. Calculate the cumulative acoustic score of each state series from the frame. When the difference between the maximum accumulated acoustic score of each state series and the starting acoustic score is larger than the threshold, the speech section detection unit 22 is the last in the series in which the maximum accumulated acoustic score is obtained. A time that is a predetermined time later than the time when the voice was in the voice state is set as the speech start time.
The speech section detection unit 22 further calculates the accumulated acoustic score of each state series from the input frame at the start of processing to the current input frame for the input frame after the utterance start time is detected in the same manner as described above. The voice section detection unit 22 sets a threshold value as a difference between the maximum accumulated acoustic score in each state series and the maximum cumulative acoustic score in the state series from the voice to the silent end of each state series. Judge whether it exceeded. When a predetermined time has elapsed after the threshold value is exceeded, the voice section detection unit 22 sets a time that is a predetermined time later than the elapsed time as the utterance end time.
The voice section detection unit 22 outputs voice section data in which input frames of a section from the utterance start time to the utterance end time are collected to the voice recognition unit 23. Further, the speech segment detection unit 22 outputs information indicating the identified speech segment to the prosodic feature value extraction unit 27.

  The speech recognition unit 23 uses the speech segment data output from the speech segment detection unit 22 as a statistical acoustic model for speech recognition stored in the acoustic model storage unit 20 and a statistical model stored in the language model storage unit 21. Using a language model, speech recognition is performed using conventional technology (step S110). In the present embodiment, the speech recognition unit 23 uses HMM and GMM for the statistical acoustic model. In the present embodiment, the speech recognition unit 23 obtains a recognition result by multipath speech recognition using a word n-gram language model as a statistical language model. This recognition result is a segmentation in units of words, and the speech recognition unit 23 gives each word time information when the word is uttered. The voice recognition unit 23 outputs voice recognition result data D2 in which a voice recognition result is set.

  On the other hand, the non-speech section detection unit 24 detects and cuts out a non-speech section composed of non-speech including background sound or the like in the input sound data D1 (step S115). In the present embodiment, the non-speech section detection unit 24 detects a non-speech section including an overlap with a portion to be converted into text by speech recognition. The non-speech section detection unit 24 detects a non-speech section using the GMM and HMM for non-speech section detection stored in the acoustic model storage unit 20 by the same algorithm as the speech section detection unit 22. However, the difference is that the speech section detection unit 22 uses the speech section as a detection target, whereas the non-speech section detection unit 24 uses the non-speech speech section as the detection target. In addition, an HMM for non-voice interval detection is used instead of the state transition network for voice interval detection.

  FIG. 4 is a diagram illustrating a non-voice segment detection HMM stored in the acoustic model storage unit 20. In the present embodiment, the configuration of the HMM is a so-called ergodic HMM. As shown in the figure, this ergodic HMM is an HMM that represents three classes of transitions: voice, non-voice, and silence. Each transition is given a transition probability obtained by learning.

  FIG. 5 is a diagram showing a non-speech section detection processing flow of the non-speech section detection unit 24, and shows detailed processing in step S115 of FIG. First, every time the input voice data D1 is input, the non-voice section detection unit 24 divides the input voice data D1 into input frames that are frames of one processing unit at a predetermined time interval. The divided input frame is a unit for processing the acoustic feature amount, and is usually set to a unit of 10 milliseconds or a length close to 10 milliseconds.

  The non-speech section detection unit 24 acquires a predetermined number of input frames in order from the earliest time among the input frames not yet processed (step S205). The non-speech section detection unit 24 calculates the acoustic feature amount of each acquired input frame. The non-speech section detection unit 24 reads from the acoustic model storage unit 20 each voice, non-speech, and silent GMM that is each state of the HMM. The non-speech section detection unit 24 compares the read GMM and the acoustic feature quantity of each input frame, calculates the acoustic score of each input frame, and makes a transition between HMM states if necessary (step) S210). When the predetermined number of input frames necessary for the traceback are not processed (step S215: NO), the non-speech section detection unit 24 returns to step S205 to acquire a new input frame, and calculates the acoustic score. I do.

  When the predetermined number of input frames necessary for the traceback are processed (step S215: YES), the non-voice section detection unit 24 obtains a list of state series up to the current state by traceback (step S220). ). That is, the non-speech section detection unit 24 traces the record of the state transition from the current state toward the start state, and uses the ergodic HMM shown in FIG. The cumulative acoustic score of each state sequence up to the state is calculated. At this time, the non-speech section detection unit 24 sorts the series in descending order of the accumulated acoustic score.

  The non-speech section detection unit 24 compares the first rank sequence and the second rank sequence from the HMM state sequence obtained by the traceback (step S225). The non-speech section detection unit 24 determines that the section is not fixed when the accumulated acoustic score difference is equal to or less than a predetermined threshold (step S230: NO), and returns to step S205 to perform a new input frame. Calculate the acoustic score. If the non-speech section detection unit 24 determines that the difference between the accumulated acoustic scores exceeds a predetermined threshold (step S230: YES), the non-speech section detection unit 24 sets the first series as a confirmed section. The non-speech section detection unit 24 outputs a frame sequence in which the frames of the non-speech determined section are collected in order of time as non-speech section frame data (step S235). Each frame sequence is given information on one or both of the start time and end time of the frame.

  In FIG. 3, the metaphor recognition processing unit 2 performs the following processing from step S125 to step S140 for each of the N non-speech intervals detected by the non-speech interval detection unit 24. The metaphor recognition processing unit 2 sets n = 1 as an initial value (step S120).

  The acoustic feature amount extraction unit 25 extracts the acoustic feature amount of each frame included in the nth non-speech segment frame data output from the non-speech segment detection unit 24 (step S125). In the present embodiment, a log mel filter bank output generally used in speech recognition is used as an acoustic feature amount. However, the acoustic feature quantity extraction unit 25 performs normalization of the mean zero and the variance 1 in advance on the log mel filter bank output. In speech recognition, usually, speech data is subjected to discrete Fourier transform, passed through a mel filter bank, logarithmically transformed, and a mel cepstrum coefficient (MFCC) is obtained as an acoustic feature value by discrete cosine transform. However, the acoustic feature amount extraction unit 25 does not perform discrete cosine transform because the sound frequency is directly set as the acoustic feature amount.

The linguistic feature amount extraction unit 26 extracts linguistic feature amounts necessary for the syllable recognition unit 28 from the speech recognition result indicated by the speech recognition result data D2 (step S130).
FIG. 6 is a diagram illustrating a feature amount extraction section. The language feature quantity extraction unit 26 starts the start point (start time) of the language feature quantity extraction section based on the start (start time) and end (end time) of the nth non-speech section identified by the non-speech section detection unit 24. ) And end point (end time). That is, the linguistic feature quantity extraction unit 26 starts from the time shifted by K time units (seconds) before the start of the non-speech segment, and shifts backward by K time units (seconds) from the end of the non-speech segment. The section whose end point is is a language feature amount extraction section.
This is based on the assumption that the linguistic context that strongly affects the estimation of the non-speech linguistic expression is limited to the non-speech interval and its surroundings. In this embodiment, this context is the word frequency distribution. It shall be obtained based on

  The language feature quantity extraction unit 26 extracts a language feature quantity from a word string included in the language feature quantity extraction section among the word strings of the speech recognition result indicated by the speech recognition result data D2. In this embodiment, the language feature amount is determined as a relative frequency vector. When the size of the vocabulary V for speech recognition is | V |, each word included in the vocabulary V is v (vεV), and the total number of words in the language feature extraction section is M, the language feature w is given by (1).

Language feature w
= [C (v ₁ ) / M, c (v ₂ ) / M,..., C (v _{| V |} ) / M] ^T (1)

In the formula (1), T is a symbol representing transposition, and v ₁ , v ₂ ,. Further, c (v) is a function that returns the frequency of the word v in the language feature amount extraction section, and satisfies Σ _vεV c (v) = M.

  If text data such as subtitles corresponding to the speech section is input to the input speech data D1, and the start time and end time are given in advance to the text data, the text feature is extracted from the language feature amount extraction unit. 26 may be input directly. The language feature quantity extraction unit 26 extracts a language feature quantity from a word string in the language feature quantity extraction section indicated by the input text data, instead of the speech recognition result data D2. Thereby, the process of step S105 and step S110 in FIG. 3 is omissible.

  Emotions included in the spoken voice are also information necessary for obtaining an appropriate phonetic metaphor. In the present embodiment, assuming that information related to emotion is included in the prosody of the spoken voice, the prosody is extracted from the speech data, and is used as a prosodic feature amount for syllable recognition. The prosodic feature quantity extraction section from which prosodic feature quantities are to be extracted is the same as the language feature quantity extraction section, as shown in FIG.

  In FIG. 3, the prosodic feature quantity extraction unit 27 specifies the start point and the end point of the prosodic feature quantity extraction section based on the start end and the end point indicated by the voice section information output by the voice section detection unit 22. When the prosodic feature amount extraction section is different from the language feature amount extraction section, the value of K in FIG. 6 may be changed. The prosody feature quantity extraction unit 27 uses the fundamental frequency of the voiced sound (mainly vowel) that is the prosodic information of the spoken voice from the input speech data D1 corresponding to the specified prosodic feature quantity extraction section (the lowest of the vocal cord frequencies). Point out) and extract power. The prosodic feature quantity extraction unit 27 performs logarithmic transformation on the extracted fundamental frequency and power, respectively, and obtains a delta fundamental frequency feature quantity vector and a delta power feature quantity vector, which are a series of time change quantities. These vectors are the values of the corresponding delta fundamental frequency feature quantity and delta power feature quantity for the voiced sound section, and are zero for the other sections. The prosodic feature quantity extraction unit 27 concatenates these vectors into a prosodic feature quantity (step S135).

  The phonetic recognition unit 28 generates an nth input feature amount (step S140). The acoustic feature quantity constituting the input feature quantity is an acoustic feature quantity extracted from each frame by the acoustic feature quantity extraction unit 25 for the nth non-voice section. The language feature quantity constituting the input feature quantity is a language feature quantity extracted from the language feature quantity extraction section corresponding to the nth non-speech section by the language feature quantity extraction unit 26. The prosodic feature value constituting the input feature value is a prosodic feature value extracted from the prosodic feature value extraction section corresponding to the nth non-voice section by the prosodic feature value extraction unit 27.

  The metaphor recognition processing unit 2 adds 1 to n when all the N non-speech segments detected by the non-speech segment detection unit 24 have been completed, and adds n to the nth The processing from step S125 is repeated for the frame. When the processes from step S125 to step S140 are completed for all N non-speech sections detected by the non-speech section detection unit 24, the metaphor recognition processing unit 2 performs the process from step S150 (step S145).

  The metaphor recognition unit 28 performs the following metaphor recognition for each of the N input feature values using the metaphor recognition model stored in the metaphor recognition model storage unit 55 (step S150). In conventional speech recognition, text is estimated from speech, but in metaphor recognition, a set of non-speech text representations and display attributes constituting a metaphor is estimated. The non-speech text expression is a word string for expressing non-speech. The display attribute is one or more of fonts (fonts) such as typeface and character design, size, color, display operation, and the like. In this embodiment, a neural network is used as a statistical phonetic recognition model for recognizing text representations and display attributes that make up a phonetic.

  FIG. 7 is a diagram showing a phonetic recognition model used in the present embodiment. As shown in the figure, the phonetic recognition model is a multi-layer neural network that receives acoustic feature values, language feature values, and prosodic feature values as inputs, and outputs posterior probabilities of text representations of metaphors and posterior probabilities of display attributes. It is. For convenience, the phonetic recognition model can be divided into a first neural network A1, a second neural network A2, a third neural network A3, and a fourth neural network A4.

  The input layer of the phonetic recognition model includes an input layer of the first neural network A1, an input layer of the second neural network A2, and an input layer of the third neural network A3. The input layer of the first neural network A1 receives a variable length acoustic feature amount as an input. The first neural network A1 is a convolutional neural network, and converts it into a fixed-length feature quantity in a hidden layer in the middle of input / output. The output of the first neural network A1 is an acoustic feature value obtained by converting the inputted acoustic feature value into a lower dimension than the input. The input layer of the second neural network A2 receives a fixed-length prosodic feature value as an input. The output of the second neural network A2 is a prosodic feature value obtained by converting the input prosodic feature value into a lower dimension than the input. The input layer of the third neural network A3 receives a fixed-length language feature quantity as an input. The output of the third neural network A3 is a language feature value obtained by converting the input language feature value into a lower dimension than the input.

  The fourth neural network A4 receives the output of the first neural network A1, the output of the second neural network A2, and the output of the third neural network A3. The output layer of the fourth neural network A4 is an output layer of the phonetic recognition model, and outputs the posterior probabilities of each phonetic component. As described above, the phonetic component is a text expression and display attributes such as font, size, color, and display operation. That is, the output layer of the phonetic recognition model outputs the posterior probability of the text representation of the phonetic and the posterior probability of each display attribute. The text representation and the output layer of each display attribute have no connection, only a connection with the hidden layer below. In the figure, the output layer includes a unit group B1 representing the posterior probability of text expression, a unit group B2 representing the posterior probability of the first display attribute (display attribute 1), and the second display attribute (display). It consists of unit group B3 representing the posterior probability of attribute 2).

  Note that the number of layers of the multilayer neural network and the number of dimensions of each layer (including the number of types of display attributes of the metaphor) are included in the number of text representations of the metaphor to be recognized and the amount of learning data. These can be arbitrary, and are determined when learning the phonetic recognition model.

  The syllable recognition unit 28 inputs the acoustic feature quantity of the input feature quantity to the input layer of the first neural network A1 of the syllable recognition model. The phonetic recognition unit 28 inputs the prosodic feature quantity of the input feature quantity to the input layer of the second neural network A2. Furthermore, the phonetic recognition unit 28 inputs the language feature quantity of the input feature quantity to the input layer of the third neural network A3. The phonetic recognition unit 28 calculates the value of the output layer by using the phonetic recognition model using the input acoustic feature, prosodic feature, and language feature. The metaphor recognition unit 28 obtains an output vector having the values of the units of the output layer of the fourth neural network A4 as elements as the output of the metaphor recognition model.

  The phonetic recognition unit 28 selects an element for each phonetic component based on the posterior probability indicated by the value of each element of the output vector of the phonetic recognition model. Note that the phonetic recognition model storage unit 55 stores the values of the phonetic component corresponding to each element of the output vector of the phonetic recognition model. The metaphor recognition unit 28 selects, for each metaphor constituent element, an element having the maximum posterior probability from among the elements of the output vector, and sets a set of values of the metaphor constituent elements corresponding to the selected element as a metaphor. .

  For example, assume that unit group B1 corresponds to text representation, unit group B2 corresponds to font, and unit group B3 corresponds to color. In this case, the output vector of the phonetic recognition model includes elements corresponding to each unit of the unit group B1, elements corresponding to each unit of the unit group B2, and elements corresponding to each unit of the unit group B3. The phonetic recognition model storage unit 55 stores text expressions associated with each element corresponding to each unit of the unit group B1. Further, the phonetic recognition model storage unit 55 corresponds to the font types (Gothic, Mincho,...) Associated with each element corresponding to each unit of the unit group B2 and each unit of the unit group B3. The color (red, blue,...) Associated with each element is stored.

  The syllable recognition unit 28 selects an element having the highest posterior probability from the element group of the output vector corresponding to the unit group B1, and the lexical recognition model storage unit 55 stores the text expression “Zawazawa” associated with the selected element. Read from. Further, the phonetic recognition unit 28 selects an element having the highest posterior probability from the element group of the output vector corresponding to the unit group B2, and stores the font “Gothic” associated with the selected element in the phonetic recognition model. Read from unit 55. Further, the phonetic recognition unit 28 selects an element with the highest posterior probability from the element group of the output vector corresponding to the unit group B3, and displays the color “red” associated with the selected element as a phonetic recognition model storage unit. Read from 55. The syllable recognition unit 28 uses the text expression “Zawazawa”, the font “Gothic”, and the color “red”, which are a set of values of these read syllable components, as a phonogram. Thus, the phonetic recognition unit 28 selects an output vector element having the maximum posterior probability for each phonetic component, and sets a phonetic composed of a set of values of the phonetic component associated with the selected element. The phonetic data D3 is output. Each metaphor is given a start time and an end time of a non-speech segment.

  Note that the phonetic recognition unit 28 may set, in the phonetic data D3, values of a predetermined number of phonetic components from those having a high posterior probability for each phonetic component. For example, the phonetic recognition unit 28 selects a predetermined number of elements from the elements of the output vector corresponding to the unit group B1 with the highest posterior probability, the text expression associated with each selected element, the posterior The rank of the text expression based on the probability is set in the phonetic data D3. Similarly, the phonetic recognition unit 28 selects a predetermined number of elements from the elements of the output vector corresponding to the unit group B2 having the highest posterior probability, and fonts associated with the selected elements and their The rank is set in the phonetic data D3. Moreover, the phonetic recognition unit 28 selects a predetermined number of elements from the elements of the output vector corresponding to the unit group B3 having the highest posterior probability, and the colors associated with the selected elements and their ranks. Is set in the phonetic data D3.

  In FIG. 3, the result editing unit 29 arranges a graphic object whose parameter is the metaphor component indicated by the metaphor data D <b> 3 in the video of the input video data D <b> 4 attached to the input audio data D <b> 1 and displays the graphic object on the display unit 3. It is displayed (step S155). At this time, the result editing unit 29 arranges the graphic object of the phonogram during the non-speech period (start time and end time) added to the phonogram in accordance with the time information of the input video data D4. The user looks at the video displayed on the display unit 3 and performs video editing work for editing a metaphor component (graphic object).

  FIG. 8 is a diagram showing an editing screen displayed on the display unit 3. The result editing unit 29 superimposes the video of the input video data D4 and the graphic object 102 of the metaphor arranged in the video on the video display screen 101 of the editing screen shown in FIG. Further, the result editing unit 29 causes the recognition result display screen 103 to display a list of text representations and display attribute values that are the metaphor constituent elements indicated by the metaphor data D3 displayed on the video display screen 101. Further, the result editing unit 29 displays the timeline of the input video data D4 on the timeline screen 104. On the timeline screen 104, as with a normal commercial video editing application, each track of video and audio and the current playback position are displayed. Each track represents the presence of video data and audio data. In the present embodiment, as an additional track on the timeline screen 104, the non-speech section set in the phonogram data D3 and the text representation of the phonogram in the non-speech section are added as new tracks.

  The user (operator) arranges the recognized metaphor on the screen in accordance with the production intention while viewing the recognition result of the metaphor displayed on the display unit 3. When the text representation of the phonetic does not meet the production intention, the user corrects the text representation on the recognition result display screen 103. The result editing unit 29 replaces the text representation of the graphic object 102 of the currently displayed phonogram with the corrected text representation input by the input unit 4. Similarly, the display attribute of the phonetic can be changed. For example, the user corrects display attributes such as font, color, and size on the recognition result display screen 103. The result editing unit 29 replaces the display attribute of the graphic object 102 of the metaphor currently displayed with the corrected display attribute input by the input unit 4. Further, the result editing unit 29 does not arrange the graphic object on the screen when the user inputs the text expression as empty by the input unit 4.

At the time of correction, the user may arbitrarily input a text expression or a value of each display attribute by the input unit 4, and display the corrected display from the values of each display attribute prepared in advance by the result editing unit 29. An attribute value may be selected. Alternatively, when a plurality of text expressions and display attributes are set in order of posterior probabilities in the phonetic data D3, the result editing unit 29 displays them in a list on the display unit 3, and the user modifies the list from the list. The text representation and the value of each display attribute may be selected by the input unit 4.
The result editing unit 29 generates and outputs completed video data D5 for displaying the corrected metaphor on the display screen of the input video data D4. In addition, the result editing unit 29 outputs the corrected syllable data D10 indicating the corrected syllable, and the acoustic feature value, prosodic feature value, and language feature value when the corrected syllable data D10 is obtained. To do.
The above-described video editing work is realized as dedicated computer software or plug-in software for commercially available software used for production of broadcast programs.

Next, the phonetic recognition model learning process in the phonetic recognition device 1 will be described.
The phonetic recognition model used by the phonetic recognition unit 28 for phonetic recognition needs to be learned in advance by statistical means. Therefore, learning data used for learning the phonetic recognition model is stored in the spoken language resource storage unit 50. The learning data includes audio data, text associated with the audio data (for example, transcription and subtitles of audio), and information related to a metaphor.

FIG. 9 is a diagram illustrating a phonetic table included in learning data, and FIG. 10 is a diagram illustrating a text table included in learning data.
As shown in FIG. 9, the metaphor table included in the learning data includes an audio file, a text file, a start time and an end time of the metaphor, a text representation of the metaphor, and display attributes (fonts) of the metaphor. , Size, color, and the like). The start time and end time of the metaphor are indicated by the time from the beginning of the audio file. In the figure, a reference to the entity of the audio file is set as the information of the audio file, and a reference to the entity of the audio transcription or subtitle text file is set to the information of the text file. Phrase components are accumulated by collecting past production examples of human phonograms and creating a database.

  As shown in FIG. 10, the text table included in the learning data is information in which the reference name of the text file entity, the start time and end time of the utterance, and the utterance content are associated with each other. The start time and end time of the utterance are indicated by the time from the beginning of the audio file. The content of the utterance is a word string that is separated. The phonetic recognition model learning processing unit 5 extracts the prosodic feature amount from the voice file cut out based on the start time and the end time of the utterance content set in the text table.

  The metaphor recognition model learning processing unit 5 constructs the acoustic feature quantity of the non-speech section, the language feature quantity of the language feature quantity extraction section, and the prosodic feature quantity of the prosodic feature quantity extraction section from the learning data, and generates a metaphor recognition model. learn. Specifically, the phonetic recognition model learning processing unit 5 learns a phonetic recognition model as follows.

  The learning acoustic feature quantity extraction unit 51 reads the start time and the end time as non-speech intervals from the metaphor table of FIG. 9 stored in the spoken language resource storage unit 50. Further, the learning acoustic feature quantity extraction unit 51 reads the audio file of each non-speech section as non-speech data D6 from the speech language resource storage unit 50 with reference to the metaphor table. The learning acoustic feature quantity extraction unit 51 extracts the acoustic feature quantity from the non-speech data D6 by the same processing as the acoustic feature quantity extraction unit 25.

  The learning language feature quantity extraction unit 52 specifies language feature quantity extraction sections corresponding to the non-speech sections read by the learning acoustic feature quantity extraction unit 51 in the same manner as the language feature quantity extraction unit 26. The learning language feature quantity extraction unit 52 reads the text data D7 of the utterance content corresponding to the specified language feature quantity extraction section from the text table of FIG. 10 stored in the spoken language resource storage unit 50. The learning language feature quantity extraction unit 52 extracts a language feature quantity based on the relative frequency from the text data D7 in the same manner as the language feature quantity extraction unit 26.

  The prosodic feature quantity extraction unit 53 for learning specifies prosodic feature quantity extraction sections corresponding to the non-speech sections read by the learning acoustic feature quantity extraction unit 51 in the same manner as the prosodic feature quantity extraction unit 27. The learning prosodic feature quantity extraction unit 53 cuts out the voice data D8 corresponding to the specified language feature quantity extraction section from the voice file stored in the spoken language resource storage unit 50. The learning prosodic feature quantity extraction unit 53 extracts prosodic feature quantities from the speech data D8 in the same manner as the prosodic feature quantity extraction unit 27.

  The phonetic recognition model learning unit 54 learns a neural network serving as a phonetic recognition model. First, the phonetic recognition model learning unit 54 generates an input feature value for learning for each non-speech section extracted by the learning acoustic feature amount extraction unit 51. As the acoustic feature quantity constituting the input feature quantity, the acoustic feature quantity of the non-speech section extracted by the learning acoustic feature quantity extraction unit 51 is set. As the language feature quantity constituting the input feature quantity, the language feature quantity extracted by the learning language feature quantity extraction unit 52 from the language feature quantity extraction section corresponding to the non-speech section is set. The prosodic feature value extracted by the learning prosody feature value extracting unit 53 from the prosodic feature value extracting section corresponding to the non-speech section is set as the prosodic feature value constituting the input feature value. The phonetic recognition model learning unit 54 reads the sound read from the phonetic table stored in the spoken language resource storage unit 50 corresponding to the input feature quantity for learning and the non-speech section of the input feature quantity for learning. Learn event models based on metaphors (text representation and display attributes). In other words, the data for learning the phonetic recognition model has an acoustic feature quantity, a language feature quantity, and a prosodic feature quantity corresponding to the same non-speech section as a set of inputs, and a metaphor of the non-speech section as an output. At this time, the phonetic recognition model learning unit 54 uses each element of the vector (one-dimensional array) output from the output layer of the fourth neural network A4 in the phonetic recognition model and the phonetic composition obtained by the hash function. The hash value of the element is associated.

  When N sets of metaphor recognition model learning data (sets of input features and metaphors) are given, the metaphor recognition model learning unit 54 uses these sets of data one by one to generate a metaphor recognition model. The connection weight between each layer of the neural network is learned by the error back propagation method. The phonetic recognition model learning unit 54 performs learning by the repeated error back-propagation method on N sets of data for learning the phonetic recognition model, but the discrimination performance of the verification data prepared separately from the learning data is the maximum. It is judged that learning has converged at the point.

  The phonetic recognition model learning unit 54 performs learning using the same phonetic recognition model learning data for each of the neural networks having different numbers of layers and different numbers of units. The phonetic recognition model learning unit 54 stores the neural network having the highest discrimination performance of the verification data in the phonetic recognition model storage unit 55 as a phonetic recognition model. The phonetic recognition unit 28 performs phonetic recognition processing using the phonetic recognition model stored in the phonetic recognition model storage unit 55. And when a user changes with respect to the result of having performed phonetic recognition using this phonetic recognition model, feedback learning of a phonetic recognition model is performed.

  When the result editing unit 29 receives a change in the text representation or display attribute of the phonetic phrase from the input unit 4, the result editing unit 29 changes the text representation or display attribute of the phonetic phrase set in the phonetic data D3 according to the change. The result editing unit 29 sets the modified syllable data D10 in which the changed syllable is set, and the acoustic feature value, prosodic feature value, and linguistic feature when the uncorrected syllable shown by the syllable data D3 is obtained. The amount is stored in the spoken language resource storage unit 50. The phonetic recognition model learning unit 54 uses the corrected phonetic data D10, the acoustic feature value, the prosodic feature value, and the language feature value read from the spoken language resource storage unit 50 to learn a neural network by the error back propagation method. And the phonetic recognition model currently stored in the phonetic recognition model storage unit 55 is updated.

  When outputting the corrected syllable data D10, the result editing unit 29 may output the user profile data D11 of the user who edited the syllable in association with each other. The user profile data D11 includes, for example, a user ID, a group ID, a phonetic list created by the user, and the like. The user ID is a unique number for identifying the user who is the creator. The group ID is the number of a group that produces a similar broadcast program. The group ID may represent a program, for example, and may represent a program genre such as a documentary or a historical drama. The phonetic list created by the user is a production history that the user has done in the past. The production history holds the serial number of the phonetic data stored in the spoken language resource storage unit 50 and the name of the broadcast program that produced the phonetic data.

  When the corrected syllable data D10 is stored in the spoken language resource storage unit 50, user profile data D11 associated with this data is also stored. The phonetic recognition model learning unit 54 selects a set of the corrected phonetic data D10, the acoustic feature value, the prosodic feature value, and the language feature value for each category obtained from the user profile data D11, and feeds back the phonetic recognition model. Used for learning. The category is indicated by one or more of a user ID, a group ID, and a phonetic list. As a result, it becomes possible to learn a metaphor recognition model that matches the tendency and preferences of the user's performance (what kind of metaphor they liked and used), the genre of the program, and the like.

  According to the above-described embodiment, the phonetic recognition apparatus 1 can support the production of a phonetic by using a pattern recognition (machine learning) technique, and can provide a new program production method different from the conventional one at a low cost. Moreover, the syllable recognition device 1 can provide a service with added value to the conventional captions to people who need captions, such as persons with hearing disabilities and elderly people, by supporting the production of the metaphors.

  The above-described phonetic recognition apparatus 1 has a computer system inside. The operation process of the phonetic recognition apparatus 1 is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing the program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 ... Metaphor recognition apparatus, 2 ... Metaphor recognition process part, 3 ... Display part, 4 ... Input part, 5 ... Metaphor recognition model learning process part, 20 ... Acoustic model storage part, 21 ... Language model storage part, 22 ... Speech segment detection unit, 23... Speech recognition unit, 24 .. non-speech segment detection unit, 25... Acoustic feature quantity extraction unit, 26 .. language feature quantity extraction unit, 27 .. prosodic feature quantity extraction unit, 28. , 29 ... Result editing unit, 50 ... Spoken language resource storage unit, 51 ... Learning acoustic feature quantity extraction unit, 52 ... Learning language feature quantity extraction unit, 53 ... Learning prosodic feature quantity extraction unit, 54 ... Phrase recognition Model learning unit, 55 ... Phonetic recognition model storage unit

Claims (7)

A non-speech section detection unit that detects a non-speech section in the speech data by collating a statistical acoustic model for detecting a non-speech section with speech data;
An acoustic feature quantity extraction unit that extracts an acoustic feature quantity from the voice data in the non-speech section detected by the non-speech section detection unit;
The speech data of the specified language feature quantity extraction section is specified by identifying a language feature quantity extraction section that includes the non-speech section detected by the non-speech section detection section and is longer than the non-speech section by a predetermined amount. A linguistic feature quantity extraction unit that extracts linguistic feature quantities from utterance content data corresponding to
The speech data of the identified prosodic feature quantity extraction section is specified by identifying a prosodic feature quantity extraction section that includes the non-speech section detected by the non-speech section detection section and is longer than the non-speech section by a predetermined amount. A prosodic feature extraction unit for extracting prosody features from
Using a statistically learned phonetic recognition model to obtain a phonetic consisting of non-speech text representations and display attributes of the text representations with input of acoustic features, language features, and prosodic features, A posteriori probability of a metaphor is calculated from the acoustic feature amount extracted by the acoustic feature amount extraction unit, the language feature amount extracted by the language feature amount extraction unit, and the prosodic feature amount extracted by the prosodic feature amount extraction unit. A phonetic recognition unit that outputs data of a phonetic selected based on the calculated posterior probability;
A metaphor recognition device characterized by comprising: The display attribute is at least one of a font, size, color, and display operation of a character representing a text expression.
The phonetic recognition apparatus according to claim 1, wherein: The phonetic recognition model is:
First, an acoustic feature obtained from each of the frames in time order obtained by dividing the speech data of the non-speech section is input, and an acoustic feature obtained by converting the inputted acoustic feature into a lower dimension than the input is output. A neural network;
A second neural network that receives the prosodic feature value of the prosodic feature value extraction section and outputs the prosodic feature value obtained by converting the input prosodic feature value into a lower dimension than the input;
A third neural network that inputs a language feature amount of a language feature amount extraction section and outputs a language feature amount obtained by converting the input language feature amount into a lower dimension than the input;
The acoustic feature quantity that is the output of the first neural network, the prosodic feature quantity that is the output of the second neural network, and the language feature quantity that is the output of the third neural network are input. A fourth neural network that outputs the posterior probabilities of each of the textual representation of the phonetic and the display attributes;
The acoustic feature quantity extraction unit extracts an acoustic feature quantity from each of the frames obtained by dividing the voice data of the non-speech section detected by the non-speech section detection unit,
The phonetic recognition unit includes the acoustic feature amount extracted by the acoustic feature amount extraction unit from each of the frames in time order in the first neural network, and the prosodic feature amount extracted by the prosody feature amount extraction unit. The linguistic feature quantity extracted by the linguistic feature quantity extraction unit is inputted to the neural network of No. 2 to the third neural network, and the text expression of the metaphor and the posterior of the display attributes which are the output of the fourth neural network Calculate the probability,
The metaphor recognition apparatus according to claim 1, wherein the syllable recognition apparatus according to claim 1. A result editing unit for generating video data for displaying the metaphor selected by the metaphor recognition unit on video data corresponding to the audio data;
The phonetic recognition apparatus according to any one of claims 1 to 3, characterized in that: The result editing unit receives video correction instructions for the phonetic phrase selected by the phonetic recognition unit, and displays video data that is corrected based on the correction command and superimposed on video data corresponding to the audio data Produces
The phonetic recognition model based on the acoustic feature quantity, the prosodic feature quantity, the language feature quantity and the corrected phonetic that are input to the phonetic recognition model when the phonetic is obtained A phonetic recognition model learning unit for updating
The metaphor recognition apparatus according to claim 4, wherein: A speech section detecting unit for detecting a speech section in the speech data in comparison with a statistical acoustic model for speech section detection;
A voice recognition unit that recognizes the voice data in the voice section detected by the voice section detection unit and outputs data of speech content obtained as a result of the voice recognition;
The language feature amount extraction unit extracts a language feature amount in the language feature amount extraction section from the utterance content data output by the speech recognition unit.
The phonetic recognition apparatus according to any one of claims 1 to 5, wherein: Computer
A non-speech section detecting means for collating a statistical acoustic model for detecting a non-speech section with speech data and detecting a non-speech section in the speech data;
Acoustic feature quantity extraction means for extracting an acoustic feature quantity from the speech data in the non-speech section detected by the non-speech section detection means;
The speech data of the identified language feature quantity extraction section is specified by identifying a language feature quantity extraction section that includes the non-speech section detected by the non-speech section detection unit and is longer than the non-speech section by a predetermined amount. Language feature extraction means for extracting a language feature from utterance content data corresponding to
The speech data of the specified prosodic feature quantity extraction section is specified by identifying a prosodic feature quantity extraction section that includes the non-speech section detected by the non-speech section detection means and that is longer than the non-speech section by a predetermined amount. A prosodic feature extracting means for extracting prosody features from
Using a statistically learned phonetic recognition model to obtain a phonetic consisting of non-speech text representations and display attributes of the text representations with input of acoustic features, language features, and prosodic features, A posteriori probability of a metaphor is calculated from the acoustic feature extracted by the acoustic feature extractor, the language feature extracted by the language feature extractor, and the prosodic feature extracted by the prosodic feature extractor. A phonetic recognition means for outputting data of a phonetic selected based on the calculated posterior probability;
A program for functioning as a syllable recognition device.

Images