JP2019097016A - Corpus generation device, corpus generation method, and program - Google Patents

Abstract

To provide a device for generating a corpus for lip reading.SOLUTION: A corpus generation device 1 comprises: a moving picture acquisition unit 10 that acquires a moving picture with subtitles from a website; a lip moving picture generation unit 12 that recognize a speaker included in the moving picture and generates a lip moving picture of the speaker; a voice recognition unit 13 that recognizes a voice included in the moving picture and obtains morphemes composing the voice; a text alignment unit 14 that, on the basis the morphemes obtained by the voice recognition unit and text data of the subtitles in the moving picture, obtains a time at which a voice corresponding to the text is obtained, thereby generating the text with the time; a dividing unit 15 that divides the text with the lip moving picture, the voice and the time into data per predetermined unit time; and a corpus storage unit 16 that stores the text with the lip moving picture, the voice, and the time.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for generating a corpus for lip reading.

  Lip reading is a technology for reading the contents of speech from lip movements. Until around the first half of 2016, the recognition accuracy of lip reading was low, but recent advances in machine learning technology such as deep learning have suggested new lip reading technology and its accuracy has also improved . For example, Non-Patent Document 1 discloses a technique of taking an image as an input and estimating a uttered phrase.

Yannis M. Assael et al. "LIPNET: END-TO-END SENTENCE-LEVEL LIPREADING" [online] Cornel University Library, [October 27, 2017 search], Internet (URL: https://arxiv.org/pdf/ 1611.01599. Pdf)

  The technique described in Non-Patent Document 1 is an estimation method using a neural network model, and it is necessary to perform learning of a neural network in advance using teacher data. Although learning of a neural network requires a huge amount of teacher data, it is not easy to create teacher data.

  In view of the above background, the present invention aims to provide an apparatus for generating a corpus for lip reading.

  The corpus generation device according to the present invention comprises: a moving image acquisition unit that acquires a subtitled moving image from an external resource; a lip moving image generation unit that recognizes a speaker included in the moving image and generates a speaker's lip moving image; Corresponds to text based on a voice recognition unit for performing voice recognition of voice contained in a moving image to obtain a morpheme forming the voice, a morpheme obtained by the voice recognition unit, and text data of subtitles in the moving image A text alignment unit for generating a time-added text by obtaining a time when a sound to be emitted is generated; and a storage unit for storing the lip moving image and the time-added text.

  With this configuration, a corpus is generated based on a moving image acquired from an external resource, so a large amount of data such as a website on the Internet can be used for corpus generation. For example, although a database such as GRID is limited in terms of words and grammar, a moving image acquired from an external resource is capable of generating a versatile corpus since free speech is made by various people. . The text with a time may have a time in units of sentences, or may have a time for each word or character.

  The corpus generation device according to the present invention may include a dividing unit that divides the lip moving image and the time-added text into data of a predetermined unit time. It is possible to perform appropriate learning using a corpus by dividing it into the same time unit as when performing lip reading inference.

  In the corpus generation device of the present invention, the text alignment unit may convert kanji contained in the subtitles into hiragana to generate a time-added text corresponding to the hiragana. According to this configuration, a corpus can be generated also from a moving image including kanji in subtitles.

  The corpus generation device according to the present invention includes an utterance sound estimation unit that estimates a sound uttered from a lip animation, and the lip animation generation unit is configured to estimate the utterance sound when a plurality of persons appear in the animation. However, a person who matches the text data of the subtitle may be recognized as a speaker in the moving image. With this configuration, it is possible to generate a corpus also from a video in which a plurality of people are talking.

  In the corpus generation device of the present invention, the storage unit may store the voice together with the lip moving image and the time-added text. With this configuration, it is possible to generate a corpus useful for generating a model that infers speech sounds from lip images and speech.

  A corpus generation apparatus according to another aspect of the present invention includes: a moving image acquisition unit that acquires a moving image from an external resource; a lip moving image generation unit that recognizes a speaker included in the moving image and generates a lip moving image of the speaker; A voice recognition unit for performing voice recognition of voice included in the moving image to obtain morphemes forming the voice, and text data representing the morpheme obtained by the voice recognition unit are generated, and sounds corresponding to each morpheme are emitted A text alignment unit for obtaining a timed text and generating a timed text, and a storage unit for storing the lip moving image and the timed text.

  The corpus generation method according to the present invention is a method in which the corpus generation device generates a corpus using subtitled moving images acquired from an external resource, wherein the corpus generation device generates a subtitled moving image from an external resource. Acquiring, the corpus generating device recognizes a speaker included in the moving image, and generating a lip moving image of the speaker, and the corpus generating device performs speech recognition of speech included in the moving image Determining a morpheme forming the voice, and the time when the sound corresponding to the text is emitted based on the morpheme determined by the voice recognition and the text data of the subtitle in the moving picture, the corpus generation device Determining a timed text, and the corpus generation device storing the lip moving image and the timed text in a storage unit. Tsu and a flop.

  The program according to the present invention is a program for generating a corpus using subtitled moving images acquired from an external resource, and the computer acquires the subtitle moving image from the external resource, and Recognizing a speaker included and generating a lip moving image of the speaker, performing voice recognition of a voice included in the moving image to obtain a morpheme constituting the voice, and a morpheme obtained by the voice recognition Obtaining a time when a sound corresponding to the text is emitted based on the text data of subtitles in the moving image, generating a time-added text, and storing the lip moving image and the time-added text in a storage unit And do.

  The present invention can generate a corpus based on a moving image acquired from an external resource such as a website on the Internet.

It is a figure showing composition of a corpus generation device of a 1st embodiment. It is a figure which shows operation | movement of the corpus production | generation apparatus of embodiment. It is a figure which shows an example of the model of the neural network for lip reading. It is a figure which shows the structure of the corpus generation apparatus of 2nd Embodiment. FIG. 11 is a diagram showing an operation of recognizing a speaker in the second embodiment.

  Hereinafter, a corpus generation device according to an embodiment of the present invention will be described with reference to the drawings. The corpus generation device according to the embodiment uses a moving image acquired from a website on the Internet as an external resource. The corpus generation device of the present embodiment generates a corpus from subtitled moving images acquired from a website. Here, the subtitled moving image is a moving image having the content spoken by the speaker as text data of subtitles without giving a single word.

  Here, a corpus for lip reading will be described. What is required for a lip reading corpus is text data indicating what lip animation and what it says in the lip animation. The text data needs to indicate which sound is output when the lip is in shape, so it needs to be synchronized with the lip video, but in the video with subtitles, the captioned content is indicated by subtitles However, subtitles do not have data on the timing of sound generation. In this embodiment, a corpus is generated in synchronization with subtitle text data and lip animation.

First Embodiment
FIG. 1 is a diagram showing the configuration of a corpus generation device 1 according to the embodiment. The corpus generation device 1 includes the moving image acquisition unit 10, accesses an external resource, and acquires data of a subtitled moving image. The subtitled moving image data is composed of video data, audio data, and subtitle text data.

  The corpus generation device 1 includes a speaker recognition unit 11, a lip moving image generation unit 12, a speech recognition unit 13, and a text alignment unit 14. The speaker recognition unit 11 has a function of performing image processing of video data to recognize the face of a person appearing in the video and its organ points (eye, nose, mouth, etc.). Then, the speaker recognition unit 11 recognizes a person whose lips are open or closed as a speaker. A method of recognizing a speaker when the lips of a plurality of persons are simultaneously moving will be described in the second embodiment.

  The lip moving image generation unit 12 has a function of generating a lip moving image obtained by cutting out the area of the lip of the speaker recognized by the speaker recognition unit 11. The region of the lip cut out in this embodiment has a distance of “3” to the left and right from the center of the lip, assuming that each distance from the center of the lip to the left and right ends of the lip is “1” Similarly, in the vertical direction, when each distance from the center of the lip to the upper and lower ends of the lip is "1", this is a region having a distance of "3" from the center of the lip up and down. What has been described here is an example, and the area of the lip image can be set as appropriate. The lip moving image generation unit 12 inputs the generated lip moving image to the dividing unit 15.

  The speech recognition unit 13 has a function of performing speech recognition processing of speech data to obtain morphemes and sentences constituting speech. The text alignment unit 14 has a function of synchronizing the text data of subtitles and the timing of generation of sound. That is, the text alignment unit 14 adds time to the text data by collating the text data with the morphemes and sentences constituting the voice obtained by voice recognition. Specifically, data indicating the time when generation of each sentence constituting the text data is started and the time when the generation is finished is attached.

  Even in the case where an error is included in part of the recognition result by the speech recognition unit 13, the text alignment unit 14 can allocate time to each sentence of the text data. For example, there is a mistake between "Tenki" and "Kenki" such that the subtitle data is "Today's weather was fine." While the speech recognition result is "Jonjitsu no Kenkenhaha." Even so, because the degree of agreement between the two sentences was high, a sentence was uttered to the subtitle data "Today's weather was fine." The time can be attached.

  By adding time to the text data by the text alignment unit 14, the timing of the voice and the text data can be matched. And since the audio and the video are synchronized, it is possible to synchronize the text data and the video. That is, by synchronizing lip animation and text data, a corpus for lip reading can be generated.

  The dividing unit 15 divides the generated lip moving image, voice, and time-added text, and generates data of a predetermined unit time. Here, the unit time is the same unit time as when inferring lip reading. For example, the unit time is 3 seconds. If one sentence is within 3 seconds, the blank data is filled so as to be 3 seconds of unit time. Conversely, if one sentence is 3 seconds or more, the sentence is separated at a good place of the grammatical division, and then the blank data is filled so as to be 3 seconds of unit time. The set of lip moving images, speech and time-added text of unit time generated in this manner is stored in the corpus storage unit 16. Here, an example is given where segmentation is performed when the sentence length exceeds unit time, but when the sentence length exceeds unit time, a corpus (for example, 5 seconds) in which the unit time is extended is generated May be

  FIG. 2 is a diagram showing the flow of the operation of the corpus generation device 1. In FIG. 2, a bold frame indicates the process of the corpus generation device 1, and a narrow frame indicates data. The corpus generation device 1 acquires a subtitled video D10 from a website on the Internet. The subtitled moving image D10 includes data of a video D11, an audio D12, and a subtitle D13.

  The corpus generation device 1 recognizes the face of a person appearing in the image D11 as processing on the image D11, and recognizes the organ point (S10). Subsequently, the corpus generation device 1 specifies a person who emits a speech, that is, a speaker (S11). In the process of speaker recognition (S11), the dotted line is described from the data of the voice D12 because the data of the voice D12 is used for the recognition process of the speaker when a plurality of persons are speaking at the same time Because there is This will be described in the second embodiment. In the present embodiment, it is assumed that only one person appears in the image D11, or even if a plurality of people appear in the image D11, only one person speaks.

  Further, the corpus generation device 1 performs text alignment as processing for text data of the subtitle D13 (S12). The voice D12 data is used to perform text alignment. As described above, voice recognition of the voice D12 is performed to obtain morphemes constituting the voice, and time is added to the text data using the voice recognition result. The text alignment process (S12) generates time-added text D15.

  The corpus generation device 1 divides the generated lip moving image S14, the speech D12, and the time-added text D15 into sentences for each sentence, and processes it so as to be a predetermined unit time (fills the blank data). A moving image D16, a unit time audio D17, and a unit time text D18 are generated. In addition, in FIG. 2, the lip moving image D16 and the audio D17 are also described as "with time" because the moving image and the audio originally have a time for indicating the reproduction timing, so this is described in a confirmed manner. It is

  The configuration of the corpus generation device 1 according to the present embodiment has been described above. The hardware of the corpus generation device 1 described above is provided with a CPU, a RAM, a ROM, a hard disk, a display, a keyboard, a mouse, a communication interface, etc. Computer. A program having modules for realizing the functions of the above-described speaker recognition unit 11, lip moving image generation unit 12, speech recognition unit 13, text alignment unit 14, and division unit 15 is stored in RAM or ROM, and the CPU By executing the program, the function of the above-described corpus generation device 1 is realized. Such programs are also included in the scope of the present invention. Moreover, an example of the moving image acquisition unit 10 is a communication interface.

  The configuration and operation of the corpus generation device 1 according to the first embodiment have been described above. The corpus generation device 1 according to the present embodiment can generate a corpus based on a moving image acquired from an external resource such as a website on the Internet. By this, it is not necessary to create a moving image for generating a corpus, and it is possible to generate a highly versatile corpus by using a large amount of resources on a website. The generated corpus can be used for learning for lipreading inference, and also for verification of the learned model.

  Here, learning of a neural network for inference of lip reading will be described. FIG. 3 is a diagram showing a model of a neural network which outputs a text of what words are issued with lip video and voice as input. The neural network model has (1) a speech processing unit and (2) an image processing unit. (1) Each of the audio processing unit and (2) image processing unit has three layers consisting of a combination of an STCNN (Spatiotemporal Convolutional Neural Network) layer and a Spatial pooling layer, and a GRU (Gated Recurrent Unit) layer at the latter stage Has two layers.

  Subsequently, (1) the network of the audio processing unit and (2) the image processing unit are merged, and (3) the language processing unit is input. (3) The language processing unit has two GRU layers, and is configured to include a Liner layer and a CTC (Connectionist Temporal Classification) Loss layer at the subsequent stage. The lip video of the corpus is input to the image processing unit, and the voice is input to the voice processing unit. As a result, learning is performed by feeding back the error between the obtained text data and the timed text of the corpus to the neural network by the back error propagation method.

Second Embodiment
FIG. 4 is a diagram showing the configuration of the corpus generation device 2 according to the second embodiment. The basic configuration of the corpus generation device 2 of the second embodiment is the same as that of the corpus generation device 1 of the first embodiment, but the corpus generation device 2 of the second embodiment has subtitles It has a configuration for recognizing a speaker corresponding to subtitle text from among a plurality of persons when a plurality of persons shown in a video are talking at the same time.

  The corpus generation device 2 has a speech sound estimation unit 17. The speech sound estimation unit 17 has a function of estimating the speech sound by lip reading from the lip image of the person speaking in the moving image. The speech sound estimation unit 17 reads a neural network model for inference of lip reading from the model storage unit 18, inputs lip video to the read neural network model, and estimates speech sound. The neural network model stored in the model storage unit 18 is one learned by the learning unit 19 using the corpus stored in the corpus storage unit 16.

  FIG. 5 is a diagram showing processing of specifying a speaker from among a plurality of persons by the corpus generation device 2 according to the second embodiment. This operation is a flow corresponding to the speaker recognition (S11) shown in FIG. 2, and as shown by a dotted line in FIG.

  The speaker recognition unit 11 specifies the region of each lip of a plurality of persons in the moving image (S20). The speaker recognition unit 11 inputs data of the identified region of the plurality of lips to the speech sound estimation unit 17. The speech sound estimation unit 17 estimates speech sounds based on the movements of the lips of a plurality of persons (S21 to S22), and inputs the estimated speech sounds to the speaker recognition unit 11. The speaker recognition unit 11 compares a plurality of utterance sounds with text data of subtitles, and recognizes a person who is uttering corresponding to subtitles as a speaker (S23). As can be understood from the above description, the accuracy of the lip reading by the speech sound estimation unit 17 may be as long as it is possible to specify the speaker.

  The configuration has been described above in which the corpus generation device 2 of the second embodiment is added from the corpus generation device 1 of the first embodiment. The corpus generation device 2 according to the second embodiment can recognize the speaker even in the case of a moving image spoken by a plurality of persons, so the range of the moving image which can be used for corpus generation is expanded.

As mentioned above, although the corpus generation apparatus and the corpus generation method of the present invention have been described in detail by taking the embodiment, the present invention is not limited to the above embodiment.
In the second embodiment described above, the configuration for recognizing a person corresponding to the text data of a subtitle as a speaker when a plurality of persons appearing in a video are speaking has been described. The method of recognizing C. is not limited to the method described in the second embodiment. For example, for the image with subtitles, the question "Who is the person corresponding to the subtitles?" And the problem of selecting the person who is speaking are automatically generated, The user may be inquired (so-called crowdsourcing), and the speaker may be identified based on the counting result.

  Although the above embodiment has been described by way of an example in which a corpus consisting of lip video, voice, and timed text as a set is generated, the lip video and timed text are set without storing voice data. You may generate a corpus that

  In the above-described embodiment, an example is given in which a captioned moving image is used as a material for corpus generation. However, if the accuracy of speech recognition by the speech recognition unit 13 is high, a corpus can be generated without captions. . That is, by using data of morphemes that constitute speech recognized by the speech recognition unit 13 as time-added text, collation with subtitle text data can be omitted.

  In the above embodiment, although the data indicating the start time and the end time of the sentence is attached to the time-added text, the start time and the end time of the utterance of each word may be attached instead of each sentence. The occurrence time may be attached to each character.

  The present invention is useful as an apparatus or the like for generating a corpus for lipreading that estimates speech sound from lip moving images.

1, 2 corpus generation apparatus 10 moving image acquisition unit 11 speaker recognition unit 12 lip moving image generation unit 13 speech recognition unit 14 text alignment unit 15 division unit 16 corpus storage unit 17 speech sound estimation unit 18 model storage unit 19 learning unit

Claims (8)

A video acquisition unit that acquires subtitled videos from external resources,
A lip moving image generating unit that recognizes a speaker included in the moving image and generates a lip moving image of the speaker;
A voice recognition unit which performs voice recognition of voice included in the moving image to obtain morphemes constituting the voice;
A text alignment unit for generating a time-added text by obtaining a time at which a sound corresponding to a text is emitted, based on the morpheme obtained by the voice recognition unit and the text data of subtitles in the moving image;
A storage unit for storing the lip moving image and the time-added text;
A corpus generating device comprising:   The corpus generation device according to claim 1, further comprising: a division unit configured to divide the lip moving image and the time-added text into data of a predetermined unit time.   3. The corpus generation device according to claim 1, wherein the text alignment unit converts kanji contained in the subtitles into hiragana characters to generate time-added text corresponding to the hiragana characters. A speech sound estimation unit that estimates the sound uttered from the lip video;
The lip moving image generation unit is configured to recognize a person whose estimated speech sound matches the text data of the subtitle as a speaker in the moving image, when a plurality of persons appear in the moving image. The corpus generation device according to any one of 1 to 3.   The corpus generation apparatus according to any one of claims 1 to 4, wherein the sound is stored in the storage unit together with the lip moving image and the time-added text. A video acquisition unit that acquires videos from external resources,
A lip moving image generating unit that recognizes a speaker included in the moving image and generates a lip moving image of the speaker;
A voice recognition unit which performs voice recognition of voice included in the moving image to obtain morphemes constituting the voice;
A text alignment unit that generates text data representing morphemes determined by the voice recognition unit, determines times at which sounds corresponding to the morphemes are emitted, and generates a time-added text;
A storage unit for storing the lip moving image and the time-added text;
A corpus generating device comprising: A method of generating a corpus using a movie with subtitles acquired from an external resource, wherein the corpus generation device generates a corpus,
The corpus generation device acquiring a subtitled moving image from an external resource;
The corpus generation device recognizes a speaker included in the moving image and generates a lip image of the speaker;
The corpus generation device performs speech recognition of speech included in the moving image to obtain morphemes constituting the speech;
The corpus generation device determines a time when a sound corresponding to a text is emitted based on a morpheme obtained by the speech recognition and text data of a subtitle in the moving image, and generates a time-added text;
Storing the lip moving image and the time-added text in a storage unit;
A corpus generation method comprising: A program for generating a corpus using subtitled moving images acquired from an external resource, the computer comprising:
Obtaining a subtitled video from an external resource;
Recognizing a speaker included in the video, and generating a lip video of the speaker;
Performing voice recognition of voice included in the moving image to obtain morphemes constituting the voice;
Determining a time at which a sound corresponding to the text is emitted based on the morpheme determined by the voice recognition and the subtitle text data in the moving image, and generating a time-added text;
Storing the lip moving image and the time-added text in a storage unit;
Program to run.

Images