JP6127811B2 - Image discrimination device, image discrimination method, and image discrimination program - Google Patents

Description

  The present invention relates to an image discrimination device, an image discrimination method, and an image discrimination program.

  In recent years, lip-reading technology that estimates the utterance content from an image of a person's mouth has attracted attention. In the lip reading technique, for example, dictionary information in which character information of a predetermined unit such as a word is associated with shape information indicating the shape of the mouth when the character information is pronounced is used. In the lip reading process, the input image showing the mouth area is compared with the shape information included in the dictionary information, and the character information associated with the shape information having a high degree of coincidence with the mouth area in the input image Is determined as the utterance content.

As the shape information included in the dictionary information, an image obtained by photographing the mouth area, a measured value of the shape feature value of the mouth area, or the like can be used.
It is also considered that the lip reading technique is used for operating a device such as a portable terminal. For example, there has been proposed a mobile phone that determines a word uttered from an image obtained by photographing an operator's mouth and executes a process associated with the determination result.

  Further, as an example of a technique for specifying a mouth area, a template matching technique using a template having color information and shape information is known.

JP 2012-59017 A JP 2012-118679 A JP 2006-12093 A

  By the way, as a method of generating the dictionary information as described above, for example, a number of subjects can pronounce a character string such as a word, and an image of the mouth area at the time of pronunciation is taken, and the dictionary information is based on the obtained image. The method of generating is conceivable. However, this method has a problem that it takes time and effort to shoot.

  On the other hand, it is also conceivable to collect various video contents that are disclosed on the network space or recorded on a recording medium, and generate dictionary information based on the video contents. However, since the contents of the video content collected in this way are various, there is a problem that an operation for extracting an appropriate part for generating dictionary information from the video content becomes enormous.

  In one aspect, an object of the present invention is to provide an image discriminating apparatus, an image discriminating method, and an image discriminating program capable of increasing efficiency in generating dictionary information for lip reading processing.

  In one proposal, an image discrimination device having a detection unit and a discrimination unit is provided. The detection unit detects a mouth area from each frame of the input moving image in which a scene during a period in which the character string is pronounced is shown. The discriminating unit indicates the shape of the mouth when the character string is pronounced when the number of frames in which the mouth area is not detected among the frames of the input moving image is equal to or less than a predetermined number. It is determined as a moving image for generating dictionary information.

Further, in one proposal, an image discrimination method is provided in which processing similar to that of the image discrimination device is executed.
Furthermore, in one proposal, an image discrimination program is provided that causes a computer to execute the same processing as that of the image discrimination device.

  According to one aspect, it is possible to improve the efficiency of generating dictionary information for lip reading processing.

It is a figure which shows the structural example and processing example of the image discrimination | determination apparatus which concern on 1st Embodiment. It is a figure which shows the hardware structural example of the work assistance apparatus which concerns on 2nd Embodiment. It is a figure which shows the outline | summary of the procedure of a lip reading process. It is a block diagram which shows the example of the function with which a work assistance apparatus is provided. It is a figure shown about the example of a process of a speech image file generation part. It is a figure shown about the other example of a process of an utterance image file generation part. It is a figure shown about the example of a process of a word division part and a word area extraction part. It is a figure shown about the process example of a mouth area | region detection part. It is a figure shown about the example of a process of a determination part. It is a flowchart which shows the process example of a work assistance apparatus. It is a figure which shows the outline | summary of the search process of the mouth area | region using the template prepared according to the character. It is a figure which shows an example of the use switching timing of a template. It is a figure which shows the example which uses the template obtained by interpolation. It is a figure which shows the example of an interpolation process at the time of using a feature-value vector. It is a figure which shows the process example when multiple mouth area | regions are detected. It is a figure which shows the other example of a process when multiple mouth area | regions are detected. It is a figure which shows the example of a detection process according to the magnitude | size of a mouth area | region.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example and a processing example of the image discrimination device according to the first embodiment.

  The image discriminating apparatus 1 supports the operation of generating dictionary information for lip reading processing. The dictionary information is information in which shape information indicating the shape of a mouth when a character string is pronounced is associated with a meaningful character string such as a word. In the lip reading process, for example, the input image showing the mouth area and the shape information included in the dictionary information are compared, and the character string associated with the shape information having a high degree of coincidence with the mouth area in the input image, Determined as utterance content.

  The image discrimination device 1 includes a detection unit 2 and a discrimination unit 3. The processing of the detection unit 2 and the determination unit 3 is realized by, for example, a processor included in the image determination device 1 executing a predetermined program.

  The detection unit 2 detects a mouth area from each frame of an input moving image in which a scene during a period in which a certain character string is pronounced is shown. As a method for detecting the mouth region, various methods such as a template matching method can be used. Moreover, as an input moving image, what was extracted from the moving image published on networks, such as the internet, can be used, for example. It is assumed that the character string pronounced in the scene shown in the input moving image at the time of detection by the detection unit 2 is known.

  In the example of FIG. 1, the input moving image 4 is obtained reflects the scene periods character string "Hello" is pronounced. Here, for easy understanding, it is assumed that the input moving image 4 includes five frames.

  The discriminating unit 3 counts the number of frames in which the mouth region is not detected by the detecting unit 2 among the frames of the input moving image. When the number of frames in which no mouth area is detected is equal to or smaller than a predetermined determination threshold, the input moving image is determined as a moving image for generating dictionary information. For example, when the number of frames in which no mouth area is detected is equal to or less than a predetermined determination threshold, the determination unit 3 may register the input moving image as mouth shape information in the dictionary information. The determination threshold value is an integer of 0 or more.

  In the example shown in FIG. 1, the mouth area is detected from the first, second, fourth, and fifth frames of the frame of the input moving image 4, but the mouth area is detected from the third frame. It is assumed that there was not (step S1). Here, if the determination threshold value is “0”, the determination unit 3 determines that the number of frames in which no mouth area is detected is “1” and is greater than the determination threshold value “0” (step S2). In this case, the determination unit 3 determines that the input moving image 4 is not a moving image for generating dictionary information.

  Here, it is preferable that a moving image used for generating mouth shape information included in the dictionary information includes mouth regions in as many frames as possible. Thereby, in the lip reading process, it is possible to improve the accuracy of the coincidence determination when the generated shape information is compared with an image showing a mouth.

  With the above-described processing of the image discrimination device 1, it is possible to accurately determine whether or not the input moving image is appropriate as a moving image used for generating dictionary information for executing highly accurate lip reading processing. Therefore, the efficiency of the dictionary information generation work can be increased.

  In particular, when an input video image extracted from a large amount of video images already distributed in the world, such as a video image published on the network, the corresponding character string is included in the input video image. Despite the pronounced scene, the mouth of the person who pronounced it is not always reflected. In addition, there may be cases where a face or a person is not reflected in the mouth. The above processing of the image discriminating apparatus 1 can remarkably improve the dictionary information generation work when such an input moving image is used.

[Second Embodiment]
FIG. 2 is a diagram illustrating a hardware configuration example of the work support apparatus according to the second embodiment. The work support apparatus 100 is an apparatus for supporting the work of generating dictionary information used for lip reading processing. The work support apparatus 100 is realized, for example, as a computer as shown in FIG.

  The work support apparatus 100 is entirely controlled by a processor 101. The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may be a combination of two or more elements among CPU, MPU, DSP, ASIC, and PLD.

A RAM (Random Access Memory) 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 108.
The RAM 102 is used as a main storage device of the work support apparatus 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 stores various data necessary for processing by the processor 101.

  Peripheral devices connected to the bus 108 include an HDD (Hard Disk Drive) 103, a graphic processing device 104, an input interface 105, a reading device 106, and a communication interface 107.

  The HDD 103 is used as an auxiliary storage device of the work support apparatus 100. The HDD 103 stores an OS program, application programs, and various data. As the auxiliary storage device, other types of nonvolatile storage devices such as SSD (Solid State Drive) can be used.

  A display device 104 a is connected to the graphic processing device 104. The graphic processing device 104 displays an image on the screen of the display device 104a in accordance with an instruction from the processor 101. Examples of the display device include a display device using a CRT (Cathode Ray Tube) and a liquid crystal display device.

  An input device 105 a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. Examples of the input device 105a include a keyboard and a pointing device. Examples of pointing devices include a mouse, a touch panel, a tablet, a touch pad, and a trackball.

  A portable recording medium 106 a is detached from the reading device 106. The reading device 106 reads the data recorded on the portable recording medium 106 a and transmits it to the processor 101. Examples of the portable recording medium 106a include an optical disk, a magneto-optical disk, and a semiconductor memory.

  The communication interface 107 transmits / receives data to / from other devices via the network 107a. The network 107a may be connected to the Internet, for example.

With the hardware configuration described above, the processing function of the work support apparatus 100 can be realized.
Next, FIG. 3 is a diagram showing an outline of the procedure of the lip reading process. With reference to FIG. 3, description will be given of dictionary information to be generated in this embodiment and how dictionary information is used in lip reading processing.

  For example, a record 201 corresponding to one word is registered in the dictionary information 200. In each record 201, text information (or word identification information) indicating a word is registered in association with a moving image in which a mouth area is captured when the word is pronounced. Also, a plurality of moving images can be associated with each record 201 and registered.

  For example, the lip reading engine 210 performs the following lip reading process using the dictionary information 200. A moving image 221 to be processed is input to the lip reading engine 210. The moving image 221 to be processed includes an unknown person's mouth area. The lip reading engine 210 performs matching between the mouth area image in the moving image 221 to be processed and the moving image registered in the dictionary information 200, and calculates the similarity. Then, the lip reading engine 210 determines the moving image having the highest similarity among the moving images in the dictionary information 200, and sets the text information of the word associated with the determined moving image as the moving image 221 to be processed. It is output as an estimation result 222 of the content uttered by the reflected person.

  Here, as described above, in the dictionary information 200, it is possible to register a plurality of moving images in association with one word. For example, it is possible to register a moving image in which a mouth area is photographed when different people pronounce the same word in one record 201. Also, for example, it is possible to register a moving image obtained by photographing mouth areas when the same person pronounces the same word from different angles, in one record 201.

  As described above, in the dictionary information 200, by associating many moving images with one word and using these moving images for matching of the lip reading process, the lip reading accuracy can be improved.

  By the way, as a method of preparing a moving image to be registered in the dictionary information 200, for example, there is a method in which a large number of subjects pronounce a word and an image of a mouth region at the time of pronunciation is taken. There is a problem that is large. In particular, as the number of moving images registered in the dictionary information 200 in order to improve the lip reading accuracy as described above increases, the time and effort of shooting increases.

  On the other hand, the inventor of the present case has studied a method of collecting various moving images that have already been shot and generating a moving image to be registered in the dictionary information 200 using the collected moving images. As the moving image to be collected, for example, a moving image published on a network such as the Internet, a moving image recorded on a recording medium such as an optical disk, and the like can be considered. By such a method, it is possible to save the trouble of performing a photographing operation for generating the dictionary information 200.

  However, in a moving image collected by such a method, even if a scene in which the target word is pronounced is shown, the mouth area of the person who pronounced the word is not always shown. Even if the mouth area is shown, the mouth area of the person who pronounced the word is not necessarily shown throughout the entire period from the start point to the end point of the pronunciation of the word.

  In order to be used as a moving image of the dictionary information 200, the mouth area of the person who pronounced is shown for as long a period as possible from the start to the end of pronunciation of the desired word (most preferably the entire period). desirable. Thereby, the accuracy of the lip reading process can be increased. However, there is a problem that the labor for extracting a moving image suitable for use as the moving image of the dictionary information 200 from the collected moving image is very large.

  Therefore, the work support apparatus 100 according to the present embodiment automates at least a part of the work of extracting a moving image suitable for use as the moving image of the dictionary information 200 from the moving image collected in this way. , Improve its working efficiency. Thereby, the development work of the lip reading engine 210 is made efficient, the lip reading processing accuracy is improved, and the development cost is reduced.

  In the description of FIG. 3 above, one or more moving images are registered in each record 201 of the dictionary information 200, but a corresponding word is pronounced in each record 201 of the dictionary information 200. Information indicating the shape of the mouth area at that time may be registered. A moving image is an example of information indicating such a shape. Another example of information indicating the shape of the mouth area is a feature amount of the mouth area.

  However, information indicating the shape of the mouth area is generally generated based on a moving image in which the mouth area is photographed. Accordingly, even if information other than a moving image is registered as information indicating the shape of the mouth area in the dictionary information 200, a moving image in which the mouth area is photographed is necessary to generate such dictionary information 200. There will be no change.

  In the lip reading process of the present embodiment, recognition is performed in units of words as described in FIG. This method can improve the recognition accuracy, for example, as compared with a method of performing recognition in units of one character (syllable unit).

In addition, the “word” referred to in the present embodiment includes, for example, a “sentence” including a word and an accompanying word that follows the word.
In addition, the present embodiment can be modified so as to be recognized in units larger than words (or phrases) (for example, a plurality of phrases). In this case, in the dictionary information 200, for example, one or more pieces of information indicating the shape of the mouth area may be registered in association with each character information of a plurality of phrases.

  FIG. 4 is a block diagram illustrating an example of functions provided in the work support apparatus. The work support apparatus 100 includes a moving image collection unit 111, an utterance image file generation unit 112, a word division unit 113, a word section extraction unit 114, a mouth area detection unit 115, and a determination unit 116. The processing of each processing block is realized by, for example, the processor 101 included in the work support apparatus 100 executing a predetermined program.

  The moving image collection unit 111 collects moving images from the network 107a. For example, the moving image collection unit 111 accesses a Web server that provides a Web site that distributes moving image content free of charge, such as a moving image posting site, and downloads moving image data. The collected moving image data is stored in the HDD 103 of the work support apparatus 100, for example.

  The utterance image file generation unit 112 extracts a section of a scene in which a person utters a word (hereinafter referred to as “speech section”) from each of the moving images collected by the moving image collection unit 111. The utterance image file generation unit 112 cuts out the extracted moving image of the utterance section and generates the utterance image file 130. In the utterance image file 130, utterance content text 131 indicating the utterance content and moving image data 132 are stored in association with each other. The generated utterance image file 130 is temporarily stored in the RAM 102 of the work support apparatus 100, for example.

  As will be described later, in the present embodiment, it is assumed that subtitle text is basically added to the moving image collected by the moving image collection unit 111. For example, caption text is added so that one sentence (caption) is displayed for a plurality of adjacent frames in a certain range in the collected moving image.

The word dividing unit 113 divides the utterance content text 131 of each of the generated utterance image files 130 in units of words using morphological analysis or the like.
The word segment extraction unit 114 cuts out a segment in which each word divided by the word segmentation unit 113 is pronounced (hereinafter referred to as “word segment”) from the moving image data 132 of each generated utterance image file 130. The word image file 140 is generated. In the word image file 140, word text 141 indicating a pronounced word and moving image data 142 are stored in association with each other. The generated word image file 140 is temporarily stored in the RAM 102 of the work support apparatus 100, for example.

The mouth area detection unit 115 detects the mouth area by image processing from each frame of the image based on the moving image data 142 of each of the generated word image files 140.
The determination unit 116 counts the number of frames in which the mouth region is not detected by the mouth region detection unit 115 for the moving image data 142 of each word image file 140. The determination unit 116 uses the moving image data 142 to be detected to generate the dictionary information 200 described above when the number of frames in which no mouth area is detected is equal to or less than a predetermined determination threshold. Determine that it is possible. In this case, the determination unit 116 generates a dictionary candidate file 150 to be used for generating the dictionary information 200 based on the corresponding word image file 140.

  The dictionary candidate file 150 includes word text 151, moving image data 152, and mouth area coordinates 153 that indicate areas in which mouth areas are detected in each frame of the moving image data 152. Among these, the word text 151 and the moving image data 152 are the same as the word text 141 and the moving image data 142 of the corresponding word image file 140. The generated dictionary candidate file 150 is stored in the HDD 103 of the work support apparatus 100, for example.

  The work support apparatus 100 may not include the moving image collection unit 111 and the utterance image file generation unit 112. In this case, the utterance image file 130 is stored in the work support apparatus 100 via a network or via a portable recording medium.

  In addition to the moving image collection unit 111 and the utterance image file generation unit 112, the work support apparatus 100 may not further include the word division unit 113 and the word section extraction unit 114. In this case, the word image file 140 is stored in the work support apparatus 100 through a network or a portable recording medium.

Next, processing of each processing block will be described.
First, FIG. 5 is a diagram illustrating a processing example of the utterance image file generation unit.
As described above, subtitle text is basically added to the moving image collected by the moving image collection unit 111. The subtitle text is added as text data, for example, in the header area of the collected moving image data, and the utterance image file generation unit 112 can recognize the content of the subtitle text and its display period. To do.

  The utterance image file generation unit 112 specifies a period during which the same caption text is displayed from the collected moving images (hereinafter referred to as “same caption display period”). The utterance image file generation unit 112 extracts the moving image data of the specified period as the moving image data 132 of the utterance image file 130 and associates it with the utterance content text 131 describing the displayed subtitle text. Thereby, the utterance image file 130 is generated.

  In the example of FIG. 5, the period T1 in which the subtitle text “Good morning, today” is displayed is specified, and the moving image data 132 is generated by cutting out the frame of the period T1. Then, an utterance image file 130 including the extracted moving image data 132 and the utterance content text 131 describing the text information “Good morning, today” is output.

  Note that the utterance image file generation unit 112 can also generate the utterance image file 130 based on the moving image collected by the moving image collection unit 111 and without subtitle text added. An example of this case will be described with reference to FIG.

FIG. 6 is a diagram illustrating another processing example of the utterance image file generation unit.
For example, the utterance image file generation unit 112 detects an utterance section in which a person utters a word based on an audio signal in a moving image collected by the moving image collection unit 111. The detection of the utterance period can be performed, for example, by analyzing the frequency spectrum of the audio signal. The utterance image file generation unit 112 further performs voice recognition based on the detected voice signal of each utterance section, and acquires text indicating the utterance content. By such a method, the utterance image file 130 can be generated even when subtitle text is not added to the collected moving images.

  In the example of FIG. 6, it is assumed that speech sections A1 and A2 are detected from the audio signal in the collected moving image. It is assumed that the utterance contents “Good morning” and “Kyoha” are recognized by voice recognition from the respective voice signals in the utterance sections A1 and A2. In this case, the utterance image file generation unit 112 outputs the utterance image file 130 including the moving image data 132 obtained by cutting out the frame in the utterance section A1 and the utterance content text 131 in which the text information “Good morning” is described. Further, the utterance image file generation unit 112 outputs an utterance image file 130 including moving image data 132 obtained by cutting out a frame in the utterance section A2 and utterance content text 131 in which text information “Kyoha” is described.

  Note that at least a part of the speech segment and speech content extraction work from the speech image file 130 or from the moving image collected by the moving image collection unit 111 may be performed by an operation by an operator. Further, for example, with respect to the word image file 140 generated by the processing of the word dividing unit 113 and the word section extracting unit 114, the start time and end time of the moving image and the utterance content are corrected by an operation by the operator. May be.

Next, FIG. 7 is a diagram illustrating a processing example of the word dividing unit and the word section extracting unit.
The word dividing unit 113 divides the utterance content text 131 of each of the generated utterance image files 130 in units of words using morphological analysis or the like.

  The word segment extraction unit 114 cuts out a segment in which each word divided by the word segmentation unit 113 is pronounced (hereinafter referred to as “word segment”) from the moving image data 132 of each generated utterance image file 130. The word image file 140 is generated. In the word image file 140, word text 141 indicating a pronounced word and the extracted moving image data 142 are stored in association with each other. The generated word image file 140 is temporarily stored in the RAM 102 of the work support apparatus 100, for example.

  For example, the word section extraction unit 114 performs voice recognition based on a voice signal included in the moving image data 132 and detects a start time and an end time of a section in which each word is pronounced. Then, the word segment extraction unit 114 cuts out frames displayed from the start time to the end time from the moving image data 132 to generate moving image data 142 of the word image file 140.

  For example, when the utterance image file generation unit 112 generates the utterance image file 130 using voice recognition, at the time of the voice recognition, in units of words or syllables in the moving image data 132 in the utterance image file 130. In some cases, the pronunciation start time and the pronunciation end time are detected. In this case, the word segment extraction unit 114 can detect a word segment in the moving image data 132 based on the pronunciation start time and the pronunciation end time detected by the utterance image file generation unit 112.

  FIG. 7 shows an example in which the word image file 140 is generated from the utterance image file 130 in which the utterance content “Good morning, today” is described. The word dividing unit 113 performs morphological analysis on the utterance content “Good morning, today” and divides the utterance content into words (phrases) “Good morning” and “Kyoha”.

  The word segment extraction unit 114 cuts out a word segment pronounced as “good morning” and a word segment pronounced as “Kyoha” from the moving image data 132 of the utterance image file 130. In the example of FIG. 7, it is determined that the period from time T11 to time T12 is a word segment that is pronounced “good morning”. Then, the frame displayed from time T11 to time T12 in the moving image data 132 of the utterance image file 130 is cut out as moving image data 142 of the word image file 140. Further, in the example of FIG. 7, it is determined that the period from time T13 to time T14 is a word section pronounced “Kyoha”. Then, a frame displayed from time T13 to time T14 in the moving image data 132 of the utterance image file 130 is cut out as moving image data 142 of the word image file 140.

Next, FIG. 8 is a diagram illustrating a processing example of the mouth area detection unit.
The mouth area detection unit 115 detects a mouth area from each frame of the moving image data 142 included in the word image file 140 by image processing. For example, a template matching method can be used for detecting the mouth region.

  When the template matching method is used, the mouth area detection unit 115 compares each frame of the moving image data 142 with the template of the mouth area. A template is image information including a shape pattern of the mouth area at the time of utterance, for example, an image in which the mouth area at the time of utterance is photographed or only a characteristic shape pattern of the mouth area from such a photographed image. Are extracted images.

  For example, as illustrated in FIG. 8, the mouth area detection unit 115 aligns the upper left of the detection target frame 231 and the upper left of the template 232, and starts the template 232 one pixel at a time from such a positional relationship as a starting point. While moving, the similarity between the template 232 and the area of the frame 231 that overlaps the template 232 is calculated. When the right end of the template 232 reaches the right end of the frame 231, the mouth area detection unit 115 moves the template 232 downward by one pixel and performs the same processing. In this way, the mouth area is searched from the entire area of the frame 231.

  Further, the mouth area detection unit 115 searches the mouth area from the entire area of the frame 231 in the same manner as described above by changing the size of the template 232 (for example, by reducing it by one level). The size change of the template 232 may be performed in three or more stages.

  The mouth area detection unit 115 determines the area 233 of the frame 231 corresponding to the position of the template 232 for which the highest similarity is calculated among the similarities calculated in this way as the mouth area. However, the mouth area detection unit 115 determines that the mouth area is not detected from the frame when the calculated maximum value of the similarity is lower than a predetermined determination threshold value.

Note that the template may be, for example, image data with only luminance information. In that case, the luminance information of the frame is compared with the template.
Moreover, when calculating the similarity between the template and the region overlapping the template in the frame, for example, the similarity between the feature amounts of both may be calculated. An example of the feature quantity is a feature quantity vector. For example, if the feature quantity vector of the template is X and the feature quantity vector of the region overlapping the template in the frame is Y, the distance D indicating the similarity can be obtained as D = | X−Y |.

  Note that face detection can also be used to detect the mouth area. For example, a face is detected from the frame, and a part of the area where the face is detected is set as a mouth area search area using the template as described above. According to this method, the search area of the mouth area using the template is limited, and the processing load can be reduced. However, face detection can be used only when almost the entire face is shown in the frame.

FIG. 9 is a diagram illustrating a processing example of the determination unit.
The determination unit 116 counts the number of frames in which the mouth region is not detected by the mouth region detection unit 115 among the frames included in the moving image data 142 of the word image file 140. Here, the number of frames included in the moving image data 142 is N, the number of frames in which the mouth area is detected among the frames of the moving image data 142 is R, and the determination threshold is K (where K is an integer of 0 or more) ). At this time, the determination unit 116 determines that the moving image data 142 can be used for generating the dictionary information 200 when (N−R) ≦ K. In this case, the determination unit 116 generates a dictionary candidate file 150 to be used for generating the dictionary information 200 based on the corresponding word image file 140.

  FIG. 9A shows an example of determination based on the word image file 140 in the word period pronounced “Good morning”, and FIG. 9B shows the words in the word interval pronounced “Kyoha”. An example of determination based on the image file 140 is shown. In order to simplify the description, it is assumed that the moving image data 142 in the case of FIG. 9A includes 4 frames, and the moving image data 142 in the case of FIG. 9B includes 5 frames. . In addition, a rectangular area indicated by a thick line in each frame is an area where the mouth is detected.

  Here, the determination threshold value K is set to “0” as an example. In the example of FIG. 9A, it is assumed that the mouth area is detected from all frames. In this case, since the number of frames in which the mouth area is not detected (the above “N−R”) is equal to or less than the determination threshold value “0”, the determination unit 116 converts the target moving image data 142 into the dictionary. It is determined that the information 200 can be used for generation.

  In this case, the determination unit 116 generates a dictionary candidate file 150 in which “good morning” is described as the word text 151 and the moving image data 142 that is the detection target of the mouth area is stored as the moving image data 152, and is stored in the HDD 103 or the like. save. In addition, information indicating the coordinates of the mouth area detected for each frame in the moving image data 152 is registered in the mouth area coordinates 153 of the generated dictionary candidate file 150.

  The moving image registered in the dictionary candidate file 150 is, for example, a moving image in which only the mouth area is shown in each frame. The mouth area coordinates 153 of the dictionary candidate file 150 are used to extract a mouth area from each frame when generating a moving image to be registered in the dictionary information 200 from the dictionary candidate file 150, for example. The determination unit 116 may directly generate a moving image to be registered in the dictionary information 200.

  On the other hand, in the example of FIG. 9B, the mouth area is detected from the first, second, fourth, and fifth frames, but the mouth area is not detected from the third frame. In this example, since the number of frames in which no mouth area is detected (the above “N−R”) is larger than the determination threshold value “0”, the determination unit 116 stores the target moving image data 142 in the dictionary. It is determined that the information 200 cannot be used for generation. In this case, the dictionary candidate file 150 is not generated.

  As described above, the moving image registered in the dictionary information 200 shows the mouth area of the person who pronounced the sound as long as possible from the start to the end of the corresponding word (most preferably the entire period). It is desirable. Thereby, the accuracy of the lip reading process can be increased. For example, when the mouth area is photographed to generate the dictionary information 200, the mouth area is surely reflected in the photographed moving image. However, in the moving image data 142 of the word image file 140 generated from the moving images collected by the moving image collecting unit 111, the mouth area is not always shown in each frame.

  As a result of the processing of the determination unit 116 described above, a moving image in which a mouth area is detected in many sections where a word is pronounced is selected, and the selected moving image is used to generate the dictionary information 200. Can be saved. Thereby, the generated dictionary information 200 can be optimized and the accuracy of the lip reading process using the dictionary information 200 can be improved. And the efficiency of the operation | work which extracts the moving image suitable for using as a moving image of such dictionary information 200 can be improved.

  The determination threshold value K is preferably “0”. On the other hand, when the determination threshold value K is “1” or more, for example, when the dictionary candidate file 150 is generated based on a moving image including a frame in which no mouth area is detected, the mouth area is detected. The correction may be performed so that the frame that is not included includes the mouth region. As a correction method, for example, a method in which a frame in which the mouth area is not detected is replaced with a frame immediately before or immediately after the mouth area is detected.

Next, the process of the work support apparatus 100 described above will be described using a flowchart. FIG. 10 is a flowchart illustrating a processing example of the work support apparatus.
[Step S11] The utterance image file generation unit 112 extracts a scene section (utterance section) in which a person uttered a word from the moving images collected by the moving image collection unit 111. The utterance image file generation unit 112 cuts out the extracted moving image of the utterance section and generates the utterance image file 130. In the utterance image file 130, utterance content text 131 indicating the utterance content and moving image data 132 are stored in association with each other.

[Step S12] The word dividing unit 113 divides the utterance content text 131 of the generated utterance image file 130 in units of words using morphological analysis or the like.
[Step S13] The word section extraction unit 114 cuts out a section (word section) in which each word obtained by dividing in step S12 is pronounced from the moving image data 132 of the utterance image file 130, and the word image file 140 is extracted. Is generated. In the word image file 140, word text 141 indicating a pronounced word and moving image data 142 are stored in association with each other.

  [Step S14] The determination unit 116 determines whether or not the processing in steps S15 to S18 has been executed for all the word image files 140 generated in step S13. If all the word image files 140 have been processed, the process is terminated. On the other hand, if there is a word image file 140 that has not been processed, the process of step S15 is executed for one of the word image files 140 that has not been processed.

  [Step S15] The mouth area detection unit 115 determines whether the mouth area detection processing in step S16 has been executed from all the frames of the moving image data 142 in the word image file 140 to be processed. When the detection process from all frames is completed, the process of step S17 is executed. On the other hand, if there is a frame that has not been subjected to the detection process, the process of step S16 is executed with the next frame in the moving image data 142 as the target of the detection process.

  [Step S16] The mouth area detection unit 115 detects the mouth area by image processing from the search target frame, and temporarily stores the detection result in the RAM 102 or the like. When the mouth area is detected, the detection result includes coordinate information of the detected mouth area. On the other hand, when the mouth area is not detected, the detection result describes, for example, that the mouth area is not detected.

  [Step S17] The determination unit 116 counts the number of frames in which the mouth area is not detected by the process of step S16 among the frames of the moving image data 142 of the word image file 140 to be processed. When the number of frames in which the mouth area is not detected is equal to or smaller than a predetermined determination threshold value, the determination unit 116 performs the process of step S18. On the other hand, when the number of frames in which no mouth area is detected is larger than the determination threshold, the process of step S18 is skipped and the process of step S14 is executed.

  [Step S <b> 18] The determination unit 116 generates a dictionary candidate file 150 to be used for generating the dictionary information 200 based on the word image file 140 to be processed, and stores it in the HDD 103.

Next, some modified examples based on the second embodiment will be described.
<Modification 1: Example of mouth region detection processing using a template for each character>
In FIG. 8, the mouth area detection process using only one type of template has been described. However, the shape of the mouth region varies depending on the character to be uttered. For this reason, by detecting the mouth area using an appropriate template for each character, the calculated value of the similarity when matching is increased, and as a result, the determination accuracy of the mouth area can be increased.

  FIG. 11 is a diagram showing an outline of mouth area search processing using a template prepared for each character. A template database (DB) 160 illustrated in FIG. 11 is stored in the HDD 103 of the work support apparatus 100, for example. In the template database 160, a template for a mouth area when a character is pronounced is prepared for each character (kana).

  The moving image data 142 to be detected by the mouth area detection unit 115 is associated with a word text 141 indicating the content of the utterance. Therefore, the mouth area detection unit 115 can easily specify a template to be used for the mouth area detection process from the character-specific templates. In the example of FIG. 11, the word text 141 describes the word “Good morning”. In this case, the mouth area detection unit 115 reads out templates (templates # 5, # 26, # 38, and # 3 in FIG. 11) corresponding to “o”, “ha”, “yo”, and “u” from the template database 160, respectively. Detect the mouth area using these templates.

  FIG. 12 is a diagram illustrating an example of template use switching timing. The mouth area detection unit 115 detects the mouth area using the templates read from the template database 160 in the order of appearance of characters. However, it is necessary to determine when to switch the template to be used.

  As an example of a method for determining the switching timing, there is a method in which the number of frames is equally divided by the number of characters and one template is assigned to each divided frame. Further, as a modified example of this method, there is a method in which the same number of frames are basically assigned to one template, but the number of frames to be assigned is smaller than other characters only for the last character. This method utilizes the fact that the period in which the last character in a word (or phrase) is pronounced may be shorter than the period in which each preceding character is pronounced.

  In the example of FIG. 12, templates corresponding to “o”, “ha”, and “yo” are used in four frames, respectively, among the frames of the moving image to be processed, and the mouth area is detected. However, the template corresponding to the last character “U” is used only in the last one frame. By such a method, it is possible to detect the mouth region by appropriately using a template corresponding to the character to be uttered.

For example, the ratio of the number of frames assigned to each character included in a word to the number of frames corresponding to the entire word may be determined for each word.
FIG. 13 is a diagram illustrating an example in which a template obtained by interpolation is used.

  In a period of transition from a state where a certain character is uttered to a state where a next character different from that is uttered, the shape of the mouth region changes from a shape corresponding to the previous character to a shape corresponding to the next character. For this reason, there is a possibility that the mouth region cannot be detected with high accuracy even if a template corresponding to either the previous character or the next character is used.

  Therefore, the mouth area detection unit 115 determines from the template corresponding to the previous character and the template corresponding to the next character during the transition from the state of speaking a certain character to the state of speaking the next character different from that. Alternatively, a template may be generated by interpolation, and the mouth area may be detected using the generated template.

  In the example of FIG. 13, a template used when continuously pronounced “o” and “ha” is shown. In FIG. 13, for a frame corresponding to a time (or a predetermined period) between a period in which “o” is pronounced and a period in which “ha” is pronounced, a template 161 corresponding to “o” and “ha”. The mouth region is detected using the interpolation template 163 generated by interpolation from the template 162 corresponding to the. Thereby, the detection precision of a mouth area | region can be improved.

  Further, instead of interpolating the template itself, the template feature amount used for similarity calculation may be interpolated. Hereinafter, a case where a feature quantity vector is used as an example of the feature quantity will be described.

  FIG. 14 is a diagram illustrating an example of interpolation processing in the case where a feature amount vector is used. In the example of FIG. 14, when the mouth area detection unit 115 detects the mouth area from two frames immediately before and after the boundary at which the sounded characters are switched, the mouth area detecting unit 115 calculates the feature vector of the template corresponding to each character before and after the switching. Interpolated feature vector obtained by interpolation is used.

  For example, as shown in FIG. 14, the frame corresponding to the frame number f = 0, 1 is included in the period in which “o” is pronounced, and the frame corresponding to the frame number f = 2, 3 is “ha”. It is included in the period when is pronounced. In this case, when the mouth area is detected from the frame corresponding to the frame number f = 1, 2, the feature value vector X_a of the template corresponding to “O” and the feature value vector X_b of the template corresponding to “ha” An interpolation feature vector obtained by interpolation based on the above is used.

  Correction coefficients w_a and w_b are used for calculation of the interpolation feature vector. The correction coefficient w_a is calculated by “−f / u + 1”, and the correction coefficient w_b is calculated by “f / u”. However, the variable u is a value obtained by adding the number of frames “2” immediately before and immediately after the number of frames in which the interpolation feature vector is used, and is “3” in the example of FIG.

  By such calculation, correction coefficients w_a and w_b are obtained for each frame position. The interpolation feature vector to be used is calculated by “w_a × X_a + w_b × X_b”. For each frame corresponding to the frame number f = 1, 2, the similarity is calculated using the interpolation feature vector calculated in this way and the frame feature vector, thereby improving the detection accuracy of the mouth region. Can be improved.

  In the first modification described above, an individual template is prepared for each character (kana). For example, the same template is used for characters whose mouth region shape is similar when pronounced. May be prepared. Further, for example, a template may be prepared for each vowel instead of each character (kana).

<Modification 2: Processing example (1) when a plurality of mouth regions are detected>
FIG. 15 is a diagram illustrating a processing example when a plurality of mouth areas are detected. For example, when a plurality of mouth areas are detected from the same frame, the mouth area detection unit 115 determines the detection area having the maximum calculated similarity value as the mouth area. Thereby, the detection accuracy of the mouth area can be improved.

  This method is particularly suitable when a mouth region is detected using a template corresponding to a pronounced character as shown in FIG. This is because, in a frame, an area similar to the shape of the mouth area when the corresponding character is pronounced is more likely to be pronounced.

  In the example of FIG. 15, two mouth areas having a similarity equal to or greater than a predetermined value are detected from the same frame, the similarity calculated for one area is “0.8”, and the other area is calculated for the other area. Assume that the similarity is “0.9”. However, it is assumed that the size of the detection region with the similarity of “0.9” is smaller than the detection region with the similarity of “0.8”.

  In this case, the mouth area detection unit 115 determines a detection area having a high degree of similarity as the mouth area. Thus, when a plurality of mouth areas are detected, the detection area having the maximum similarity is the mouth area even if the size of the detection area having the maximum similarity is smaller than at least one other detection area. It is determined.

<Modification 3: Processing example (2) when a plurality of mouth areas are detected>
FIG. 16 is a diagram illustrating another example of processing when a plurality of mouth areas are detected. The processing example in FIG. 16 is different from the processing example in FIG. 15, and the determination unit 116 has at least one frame in which a plurality of mouth areas are detected from the frames of the moving image data 142 of the word image file 140. If it is determined that the moving image data 142 is not suitable for generating the dictionary information 200, the dictionary candidate file 150 based on the moving image data 142 is not stored.

  When the mouths of a plurality of persons are shown in the same frame, it may be difficult to specify which of them pronounces the target character. For this reason, the quality of the dictionary information 200 can be improved by not using the moving image data 142 including the frame in which the mouths of a plurality of people are reflected for the generation of the dictionary information 200.

  For example, when the number of frames in which a plurality of mouth areas are detected from the frames of the moving image data 142 is equal to or greater than a predetermined determination threshold value of 2 or more, the determination unit 116 determines whether the dictionary information 200 It may be determined that it is not suitable for generation. In this case, when the number of frames in which a plurality of mouth areas are detected is smaller than the determination threshold, the determination unit 116 determines the similarity for frames in which a plurality of mouth areas are detected as shown in FIG. May be determined as the mouth region.

<Modification 4: Example of detection processing according to the size of the mouth area>
FIG. 17 is a diagram illustrating a detection processing example according to the size of the mouth area. Even if the mouth area is detected from the frame, if the size of the detected mouth area is too small, it may be unsuitable as an image used for generating the dictionary information 200. For example, if the detected mouth area is too small, it is difficult to understand the change in mouth shape between frames.

  Therefore, the mouth area detection unit 115 determines that the mouth area has not been detected when the area ratio of the detected mouth area to the frame is equal to or less than a predetermined determination threshold value. Thereby, the quality of the generated dictionary information 200 can be improved. The area ratio is calculated by “area of mouth region (number of pixels) / area of frame (number of pixels)”.

  In the example of FIG. 17, it is assumed that the mouth area is detected in any of the first to fourth frames. The area ratio of the mouth area detected in each frame is assumed to be “0.05”, “0.04”, “0.03”, and “0.01” in order from the first. Here, when the determination threshold is “0.02,” the mouth area detection unit 115 determines that the mouth area is detected from each of the first to third frames, but from the fourth frame, the mouth area is detected. It is determined that the area has not been detected. In this case, the determination unit 116 counts “1” as the number of frames in which the mouth area is not detected among the frames of the corresponding moving image.

The mouth area detection unit 115 may determine whether or not the mouth area is detected based on the area (number of pixels) of the detected mouth area instead of the area ratio.
Note that the processing functions of the devices (the image determination device 1 and the work support device 100) described in each of the above embodiments can be realized by a computer. In that case, a program describing the processing contents of the functions that each device should have is provided, and the processing functions are realized on the computer by executing the program on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic storage device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic storage device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Optical discs include DVD (Digital Versatile Disc), DVD-RAM, CD-ROM (Compact Disc-Read Only Memory), CD-R (Recordable) / RW (Rewritable), and the like. Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the program, for example, a portable recording medium such as a DVD or a CD-ROM in which the program is recorded is sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. In addition, each time a program is transferred from a server computer connected via a network, the computer can sequentially execute processing according to the received program.

DESCRIPTION OF SYMBOLS 1 Image discrimination device 2 Detection part 3 Discrimination part 4 Input moving image

Claims (9)

A detection unit for detecting a mouth area from each frame of an input moving image in which a scene of a period during which a character string is pronounced is reflected;
A dictionary indicating the shape of the mouth when the character string is pronounced when the number of frames in which the mouth area is not detected among the frames of the input moving image is equal to or less than a predetermined number A discriminating unit for discriminating a moving image for generating information;
An image discrimination device characterized by comprising: The discrimination unit
Referring to the moving image including the scene where the utterance is performed and the character information indicating the utterance content associated with the moving image, the character information is divided into character strings in units of words or phrases,
From the moving image, the pronunciation interval where the divided character string is pronounced is extracted for each divided character string,
The dictionary information is generated from each of the moving images corresponding to the sound generation interval, and the moving image corresponding to the sound generation interval in which the number of frames in which the mouth area is not detected by the detection unit is equal to or less than the predetermined number is generated. To select as a moving image for
The image discriminating apparatus according to claim 1.   3. The image discrimination device according to claim 2, wherein the process of extracting the pronunciation period for each of the divided character strings is performed by a voice recognition process based on a voice synchronized with a moving image corresponding to the pronunciation period. .   The detection unit determines, for each frame in the moving image, which character in the characters constituting the character string is pronounced, and the mouth shape pattern when each of the plurality of characters is pronounced. 4. A mouth region is detected by selecting a template corresponding to a pronounced character for each frame from a plurality of templates included, and detecting a mouth region by template matching using the selected template. The image discrimination device according to claim 1.   The detection unit detects a mouth area by template matching, and when a plurality of mouth areas are detected from one frame, an area having a maximum similarity to the template among the detected areas, The image discriminating apparatus according to claim 1, wherein the image discriminating apparatus is determined to be a mouth area.   The discriminating unit generates the dictionary information for the input moving image when the number of frames in which a plurality of mouth areas are detected is greater than or equal to a predetermined threshold among the frames of the input moving image. 6. The image discriminating apparatus according to claim 1, wherein the image discriminating apparatus discriminates that the moving image is not a moving image.   When the mouth area is detected from the frame, the detection unit determines that the ratio of the size of the detected mouth area to the size of the frame is equal to or less than a predetermined ratio. The image determination apparatus according to claim 1, wherein it is determined that a region has not been detected. Computer
Detect the mouth area from each frame of the input video that shows the scene of the period when the character string was pronounced,
A dictionary indicating the shape of the mouth when the character string is pronounced when the number of frames in which the mouth area is not detected among the frames of the input moving image is equal to or less than a predetermined number Discriminate from moving images to generate information,
An image discrimination method characterized by the above. On the computer,
Detect the mouth area from each frame of the input video that shows the scene of the period when the character string was pronounced,
A dictionary indicating the shape of the mouth when the character string is pronounced when the number of frames in which the mouth area is not detected among the frames of the input moving image is equal to or less than a predetermined number Discriminate from moving images to generate information,
An image discrimination program for executing a process.

Images