JP2005135432A - Image recognition apparatus and image recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image recognition apparatus which can quickly and highly accurately recognize shapes and motions of human lips. <P>SOLUTION: The image recognition apparatus is provided with; an image acquisition part for acquiring a range image stream for an object material; a mouth part extraction part for extracting a mouth part from the range image stream acquired by the image acquisition part; and an image recognition part for recognizing at least shapes or motions of lips on the basis of the range image stream of the mouth part extracted by the mouth part extraction part. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image recognition apparatus and an image recognition method for recognizing the shape and / or movement of an image based on an acquired distance image.

  Conventionally, when performing image processing such as reading a lip by recognizing the shape and movement of a human lip, or determining the face orientation, facial expression, etc., first, using a CCD camera or other imaging device, A pre-processing is performed in which an extra portion such as a background is removed from the image and only an object to be recognized, such as only the lips and only the face, is cut out. Then, by using the processed image, the shape and movement are recognized.

  First, the pre-processing part of extracting the recognition target will be described.

  In the conventional method, in the process of cutting out only the part of the object to be acquired from the image photographed by the camera, the object is cut out by using some difference between the object and the other part as a clue. As a clue, a method using a change in hue, a method using a difference image, a method using a marker, a method using a chroma key, and the like have been used. These will be described by taking as an example a case where only the lip portion is cut out from an image showing a person.

  In the method using the change in hue, a rapid change in hue (pixel value) is used in which the lip portion is almost uniformly red and the surrounding skin portion is substantially uniformly colored. Thus, only the lip portion was discriminated and cut out.

  However, with this method, it is possible to extract skillfully and reliably in an environment that shows a hue different from normal, such as the hue changing due to a shadow on the skin or lip depending on the lighting conditions. There was a problem that it was impossible. In addition, in order to stably obtain the shape of the lips, it may be necessary to emphasize the hue change by using a lipstick of a specific color.

  In the method using the difference image, when the speaker is talking, only the lip part is moving in the face, and the current frame and the next frame are used. By taking a difference image, a moving part is acquired and used as a lip part.

  However, with this method, in an environment where something is moving in the background, unnecessary parts other than the lips are extracted, and when the lips are not moving, extraction is not possible. It was very difficult to always extract only the lip part reliably.

  In the method using the marker, some markers are pasted around the lips as feature points, and the lip portion is extracted based on the movement of the feature points.

  However, this method has a problem in that the environment in which it can be used is limited because a marker or the like must be put on the face.

  In the method using the chroma key, for example, a person is placed in front of a screen of a color that does not appear much on the face of the person, such as blue, and the blue is removed from the image obtained with a camera etc. It was extracted.

  However, in this method, since the background color is compelled, there are problems such that it can be used only in a specific situation and only a part of the inside of the face such as the lips cannot be extracted.

  As described above, according to the conventional method, it is very difficult to reliably cut out only a portion of an object to be acquired from an image captured by a camera.

  Next, a part for recognizing the shape, movement, etc. of the object from the image obtained by cutting out the object will be described.

  Conventionally, an image of a cut object includes only two-dimensional information. This is because it is difficult to acquire a three-dimensional shape with a conventional imaging device, and even if there are imaging devices that acquire a three-dimensional shape, they are not suitable for real-time recognition such as movement. It is. In addition, such a three-dimensional imaging device is very expensive and cannot be used easily. For this reason, in conventional image processing, efforts have been made to somehow recognize originally three-dimensional objects such as the shape and movement of human faces and lips using only two-dimensional information.

  However, since 3D shapes and movements were originally used as 2D information, necessary information was lost, and although various measures were taken, only simple shapes and movements could be recognized. However, it was impossible.

  Further, as described above, since it is very difficult to reliably perform the operation of cutting out only the target object from the image, the uncertainties of the cutting are greatly related to the factors that lower the recognition rate.

  As described above, the conventional method has various problems in both the method for extracting an object from an image and the method for recognizing an image.

  As described above, conventionally, it is very difficult to reliably cut out only a portion of an object desired to be acquired from an image photographed by a camera, which has been a cause of a decrease in the recognition rate of image recognition.

  In addition, because the image was acquired as two-dimensional information using a camera or the like due to various restrictions, the recognition of the three-dimensional shape and the three-dimensional movement can be performed only from the two-dimensional information, and the simple shape and movement can be recognized. There was a problem that could only be done.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image recognition apparatus and an image recognition apparatus method capable of recognizing the shape and movement of a human face and lips with high speed and high accuracy. .

  An image recognition apparatus according to the first aspect of the present invention includes an image acquisition unit for acquiring a distance image with respect to a target object, an oral part extraction unit that extracts an oral part from the distance image acquired by the image acquisition unit, and the oral part And an image recognition means for recognizing the shape of the lips based on the distance image of the oral cavity portion extracted by the extraction means.

  The image recognition apparatus according to the second aspect includes an image acquisition unit for acquiring a distance image stream (a moving image of a distance image) for a target object, and a distance image stream (a moving image of a distance image) acquired by the image acquisition unit. ) And an image recognition means for recognizing at least one of the shape of the lips and the movement of the lips based on the distance image stream of the oral portions extracted by the oral portion extraction means It was characterized by comprising.

  According to the present invention, since a necessary part is extracted from a distance image with respect to a target object, and recognition processing is performed based on the extracted distance image of the extracted part, speaker's lip recognition (for example, oral cavity shape, movement, , Etc.) can be performed at high speed and with high accuracy.

  An image recognition apparatus according to a third aspect includes an image acquisition unit for acquiring a distance image for a target object, a face part extraction unit for extracting a face part from the distance image acquired by the image acquisition unit, and the face part And image recognition means for recognizing the shape of the face based on the distance image of the face portion extracted by the extraction means.

  An image recognition apparatus according to a fourth aspect of the present invention includes an image acquisition unit for acquiring a distance image stream (a moving image of a distance image) for a target object, and a distance image stream (a moving image of a distance image) acquired by the image acquisition unit. ), And an image recognition unit for recognizing at least one of the shape of the face and the movement of the face based on the distance image stream of the face part extracted by the face part extraction unit. It was characterized by comprising.

  According to the present invention, since a necessary part is extracted from a distance image with respect to a target object and recognition processing is performed based on the extracted distance image of the extracted part, speaker face recognition (for example, face shape, motion, etc.) Can be performed at high speed and with high accuracy.

  According to a fifth aspect of the present invention, in the image recognition device according to any one of the first to fourth aspects, the direction of the speaker's face is determined based on the shape information or the movement information obtained by the image recognition means. It further comprises direction identifying means for identifying.

  According to a sixth aspect of the present invention, in the image recognition apparatus according to the first or second aspect of the present invention, the direction identifying means for identifying the direction of the speaker's face based on the lip shape or lip movement recognized by the image recognizing means. Is further provided.

  According to a seventh aspect of the present invention, in the image recognition apparatus according to the third or fourth aspect, the direction identifying means for identifying the orientation of the speaker's face based on the face shape or the face movement acquired by the image acquiring means. Is further provided.

  The present invention 8 is the image recognition apparatus according to the first or second invention, wherein the face part extracting unit extracts a face part from the distance image acquired by the image acquiring unit, and the face part extracted by the face part extracting unit And a direction identification means for identifying the direction of the speaker's face based on the distance image.

  According to the present invention, it is possible to identify the direction in which the speaker is facing along with the lip recognition or face recognition of the speaker.

  A ninth aspect of the present invention is the image recognition apparatus according to the first, second, or eighth aspect, wherein the speech recognition means for recognizing the input voice and the start of the conversation of the speaker based on the recognition result by the image recognition means When the end of the conversation of the speaker is detected based on the control for starting the speech recognition by the speech recognition unit and the recognition result by the image recognition unit when the speech recognition unit is detected, the speech recognition by the speech recognition unit is terminated. Control means for performing at least one of the controls is further provided.

  According to the tenth aspect of the present invention, in the image recognition apparatus according to the fifth or eighth aspect, the voice recognition means for recognizing the input voice and the voice by the voice recognition means when the identification result by the direction identification means is the front. Control means for performing at least one of control for starting recognition and control for ending speech recognition by the speech recognition means when the identification result by the direction identification means is not the front is further provided.

  According to the present invention, voice recognition can be controlled in accordance with the lip recognition result or the direction in which the speaker is facing, along with the speaker's lip recognition or face recognition.

  According to the eleventh aspect of the present invention, in the image recognition apparatus according to the first, second, eighth, or ninth aspects, predetermined information is obtained by a predetermined output form (sound, image, other forms, or a combination of plural forms). Information presenting means for presenting and detecting at least one of the start and end of the conversation of the speaker based on the recognition result by the image recognizing means, and according to the detection result, the information presenting means by the information presenting means At least one of the control for starting the information, the control for ending the information presentation by the information presentation means, and the control for changing at least a part of the output form used for the information presentation performed by the information presentation means And a control means for performing the above.

  The present invention 12 presents predetermined information in a predetermined output form (sound, image, other forms, or a combination of plural forms) in the image recognition apparatus according to the fifth, eighth or tenth invention. Information presentation means, control for starting information presentation by the information presentation means and control for ending information presentation by the information presentation means in accordance with the relationship between the direction related to the identification result by the direction identification means and the front direction And a control means for performing at least one control among the controls for changing at least a part of the output form used for the information presentation performed by the information presenting means.

  According to the present invention, it is possible to control information presentation according to the lip recognition result and the direction in which the speaker is facing along with the speaker's lip recognition or face recognition.

  The present invention 13 is the image recognition apparatus according to the first, second, sixth or eighth invention, wherein the speech recognition means for recognizing the input speech and the speaker's recognition result based on the recognition result by the image recognition means. It further comprises voice recognition starting means for detecting the start of conversation (acting by the speaker) and starting voice recognition by the voice recognition means when the start of the conversation is detected.

  According to the fourteenth aspect of the present invention, in the image recognition apparatus according to the first, second, sixth, eighth or thirteenth aspects, the speech recognition means for recognizing the input voice and the recognition result by the image recognition means. And further comprising voice recognition ending means for detecting the end of the conversation of the speaker (execution of the speaker's action) and ending the voice recognition by the voice recognition means when the end of the conversation is detected. To do.

  According to a fifteenth aspect of the present invention, in the image recognition device according to the fifth, sixth, seventh, or eighth aspects, the speech recognition means for recognizing the input speech and the identification result by the direction identifying means are in front And voice recognition starting means for starting voice recognition by the voice recognition means.

  According to a sixteenth aspect of the present invention, in the image recognition device according to the fifth, sixth, seventh, eighth, or fifteenth aspects, the voice recognition means for recognizing the input voice and the identification result by the direction identification means are in front ( And voice recognition end means for ending voice recognition by the voice recognition means when it is not a speaker's action).

  Invention 17 is an image recognition apparatus according to Invention 1, Invention 2, Invention 6, Invention 8, Invention 13 or Invention 14, and a predetermined output form (sound, image, other form, or a combination of plural forms) Information) presenting means for presenting predetermined information and the start of the conversation of the speaker (execution of the speaker) is detected based on the recognition result by the image recognition means, and the start of the conversation is detected And an information presentation start means for starting the information presentation by the information presentation means.

  According to the eighteenth aspect of the present invention, in the image recognition device according to the first, second, sixth, eighth, thirteenth, fourteenth, or eighteenth aspects, a predetermined output form (sound, image, other form, or a plurality of forms) Information presenting means for presenting predetermined information by a combination of the above and the end of the speaker's conversation (execution of the speaker's action) is detected based on the recognition result by the image recognition means, and the end is detected And an information presentation end means for ending the information presentation by the information presenting means when the information is presented.

  In the image recognition device according to the fifth, sixth, seventh, eighth, fifteenth, or sixteenth invention, the present invention 19 is a predetermined output form (sound, image, other form, or a combination of plural forms) Information presentation means for presenting predetermined information, and information presentation start for starting information presentation by the information presentation means when the identification result by the direction identification means is front (acting by the speaker) And a means.

  In the image recognition apparatus according to the fifth, sixth, seventh, eighth, fifteenth, sixteenth, or twentyth invention, the present invention 20 has a predetermined output form (sound, image, other form, or plural forms). Information presenting means for presenting predetermined information by information) and information for terminating the information presentation by the information presenting means when the identification result by the direction identifying means is not the front (acting of the speaker) And a presentation ending unit.

  Invention 21 is an image recognition apparatus according to Invention 1, Invention 2, Invention 6, Invention 8, Invention 13, Invention 14, Invention 17 or Invention 18, in a predetermined output form (sound, image, or other form, or Information presenting means for presenting predetermined information by combining a plurality of forms) and detecting the start of the conversation of the speaker (performing the speaker's action) based on the recognition result by the image recognition means, It further comprises information presentation switching means for switching information presentation by the information presentation means to information presentation in a different output form when the start of conversation is detected.

  Invention 22 is an image recognition apparatus according to Invention 1, Invention 2, Invention 6, Invention 8, Invention 13, Invention 14, Invention 17, Invention 18 or Invention 21, in a predetermined output form (sound, image, or other). Information presenting means for presenting predetermined information in a form or a combination of a plurality of forms) and detecting the end of the conversation of the speaker (execution of the speaker's action) based on the recognition result by the image recognition means And an information presentation switching means for switching the information presentation by the information presentation means to the information presentation in a different output form when the end of the conversation is detected.

  In the image recognition apparatus according to the fifth, sixth, seventh, eighth, fifteenth, sixteenth, nineteenth, or twentyth invention, the present invention 23 is a predetermined output form (sound, image, or other form, or Information presenting means for presenting predetermined information by combining a plurality of forms) and information by the information presenting means when the identification result by the direction identifying means is the front (acting of the speaker) It further comprises information presentation switching means for switching the presentation to the information presentation in an output form different from the previous one.

  The present invention 24 is an image recognition apparatus according to the invention 5, the invention 7, the invention 8, the invention 15, the invention 16, the invention 19, the invention 20, or the invention 23. Information presenting means for presenting predetermined information in the form or a combination of plural forms), and the identification result by the direction identifying means during the information presenting by the information presenting means is the front (acting of the speaker) The information presenting means further comprises information presenting switching means for switching the information presenting by the information presenting means only to the information presenting in the output form different from the previous one.

  According to a twenty-fifth aspect of the present invention, in the image recognition apparatus according to any one of the first to twelfth aspects of the present invention, the image recognition apparatus further includes a communication unit for communicating the obtained predetermined information.

  According to the present invention, desired information such as a recognition result can be given to an external device.

  According to a twenty-sixth aspect of the present invention, in the image recognition apparatus according to the first or second aspect of the present invention, the image recognition apparatus further comprises a communication means for communicating lip shape information or lip movement information recognized by the image recognition means. And

  According to a twenty-seventh aspect of the present invention, in the image recognition apparatus according to the third or fourth aspect of the present invention, the image recognition apparatus further includes a communication unit for communicating information on a face shape or face movement recognized by the image recognition unit. And

  According to a twenty-eighth aspect of the present invention, in the image recognition apparatus according to the fifth, sixth, seventh or eighth aspect of the present invention, the image recognition apparatus further comprises a communication means for communicating information about a speaker's face orientation identified by the direction identification means. It is characterized by that.

  The present invention 29 is the image recognition apparatus according to the ninth, tenth, eighth, ninth, tenth, or eleventh invention, further comprising a communication means for communicating a recognition result by the voice recognition means. And

  The present invention 30 is the information presented by the information presenting means in the image recognition device according to the invention 11, the invention 12, the invention 12, the invention 14, the invention 15, the invention 16, the invention 17, the invention 18, or the invention 19. It further has a communication means for communicating.

  In each of the present invention described above, a configuration in which the image acquisition unit is omitted and a distance image with respect to the target object is given from the outside is also established.

  The image recognition method according to the present invention 31 is characterized in that an oral part is extracted from a given distance image with respect to a target object, and the shape of the lips is recognized based on the extracted distance image of the oral part. .

  The image recognition method according to the thirty-second aspect of the present invention extracts a mouth part from a given distance image stream with respect to a target object, and based on the extracted distance image stream of the mouth part, at least lip shape and lip movement It is characterized by recognizing one.

  The present invention 33 is a computer that records a program including a procedure for causing a computer to extract an oral part from a distance image with respect to a given target object and to recognize the shape of the lips based on the extracted distance image of the oral part. The gist is a readable recording medium.

  The present invention 34 causes a computer to extract a mouth part from a distance image stream for a given target object, and to recognize at least one of the shape of the lips and the movement of the lips based on the extracted distance image stream of the mouth part. A gist is a computer-readable recording medium on which a program including the above procedure is recorded.

  The present invention relating to the apparatus is also established as an invention relating to a method, and the present invention relating to a method is also established as an invention relating to an apparatus.

  Further, the present invention relating to an apparatus or a method has a function for causing a computer to execute a procedure corresponding to the invention (or for causing a computer to function as a means corresponding to the invention, or for a computer to have a function corresponding to the invention. It can also be realized as a computer-readable recording medium on which a program (for realizing) is recorded.

  According to the present invention, since a necessary part is extracted from a distance image with respect to a target object and recognition processing is performed based on the extracted distance image of the extracted part, the shape and movement of a human face and lips can be detected at high speed and with high accuracy. Can be recognized.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
First, a first embodiment of the present invention will be described.

  FIG. 1 is an overall configuration diagram of an image recognition apparatus according to a first embodiment of the present invention.

  The image recognition apparatus according to the present embodiment includes an image acquisition unit 1 for acquiring a distance image stream, and an oral cavity that extracts only the oral cavity part from all or part of the distance image stream of the face acquired by the image acquisition unit 1. A part extraction unit 2 and an image recognition unit 3 for recognizing the shape of the lips and / or the movement of the lips from the extracted distance image stream of the oral cavity.

  The image acquisition unit 1 sets all or part of a human face as an image recognition target object as an image having a depth value reflecting the three-dimensional shape (hereinafter referred to as a distance image) at predetermined time intervals (for example, 1 / (For example, it can be realized by using the image acquisition method of Japanese Patent Application No. 9-299648). For example, the image acquisition unit 1 schematically irradiates light on a target object, extracts a spatial intensity distribution of reflected light from the target object, and sets the intensity value of each pixel to a value indicating depth or distance. A distance image is generated by the conversion. By capturing an image of a face using the image acquisition unit 1, a moving image (hereinafter referred to as a distance image stream) based on a distance image of all or part of the face can be obtained. Details of the image acquisition unit 1 will be described later.

  FIG. 2 shows an example of the face distance image (for one frame in the distance image stream) acquired by the image acquisition unit 1. The distance image is a three-dimensional image having depth information, and is, for example, an image having 64 pixels in the x-axis (horizontal) direction, 64 pixels in the y-axis (vertical) direction, and 256 gradations in the z-axis (depth) direction. FIG. 2 represents the distance value of the distance image, that is, the gradation in the z-axis direction in gray scale. In the distance image, the closer the color is to white, the closer the distance is, and the closer the color is to black, the farther the distance is. Further, a place where the color is completely black indicates that there is no image, or even if it is present, it is the same as not far away. For example, FIG. 2 shows a state where the lip portion is white and the oral portion inside thereof is black.

  Note that the light-receiving surface in the image acquisition unit 1 or the housing that houses the light-receiving surface may be appropriately installed according to the purpose of the image recognition apparatus. For example, when the image recognition apparatus has a display device, when a human face as a target object faces the front with respect to the display device, the light receiving surface also faces the front. Thus, it is provided in the housing of the image recognition apparatus.

  Next, the oral cavity extraction unit 2 will be described.

  The oral cavity extraction unit 2 extracts only the oral cavity from the distance image stream of all or part of the face acquired by the image acquisition unit 1.

  When the peripheral portion of the human lip is viewed three-dimensionally, the local shape varies depending on the person, and the same person has various shapes depending on the situation. However, as a general rule, there is a characteristic that is uniquely determined regardless of a person or a situation, that “the lip portion has a slightly convex shape and the oral cavity portion inside thereof has a large concave shape”.

  FIG. 3 is a three-dimensional view of the face distance image when the lips are closed, and FIG. 4 is a three-dimensional view of the face distance image when the lips are open. 3 and 4, it can be seen that the three-dimensional features of the oral cavity as described above can be clearly seen.

  If the feature of the three-dimensional shape of the oral cavity is positively used, it is easy to construct a distance image stream in which only the oral cavity is extracted from the facial distance image stream.

  Below, how the oral cavity part is extracted by the oral cavity extraction part 2 is demonstrated concretely.

  A distance image (hereinafter also referred to as an original image) acquired by the image acquisition unit 1 represents a three-dimensional shape of a face. By obtaining a second-order differential image of this distance image, it is possible to know the state of inclination change in the original image. If this is used, the edge portion of the original image can be extracted. Note that the edge here refers to a portion having a change in inclination such as a boundary between the face and the background or a boundary between the lips and the skin.

  FIG. 5 shows an example of a specific processing flow for edge extraction.

  First, a Gaussian Laplacian filter proposed by Marr-Holdreth is applied to the original image (step S100).

  Next, the zero cross point is obtained (step S101). At this time, for example, a point where the pixel value in the vicinity of 4 of the target pixel is positive may be set as the zero cross point.

  If it is a zero cross point, a Sobel operator ((a) in the figure corresponds to the X direction and (b) corresponds to the Y direction) as shown in FIG. 6 is performed to determine the intensity of the pixel (step S102). ).

  If this intensity is greater than or equal to a certain threshold value, it is regarded as a constituent point of the edge (step S103).

  With the above processing, only the edge portion can be extracted from the original image.

  Here, as a method of edge extraction, a method using a Gaussian Laplacian filter and a Sobel operator has been described. However, the method is not limited to this, and can be realized by using another method such as a method using Hough transform. May be.

  By applying the processing described above to the distance image, only the edge portion can be extracted from the distance image of the face. Furthermore, by using this edge information and information on the shape of the lip (the largest one among the loop-shaped (hole) edges), only the edge of the lip can be extracted.

  In this method, the edge is extracted based on the actual three-dimensional shape of the face. Therefore, compared to the conventional method of extracting an edge from a two-dimensional image using a change in hue, the edge is extracted. Only the oral cavity can be reliably cut out without erroneous recognition (excess edge extraction). This is because the three-dimensional shape is deeply related to the actual edge, whereas the method using the hue change is a means for judging the portion having a different hue as the edge, but it is not definitive. It is.

  With the above processing, it is possible to acquire a distance image stream of only the lips from the distance image stream of the face.

  In addition, although the method of using the change of inclination was demonstrated here as a method of extracting an oral cavity part from the distance image of a face, it is not limited to this. For example, using the geometric shape of the “dent” of the oral cavity (depth Z value is below a certain value), the “dent” portion may be extracted by providing a threshold, or geometric inference You may extract by doing. Alternatively, a “dent” -shaped template for the oral cavity may be prepared in advance and obtained by pattern matching with the template. An edge can also be obtained by performing a filtering process using a bandpass filter using distance information. An extraction method using another three-dimensional shape may be used.

  Next, the image recognition unit 3 will be described.

  The image recognizing unit 3 recognizes the shape and / or movement of the lips based on the distance image stream of the oral cavity extracted by the oral cavity extracting unit 2.

  First, recognition of the shape of the lips will be described.

  In the image recognizing unit 3, various templates such as “A”, “I”,... Are prepared in advance, and these are compared with the shape of the lip obtained by the oral part extracting unit 2 to determine the similarity. Recognition is performed using template matching or the like in which the highest similarity is calculated and adopted as the recognition result.

  FIG. 7 shows an example of the template matching processing flow in the image recognition unit 3.

  First, the extracted distance image (original image) of the oral cavity is normalized in accordance with the direction and size of the template (step S200).

  Next, a template k to be compared with the original image is selected from various prepared templates (step S201).

Next, the Hamming distance between the original image and the template is calculated (step S202). The Hamming distance (H) is calculated by, for example, H = Σ _i Σ _j | d (i, j) −t _k (i, j) | Here, i and j are the x and y coordinates of each pixel, d (i, j) is the distance value at the coordinates (i, j) of the original image, and t _k (i, j) is the coordinates of the template k ( i, j) is the distance value.

  Here, although one method of deriving the Hamming distance has been described, the deriving of the Hamming distance is not limited to this, and other calculation formulas may be used.

  Since these processes are performed for all templates, it is determined whether the calculation of the Hamming distance is completed for all templates (step S203).

  If there is a template for which the calculation of the Hamming distance is not yet completed, the process returns to step S201.

  When calculation of the Hamming distance from the original image is completed for all templates, they are compared to find the template with the smallest value. Then, the content expressed by this template is set as a recognition result (step S204). For example, if the selected template has a lip shape when “ta” is pronounced, it is recognized that the pronunciation (lip shape) of the distance image of the original image is “ta”.

  The speaker's utterance content is recognized by sequentially performing the above processing on all the distance images included in the distance image stream.

  In the following, in order to distinguish it from speech recognition, the utterance content of the speaker is recognized from the lip shape (the person who is the recognition target does not actually make a voice, just like when speaking actually (Including recognition based on the distance image obtained when moving) is called lip recognition.

  Next, recognition of lip movement will be described.

  When recognizing the movement of the lips, for example, a sequence of templates representing movements such as “open and closed mouth”, “yawning” (divide the movement into each frame, 1 each As a template, a series of motion templates is prepared), and in the same manner as described above, template matching is sequentially performed with the template column for all the distance images included in the distance image stream. By doing so, it is also possible to perform lip recognition for movement.

  Unlike the conventional image recognition, the result of lip recognition obtained by the above method is the result of recognition using the actual three-dimensional shape of the lips. Conventionally, since recognition was performed using a two-dimensional lip shape extracted from an image of a normal video camera or the like, the recognition had to be performed only from the planar movement of the lips. Since three-dimensional information can be used, more information can be recognized than in the past. Therefore, the image recognition apparatus according to the present embodiment is used even in the case where recognition is impossible in the past, such as when the lip shape when viewed from the front is almost the same and the shape of the lips in the depth direction is different. Can be recognized. In addition, since there are more clues to identify, there is an advantage that the recognition rate is higher than the conventional one, and it is difficult to make erroneous recognition.

  Here, the method of calculating the similarity between the original image and the template by obtaining the Hamming distance between the original image and the template has been described. However, the calculation of the similarity is not limited to this. Any method can be used, such as a method using a DP matching method, a KL conversion method, or the like, or a method of calculating a degree of similarity by Fourier-transforming an original image and obtaining a correlation with respect to the image after the Fourier transform.

  In addition, here, as a method of recognizing the shape and movement of the lips from the distance image stream of the oral cavity, the method of performing template matching has been described, but the present invention is not limited to this, for example, from the shape of the lips, Recognition may be performed by other methods such as obtaining the movement of the muscle and using the shape change as a clue to analogize the pronunciation content from the muscle model.

  As described above, according to the present embodiment, by using the lip distance image, it is possible to easily extract the lip portion without much calculation cost. Furthermore, with regard to lip recognition, by using the extracted three-dimensional shape information of the lip, it is impossible to recognize the shape, which was difficult to distinguish with the conventional method (there were many misrecognitions), or impossible with the conventional method. It is possible to recognize the shape as if it were.

  The lip shape recognition result, the lip movement recognition result, or the combination of the lip shape recognition result and the lip movement recognition result obtained as described above can be used for various subsequent processing. it can. It should be noted that whether the image recognition unit 3 is provided with both functions of lip shape and movement recognition or either one can be appropriately designed according to the purpose of the system.

  The present embodiment is not limited to the configuration described above, and can be implemented with various modifications. Below, some modifications of this embodiment are shown.

(Modification 1 of the first embodiment)
Instead of the lip extraction unit 2, a face extraction unit for extracting only the face part from the distance image stream obtained by the image acquisition unit 1 may be provided.

  The image recognition unit 3 prepares a template of a person's face shape such as Mr. A and Mr. B in advance, and uses the distance image of the face part extracted by the face part extraction unit 5 using them. By performing matching, it is possible to recognize who is the person imaged by the image recognition apparatus of the present embodiment.

  Thereby, for example, by placing the image recognition device of this embodiment (or at least the light emitting element and the light receiving element part of the image acquisition unit 1) near an automatic door or the like and recognizing the face of a person passing there, It can be used for simple security checks, such as opening the door only when it recognizes a specific person.

(Modification 2 of the first embodiment)
This embodiment is also effective for nursing a seriously ill patient from the medical aspect. Conventionally, when a patient in a hospital room or home of a home caregiver has something abnormal, a push button type buzzer at the bedside informs the nurse or doctor. However, the push button type buzzer is dangerous because there is often no room for pressing the button when the patient is weak. By placing the image recognition apparatus of the first embodiment in such a place, even if it is weak because of illness and cannot speak so much, the sick person's slight voice and subtle lip movement Can tell you what you want to convey.

  If you push this forward and a sick person who does not normally move the lips moves the lips, the medical condition may have suddenly changed. In such a case, the movement of the lips can be converted into some sound and used as a warning sound, so that doctors and nurses can take measures to rush to hospital rooms and home nurse homes. it can.

  In such a case, as illustrated in FIG. 8, the sound presenting unit 4 is provided that directly converts the lip recognition result into a sound and presents it or presents some sound according to the result.

(Modification 3 of the first embodiment)
As illustrated in FIG. 9, a face extraction unit 5 for extracting a distance image stream of only the face is added to the configuration of the second modification of the first embodiment (FIG. 8). By using the 3D shape information of the face, for example, waving the face up and down, recognizing the gesture of the face, laughing, angry, troubled, etc. It is possible to recognize facial expressions.

  At that time, the image recognition unit 3 may, for example, whisper: shake the face several times up and down, refuse: shake the face several times to the left and right, rejoice: make a big mouth, narrow eyes, surprise: open eyes, etc. In this way, templates for obtaining gestures and facial expressions are prepared, and template matching is performed using them to recognize facial gestures and facial expressions.

  Then, according to the recognized facial expression, when voice conversion is performed on the movement of the lips, it is possible to change the type and pitch of the voice to be converted.

  Also, for example, even if the movement of the lips is the same, it is possible to change such as a voice without a dog if affirmative, a cry of a chicken if negative, or a voice without a cat if happy. In this way, for example, it becomes possible to allow children to study English word generation and the like so as not to get bored.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the present embodiment, description will be made centering on portions that differ from the first embodiment.

  FIG. 10 is an overall configuration diagram of an image recognition apparatus according to the second embodiment of the present invention.

  As shown in FIG. 10, the image recognition apparatus according to the present embodiment has a recognition result of the shape or movement of the lips obtained by the image recognition unit 3 with respect to the configuration of the image recognition apparatus according to the first embodiment. In addition, a direction identification unit 6 for identifying the direction in which the speaker's face is facing is added.

  This makes it possible to recognize not only the content of the speaker's speech but also the direction in which the speaker is speaking.

  Next, the direction identification unit 6 will be described.

  The direction identification unit 6 identifies the direction in which the speaker's face is facing based on the recognition result of the lip shape or movement obtained by the image recognition unit 3. At that time, the direction of the speaker's face is calculated by using the three-dimensional shape of the lips.

  In the following, an example of a specific method for obtaining the direction in which the speaker's face is facing will be described using the flowchart of the processing shown in FIG.

  First, a certain pixel X (for example, coordinate value (i, j)) in the lip distance image is selected (step S300).

  Next, a pixel Y (for example, coordinate values (i−1, j)) adjacent to the pixel X is selected (step S301).

  Next, as shown in FIG. 12 ((a) is a diagram showing eight adjacent pixels, (b) is a diagram for explaining a gradient vector g and a normal vector p orthogonal thereto) Based on the difference d (i, j) −d (i−1, j) of the distance value from the coordinate value (i−1, j)), an inclination vector g between the two pixels is obtained (step 302). .

  A normal vector p that is on the same plane as the two pixels X and Y and is orthogonal to the gradient vector g obtained in step S302 is obtained (step S303).

  It is determined whether the normal vector calculation has been completed for all the pixels Y adjacent to the pixel X (step S304).

  If the process has not been completed for all adjacent pixels, the process returns to step S301. If all the processing has been completed, the average P = Σp of the normal vectors is calculated and set as the normal vector P of the pixel X (step S305).

  It is determined whether or not the above processing has been performed for all the pixels in the distance image (step S306). If not, the process returns to step S300.

When the calculation of the normal vector P is completed for all the pixels, the average P _lip = Σp of the normal vectors of each pixel is calculated, and this is set as the normal vector of the lips (step S307).

Since the lip is substantially at the center of the face and has a substantially symmetrical shape, the direction of the normal vector of the lip and the direction of the normal vector of the face are approximately the same. Therefore, P _lip obtained in step S307 becomes the normal vector of the face. That is, the direction in which the speaker is facing can be identified using the normal vector P _lip as the face direction.

  Here, as a means for obtaining the direction in which the lips are facing, the method of calculating the normal vector of the lips from the distance image has been described, but the present invention is not limited to this, and the ratio of the size of the lips, Other methods such as analogizing the direction of the lips from the change in shape may be used.

  As described above, according to the present embodiment, it is possible to simultaneously recognize which direction the speaker is facing, what kind of talk he is talking about, what kind of lip movement, etc. Is possible.

  The present embodiment is not limited to the configuration described above, and can be implemented with various modifications. Below, some modifications of this embodiment are shown.

(Modification 1 of 2nd Embodiment)
As shown in FIG. 13, instead of the oral cavity extraction unit 2, a facial part extraction unit 5 for extracting only the facial part from the distance image stream of all or part of the face acquired by the image acquisition unit 1 is placed. Also good. In this case, the distance image stream of the face part extracted by the face part extraction part 5 is input to the image recognition part 3.

  Then, in the image recognition unit 3, for example, whisper: shake the face several times up and down, refuse: shake the face several times left and right, rejoice: big mouth, narrow eyes, surprise: open eyes, etc. Prepare a template to obtain gestures and facial expressions, and use them to perform template matching with the distance image stream of the input face. It is possible to recognize facial expression changes such as being surprised or troubled.

  The direction identification unit 6 identifies the direction in which the speaker's face is facing based on the recognition result of the shape and movement of the face obtained by the image recognition unit 3.

  By deforming in this way, it is possible to recognize in which direction the target person is facing and what kind of facial movement (such as a gesture or facial expression change).

(Modification 2 of the second embodiment)
In the second embodiment, the direction of the speaker is identified by the direction identification unit 6 based on the recognition result of the image recognition unit 3, but the image acquisition unit 1 as shown in FIG. A face extraction unit 5 for newly extracting only the face portion from the acquired distance image stream of the face (this includes the background etc.) is newly added, and the face extracted by the face extraction unit 5 Based on the distance image stream, the direction identification unit 6 may identify the direction in which the speaker is facing. In this case, the direction identification unit 6 calculates the normal direction of the face (for example, the average of the normal directions of the pixels constituting the face) from the distance image stream of the face extracted by the face extraction unit 5. Get the direction the speaker is facing.

  By doing in this way, in the second embodiment, the direction in which the face is facing is obtained from the direction in which the lips are facing, but since the direction in which the face is facing can be obtained directly, more It is possible to obtain a fine and delicate face orientation.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the present embodiment, description will be made centering on parts that are different from the second embodiment.

  FIG. 15 is an overall configuration diagram of an image recognition apparatus according to the third embodiment of the present invention.

  As shown in FIG. 15, the image recognition apparatus according to the present embodiment has a voice recognition unit 7 that recognizes the content of a speaker's speech in comparison with the configuration of the image recognition apparatus according to the second embodiment or the configuration of a modification thereof. And a voice recognition start unit 8 for instructing the voice recognition unit 7 to start voice recognition based on the direction of the face of the speaker obtained by the direction identification unit 6. It has become.

  This makes it possible to perform speech recognition according to the direction in which the speaker's face is facing.

  Next, the voice recognition unit 7 will be described.

  The voice recognition unit 7 recognizes the content of voice input using a voice input device such as a microphone. The voice recognition unit 7 can use various recognition methods. For example, it may be realized using a hidden Markov model. By performing speech recognition, it is possible to recognize the content of the speaker's conversation.

  Next, the voice recognition start unit 8 will be described.

  The voice recognition start unit 8 instructs the voice recognition unit 7 to start voice recognition based on the result obtained by the direction identification unit 6. Here, for example, when the speaker faces the front (with respect to the image recognition apparatus of the present embodiment (that is, with respect to the light receiving element portion of the image acquisition unit 1; hereinafter the same)), the speaker's action is At this time, the voice recognition unit 7 is instructed to start voice recognition.

  As described above, according to the present embodiment, it is possible to start speech recognition according to the operation of the speaker. For example, voice recognition can be started when the speaker is facing the front (relative to the image recognition device of the present embodiment).

  Further, according to the present embodiment, since the result of lip recognition (lip reading) by the image recognition unit 3 can also be obtained, it is possible to perform voice recognition and lip recognition (lip reading) at the same time, and combine the results of these two recognitions. By using this method, a recognition result having a higher recognition rate can be obtained for the conversation contents of the speaker.

  This is very effective in the following situations. For example, in a place where there is a lot of noise such as a construction site where it is difficult to hear the voice, the recognition rate decreases when only voice recognition is performed, and depending on the case, it may not be recognized at all, but as in the third embodiment. If the lip recognition is performed at the same time, the lip recognition is not affected by noise, so that the recognition rate does not decrease, and a high recognition rate can be maintained as a whole. Also, even in a quiet place such as a library where you cannot make a loud voice, speech recognition alone must be recognized with a small amount of speech, so the recognition rate may be reduced. If done, a high recognition rate can be maintained as a whole.

  In addition, when two people are talking, in the conventional voice recognition, it is difficult to determine a recognition target because a plurality of voices are input at the same time. Among them, for example, it is easy to determine the recognition target and recognize the lips at the same time, such as recognizing only the person facing the front with respect to the image recognition apparatus of the present embodiment. Therefore, the recognition target can be determined using the information.

  The present embodiment is not limited to the configuration described above, and can be implemented with various modifications. Below, some modifications of this embodiment are shown.

(Modification 1 of 3rd Embodiment)
In the third embodiment, the example in which the voice recognition unit 7 and the voice recognition start unit 8 are placed and the voice recognition is started based on the result obtained by the direction identification unit 6 has been described. However, the third embodiment is limited to this. It is not limited to speech recognition, and any other recognition means may be used.

(Modification 2 of the third embodiment)
In the third embodiment, an example of using the voice recognition start instruction according to the direction of the speaker's face is shown. However, as shown in FIG. A speech recognition end unit 9 for instructing the end of speech recognition may be placed in the speech recognition unit 7 and used for instructing the end of speech recognition.

  In such a case, it is possible to end the speech recognition according to the operation of the speaker. For example, the voice recognition can be terminated when the speaker turns away from the face (relative to the image recognition apparatus of the present embodiment).

  Of course, the voice recognition end unit 9 may be further provided in FIG. 15 and used for both the start and end instructions of voice recognition.

(Modification 3 of the third embodiment)
Instead of obtaining the direction in which the speaker's face is directed by the direction identification unit 6 and using it as an instruction to start speech recognition, the recognition result obtained by the image recognition unit 3 is used as shown in FIG. Alternatively, a new voice recognition start unit 8 may be provided for detecting the movement of the lips at the beginning of the conversation and instructing the voice recognition unit 7 to start voice recognition based on the detected movement.

  In this case, the voice recognition start unit 8 starts from the result of the lip recognition obtained by the image recognition unit 3 (the point where the lip starts to move slightly when starting to speak a word, At this time, the voice recognition unit 7 is instructed to start voice recognition.

  Similarly, instead of the voice recognition start unit 8 of the third modification, a voice recognition end unit 9 that detects a point at which the movement of the lips ends may be placed and used for an instruction to end the voice recognition.

  Of course, similarly, in addition to the voice recognition start unit 8 of the third modification, a voice recognition end unit 9 for detecting a point at which the movement of the lips ends is placed and used for both the start and end instructions of voice recognition. Also good.

  In the conventional method, since it takes time to calculate the movement of the lip, it is difficult to use the detection of the movement of the lip for such real-time processing. However, the image recognition apparatus according to the present embodiment is difficult to use. Then, as described in the first embodiment, since it is possible to extract the lip portion without requiring much calculation cost, it is possible to detect such movement of the lip sufficiently in real time.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the present embodiment, description will be made centering on portions that differ from the first embodiment.

  FIG. 18 is an overall configuration diagram of an image recognition apparatus according to the fourth embodiment of the present invention.

  As shown in FIG. 18, the image recognition apparatus according to the present embodiment includes an information presentation unit 10 that presents various types of information and a direction identification unit 6 with respect to the configuration of the image recognition apparatus according to the second embodiment. An information presentation start unit 11 for instructing the information presentation unit 10 to start information presentation based on the obtained direction of the speaker's face is added.

  As a result, various types of information can be presented according to the direction in which the speaker's face is facing.

  Next, the information presentation unit 10 will be described.

  The information presentation unit 10 presents some information to the target person (speaker). The information presenting unit 10 includes at least one information presenting device such as a display (presenting images, characters, etc.), a speaker (presenting sound), a force feedback device (presenting touch), and the like. Information can be presented.

  Next, the information presentation start part 11 is demonstrated.

  The information presentation start unit 11 plays the same role as the voice recognition start unit 8 in the third embodiment described above. Based on the result obtained by the direction identification unit 6, the information presentation unit 10 An instruction to start the presentation is given.

  According to this embodiment, it is possible to start information presentation according to the operation of the speaker. For example, when the speaker faces the front (with respect to the image recognition apparatus of the present embodiment), it can be regarded as the start of the speaker's action and information presentation can be started.

  Further, since the result of lip recognition (lip reading) by the image recognition unit 3 is also obtained, it is possible to start presenting information according to the content of the conversation of the speaker.

  The present embodiment is not limited to the configuration described above, and can be implemented with various modifications. Below, some modifications of this embodiment are shown.

(Modification 1 of 4th Embodiment)
As in the case of the second modification of the third embodiment, an information presentation end unit may be placed instead of the information presentation start unit 11 or in addition to the information presentation start unit 11 to give an instruction to end the presentation.

(Modification 2 of the fourth embodiment)
Similar to the third modification of the third embodiment, the movement of the lip at the beginning of the conversation is detected from the recognition result obtained by the image recognition unit 3, and information is presented to the information presentation unit 10 based on the detected movement. A new information presentation start unit 11 for instructing the start of the information may be placed.

  In this way, for example, by using speech synthesis as a method of presenting information, based on the recognition result of the shape and movement of the lips, the recognition content is provided by speech synthesis, so Even if you can't speak a word, you can just speak to the image recognition device of this embodiment by voice synthesis, just by talking to the mouth (moving your lips as if you are actually speaking) In other words, so-called voice synchronization (lip sync) is possible.

  Of course, as in the case of the third modification of the third embodiment, an information presentation end unit is placed instead of or in addition to the information presentation start unit 11 of this modification, and an instruction to end the presentation is given. You may do it.

(Modification 3 of the fourth embodiment)
As shown in FIG. 19, an information presentation switching unit 12 for switching the type of information to be presented is placed instead of the information presentation starting unit 11, and the information presentation form is switched depending on the direction in which the speaker is facing. May be.

  As switching of this information presentation form, (1) adding different information presentation forms, (2) when providing a plurality of information presentation forms, at least one information presentation form is canceled, (3) When one or a plurality of information presentation forms are provided, some or all of them may be changed to different information presentation forms (including the case where the number of information presentation forms changes). .

  In this way, when the speaker's face is not facing (toward the image recognition apparatus of the present embodiment), the information presentation only by voice is performed, and when the speaker's face is facing, the information presentation switching unit 12, it is possible to switch from presenting only sound to presenting information using composite media such as images in addition to sound.

  For example, to explain an exhibit such as a museum or an art gallery, you usually read the explanation in speech and the visitor looks at the exhibit (or speaks more). It can be used such that the explanation video starts on the display placed next to the exhibit.

(Modification 4 of the fourth embodiment)
By combining the voice recognition unit, the voice recognition start unit, the voice recognition end unit, and the like described in the third embodiment with the fourth embodiment, the raw voice of the speaker and the image information generated by the information presentation unit 10 Can be presented in combination.

  For example, using the distance image stream of the oral cavity extracted by the oral cavity extraction unit 2, the information presentation unit 10 performs three-dimensional CG synthesis on the shape, and the raw speech of the speaker acquired by the speech recognition unit is used. By combining, it is possible to provide a three-dimensional CG in which the lips move in synchronism (lip sync) with the voice of the speaker.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described.

  The image recognition apparatus according to the fifth embodiment includes a communication unit that performs communication with the outside in each of the configurations of the image recognition apparatus according to the first, second, third, or fourth embodiment and various modifications thereof. (Not shown) is added.

  Thereby, the desired information obtained in the first, second, third, or fourth embodiment or its modification can be communicated to the outside.

  The communication unit communicates the input data to the outside using a communication path such as a telephone line. By adding this, for example, in the first embodiment, the result of the lip recognition is changed to the second lip recognition result. In the embodiment, the result of lip recognition and the direction in which the speaker is facing, in the third embodiment, the result of lip recognition and speech recognition, in the fourth embodiment, the result of lip recognition and the presented information Can be communicated with each other.

  As described above, according to the present embodiment, the result obtained by the image recognition apparatus (the lip recognition result based on the first embodiment, the talk based on the second embodiment, The person's direction and lip recognition result, the lip and voice recognition result based on the third embodiment, and the lip recognition result and presentation information (based on the fourth embodiment) are communicated via the Internet or the like. Is possible.

  For example, in the case of the fourth modification of the fourth embodiment, a three-dimensional CG in which the lips move in synchronism with the voice of the speaker (lip sync) and the lips move is obtained. Send it to the other party, send the speaker's remarks and send only the lip of the 3D CG in real time using the communication unit, and combine it with the face sent in advance at the destination. It is possible to perform three-dimensional CG audio synchronization (lip sync) without applying a load on the road (that is, without making the communication path a bottleneck). This is very effective when performing real-time processing using relatively large data such as voice and CG on the Internet or the like where speed bottlenecks are likely to occur in the communication path.

  Below, the structure of the image acquisition part 1 in each above embodiment is demonstrated in detail.

  FIG. 20 shows a configuration example of the image acquisition unit 1. The image acquisition unit 1 includes a light emitting unit 101 for irradiating a target object with light, a reflected light extraction unit 102 for extracting reflected light from the target object as an image, and information on the reflected light imaged. The distance image generating unit 103 for generating the distance image and the timing control unit 104 for controlling the operation timing of each of these units are configured.

  The light emitting unit 101 has a light emitting element and emits light whose intensity varies with time according to the timing signal generated by the timing control unit 104. The light emitted from the light emitting unit 101 is reflected by the target object in front of the light emitting element of the light emitting unit 101 and then enters the light receiving surface of the reflected light extracting unit 102.

  The reflected light from the object decreases significantly as the distance of the object increases. When the surface of the object scatters light uniformly, the amount of received light per pixel of the reflected light image decreases in inverse proportion to the square of the distance to the object. Therefore, when an object is present in front of the light receiving surface, the reflected light from the background becomes small enough to be ignored, and a reflected light image from only the object can be obtained.

  For example, when a human face portion exists in front of the light receiving surface, a reflected light image from the face is obtained. At this time, each pixel value of the reflected light image represents the amount of reflected light received by the unit light receiving unit corresponding to the pixel. The amount of reflected light is affected by the nature of the object (specularly reflecting, scattering, absorbing, etc.), the direction of the object, the distance of the object, etc. In some cases, the amount of reflected light is closely related to the distance to the object. Since the face and the like have such properties, the reflected light image when the face is the target object reflects the three-dimensional shape of the face, the distance of the face, the inclination of the face (partly different distance), and the like. .

  The reflected light extraction unit 102 has a light receiving element that detects the amount of light arranged in a matrix, and extracts a spatial intensity distribution of reflected light from the target object of light emitted from the light emitting unit 101. Since the spatial intensity distribution of the reflected light can be grasped as an image, it is hereinafter referred to as a reflected light image.

  Here, in the light receiving element of the reflected light extraction unit 102, it is generally assumed that not only reflected light from the target object of the light emitted from the light emitting unit 101 but also external light such as illumination light and sunlight is simultaneously received. The Therefore, in the reflected light extraction unit 102 of this configuration example, the difference between the amount of light received when the light emitting unit 101 emits light and the amount of light received when the light emitting unit 101 does not emit light is taken. Thus, only the component of the reflected light from the target object of the light from the light emitting unit 101 is extracted. The timing of this light reception is also controlled by the timing control unit 104.

  Then, the reflected light amount (analog signal) corresponding to each pixel of the reflected light image after external light correction obtained by the reflected light extraction unit 102 is amplified as necessary, and then A / D converted, thereby digitizing. A reflected light image is obtained.

  The distance image generation unit 103 converts the received light amount value (digital data) of each pixel of the reflected light image obtained by the reflected light extraction unit 102 into a distance value, thereby obtaining a distance image (for example, 64 pixels × 64). Pixel, 256 gradation image).

  Next, FIG. 21 shows a more detailed configuration example of the image acquisition unit 1.

  The light emitted from the light emitting unit 101 is reflected by the target object 106 and imaged on the light receiving surface of the reflected light extracting unit 102 by the light receiving optical system 107 such as a lens.

  The reflected light extraction unit 102 detects the intensity distribution of the reflected light, that is, the reflected light image. The reflected light extraction unit 102 includes a first light receiving unit 121 and a second light receiving unit 122 provided for each pixel (unit light receiving unit), and one for all pixels (or for each of a plurality of pixels or each It is configured using a difference calculation unit 123 provided for each pixel).

  The first light receiving unit 121 and the second light receiving unit 122 receive light at different timings. Then, the timing control unit 104 performs these operations so that the light emitting unit 101 emits light when the first light receiving unit 121 receives light and the light emitting unit 101 does not emit light when the second light receiving unit 122 receives light. Control the operation timing. As a result, the first light receiving unit 121 receives light reflected from the light emitting unit 101 and other external light such as sunlight and illumination light. On the other hand, the second light receiving unit 122 receives only external light. Since the timings at which they are received are different, but close to each other, fluctuations in external light and displacement of the target object during this period can be ignored.

  Therefore, if the difference calculation unit 123 calculates the difference between the image received by the first light receiving unit 121 and the image received by the second light receiving unit 122, only the component of the reflected light from the target object is extracted. When one difference calculation unit 123 is shared by a plurality of pixels, the difference is calculated sequentially.

  In addition, various things can be considered about the actual structure of the 1st light-receiving part 121 of the unit light-receiving part, and the 2nd light-receiving part 122. FIG. For example, instead of providing a light receiving element in each of the first light receiving unit 121 and the second light receiving unit 122, one photoelectric conversion element (for example, a photodiode) is provided for each unit light receiving unit, and the first light receiving unit is provided. A charge storage element (for example, a capacitor) that stores the charge amount corresponding to the amount of received light is used for each of the first light receiving unit 121 and the second light receiving unit 122. One method can be considered.

  As described above, the reflected light extraction unit 102 outputs the reflected light amount of each pixel of the reflected light image after performing external light correction. Here, it is assumed that the amount of reflected light of each pixel is output sequentially.

  The output from the reflection extraction unit 102 is amplified by the amplifier 131, converted into digital data by the A / D converter 132, and then stored as image data in the memory 133. Then, the data accumulated from this memory is read out at an appropriate timing and is given to the distance image generation unit 103.

  The distance image generation unit 103 generates a distance image based on the reflected light image obtained by the reflected light extraction unit 102. For example, the reflected light amount of each pixel of the reflected light image is converted into digital data of a predetermined gradation (for example, 256 gradations). In this conversion, for example, (1) correction for a nonlinear factor that the amount of light received by the light receiving element has nonlinearity with respect to the distance to the target object (inversely proportional to the square of the distance to the target object) is performed. Processing, or (2) processing for correcting variations and nonlinearity of the characteristics of the light receiving elements corresponding to each pixel, or (3) processing for removing background and noise (for example, a level of a pixel having a received light amount equal to or less than a reference value) The key may be set as appropriate.

  When extracting a three-dimensional shape of a face, it is desirable to obtain distance information with high resolution. In this case, it is desirable to use a logarithmic amplifier as the amplifier 131. The amount of light received on the light receiving surface is inversely proportional to the square of the distance to the target object, but when a logarithmic amplifier is used, the output is inversely proportional to the distance. By doing so, the dynamic range can be used effectively.

  Now, in the configuration as described above, assuming that reflected light is obtained for all pixels by one light emission, the second control is performed without light emission → light reception by the first light receiving unit → light emission under the control of the timing control unit 104. Light reception by the light receiving unit → difference calculation → digitalization → distance image generation (or light reception by the second light receiving unit without light emission → light emission → light reception by the first light receiving unit → difference calculation → digitalization → distance image generation) A series of processing is advanced, and thereby a single distance image is obtained. Further, a range image stream can be obtained by repeating this series of processing (for example, every 1/60 seconds).

  The light emitting unit 101 preferably emits near infrared light that is invisible to human eyes. In this way, even if light is irradiated, since humans cannot see the light, it is possible not to feel dazzling. In this case, it is preferable to provide a near-infrared light passing filter in the light receiving optical system. Since this filter passes near-infrared light, which is the emission wavelength, and blocks visible light and far-infrared light, much of the external light can be cut off. However, visible light may be used as long as it is not dazzling to human eyes (for example, the light emission amount is not so large or the optical system does not directly enter human eyes). . A method using electromagnetic waves, ultrasonic waves, or the like is also conceivable.

  In the above description, the difference between the two types of received light amounts due to the difference in the presence or absence of light emission of the light emitting unit 101 is taken as an analog signal as external light correction. However, the difference is calculated after each of the two types of received light amounts is digitized. There is also a way to make it.

  Note that the above-described light-receiving surface or a housing that houses the light-receiving surface may be installed as appropriate according to the purpose of the image recognition apparatus. For example, when the image recognition apparatus has a display device, when a human face as a target object faces the front with respect to the display device, the light receiving surface also faces the front. Thus, it is provided in the housing of the image recognition apparatus.

  It should be noted that the above embodiments and their modifications can be implemented in combination as appropriate.

  Further, in each of the above-described embodiments and modifications thereof or a combination thereof as appropriate, the shape and / or movement is recognized from the distance image stream, or various processes are performed based on the recognition result. However, an embodiment in which the shape is recognized from the distance image or various processes are performed based on the recognition result is also possible.

  Further, each of the above-described embodiments and modifications thereof, or a combination of them as appropriate, omits the image acquisition unit 1 or a portion from which the reflected light image is extracted, and is based on a given distance image or its stream, or A distance image or its stream is generated from a given reflected light image or its stream, and shape and / or movement is recognized based on the generated distance image or its stream, or various types of motions based on the recognition result. It is also possible to configure as an apparatus that performs processing.

  Each of the above functions can be realized as software except for the element portion. Further, the present invention can also be implemented as a machine-readable medium in which a program for causing a computer to execute each procedure or means described above is recorded.

  The present invention is not limited to the embodiment described above, and can be implemented with various modifications within the technical scope thereof.

1 schematically shows a configuration example of an image recognition apparatus according to a first embodiment of the present invention. Diagram for explaining the distance image Diagram for explaining the distance image Diagram for explaining the distance image Flow chart showing the flow of edge extraction processing Illustration for explaining the Sobel operator Flow chart showing the flow of template matching process The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 2 of the 1st Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 3 of the 1st Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the 2nd Embodiment of this invention. Flow chart showing the flow of processing for obtaining the direction of the speaker's face Diagram for explaining the normal direction of a pixel The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 1 of the 2nd Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 2 of the 2nd Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the 3rd Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 2 of the 3rd Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 3 of the 3rd Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the 4th Embodiment of this invention. The figure which shows schematically the structural example of the image recognition apparatus which concerns on the modification 3 of the 4th Embodiment of this invention. The figure which shows the structural example of an image acquisition part. The figure which shows the more detailed structural example of an image acquisition part.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image acquisition part, 2 ... Oral part extraction part, 3 ... Image recognition part, 4 ... Sound presentation part, 5 ... Face part extraction part, 6 ... Direction identification part, 7 ... Voice recognition part, 8 ... Voice recognition start part DESCRIPTION OF SYMBOLS 9 ... Voice recognition end part, 10 ... Information presentation part, 11 ... Information presentation start part, 12 ... Information presentation switching part, 101 ... Light emission part, 102 ... Reflected light extraction part, 103 ... Distance image generation part, 104 ... Timing Signal generating unit 107 ... Light receiving optical system 121 ... First light receiving part 122 ... Second light receiving part 123 ... Difference calculating part 131 ... Amplifier 131, 132 ... A / D converter, 133 ... Memory

Claims (18)

An image acquisition means for acquiring a distance image for the target object;
Oral part extraction means for extracting the oral part from the distance image acquired by the image acquisition means,
An image recognition means for recognizing the shape of the lips based on the distance image of the oral cavity portion extracted by the oral cavity extraction means;
Speech recognition means for recognizing input speech;
Control for starting speech recognition by the speech recognition means when the start of the conversation of the speaker is detected based on the recognition result by the image recognition means and termination of the conversation of the speaker based on the recognition result by the image recognition means An image recognition apparatus comprising: a control unit that performs at least one control of terminating the voice recognition performed by the voice recognition unit when the voice is detected. Image acquisition means for acquiring a distance image stream for the target object;
An oral cavity extraction means for extracting an oral cavity portion from the distance image stream acquired by the image acquisition means;
Image recognition means for recognizing the movement of the lips based on the distance image stream of the oral cavity portion extracted by the oral cavity extraction means;
Speech recognition means for recognizing input speech;
Control for starting speech recognition by the speech recognition means when the start of the conversation of the speaker is detected based on the recognition result by the image recognition means and termination of the conversation of the speaker based on the recognition result by the image recognition means An image recognition apparatus comprising: a control unit that performs at least one control of terminating the voice recognition performed by the voice recognition unit when the voice is detected. An image acquisition means for acquiring a distance image for the target object;
Oral part extraction means for extracting the oral part from the distance image acquired by the image acquisition means,
An image recognition means for recognizing the shape of the lips based on the distance image of the oral cavity portion extracted by the oral cavity extraction means;
Information presenting means for presenting information to be presented to the user using all or part of a plurality of output forms including a sound output form and an image output form;
Based on the recognition result by the image recognition means, at least one of the start and the end of the conversation of the speaker is detected, and control for starting the information presentation by the information presentation means according to the detection result and the information presentation means Image recognition comprising: control for terminating information presentation; and control means for performing at least one control among addition, stop, or change of an output form used for information presentation performed by the information presentation means apparatus. Image acquisition means for acquiring a distance image stream for the target object;
An oral cavity extraction means for extracting an oral cavity portion from the distance image stream acquired by the image acquisition means;
Image recognition means for recognizing the movement of the lips based on the distance image stream of the oral cavity portion extracted by the oral cavity extraction means;
Information presenting means for presenting information to be presented to the user using all or part of a plurality of output forms including a sound output form and an image output form;
Based on the recognition result by the image recognition means, at least one of the start and the end of the conversation of the speaker is detected, and control for starting the information presentation by the information presentation means according to the detection result and the information presentation means Image recognition comprising: control for terminating information presentation; and control means for performing at least one control among addition, stop, or change of an output form used for information presentation performed by the information presentation means apparatus. An image acquisition means for acquiring a distance image for the target object;
A face extraction means for extracting a face portion from the distance image acquired by the image acquisition means;
An image recognition means for recognizing the shape of the face based on the distance image of the face part extracted by the face extraction means;
Speech recognition means for recognizing input speech;
Control for starting speech recognition by the speech recognition means when the start of the conversation of the speaker is detected based on the recognition result by the image recognition means and termination of the conversation of the speaker based on the recognition result by the image recognition means An image recognition apparatus comprising: a control unit that performs at least one control of terminating the voice recognition performed by the voice recognition unit when the voice is detected. Image acquisition means for acquiring a distance image stream for the target object;
Face part extracting means for extracting a face part from the distance image stream acquired by the image acquiring means;
Image recognition means for recognizing the movement of the face based on the distance image stream of the face extracted by the face extraction means;
Speech recognition means for recognizing input speech;
Control for starting speech recognition by the speech recognition means when the start of the conversation of the speaker is detected based on the recognition result by the image recognition means and termination of the conversation of the speaker based on the recognition result by the image recognition means An image recognition apparatus comprising: a control unit that performs at least one control of terminating the voice recognition performed by the voice recognition unit when the voice is detected. An image acquisition means for acquiring a distance image for the target object;
A face extraction means for extracting a face portion from the distance image acquired by the image acquisition means;
An image recognition means for recognizing the shape of the face based on the distance image of the face part extracted by the face extraction means;
Information presenting means for presenting information to be presented to the user using all or part of a plurality of output forms including a sound output form and an image output form;
Based on the recognition result by the image recognition means, at least one of the start and the end of the conversation of the speaker is detected, and control for starting the information presentation by the information presentation means according to the detection result and the information presentation means Image recognition comprising: control for terminating information presentation; and control means for performing at least one control among addition, stop, or change of an output form used for information presentation performed by the information presentation means apparatus. Image acquisition means for acquiring a distance image stream for the target object;
Face part extracting means for extracting a face part from the distance image stream acquired by the image acquiring means;
Image recognition means for recognizing the movement of the face based on the distance image stream of the face extracted by the face extraction means;
Information presenting means for presenting information to be presented to the user using all or part of a plurality of output forms including a sound output form and an image output form;
Based on the recognition result by the image recognition means, at least one of the start and the end of the conversation of the speaker is detected, and control for starting the information presentation by the information presentation means according to the detection result and the information presentation means Image recognition comprising: control for terminating information presentation; and control means for performing at least one control among addition, stop, or change of an output form used for information presentation performed by the information presentation means apparatus.   9. The apparatus according to claim 1, further comprising direction identification means for identifying a direction of a speaker's face based on the shape information or the movement information obtained by the image recognition means. The image recognition apparatus of any one of Claims.   The image recognition apparatus according to claim 1, further comprising communication means for communicating the obtained predetermined information. An image acquisition step for acquiring a distance image for the target object;
Oral part extraction step for extracting the oral part from the distance image acquired by the image acquisition step;
An image recognition step for recognizing the shape of the lips based on the distance image of the oral cavity part extracted by the oral cavity extraction step;
A speech recognition step for recognizing the input speech by speech recognition means;
Control of starting speech recognition by the speech recognition means when the start of the speaker's conversation is detected based on the recognition result of the image recognition step and termination of the speaker's conversation based on the recognition result of the image recognition step And a control step of performing at least one of the controls for ending the speech recognition by the speech recognition means when a speech is detected. An image acquisition step for acquiring a distance image stream for the target object;
Oral part extraction step for extracting the oral part from the distance image stream acquired by the image acquisition step,
An image recognition step for recognizing the movement of the lips based on the distance image stream of the oral cavity portion extracted by the oral cavity extraction step;
A speech recognition step for recognizing the input speech by speech recognition means;
Control of starting speech recognition by the speech recognition means when the start of the speaker's conversation is detected based on the recognition result of the image recognition step and termination of the speaker's conversation based on the recognition result of the image recognition step And a control step of performing at least one of the controls for ending the speech recognition by the speech recognition means when a speech is detected. An image acquisition step for acquiring a distance image for the target object;
Oral part extraction step for extracting the oral part from the distance image acquired by the image acquisition step;
An image recognition step for recognizing the shape of the lips based on the distance image of the oral cavity part extracted by the oral cavity extraction step;
An information presenting step for presenting information to be presented to the user using all or part of a plurality of output forms including an output form by sound and an output form by an image by an information presenting unit;
Based on the recognition result of the image recognition step, at least one of the start and end of the conversation of the speaker is detected, and in accordance with the detection result, the information presentation means starts the information presentation and the information presentation means An image recognition method comprising: control for terminating information presentation; and a control step for performing at least one control among control relating to addition, cancellation, or change of an output form used for information presentation performed by the information presentation means. . An image acquisition step for acquiring a distance image stream for the target object;
Oral part extraction step for extracting the oral part from the distance image stream acquired by the image acquisition step,
An image recognition step for recognizing the movement of the lips based on the distance image stream of the oral cavity portion extracted by the oral cavity extraction step;
An information presenting step for presenting information to be presented to the user using all or part of a plurality of output forms including an output form by sound and an output form by an image by an information presenting unit;
Based on the recognition result of the image recognition step, at least one of the start and end of the conversation of the speaker is detected, and in accordance with the detection result, the information presentation means starts the information presentation and the information presentation means An image recognition method comprising: control for terminating information presentation; and a control step for performing at least one control among control relating to addition, cancellation, or change of an output form used for information presentation performed by the information presentation means. . A computer-readable recording medium storing a program for causing a computer to function as an image recognition device,
An image acquisition step for acquiring a distance image for the target object;
Oral part extraction step for extracting the oral part from the distance image acquired by the image acquisition step;
An image recognition step for recognizing the shape of the lips based on the distance image of the oral cavity part extracted by the oral cavity extraction step;
A speech recognition step for recognizing the input speech by speech recognition means;
Control of starting speech recognition by the speech recognition means when the start of the speaker's conversation is detected based on the recognition result of the image recognition step and termination of the speaker's conversation based on the recognition result of the image recognition step A computer-readable recording medium on which is recorded a program for causing a computer to execute a control step of performing at least one control of control for ending speech recognition by the speech recognition means when a sound is detected. A computer-readable recording medium storing a program for causing a computer to function as an image recognition device,
An image acquisition step for acquiring a distance image stream for the target object;
Oral part extraction step for extracting the oral part from the distance image stream acquired by the image acquisition step,
An image recognition step for recognizing the movement of the lips based on the distance image stream of the oral cavity portion extracted by the oral cavity extraction step;
A speech recognition step for recognizing the input speech by speech recognition means;
Control of starting speech recognition by the speech recognition means when the start of the speaker's conversation is detected based on the recognition result of the image recognition step and termination of the speaker's conversation based on the recognition result of the image recognition step A computer-readable recording medium on which is recorded a program for causing a computer to execute a control step of performing at least one control of control for ending speech recognition by the speech recognition means when a sound is detected. A computer-readable recording medium storing a program for causing a computer to function as an image recognition device,
An image acquisition step for acquiring a distance image for the target object;
Oral part extraction step for extracting the oral part from the distance image acquired by the image acquisition step;
An image recognition step for recognizing the shape of the lips based on the distance image of the oral cavity part extracted by the oral cavity extraction step;
An information presenting step for presenting information to be presented to the user using all or part of a plurality of output forms including an output form by sound and an output form by an image by an information presenting unit;
Based on the recognition result of the image recognition step, at least one of the start and end of the conversation of the speaker is detected, and in accordance with the detection result, the information presentation means starts the information presentation and the information presentation means Records a program for causing a computer to execute a control step of ending information presentation and a control step of performing at least one control among additions, cancellations, or changes of an output form used for information presentation performed by the information presentation means Computer-readable recording medium. A computer-readable recording medium storing a program for causing a computer to function as an image recognition device,
An image acquisition step for acquiring a distance image stream for the target object;
Oral part extraction step for extracting the oral part from the distance image stream acquired by the image acquisition step,
An image recognition step for recognizing the movement of the lips based on the distance image stream of the oral cavity portion extracted by the oral cavity extraction step;
An information presenting step for presenting information to be presented to the user using all or part of a plurality of output forms including an output form by sound and an output form by an image by an information presenting unit;
Based on the recognition result of the image recognition step, at least one of the start and end of the conversation of the speaker is detected, and in accordance with the detection result, the information presentation means starts the information presentation and the information presentation means Records a program for causing a computer to execute a control step of ending information presentation and a control step of performing at least one control among additions, cancellations, or changes of an output form used for information presentation performed by the information presentation means Computer-readable recording medium.

Images