JP2011198270A - Object recognition device and controller using the same, and object recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an object recognition device correctly recognizing an object from an image, while suppressing increase in the size and the cost of the device.SOLUTION: The object recognition device 100 for recognizing the flesh-colored object (a hand) includes: a color camera 11 for generating a color image by photographing; a flesh color area detecting part 12 for detecting a flesh color area in the color image; an infrared lamp for irradiating the object with infrared light; an infrared camera 15 for generating an infrared image corresponding to the intensity of the incident infrared light; an infrared area detecting part 16 for detecting an area with the strong intensity of the infrared light, based on the infrared image; and a recognizing part 19 for outputting the recognition result of the target, based on the result of area detection in the flesh color area detecting part 12 and the result of area detection in the infrared area detecting part 16.

Description

  The present invention relates to an object recognition device that recognizes the position, shape, number, and the like of a recognition object, a control device using the device, and an object recognition method.

  2. Description of the Related Art Conventionally, an object recognition apparatus that recognizes the position, shape, and the like of a recognition target by performing image recognition on an image captured by a camera and extracting a specific area from the image is known. For example, a person's hand is photographed with a camera, the hand or finger area is extracted from the color image, the hand orientation, shape, finger position, etc. are recognized, and a control signal corresponding to the recognition result is sent to the computer. There is an object recognition device to input. With this object recognition device, the user can make a desired input to the computer by a natural operation of moving the hand in the air without using an input device such as a keyboard or a mouse.

  In this type of object recognition device, a camera captures a user's hand to generate a color image, and first extracts a skin color area from the color image, then extracts a contour line of the skin color area, Detection and estimation of hand orientation are performed (for example, see Non-Patent Document 1).

  There is also an apparatus that acquires a hand image and three-dimensional information using a camera and a three-dimensional scanner and recognizes the shape of the hand (for example, see Patent Document 1).

JP 2007-241833 A

Taehee Lee; Tobias Hollerer, “Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking”, IEEE ISWC 2007 (11th Int’l Symposium on Wearable Computers), Boston, MA Oct. 11-13, 2007 Preliminary presentation of the graduation thesis presentation of the Department of Information Science in the Department of Information Science in 2006, “Modeling of Face Area Using Geometric Features”, Yamagata University Department of Information Science

  However, in the recognition device of Non-Patent Document 1 described above, since the hand region in the image is extracted by extracting the skin color region from the color image, for example, when the background includes a region close to skin color. Even such a background may not be extracted as a hand region, and the shape and direction of the hand may not be recognized. In addition, when the hand region in the image is not skin-colored due to the influence of lighting or the like (for example, when the hand region is dark due to backlight), the hand region cannot be extracted.

  On the other hand, if a three-dimensional scanner is used as in the recognition device of the above-mentioned Patent Document 1, the above problem can be solved, and the hand is removed regardless of what color the hand is photographed in the image while eliminating the background. Can be recognized. However, since the recognition apparatus of Patent Document 1 requires a three-dimensional scanner, there is a problem that the apparatus becomes large and the manufacturing cost increases.

  Therefore, the present invention provides an object recognition apparatus that can accurately recognize an object from an image while suppressing an increase in size and cost of the apparatus.

  One aspect of the present invention is an object recognition apparatus that recognizes an object having a specific color, the object recognition apparatus detecting a color camera that generates a color image by photographing, and the region of the specific color in the color image. A specific color detection unit, an infrared lamp for irradiating an object with infrared rays, an infrared camera that generates an infrared image according to the intensity of incident infrared rays, and a region with high infrared intensity is detected based on the infrared image And a recognition unit that outputs a recognition result of the object based on a result of region detection in the specific color detection unit and a result of region detection in the infrared region detection unit.

  With this configuration, the object recognition device includes a color camera and an infrared camera, so even if an object that cannot be recognized only by detecting a specific color from the color image, the number, shape, position, etc. of the object can be recognized from the infrared image. , The recognition accuracy of the object can be improved. In addition, since such an improvement in recognition accuracy can be realized with an inexpensive configuration of a color camera and an infrared camera, an increase in size and cost of the apparatus can be suppressed.

  The target recognition device of the present invention further includes a contour extraction unit that calculates a contour of a region detected by the specific color detection unit, and extracts a contour of the target based on the determined contour. Based on the contour extracted by the contour extraction unit, a recognition result including information on at least one of the shape, position and number of the object is output.

  With this configuration, since objects other than the target can be excluded based on the contour characteristics (for example, the total length), the recognition accuracy is improved. Further, as the recognition result, information on at least one of the shape, position, and number of objects is obtained.

  In the object recognition device of the present invention, the recognition unit outputs a recognition result including information on the number of specific portions that satisfy a predetermined condition in the contour extracted by the contour extraction unit.

  With this configuration, it is possible to recognize how many specific portions (for example, fingertips) of the target are present.

  In the object recognition device of the present invention, the object is a human hand, and the specific color is a skin color.

  With this configuration, the position and shape of the user's hand can be recognized.

  In the object recognition device of the present invention, the object is a human hand, the specific color is a skin color, and the specific part is a fingertip.

  With this configuration, it is possible to recognize how many fingers of the user's hand are standing.

  Another aspect of the present invention is an object recognition apparatus for recognizing an object having a specific color, the object recognition apparatus including a first color camera that generates a first color image by photographing, and the first color camera. A specific color detection unit that detects the specific color region in the color image; a second color camera that generates a second color image by photographing; and a parallax between the first color image and the second color image. A distance estimation unit that generates a distance image, a recognition unit that outputs a recognition result of the target based on a result of area detection in the specific color detection unit and a distance image generated by the distance estimation unit; It has.

  With this configuration, the object recognition apparatus includes two color cameras and generates a distance image from the color images. Therefore, the shape and position of the object that cannot be recognized only by detecting a specific color from the first color image. Since the object can be correctly recognized based on the distance image, the recognition accuracy of the object can be improved. Further, such an improvement in recognition accuracy can be realized with an inexpensive configuration of two color cameras, so that an increase in size and cost of the apparatus can be suppressed.

  The target recognition device of the present invention further includes a contour extraction unit that calculates a contour of a region detected by the specific color detection unit, and extracts a contour of the target based on the determined contour. Based on the contour extracted by the contour extraction unit, a recognition result including information on at least one of the shape, position and number of the object is output.

  With this configuration, since the contour other than the target contour can be excluded based on the feature (for example, the total length) of the contour, the recognition accuracy is improved.

  In the object recognition device of the present invention, the specific color detection unit is configured to determine the region based on the distance image generated by the distance estimation unit, except for the region that is the specific color region but is not the target region. As a result of the detection, the contour extracting unit obtains a contour of the region with respect to the result of the region detection from which the region that is not the target region is removed, and extracts the contour.

  With this configuration, the specific color detection unit can more accurately detect the target region because the portion determined based on the distance and determined not to be the target is excluded from the region determined based only on the color information of the color image.

  In the object recognition device of the present invention, the recognition unit outputs a recognition result including information on the number of specific portions that satisfy a predetermined condition in the contour extracted by the contour extraction unit.

  With this configuration, it is possible to recognize how many specific portions (for example, fingertips) of the target are present.

  Still another aspect of the present invention is an object recognition device for recognizing an object, the object recognition device including an infrared lamp for irradiating an object with infrared rays and a first infrared ray corresponding to the intensity of incident infrared rays. A first infrared camera for generating an image, an infrared region detection unit for detecting a region having a high infrared intensity based on the first infrared image, and a second infrared image corresponding to the intensity of the incident infrared ray A second infrared camera, a distance estimation unit that estimates a distance to the target based on parallax of the first infrared image and the second infrared image, and a result of region detection in the infrared region detection unit And a recognition unit that outputs a recognition result of the object based on a result of distance estimation in the distance estimation unit.

  With this configuration, the object recognition apparatus includes two infrared cameras and estimates the distance from the parallax images to the object. Therefore, the object recognition device cannot be recognized only by detecting a region with high infrared intensity from the first infrared image. As for the shape and position of the object, the object can be correctly recognized based on the estimated distance, so that the recognition accuracy of the object can be improved. In addition, since such an improvement in recognition accuracy can be realized with an inexpensive configuration of two infrared cameras, an increase in size and cost of the apparatus can be suppressed.

  The object recognition device of the present invention further includes a contour extraction unit that obtains a contour of the region detected by the infrared region detection unit, and extracts the contour of the target based on the obtained contour, Based on the contour extracted by the contour extraction unit, a recognition result including information on at least one of the shape, position and number of the object is output.

  With this configuration, since objects other than the target can be excluded based on the contour characteristics (for example, the total length), the recognition accuracy is improved.

  In the target recognition device of the present invention, the infrared region detection unit excludes a region where the intensity of infrared light is strong but is not the target region based on the distance to the target estimated by the distance estimation unit. As a result of the region detection, the contour extraction unit obtains a contour of the region with respect to the result of the region detection from which the region that is not the target region is removed, and extracts the contour.

  With this configuration, the infrared region detection unit excludes a portion that is determined not to be a target based on a distance from a region having a high infrared intensity obtained from the first infrared image, and thus more accurately determines a target region. It can be detected.

  In the object recognition device of the present invention, the recognition unit outputs a recognition result including information on the number of specific portions that satisfy a predetermined condition in the contour extracted by the contour extraction unit.

  With this configuration, it is possible to recognize how many specific portions (for example, fingertips) of the target are present.

  Still another aspect of the present invention is a control device, which includes any one of the object recognition devices described above and a control unit that generates a control signal based on a recognition result in the recognition unit. ing.

  With this configuration, it is possible to generate a control signal that accurately reflects the position, shape, number, and the like of the target.

  Still another aspect of the present invention is an object recognition method for recognizing an object having a specific color. The method generates a color image of an object by photographing, and irradiates the object with infrared rays to make incident infrared rays. An image generation step for generating an infrared image according to the intensity of the image, a specific color detection step for detecting the specific color region in the color image, and an infrared region for detecting a region with a high infrared intensity based on the infrared image A detection step, and a recognition step of outputting a recognition result of the object based on a result of region detection in the specific color detection step and a region detection result in the region detection step.

  Still another aspect of the present invention is an object recognition method for recognizing an object having a specific color, which is obtained by photographing with a first color camera and a second color camera, respectively. And an image generation step for generating a second color image, a specific color detection step for detecting the specific color region in the first color image, and a parallax between the first color image and the second color image A distance estimation step for generating a distance image, and a recognition step for outputting the recognition result of the object based on the result of area detection in the specific color detection step and the distance image generated in the distance estimation step. Including.

  Still another aspect of the present invention is an object recognition method for recognizing an object. The method irradiates an object with infrared rays, and enters each using a first infrared camera and a second infrared camera. An image generation step of generating a first infrared image and a second infrared image according to the intensity of the infrared; an infrared region detection step of detecting a region having a high infrared intensity based on the first infrared image; A distance estimation step for generating a distance image based on the parallax between the first infrared image and the second infrared image, a result of area detection in the specific color detection step, and a distance image generated in the distance estimation step And a recognition step for outputting a recognition result of the object.

  Still another aspect of the present invention is a program, which is executed by a computer to cause the computer to perform any of the above methods.

  According to the present invention, an object can be accurately recognized from an image while suppressing an increase in size and cost of the apparatus.

The block diagram which shows the whole structure of the control apparatus in the 1st Embodiment of this invention 1 is an external front view of an imaging unit of an object recognition device according to a first embodiment of the present invention. The figure which shows the example of the color image in the 1st Embodiment of this invention The figure which shows the example of the detected skin color area | region in the 1st Embodiment of this invention The figure which shows the outline obtained by performing the outline extraction process in the 1st Embodiment of this invention The figure which shows the extracted outline in the 1st Embodiment of this invention The figure which shows the result of the finger extraction in the 1st Embodiment of this invention The figure for demonstrating the process which extracts the finger | toe in the 1st Embodiment of this invention The figure which shows the result of having performed the determination of the finger in the 1st Embodiment of this invention The figure which shows the result of having fitted the ellipse to the fingertip in the 1st Embodiment of this invention The graph which shows the change of angle (theta) i when scanning the pixel which comprises the outline of the hand in the 1st Embodiment of this invention The figure which shows the example of the infrared image in the 1st Embodiment of this invention The figure which shows the example of the infrared region detected in the 1st Embodiment of this invention The figure which shows the outline obtained by performing the outline extraction process in the 1st Embodiment of this invention The figure which shows the extracted outline in the 1st Embodiment of this invention The figure which shows the result of the finger extraction in the 1st Embodiment of this invention The block diagram which shows the whole structure of the control apparatus of the 2nd Embodiment of this invention. The block diagram which shows the whole structure of the control apparatus of the 3rd Embodiment of this invention. External appearance front view of the imaging part of the object recognition apparatus in the 3rd Embodiment of this invention The figure which shows the example of the 1st color image in the 3rd Embodiment of this invention. The figure which shows the example of the detected skin color area | region in the 3rd Embodiment of this invention The figure which shows the result of having excluded the distant area | region among the skin color area | regions in the 3rd Embodiment of this invention. The block diagram which shows the whole structure of the control apparatus of the 4th Embodiment of this invention. External appearance front view of the imaging part of the object recognition apparatus in the 4th Embodiment of this invention The figure which shows the example of the 1st infrared image in the 4th Embodiment of this invention The figure which shows the example of the infrared region detected in the 4th Embodiment of this invention The figure which shows the result of having excluded the area | region with a long distance among the infrared rays area | regions in the 4th Embodiment of this invention.

  Hereinafter, an object recognition device and a control device including the same according to an embodiment of the present invention will be described with reference to the drawings. In the following, a control device including an object recognition device that recognizes the number of fingers of a human hand and a control unit that generates a control signal for operating the computer based on the recognition result of the object recognition device will be described as an example. To do. This control device can be used as an interface to a computer.

(First embodiment)
FIG. 1 is a block diagram illustrating the overall configuration of the control device according to the first embodiment. The control device 1 includes an object recognition device 100 and a control unit 50. The object recognition apparatus 100 recognizes a human hand of skin color. The object recognition device 100 includes a color camera 11, a skin color area detection unit 12, a contour extraction unit 13, and a finger extraction unit 14. With this configuration, a finger of a hand is extracted from a color image obtained by photographing with the color camera 11. The object recognition apparatus 100 further includes an infrared lamp 20, an infrared camera 15, an infrared region detection unit 16, a contour detection unit 17, and a finger extraction unit 18. With this configuration, the finger of the hand is extracted from the infrared image obtained by photographing with the infrared camera 15.

  FIG. 2 is an external front view of the imaging unit 21 of the object recognition device 100. In FIG. 2, a lens 201 is a lens of the color camera 11, and a lens 202 is a lens of the infrared camera 15. The lens 202 of the infrared camera 15 is covered with a visible light cut filter. Around the lens 202 of the infrared camera 15, infrared LEDs 203 as a plurality of infrared lamps 20 that emit infrared light are arranged. As shown in FIG. 2, in the object recognition apparatus 100, the color camera 11 and the infrared camera 15 are provided close to each other. As a result, the captured image of the color camera 11 and the captured image of the infrared camera 15 are obtained by capturing the same subject portion at substantially the same position in the image. The user holds his / her hand over the imaging unit 21 so that the imaging unit 21 can photograph his / her hand.

  Returning to FIG. 1, the color camera 11 generates a color image by photographing the user's hand. For this purpose, the color camera 11 has an image sensor in which photoelectric conversion elements are two-dimensionally arranged. External light is incident on the image sensor via a lens or the like. The color camera 11 photoelectrically converts incident light for each pixel to generate RBG data, and outputs this as a color image. FIG. 3A is a diagram illustrating an example of a color image. In the example of FIG. 3A, the user holds the right hand over the imaging unit 21. In the example of FIG. 3A, the user's face, neck, and background are also photographed in addition to the user's hand.

  The skin color area detection unit 12 detects a skin color area in the color image generated by the color camera 11. Specifically, as described in Non-Patent Document 2, the skin color area detection unit 12 converts RGB data, which is a color image output from the color camera 11, into LUV data, and within a predetermined threshold range in the LUV space. An area having a certain value is detected as a skin color area. FIG. 3B is a diagram illustrating an example of a skin color region detected as a result of skin color detection performed on the color image of FIG. 3A by the skin color region detection unit 12. In the example of FIG. 3B, the index finger part of the user's hand is detected as a skin color area together with the skin color part of the background. In the example of FIG. 3B, the user's hand, the user's face and neck, and the skin color object in the background are also detected as the skin color region.

  The contour extracting unit 13 obtains the contour of the skin color region detected by the skin color region detecting unit 12 and extracts the contour of the hand therefrom. Here, the data indicating the contour is data in which the pixels constituting the contour are ordered. In this data, a pixel whose order is continuous with any one pixel is one of eight pixels surrounding the one pixel. The data indicating the contour is represented in the form of a vector in which the coordinate values of the pixels constituting the contour are arranged in order.

  FIG. 3C is a diagram illustrating a contour obtained by performing contour extraction processing on the skin color region of FIG. 3B detected by the skin color region detection unit 12. In the example of FIG. 3B, the hand, face and neck including the background in part, and the background object are detected as the skin color area, and accordingly, the contour extraction unit 13 also detects the part as the skin color area accordingly. The contours of the hand, face and neck, including the background, and the background object area are determined.

  The contour extraction unit 13 first obtains the total length of each contour in order to exclude contours other than the contour of the hand region from the contour thus obtained. And the thing with the longest total length is extracted as a hand outline. FIG. 3D is a diagram showing an outline extracted as having the longest overall length. That is, in FIG. 3D, the outline of the hand partially including the background is extracted, and at this stage, the skin color region of the face, the neck, and the background is excluded. The contour extracting unit 13 outputs data indicating the contour of FIG. 3D to the finger extracting unit 14.

The finger extraction unit 14 outputs an extraction result including information on the shape and position of the finger based on the hand contour extracted by the contour extraction unit 13. 4A to 4C and FIG. 5 are diagrams for explaining processing for extracting a finger. 4A to 4C are enlarged views of the fingertip portion of the contour of the hand extracted by the contour extracting unit 13. 4A to 4C, one point represents one pixel. As described above, in the data indicating the contour, the coordinate values of the pixels are arranged in order. The finger extraction unit 14 sequentially scans the pixels constituting the contour according to this order. As shown in FIG. 4A, the finger extraction unit 14, and the i-th pixel C _i of interest, (in the example of FIG. 4A, d = 30) fixed number d from the pixel by pixel C _i of pre-order An angle Θ _i of an angle formed by a line connecting _−d , a pixel C _i, and a line connecting pixel C _{i + d in} the order after d by that pixel is obtained.

Then, the finger extraction unit 14 determines whether or not the angle Θ _i is below a predetermined threshold (for example, 70 degrees). FIG. 4B is a diagram showing a result of sequentially determining the angle Θ _i in this way. In the example of FIG. 4B, a total of 11 pixels (pixels surrounded by a circle) satisfy the above condition. Finger extraction unit 14, thus to continue to scan the contour while determining whether the angle theta _i of the pixel of interest C _i is below the threshold, the pixel C _i finally angle theta _i is below a threshold If there are five or more consecutive locations, the location is determined to be the fingertip. In the example of FIG. 4B, since 11 pixels satisfying the above condition are continuous, this portion is determined as the fingertip. In the present embodiment, the number of consecutive pixels C _i whose angle Θ _i for determining the fingertip is less than the threshold is five, but this number is not limited to five.

FIG. 5 is a graph showing a change in the angle Θ _i when five fingers are detected while sequentially scanning the pixels C _i constituting the contour of the hand. In this graph, the position i in the contour is taken on the horizontal axis, and the angle Θ _i is taken on the vertical axis. In the example of FIG. 5, the threshold is set to 85 degrees. Five valley portions below this threshold (portions surrounded by circles in FIG. 5) correspond to the tip portion of the finger. Since the contour line is a closed curve that makes one round, the left corner and the right corner of the graph of FIG. 5 represent the same point on the contour line.

When the finger extraction unit 14 finds the fingertip, the finger extraction unit 14 performs ellipse fitting at the portion determined to be the fingertip. Specifically, the finger extraction unit 14 sets an ellipse that best fits the continuous pixel C _{i in} which the angle Θ _i is less than the threshold and some points around it. FIG. 4C is a diagram illustrating a result of fitting an ellipse. When the long side, the short side length, and the ratio of the fitted ellipse are within a predetermined threshold range, the finger extraction unit 14 is the finger (tip) of the part to which the ellipse is fitted. Judge. If the ellipse is too large or flat, there is a high possibility that a part that is not a finger has been erroneously detected. In this case, it is not determined to be a finger.

The finger extraction unit 14 performs the above process for all fingers. FIG. 3E is a diagram illustrating a result of finger extraction by the finger extraction unit 14. As shown in FIG. 3E, for the part where the outline of the index finger has been absorbed by the rectangular region of the background, the above condition that the pixels C _i whose angle Θ _i is less than the threshold value continues 5 or more does not satisfy the above condition. The finger is not detected. The finger extraction unit 14 outputs data representing the position (center coordinate) and shape (length and direction of the major axis and minor axis) of the ellipse fitted to the fingertip to the recognition unit 19.

  The object recognition device 100 further includes an infrared camera 15, an infrared region detection unit 16, a contour extraction unit 17, and a finger extraction unit 18. The infrared camera 15 generates an infrared image corresponding to the intensity of incident infrared light in a state where the infrared lamp 20 is irradiating infrared light in the shooting direction of the infrared camera 15. The infrared camera 15 has an image sensor having the same size and the same number of pixels as the color camera 11, and photographs a subject simultaneously with the color camera 11. Similar to the color camera 11, the infrared camera 15 includes a lens, and is configured such that infrared light that has passed through the lens enters the imaging device. The infrared camera 15 further has a visible light cut filter on the front surface of the lens, and is configured so that visible light does not reach the image sensor through the lens. The image sensor photoelectrically converts incident infrared rays and outputs data of each pixel having a pixel value corresponding to the infrared intensity as an infrared image.

  FIG. 6A is a diagram showing an infrared image captured simultaneously with the color image of FIG. In the example of FIG. 6A, an infrared image having a relatively strong infrared intensity is obtained in the areas of the user's hand, part of the arm, and part of the background (wall illuminated by outside light entering the room). In the example of FIG. 6A, for the sake of convenience, a region having a relatively high infrared intensity is surrounded by an outline, but each pixel in the infrared image actually has a pixel value corresponding to the infrared intensity. It is easy to understand if the color temperature is defined for the infrared intensity and the infrared image is expressed by the color temperature. Further, as described above, infrared light is emitted from the vicinity of the infrared camera 15 in the infrared imaging direction, but this infrared light is not so strong, and if it is separated by a certain distance, the reflected red light from there is reflected. External light is hardly detected by the image sensor.

  The infrared region detection unit 16 detects a region having a high infrared intensity (hereinafter referred to as “infrared region”) based on the infrared image generated by the infrared camera 15. Specifically, the infrared region detection unit 16 detects, as an infrared region, a region where the pixel value exceeds a predetermined threshold in the infrared image. FIG. 6B is a diagram illustrating an example of an infrared region detected as a result of performing infrared region detection on the infrared image of FIG. 6A. In the example of FIG. 6B, in addition to the user's hand, a part of the user's arm and a part of the background (a wall illuminated by outside light) are detected as an infrared region.

  The contour extracting unit 17 obtains the contour of the region detected by the infrared region detecting unit 16 and extracts the contour of the hand therefrom. The processing in the contour extraction unit 17 is the same as the processing in the contour extraction unit 13. FIG. 6C is a diagram illustrating a contour obtained by performing contour extraction processing on the infrared region of FIG. 6B detected by the infrared region detection unit 16. FIG. 6D is a diagram showing the contour extracted by the contour extracting unit 17 by removing a contour having a short overall length from the obtained contours. The infrared region of the arm portion and part of the background is not extracted as the hand contour because the entire length of the contour is relatively short. The contour extracting unit 17 outputs data indicating the contour of FIG. 6D to the finger extracting unit 18.

  The finger extraction unit 18 outputs an extraction result including information on the shape and position of the finger based on the hand contour extracted by the contour extraction unit 17. The process of extracting the finger by the finger extraction unit 18 is also the same as the process of the finger extraction unit 14 and will not be described. FIG. 6E is a diagram illustrating a result of finger extraction by the finger extraction unit 18. Here, an ellipse is fitted to the tip of the index finger. The finger extraction unit 18 outputs data representing the position (center coordinate) and shape (length and orientation of the major axis and minor axis) of the ellipse fitted to the fingertip to the recognition unit 19.

  The recognition unit 19 performs final finger recognition in consideration of the extraction result in the finger extraction unit 14 and the extraction result in the finger extraction unit 18, and outputs information on the number of fingers as the recognition result. There are various ways in which the recognition unit 19 obtains a recognition result based on two extraction results. For example, in the above example, for the other four fingers excluding the index finger, the finger detected from the color image of the color camera 11 and the finger detected from the infrared image of the infrared camera 15 are substantially the same. However, the index finger is extracted only by the infrared image and not by the color image. In such a case, the recognition unit 19 of the present embodiment recognizes that the finger extracted from at least one extraction result is the user's finger. The recognition unit 19 in another example recognizes only the finger extracted from both extraction results as the user's finger. Furthermore, the recognition unit 19 of another example recognizes that the finger extracted in both extraction results is the user's finger, and for a finger extracted only in one extraction result, the predetermined condition is satisfied. If it is satisfied, it is recognized as a finger. For example, when the positional relationship between the other fingertip detected from both detection results and the fingertip extracted only from the one extraction result satisfies a predetermined condition, it is recognized as a user's finger. .

  The recognition unit 19 may obtain a recognition result based on two extraction results using a calibration function. In this case, the user is asked to present one to five fingers in order, and the hand is presented in a shape and position corresponding to each of the plurality of control signals. Each image is captured by the infrared camera 15 to generate a color image and an infrared image. And the recognition part 19 memorize | stores these color images and infrared images as standard information corresponding to the number of predetermined fingers. When the recognition unit 19 recognizes the finger based on the two extraction results, the recognition unit 19 compares the two extraction results with the respective standard information, and adopts the detection result closer to the standard information. The recognition unit 19 may perform finger recognition based on the two extraction results by a method other than the method described above.

  The control unit 50 generates and outputs a control signal corresponding to the number of fingers. This control signal is sent to the computer to control the operation of the computer. For example, the control unit 50 outputs a control signal for holding the mouse when the number of fingers is zero (a state where the hand is clasped), and outputs a control signal for releasing the mouse when the number of fingers is five. Output. Various embodiments can be considered for the correspondence between the number of fingers and the control signal.

  The recognition unit 19 may output a recognition result including information on the number of fingers and the position of the ellipse set on the fingertip, and the control unit 50 may output a control signal corresponding to the recognition result. In this case, it is possible to generate different control signals depending on whether the hand is spread out vertically or horizontally.

  As described above, according to the object recognition device 100 of the present embodiment, a finger of a hand can be accurately detected regardless of the background color or the like with an inexpensive and simple structure of one color camera and one infrared camera. The control device 1 can generate a control signal that accurately reflects the shape and position of the user's hand.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. FIG. 7 is a block diagram showing the overall configuration of the control device according to the second embodiment of the present invention. In the control device 2 of the present embodiment, the same components as those of the control device 1 of the first embodiment are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The control device 2 includes an object recognition device 200 and a control unit 50. The object recognition device 200 includes a color camera 11, a skin color area detection unit 12, an infrared camera 15, and an infrared area detection unit 16. These structures are the same as the structure of the corresponding part of 1st Embodiment. The object recognition device 200 according to the present embodiment includes a hand region detection unit 22 that detects the user's hand region based on the detection result of the skin color region detection unit 12 and the detection result of the infrared region detection unit 16. Yes. That is, in the object recognition apparatus 100 according to the first embodiment, after the finger is extracted for each of the color image and the infrared image, the recognition unit 19 considers the result of the two finger extractions, and the final finger recognition is performed. Although the recognition result has been obtained, in the present embodiment, the skin color region is detected from the color image, the infrared region is detected from the infrared image, and the hand region is detected based on the result, and then the contour is obtained. Subsequent processing of extraction, finger extraction, and recognition is performed.

  The hand region detection unit 22 detects a region where the skin color region detected by the skin color region detection unit 12 and the infrared region detected by the infrared region detection unit 16 overlap as a hand region. That is, the hand region detection unit 22 sets a region detected in both the color image and the infrared image as a hand region. When there are a plurality of overlapping regions, only the region with the larger area may be left and the region with the smaller area may be ignored, or the plurality of regions may be output as hand regions as they are. Good.

  The contour extracting unit 13 obtains the contour of the hand region in the same manner as the contour extracting units 13 and 17 of the first embodiment, and extracts it as the contour of the user's hand. Here, when there are a plurality of regions detected as hand regions by the hand region detecting unit 22, the contour having the longest overall length is extracted as the contour of the user's hand, as in the first embodiment. The finger extraction unit 14 also detects a finger in the same manner as the finger extraction units 14 and 18 of the first embodiment, fits an ellipse to the detected finger, and outputs information on the ellipse fitted to the finger. . Based on the ellipse information obtained from the finger extraction unit 14, the recognition unit 23 outputs information including the number of fingers to the control unit 50 as a recognition result. However, unlike the recognition unit 19 of the first embodiment, the recognition unit 23 of the present embodiment does not need to determine the number of fingers or the like from the results of a plurality of finger extractions. Based on this, the number of fingers is recognized by using the calibration function described above. The control unit 50 is the same as that in the first embodiment.

  Also according to the present embodiment, the object recognition apparatus 200 has an inexpensive and simple structure of one color camera and one infrared camera, can accurately detect a finger of a hand regardless of the background color, etc. A control signal that accurately reflects the shape and position of the user's hand can be generated. Further, in the present embodiment, it is not necessary to perform contour extraction and finger extraction processing for each of the color image and the infrared image, so that the object can be recognized with fewer resources.

(Third embodiment)
Next, a third embodiment of the present invention will be described. FIG. 8 is a block diagram showing the overall configuration of the control device according to the third embodiment of the present invention. In the control device 3 of the present embodiment, the same components as those of the control devices 1 and 2 of the first and second embodiments are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The control device 3 includes an object recognition device 300 and a control unit 50. The object recognition device 300 includes a first color camera 11-1, a hand region detection unit 24, a second color camera 11-2, and a distance estimation unit 25. The first color camera 11-1 and the second color camera 11-2 are integrally configured.

  FIG. 9 is an external front view of the imaging unit 26 of the object recognition device 300. The lens 201-1 is a lens of the first color camera 11-1, and the lens 201-2 is a lens of the second color camera. As shown in FIG. 9, in the object recognition apparatus 300, the first color camera 11-1 and the second camera 11-2 are provided close to each other. As a result, the color image (first color image) obtained by the first color camera 11-1 and the color image (second color image) obtained by the second color camera 11-2 are images. The same subject portion is imaged at substantially the same position.

  The first color image obtained by the first color camera 11-1 is input to the hand region detection unit 24 and the distance estimation unit 25. The second color image obtained by the second color camera 11-2 is input to the distance estimation unit 25. The distance estimation unit 25 obtains a distance to the subject by a stereo method based on the parallax between the first color image and the second color image, and generates a distance image. A distance image is an image in which the pixel value of each pixel represents the distance to the subject appearing at that pixel.

  FIG. 10A is a diagram illustrating an example of a first color image captured by the first color camera 11-1. This color image includes the user's hand and the ceiling (upper side of the image) and the rear wall (lower side of the image) as the background. The hand region detection unit 24 performs skin color detection processing on the first color image. The skin color is detected by the same method as described in the first embodiment. FIG. 10B is a diagram illustrating a skin color area detected by the hand area detection unit 24. In the example of FIG. 10B, the ceiling overlaps with the user's hand and is detected as a skin color region.

  The hand region detection unit 24 detects only the hand portion from the detected skin color region using the distance image obtained by the distance estimation unit 25. For this reason, the hand region detection unit 24 excludes a region in which the distance in the distance image is equal to or greater than a threshold among the regions detected as the skin color region. In the example of FIG. 10B, the ceiling detected as skin color is excluded because the distance is equal to or greater than the threshold. FIG. 10C is a diagram illustrating a result of removing a far-field region from the skin-colored region. As shown in FIG. 10C, as a result of excluding the far-distant area from the area detected as the skin color, the user's hand area is detected separately from the ceiling area.

  The hand region detection unit 24 outputs the skin color region of only the detected hand part to the contour extraction unit 13 as described above. The contour extracting unit 13 extracts a contour from the skin color region and outputs the contour to the finger extracting unit 14. The finger extraction unit 14 extracts the tip of the finger from the contour of the hand, fits an ellipse to the tip of the finger, and outputs the hand contour and ellipse information to the recognition unit 23. The recognition unit 23 recognizes the number of fingers based on the information obtained from the finger extraction unit 14 and outputs a recognition result to the control unit 50.

  According to the object recognition device 300 of the present embodiment, the finger of the hand can be accurately detected regardless of the background color or the like with an inexpensive and simple structure of two color cameras. A control signal accurately reflecting the shape and position can be generated.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. FIG. 11 is a block diagram showing an overall configuration of a control device according to the fourth embodiment of the present invention. In the control device 4 of the present embodiment, the same components as those of the control devices 1, 2, and 3 of the first, second, and third embodiments are denoted by the same reference numerals, and description thereof is omitted as appropriate. . As can be seen by comparing FIG. 11 and FIG. 8, the control device 4 of the present embodiment is the first and second color cameras 11-1, 11-2 of the control device 3 of the third embodiment. Is replaced with the first and second infrared cameras 15-1 and 15-2.

  The control device 4 includes an object recognition device 400 and a control unit 50. The object recognition device 400 includes a first infrared camera 15-1, a hand region detection unit 27, a second infrared camera 15-2, and a distance estimation unit 25. The first infrared camera 15-1 and the second infrared camera 15-2 are integrally configured.

  FIG. 12 is an external front view of the imaging unit 28 of the object recognition device 300. The lens 202-1 is a lens of the first infrared camera 15-1, and the lens 202-2 is a lens of the second infrared camera 15-2. A plurality of infrared LEDs 203 are arranged around the lens 202-1 and the lens 202-2. As shown in FIG. 12, in the object recognition apparatus 300, the first infrared camera 15-1 and the second infrared camera 15-2 are provided close to each other. Thereby, an infrared image (first infrared image) obtained by the first infrared camera 15-1 and an infrared image (second infrared image) obtained by the second infrared camera 15-2 are images. The same subject portion is imaged at substantially the same position.

  The first infrared image obtained by the first infrared camera 15-1 is input to the hand region detection unit 27 and the distance estimation unit 25. The second infrared image obtained by the second infrared camera 15-2 is input to the distance estimation unit 25. The distance estimation unit 25 obtains a distance to the subject by a stereo method based on the parallax between the first color image and the second color image, and generates a distance image.

  FIG. 13A is a diagram illustrating an example of a first infrared image captured by the first infrared camera 15-1. In this infrared image, the user's hand, face, and upper body are imaged as regions with high infrared intensity. In FIG. 13A, as in FIG. 6A described in the first embodiment, for convenience, a region having a relatively high infrared intensity is surrounded by an outline, but each pixel in the infrared image actually corresponds to the infrared intensity. Have a pixel value.

  The hand region detection unit 27 performs an infrared region detection process on the first infrared image. Detection of the infrared region is performed by the same method as described in the first embodiment. FIG. 13B is a diagram illustrating an infrared region detected by the hand region detection unit 27. In the example of FIG. 13B, the user's hand, part of the face, and part of the upper body are all detected as an infrared region.

  The hand region detection unit 27 detects only the hand portion from the detected infrared region using the distance image obtained by the distance estimation unit 25. For this reason, the hand area detection unit 27 excludes an area where the distance in the distance image is greater than or equal to a threshold from the areas detected as the infrared areas. In the example of FIG. 13B, the face and upper body detected as the infrared region are excluded because the distance is equal to or greater than the threshold value. FIG. 13C is a diagram illustrating a result of excluding far-field regions from the infrared region. As shown in FIG. 13C, as a result of eliminating the region far from the region detected as the infrared region, the user's hand region is detected separately from the face and upper body region.

  The hand region detection unit 27 outputs the infrared region of only the detected hand part to the contour extraction unit 13 as described above. The contour extracting unit 13 extracts a contour from the infrared region and outputs the contour to the finger extracting unit 14. The finger extraction unit 14 extracts the tip of the finger from the contour of the hand, fits an ellipse to the tip of the finger, and outputs the hand contour and ellipse information to the recognition unit 23. The recognition unit 23 recognizes the number and position of fingers based on the information obtained from the finger extraction unit 14 and outputs the recognition result to the control unit 50.

  According to the object recognition device 400 of the present embodiment, it is possible to accurately detect a finger of a hand regardless of the background color with an inexpensive and simple structure of two infrared cameras. And a control signal accurately reflecting the position.

  As mentioned above, although embodiment of this invention was described, said description is only an illustration and this invention is implemented also in other embodiment.

  For example, the target recognition device according to the above embodiment outputs information on the number of fingers as a recognition result, but information output as a recognition result by the target recognition device is not limited to the number of fingers. For example, the target recognition device may output information on the shape and position of the finger obtained by the finger extraction unit as a recognition result. Further, the object recognition device may output information on the contour of the hand extracted by the contour extraction unit as a recognition result. In addition, the object recognition device may output information on a region detected by the skin color region detection unit or the infrared region detection unit as a recognition result. Further, the target recognition device may output information obtained by processing these pieces of information as a recognition result. For example, the target recognition device may output information on the length of the contour of the hand extracted by the contour extraction unit as the recognition result. Further, for example, the target recognition device may output information on the area of the area detected by the skin color area detection unit or the infrared area detection unit as the recognition result. Furthermore, when the object recognition apparatus has a configuration for estimating the distance to the hand as in the third and fourth embodiments, the information on the distance is added to the information on the recognition result. It may be output.

  In the above-described embodiment, the object recognition device includes one color camera and one infrared camera (first and second embodiments), or two color cameras (third embodiment), Alternatively, the two infrared cameras (fourth embodiment) in total are provided, but a total of three or more cameras may be provided.

  Furthermore, in the above embodiment, an example in which a skin color region is detected from a color image has been described. However, when the recognition target has another color, the other color may be detected.

  The present invention has an excellent effect of accurately recognizing an object from an image while suppressing an increase in size and cost of the apparatus, and is applied to an object recognition apparatus that recognizes the number, position, shape, etc. of recognition objects. can do.

1, 2, 3, 4 Control device 100, 200, 300, 400 Object recognition device 11, 11-1, 11-2 Color camera 12 Skin color area detection unit 13 Contour extraction unit 14 Finger extraction unit 15, 15-1, 15 -2 Infrared camera 16 Infrared region detection unit 17 Contour extraction unit 18 Finger extraction unit 19, 23 Recognition unit 20 Infrared lamp 21, 26, 28 Imaging unit 22, 24, 27 Hand region detection unit 25 Distance estimation unit 50 Control unit

Claims (18)

An object recognition device for recognizing an object having a specific color,
A color camera that generates color images by shooting;
A specific color detection unit for detecting a region of the specific color in the color image;
An infrared lamp for irradiating the subject with infrared rays;
An infrared camera that generates an infrared image according to the intensity of the incident infrared ray;
Based on the infrared image, an infrared region detector that detects a region having a strong infrared intensity;
A recognition unit that outputs a recognition result of the object based on a result of region detection in the specific color detection unit and a result of region detection in the infrared region detection unit;
An object recognition apparatus comprising: A contour extraction unit that obtains the contour of the area detected by the specific color detection unit and extracts the contour of the target based on the obtained contour is further provided.
The object recognition apparatus according to claim 1, wherein the recognition unit outputs a recognition result including information on at least one of the shape, position, and number of the object based on the contour extracted by the contour extraction unit.   The object recognition apparatus according to claim 2, wherein the recognition unit outputs a recognition result including information on a number of specific portions that satisfy a predetermined condition in the contour extracted by the contour extraction unit.   The object recognition apparatus according to claim 1, wherein the object is a human hand and the specific color is a skin color.   The object recognition apparatus according to claim 3, wherein the object is a human hand, the specific color is a skin color, and the specific part is a fingertip. An object recognition device for recognizing an object having a specific color,
A first color camera that generates a first color image by photographing;
A specific color detection unit for detecting a region of the specific color in the first color image;
A second color camera for generating a second color image by photographing;
A distance estimation unit that generates a distance image based on the parallax of the first color image and the second color image;
A recognition unit that outputs a recognition result of the target based on a result of area detection in the specific color detection unit and a distance image generated by the distance estimation unit;
An object recognition apparatus comprising: A contour extraction unit that obtains the contour of the area detected by the specific color detection unit and extracts the contour of the target based on the obtained contour is further provided.
The object recognition apparatus according to claim 6, wherein the recognition unit outputs a recognition result including information on at least one of the shape, position, and number of the object based on the contour extracted by the contour extraction unit. The specific color detection unit is based on the distance image generated by the distance estimation unit, except for the region of the specific color but not the target region, as a result of the region detection,
The object recognition apparatus according to claim 7, wherein the contour extraction unit obtains a contour of a region from the region detection result from which a region that is not the target region is removed, and extracts the contour.   The object recognition apparatus according to claim 7 or 8, wherein the recognition unit outputs a recognition result including information on a number of specific portions that satisfy a predetermined condition in the contour extracted by the contour extraction unit. An object recognition device for recognizing an object,
An infrared lamp for irradiating the subject with infrared rays;
A first infrared camera that generates a first infrared image according to the intensity of incident infrared rays;
Based on the first infrared image, an infrared region detection unit for detecting a region having a strong infrared intensity;
A second infrared camera that generates a second infrared image according to the intensity of the incident infrared radiation;
A distance estimation unit that generates a distance image based on the parallax of the first infrared image and the second infrared image;
A recognition unit that outputs a recognition result of the object based on a result of region detection in the infrared region detection unit and a distance image generated by the distance estimation unit;
An object recognition apparatus comprising: A contour extraction unit that obtains the contour of the region detected by the infrared region detection unit and extracts the contour of the target based on the obtained contour is further provided.
The object recognition device according to claim 10, wherein the recognition unit outputs a recognition result including information on at least one of the shape, position, and number of the object based on the contour extracted by the contour extraction unit. The infrared region detection unit is based on the distance image generated by the distance estimation unit, except for a region where the intensity of infrared light is strong but not the target region, as a result of the region detection,
The object recognition device according to claim 11, wherein the contour extraction unit obtains a contour of a region from the result of the region detection from which a region that is not the target region is removed, and extracts the contour.   The target recognition apparatus according to claim 11 or 12, wherein the recognition unit outputs a recognition result including information on a number of specific portions that satisfy a predetermined condition in the contour extracted by the contour extraction unit. An object recognition device according to any one of claims 1 to 13,
A control unit that generates a control signal according to a recognition result in the recognition unit;
A control device comprising: An object recognition method for recognizing an object having a specific color,
An image generation step of generating a color image of a target by photographing and simultaneously irradiating the target with infrared rays to generate an infrared image according to the intensity of the incident infrared rays;
A specific color detection step of detecting the specific color region in the color image;
An infrared region detecting step for detecting a region having a strong infrared intensity based on the infrared image;
A recognition step of outputting a recognition result of the object based on a result of region detection in the specific color detection step and a result of region detection in the region detection step;
An object recognition method comprising: An object recognition method for recognizing an object having a specific color,
An image generation step of simultaneously shooting with each of the first color camera and the second color camera to generate a first color image and a second color image;
A specific color detecting step of detecting a region of the specific color in the first color image;
A distance estimation step of generating a distance image based on the parallax of the first color image and the second color image;
A recognition step of outputting a recognition result of the object based on a result of area detection in the specific color detection step and a distance image generated in the distance estimation step;
An object recognition method comprising: An object recognition method for recognizing an object,
An image generation step of irradiating an object with infrared rays and simultaneously generating a first infrared image and a second infrared image corresponding to the intensity of incident infrared rays using the first infrared camera and the second infrared camera, respectively. When,
An infrared region detecting step for detecting a region having a strong infrared intensity based on the first infrared image;
A distance estimation step for generating a distance image based on the parallax of the first infrared image and the second infrared image;
A recognition step of outputting a recognition result of the object based on a result of area detection in the specific color detection step and a distance image generated in the distance estimation step;
An object recognition method comprising:   A program that, when executed by a computer, causes the computer to perform the method according to any one of claims 15 to 17.