JP2017117341A - Object detection method, device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately extract an area which indicates an object performing a touch operation to an operation object, from an image in which the operation object is imaged.SOLUTION: An acquisition part 21 acquires a depth image and a reflection intensity image. A setting part 22 approximates a function similar to a shape of a luminance histogram of an operation object which is operated by an object which is a detection object in the depth image, to a luminance histogram of the depth image, thereby estimating the luminance histogram of the operation object. An extraction part 23 extracts a candidate area indicating the object which is the detection object from the depth image, using a depth to the operation object determined based on the estimated luminance histogram of the operation object. A division part 24 divides the reflection intensity image into plural areas. An integration part 25 detects an area which is obtained by integrating the extraction area in which a contour line and a shape of the extracted candidate area are similar to each other, and the candidate area, as an area indicating the object which is the detection object.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an object detection method, an object detection apparatus, and an object detection program.

  2. Description of the Related Art In recent years, for example, on the site of a factory or the like, a method for supporting an operator's work using an augmented reality function (AR: Augmented Reality) has been proposed. Specifically, when a marker that can be identified by a camera is installed at a specific location on the site and the marker is photographed by a tablet terminal having a camera function, information associated with the marker is superimposed on the screen of the tablet terminal. An example of display is proposed. In this case, even if the operator does not carry various manuals related to the work, if he / she only brings a tablet terminal, he / she only needs to shoot the marker and refer to the manuals necessary for the place where the marker is installed. can do.

JP 2010-182014 A

Yugo Ishii, 5 others, “Proposal and Evaluation of Nuclear Power Plant Disassembly Support Method Using Augmented Reality”, Transactions of the Virtual Reality Society of Japan, Virtual Reality Society of Japan, June 30, 2008, Vol.13 , No.2, p.289-300

  However, in such a field, since an operator often works with both hands, the use of a tablet terminal for referring to a manual may hinder the work on the field. Therefore, as shown in FIG. 31, it is required to realize hands-free operation in a work environment by combining wearable devices such as a head-mounted camera 90 and a glasses-type display 91 instead of a tablet terminal.

  When a wearable device that realizes such hands-free operation is used, it is difficult to call or switch necessary information by touching the screen like a tablet terminal. Thus, for example, a gesture operation that gives an instruction to the wearable device by a hand movement may be used. In the gesture operation, the operator's hand is photographed by the head-mounted camera 90, the operation content is specified by analyzing the position and shape of the hand, and a process corresponding to the specified operation content is displayed on the glasses-type display 91. It is executed on the received information.

  As an example of work that specifies a specific location on an object, if an abnormal location such as a crack on a wall surface is found during inspection work, the location and the information on the location or object can be recorded electronically. Conceivable.

  For such work, many gesture operations for wearable devices use the shape of a finger or hand or an action in the air, and therefore there is a problem that misalignment tends to occur when a specific location is designated.

  On the other hand, when the object is in a place where the worker can reach, an operation (touch operation) in which the operator directly touches and designates the corresponding part on the object can be performed. If the touch operation is used as the gesture operation, there is an effect that the position shift of the designated portion is difficult to occur.

  In order to recognize a touch operation using a wearable device, the position of an object to be touch-operated and an object (for example, an operator's hand or finger) that performs the touch operation on the object are determined from a captured image. It is desirable to extract the shape as accurately as possible.

  As one aspect, an object of the present invention is to accurately extract a region indicating an object that performs a touch operation on an operation target from an image obtained by photographing the operation target.

  In one aspect, a distance image that represents the distance to the object with luminance corresponding to the distance and a reflection intensity image that represents the intensity of reflected light from the object with luminance are acquired. Then, by approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the detection target object in the distance image, the brightness histogram of the operation target is obtained. presume. A candidate area representing the object to be detected is extracted from the distance image using a distance to the operation target determined based on the estimated luminance histogram of the operation target. The reflection intensity image is divided into a plurality of regions based on the luminance of each pixel of the reflection intensity image. Then, an extracted area whose shape is similar to the contour line of the extracted candidate area and an area obtained by integrating the candidate areas are detected as an area representing the object to be detected.

  As one aspect, there is an effect that it is possible to accurately extract a region indicating an object that performs a touch operation on the operation target from an image obtained by photographing the operation target.

It is a figure which shows an example of a distance image. It is a figure which shows an example of a reflection intensity image. It is a figure which shows an example of a structure of an object detection system. It is a figure which shows an example of the brightness | luminance histogram of a distance image. It is a figure which shows an example of the finger and touch operation target object in a distance image. It is a figure which shows the outline of the brightness | luminance histogram and gradient change rate of a distance image. It is a figure which shows an example of the brightness | luminance histogram of a touch operation target object. It is a figure which shows an example of a structure in the case of implement | achieving an object detection apparatus with a computer. It is a flowchart which shows an example of the flow of an object detection process. It is a flowchart which shows an example of the flow of a finger candidate area | region brightness | luminance setting process in 1st Embodiment. It is a figure for demonstrating the relationship between the gradient change rate of a brightness | luminance histogram, and a peak. It is a figure for demonstrating the approximation of the Gaussian function with respect to the brightness | luminance histogram of a distance image. It is a figure which shows an example of the result of having extracted the vicinity area | region and the finger candidate area | region from the distance image. It is a figure which shows an example of the difference in the reflection intensity in a reflection intensity image. It is a figure which shows an example of the setting method of the division | segmentation target range using the brightness | luminance histogram of a reflection intensity image. It is a figure which shows an example of the flow of a setting of a division | segmentation object range. It is a flowchart which shows an example of the flow of a reflection intensity image division process. It is the figure which showed an example of the process of a division | segmentation of a reflection intensity image. It is a flowchart which shows an example of the flow of an integration process. It is a figure which shows an example of the extraction area | region which carried out incorrect coupling | bonding. It is a figure which shows an example of the excessive division | segmentation of an extraction area | region. It is a figure which shows an example of the outline of a finger candidate area | region. It is a figure which shows an example of the outline of an extraction area | region. It is a figure which shows an example of the division | segmentation object area | region which reduced the range. It is a figure which shows an example of the finger area | region integrated by the integration process. It is a flowchart which shows an example of the flow of a determination process. It is a figure which shows the other example of the outline of a finger candidate area | region. It is a figure which shows the other example of the outline of an extraction area | region. It is a flowchart which shows an example of the flow of a finger candidate area | region brightness | luminance setting process in 2nd Embodiment. It is a flowchart which shows an example of the flow of a finger candidate area | region brightness | luminance setting process in 3rd Embodiment. It is a figure which shows an example of an operator's equipment.

  Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. In addition, the same code | symbol is provided to the component or process which bears the same function through all the drawings, and the overlapping description may be abbreviate | omitted suitably.

  As a gesture operation using a wearable device, for example, the operation of information displayed on the glasses-type display 91 or the like using the shape of a finger or a hand associated with a specific operation or an operation in the air There is a method (motion operation) to specify the target object.

  However, in the case of a motion operation, for example, even though the operator does not intend to operate information, a hand movement associated with a specific operation of the wearable device is accidentally executed, causing an erroneous operation. It may be possible to end up. Also, in motion operations that move fingers and hands in the air, for example, the positions of fingers and hands are likely to change compared to operations with arms fixed. A situation that tends to occur occurs.

  Here, when the operation target is within the reach of the operator's hand, an operation (touch operation) in which a specific point is specified by the operator directly touching the operation target with a finger may be used. .

  With such touch operations, it is possible to avoid erroneous operations due to unintended movements by determining the presence or absence of touch operations on the operation target, and moreover, compared to motion operations that are the motion of fingers and hands in the air. Misalignment when specifying a specific location on the object is less likely to occur.

  In the touch operation, for example, detection processing for detecting the position and shape of the operator's hand from the captured image and determination processing for determining contact between the touch operation target and the hand are performed.

  As an example of the hand detection process, a method of detecting an area estimated to be a hand from an image captured by an optical camera using visible light based on the hand color and appearance characteristics acquired in advance is used. Sometimes. However, in this case, the color of the region corresponding to the hand of the photographed image differs from the skin color of the hand acquired in advance due to the difference in the light source, for example, whether the light source that irradiates the hand is sunlight or light from a lighting fixture. May end up. Further, the shape of the region corresponding to the hand of the photographed image may differ from the appearance of the hand acquired in advance depending on the direction of the hand with respect to the optical camera or the change in the shape of the hand such as grasping the fist. As a result, it may be difficult to detect the region corresponding to the hand from the captured image.

  Other examples of hand detection processing from captured images include a method that uses a difference in feature quantities (for example, saturation) between a hand and a background, or a feature for each frame between frames of temporally continuous captured images. There is a method of using the difference in quantity. However, since the worker wearing the optical camera moves and works in the field, the shooting location and shooting angle of the hand changes sequentially, so the method that uses the difference in the feature amount supports the hand from the shot image. In many cases, it is difficult to extract a region to be processed.

  As an example of the determination process, for example, there is a method of measuring each of the distance from the camera to the touch operation object and the finger using an optical stereo camera that captures an object from a plurality of different directions. Further, there is a method of detecting contact with a touch operation target by attaching a contact sensor to the fingertip of the worker. However, when using an optical stereo camera, the color and brightness of the object to be detected changes due to changes in the illumination position and angle accompanying the movement of the worker, making it difficult to handle and difficult to wear on the worker's body. There is a problem of becoming. Further, when a contact sensor is attached to the fingertip, it is assumed that the contact sensor hinders various operations of the operator.

  On the other hand, in recent years, distance sensors using infrared light that is less expensive than conventional ones are becoming widespread. There are multiple measurement methods for distance sensors, and one of them is based on the time difference between irradiating the object whose distance is to be measured with infrared light and receiving the reflected light from the object. There is a measurement method (TOF: Time-of-Flight method) to measure.

  A distance sensor using infrared light is less affected by the difference in the light source than an optical camera, and can accurately measure the distance to an object. The measurement result is output as a distance image based on the distance measurement result from the distance sensor to the object. The distance image is an image whose luminance changes according to the distance measurement result to the object. For example, as the distance from the distance sensor to the object increases, the luminance of the pixel corresponding to the object in the distance image increases. The distance image may be one in which the brightness of the pixel corresponding to the object in the distance image decreases as the distance from the distance sensor to the object increases. The distance image is not limited to changing the luminance of each pixel according to the distance, and pixel values other than the luminance, such as saturation and color information, may be changed according to the distance.

  FIG. 1 is a diagram illustrating an example of the distance image 3. At the time of the touch operation, the operator pays attention to the fingertip 92 that touches the touch operation target, so that the area corresponding to the fingertip 92 is often located near the center of the distance image 3 as shown in FIG. . On the other hand, the region corresponding to the hand back portion 93 from the wrist to the back of the hand is often located in the periphery of the distance image 3 because the region corresponding to the fingertip 92 comes near the center of the distance image 3.

  Further, when a touch operation is performed on the touch operation target with the fingertip 92, a difference in distance between the fingertip 92 and the touch operation target is reduced. Therefore, the luminance difference between the area corresponding to the touch operation object and the area corresponding to the fingertip 92 on the distance image 3 is also reduced. On the other hand, the difference in the distance between the touch operation object and the back of the hand 93 is larger than the difference in the distance between the touch operation object and the fingertip 92. Therefore, the luminance difference between the area corresponding to the touch operation object and the area corresponding to the hand back part 93 in the distance image 3 is compared with the luminance difference between the area corresponding to the touch operation object and the area corresponding to the fingertip 92. growing.

  That is, it is relatively easy to separate the hand back portion 93 from the touch operation target based on the brightness of the distance image 3, but separating the fingertip 92 from the touch operation target is equivalent to the touch operation target and the hand. It becomes difficult compared with the case where the back part 93 is separated. Further, when the luminance difference between the touch operation target and the fingertip 92 is small enough to be included in the inherent measurement error (noise) of the distance sensor, it becomes more difficult to separate the fingertip 92 from the touch operation target. .

  The finger region detection result 94 in FIG. 1 shows an example of an image when the finger 5 is detected from the distance image 3 based on the luminance. Here, the finger 5 refers to a portion where the fingertip 92 and the hand back portion 93 are combined. From the finger region detection result 94, it can be seen that the region corresponding to the hand back portion 93 is easy to detect from the distance image 3, but the region corresponding to the fingertip 92 is not clear and difficult to detect from the distance image 3.

  When the distance measurement method of the distance sensor is the TOF method, in addition to the distance image 3 described above, a reflection intensity image in which the intensity of reflected light is proportional to the luminance may be acquired. Since the distance image 3 obtained from the same distance sensor as the reflection intensity image uses a common irradiation unit and light receiving unit, the imaging range, that is, the angle of view is also common, and pixels at the same position in each image are It can be regarded as representing the same position of the same object.

  The reflection intensity image is an image whose luminance changes according to the intensity of the reflected light from the object. For example, as the intensity of the reflected light increases, the luminance of the pixel corresponding to the object in the reflection intensity image increases. The reflection intensity image may be one in which the luminance of the pixel corresponding to the object in the reflection intensity image decreases as the intensity of the reflected light from the object increases. In addition, the reflection intensity image is not limited to the case where the luminance of each pixel is changed according to the intensity of the reflected light, but pixel values other than the luminance, such as saturation and color information, are changed according to the intensity of the reflected light. Also good.

  The reflection intensity image uses the intensity of reflected light of infrared light applied to the object, and has a characteristic that is less susceptible to the color and appearance characteristics of the object surface than when an optical camera is used. However, the intensity of reflected light largely depends on the reflectance to infrared light based on the material of the object surface as well as the position and orientation of the object with reference to the distance sensor. When the infrared light emitted from the distance sensor is a point light source, for example, concentric irradiation unevenness may occur on the target object.

  FIG. 2 is a diagram illustrating an example of a reflection intensity image. In the reflected intensity image 4, at first glance, it appears that the fingertip 92 is clearly photographed as compared with the distance image 3 shown in FIG. For example, an example of a division result 95 in which the reflection intensity image 4 is divided into a plurality of areas based on the similarity in luminance between adjacent pixels in order to extract areas corresponding to individual objects from the reflection intensity image 4 is shown. As shown in the division result 95, a part of the member 96 may be divided as a region corresponding to the same object as the finger 5, and the shape of the region corresponding to the finger 5 may be extracted as an irregular shape such as the region 97. . The cause of such a region division failure may be the influence of the difference in the material of the object surface as described above or the irradiation unevenness of infrared light. The failure of area division includes a case where a plurality of objects are mistakenly combined as one area (erroneous combination), and a corresponding area is divided into a plurality of areas even if the object is the same (overdivision). .

  Therefore, in order to select a region having a high similarity between the shape and size of the finger 5 from the region division result of the reflection intensity image 4, the region selected from the region division result includes the presence or absence of misconnection or overdivision. In addition, a method for evaluating the similarity of the shape and size of the fingers 5 is required.

  Here, considering the influence of irradiation unevenness on the occurrence of misconnection and overdivision in region division in the reflection intensity image 4, misconnection and overdivision are likely to occur in the periphery of an image with low irradiation intensity, and the irradiation intensity is low. It tends to be difficult to occur at the center of a strong image.

  Furthermore, considering that the finger 5 at the time of the touch operation often has a region corresponding to the fingertip 92 at the center of the image, the finger 5 is positioned at the periphery of the image where misconnection and overdivision are likely to occur on the reflection intensity image 4. It is important to evaluate the similarity between the hand back portion 93 and the finger 5.

  On the other hand, in the region corresponding to the finger 5 in the distance image 3, the portion corresponding to the hand back portion 93 excluding the fingertip 92 tends to be highly similar to the finger 5. Therefore, it is considered that evaluation can be performed in consideration of misconnection and overdivision by evaluating the similarity with the region obtained from the reflection intensity image 4 on the basis of the region corresponding to the finger 5 obtained from the distance image 3. .

  Here, when comparing the shape and size of the area corresponding to the back portion 93 in the distance image 3 and the reflection intensity image 4, the angle of view and the shooting distance are the same in both images. Therefore, when no erroneous coupling or excessive division occurs in the reflection intensity image 4, the shape and size of the region corresponding to the back of the hand 93 can be regarded as the same as the region corresponding to the back of the hand 93 in the distance image 3. Approximate as much as possible. However, when viewed in pixel units, differences occur from the characteristics of each image shown below.

  For example, as a characteristic of the reflected light intensity corresponding to the luminance in the reflected intensity image 4, the intensity of the reflected light on the surface perpendicular to the irradiation direction (for example, the back of the hand 93 on the back of the hand) is strong even for an object having an approximately equal distance from the camera. Become. In addition, the intensity of the reflected light on the surface parallel to the irradiation direction (for example, the side surface of the hand back portion 93) becomes weaker than the change in distance.

  On the other hand, the distance measurement result strongly depends not on the intensity of the reflected light but on the time difference from irradiation of the object to reception of the reflected light. From these characteristics, the shape of the outline of the region on each image of the distance image 3 and the reflection intensity image 4 corresponding to the back of the hand 93 is similar as a whole, but a difference occurs in pixel units. It can be said that there is a case where similarity cannot be correctly evaluated by comparing the shapes of the regions in FIG.

  Further, similarity evaluation of regions by comparing the area (number of pixels) of the regions in the distance image 3 and the reflection intensity image 4 corresponding to the back of the hand 93 or the finger 5 is also conceivable. However, if both incorrect coupling and over-division occur in the region obtained from the reflection intensity image 4, it is considered that the area is appropriate but the shape is not appropriate. It may not be.

  Therefore, hereinafter, the object detection apparatus that can accurately extract the size, position, and shape of the finger 5 from the image will be described using the finger 5 as an example.

<First Embodiment>
FIG. 3 is a diagram illustrating a configuration example of the object detection system 1 according to the first embodiment. The object detection system 1 includes an imaging device 10, an object detection device 20, and a display device 30.

  The imaging device 10 includes an infrared light source that irradiates an object existing within the imaging range with infrared light, and a distance sensor, and outputs a distance image 3 and a reflection intensity image 4 generated by the distance sensor. Note that the imaging device 10 continuously captures an imaging range at predetermined time intervals, generates each of the distance image 3 and the reflection intensity image 4 as a time-series frame image, and sequentially outputs the generated frame images.

  As an installation form of the photographing apparatus 10, in order to realize a hands-free environment, a form that is mounted on the head of a person (user) such as an operator who uses the object detection system 1, for example, the head mounted camera 90 described above. Are used.

  In addition, although the imaging device 10 is configured to include an infrared light source and a distance sensor, the present invention is not limited thereto. For example, the infrared light source and the distance sensor are separated and each is mounted on the user's head. Good. In addition, it is assumed that the photographing apparatus 10 is attached to the user after the angle of view of the distance image 3 and the reflection intensity image 4 is adjusted so as to approach the same range as the field of view of the user wearing the photographing apparatus 10.

  The object detection device 20 receives each of the distance image 3 and the reflection intensity image 4 sequentially output as time-series frame images from the imaging device 10 as inputs. The object detection device 20 detects the position and shape (finger region) of the finger 5 using the distance image 3 and the reflection intensity image 4. As shown in FIG. 3, the object detection device 20 functionally includes an acquisition unit 21, a setting unit 22, an extraction unit 23, a division unit 24, an integration unit 25, and a determination unit 26.

  The acquisition unit 21 acquires the distance image 3 and the reflection intensity image 4 input from the imaging device 10, passes the acquired distance image 3 to the setting unit 22, and receives the acquired reflection intensity image 4 to the division unit 24. hand over.

  The setting unit 22 generates a luminance histogram indicating the luminance distribution of each pixel in the distance image 3 passed from the acquisition unit 21. Then, the setting unit 22 sets a threshold for the extraction unit 23 to extract a region that is a candidate region corresponding to the user's finger 5 from the distance image 3 based on the gradient change rate of the luminance histogram. Hereinafter, a region corresponding to the user's finger 5 in the distance image 3 or the reflection intensity image 4 is referred to as a “finger region”, and a region that is a candidate for the finger region is referred to as a “hand candidate region”. The threshold for extracting the finger candidate region is referred to as “hand candidate region luminance”.

  Here, the role of the finger candidate region and the finger candidate region luminance (hereinafter also referred to as “threshold value v2”), which is a threshold for extracting the finger candidate region, will be described in detail. In the present embodiment, the shape of the touch operation target is a plane, and it is assumed that when the user performs a touch operation on the touch operation target, the user substantially faces the touch operation target.

  Since the finger candidate region is used for determining the shape of the extraction candidate region selected from the reflection intensity image described later, the size and shape of the finger 5 on the distance image 3 are reproduced, and other objects such as a touch operation object are reproduced. It is desirable to extract as a region that does not include.

  Here, FIG. 4A is a side view of an example of the photographing environment of the distance image 3 by the photographing device 10, and shows the positional relationship between the photographing device 10, the finger 5, and the touch operation target 98. Yes.

  The threshold v2 for extracting an area including only the finger 5 as the finger candidate area is a value corresponding to the maximum distance among the distances from the imaging apparatus 10 that do not reach the touch operation target 98. It is preferable to set to.

  For example, if the threshold value v2 is too small (the shooting distance is short), the extracted finger candidate area is smaller than the original finger area. When the threshold value v2 is too large (the shooting distance is long), the finger candidate area includes the finger operation area 98 in addition to the finger 5, and thus the obtained finger candidate area has the same shape as the original finger area. May be difficult to use for determining the shape of the extraction candidate region selected from the reflection intensity image.

  Next, details of the luminance histogram in the distance image 3 including the finger 5 and the touch operation target 98 and the difference in the gradient change rate of the luminance histogram will be described.

  FIG. 4B is a diagram showing an example of a luminance histogram of the distance image 3 obtained under the shooting environment shown in FIG. 4A. The horizontal axis represents the luminance, and the vertical axis represents the number of pixels at each luminance. Show. Since the luminance of the distance image 3 corresponds to the distance from the photographing apparatus 10 to an object (photographing distance), it can be said that the luminance histogram represents the number of pixels for each photographing distance. The setting unit 22 normalizes the luminance of each pixel from 8 bits, that is, from 0 to 255 when generating the luminance histogram. Note that the luminance normalization range is an example, and the luminance may be normalized with a bit number smaller than or greater than eight bits.

  Here, assuming that the maximum value of the shooting distance is MT, when the luminance range of the distance image 3 is normalized by 8 bits (0 to 255), the luminance L with respect to a certain shooting distance T is expressed by equation (1).

    L = (T * 255) / MT (1)

  For example, when the shooting distance maximum value MT is 1500 mm, the shooting distance T is 600 mm, and the luminance range is normalized by 8 bits, the luminance L with respect to the shooting distance T is “102” from the equation (1).

  In the present embodiment, it is assumed that the touch operation target 98 has a planar shape. In many cases, when the user touches the touch operation target 98, it is considered that the user moves to a position substantially facing the touch operation target 98. For this reason, as shown in FIG. 4A, the touch operation target 98 is considered to exist in a range that can be regarded as being substantially perpendicular to the shooting direction of the shooting device 10. Therefore, the change in the area of the touch operation target 98 with respect to the change in the shooting distance becomes large at the position where the touch operation target 98 exists. That is, as shown in FIG. 4B, in the luminance histogram of the region corresponding to the touch operation target 98 in the distance image 3, the pixels corresponding to the luminance change centering on the luminance corresponding to the distance to the touch operation target 98. The change in the number will increase.

  On the other hand, since the finger 5 when performing the touch operation extends substantially along the shooting direction, the change in the area of the finger 5 with respect to the change in the shooting distance is smaller than the change in the area of the touch operation target 98. Therefore, the change in the number of pixels with respect to the luminance change in the luminance histogram of the region corresponding to the finger 5 is also reduced. In other words, in the luminance histogram of the distance image 3, the gradient change rate of the portion corresponding to the touch operation target 98 is larger than the gradient change rate of the portion corresponding to the finger 5.

  Next, estimation of the threshold value v2 will be described in detail.

  The gradient change rate of the luminance histogram of the distance image 3 is searched from the minimum value of the luminance toward the maximum value, and the luminance v0 corresponding to the shooting distance of the touch operation target 98 is the luminance at which the gradient change rate is equal to or greater than a predetermined threshold. And

  FIG. 4C is an enlarged view of the vicinity of the luminance v0 in FIG. 4B, which is a luminance histogram 82, and FIG. 4D is the luminance shown in FIG. 6 is a graph showing a gradient change rate of a histogram 82.

  As described above, the threshold value v2 is preferably set to a value corresponding to the maximum distance among the shooting distances that do not reach the touch operation target 98. However, since it is often difficult to directly determine the threshold value v2 due to the influence of noise included in the distance measurement result, a luminance value that is smaller than the luminance value v0 by a predetermined adjustment value f1 (for example, a luminance value corresponding to 30 mm) is used as the threshold value. It may be v2. The adjustment value f1 is preferably obtained in advance through actual work. The adjustment value f1 may be obtained in advance by computer simulation or the like.

  Here, in the assumed shooting environment, if the user gets too close to the touch operation target 98, the touch operation becomes difficult. Therefore, the shooting distance of the touch operation target 98 is considered to be at least 100 mm. Accordingly, the luminance corresponding to the minimum value (for example, 100 mm) of the shooting distance of the touch operation target 98 necessary for the touch operation is set as the threshold value v1. In this case, if a region including pixels from the luminance 0 to the threshold value v1 on the distance image 3 is set as the neighborhood region 81, the neighborhood region 81 includes the finger 5 and a part of the arm, but the touch operation target 98 is included. It is thought that it is not possible.

  Therefore, as shown in FIG. 4D, the threshold value v2 is set so that the luminance value v0 at which the gradient change rate of the luminance histogram 82 is larger than the predetermined gradient change rate r1 is increased from the threshold value v1 to the luminance increasing direction. This is obtained by searching and reducing the obtained luminance value v0 by the adjustment value f1. The reason why the threshold value v0 is searched from the threshold value v1 toward the direction in which the luminance increases is that the neighborhood area 81 less than the threshold value v1 is an area that includes only part of the user's fingers 5 and arms. This is because it is not necessary to start searching for the threshold value v0 from the luminance less than v1.

  However, the method for obtaining the threshold value v2 as described above is established when the area corresponding to the touch operation target 98 is larger in the distance image 3 than the user's finger area. For example, as shown in FIG. 5, a problem arises when the area difference between the finger region and the region corresponding to the touch operation target 98 is small in the distance image 3.

  FIG. 6 shows a luminance histogram when the area difference between the finger region and the region corresponding to the touch operation target 98 in the distance image 3 is small, and the gradient change rate of the luminance histogram. When the area difference between the two regions is small, the difference between the gradient change rate of the portion corresponding to the touch operation target 98 and the gradient change rate of the finger region portion is also reduced in the luminance histogram. In this case, as described above, it is assumed that the gradient change rate r1 is set as a threshold value that can identify the gradient change rate of the finger region and the gradient change rate of the portion corresponding to the touch operation target 98. When the area difference between the finger region and the region corresponding to the touch operation target 98 is small, the maximum value of the gradient change rate of the portion corresponding to the touch operation target 98 may not exceed the gradient change rate r1. Can occur. In this case, the threshold value v2 cannot be obtained based on the luminance value v0 corresponding to the gradient change rate r1.

  Therefore, the setting unit 22 in the present embodiment sets a threshold value v2 that can appropriately extract a finger candidate region even when the area difference between the finger region and the region corresponding to the touch operation target 98 is small in the distance image 3. To do.

  Specifically, the setting unit 22 approximates a function having a shape similar to the brightness histogram representing the area corresponding to the touch operation target 98 with respect to the brightness histogram of the distance image 3, and based on the approximated function. The shooting distance of the touch operation object 98 is estimated. Then, the setting unit 22 sets the threshold value v2 based on the estimated shooting distance of the touch operation target 98.

  Here, FIG. 7 shows a luminance histogram of an area corresponding to the touch operation target 98 in the distance image 3 (hereinafter referred to as “the luminance histogram of the touch operation target 98”). As described above, the touch operation target 98 exists in a range that can be regarded as being perpendicular to the shooting direction of the shooting apparatus 10 and has a flat shape. That is, the shooting distance at each position on the touch operation target 98 is substantially constant regardless of the position. Therefore, as shown in FIG. 7, the luminance histogram of the touch operation target 98 has a characteristic that the luminance corresponding to a certain shooting distance has a peak and has a shape that spreads in contrast to the left and right.

  Therefore, the setting unit 22 uses a function representing a feature of a shape spreading symmetrically around a certain peak, for example, a Gaussian function, as a function of a shape similar to the luminance histogram of the touch operation target 98. In the luminance histogram of the distance image 3, it can be said that the portion approximated to this function represents the luminance histogram of the touch operation target 98. Therefore, the setting unit 22 estimates the shooting distance of the touch operation target 98 using the peak and variance of the approximate function. Hereinafter, an example in which a Gaussian function is used as a function having a shape similar to the luminance histogram of the touch operation target 98 will be described.

  The extraction unit 23 extracts a finger candidate region from the distance image 3 based on the finger candidate region luminance (threshold value v2) set by the setting unit 22. The extraction unit 23 delivers the information regarding the extracted finger candidate area to the integration unit 25.

  The dividing unit 24 divides the reflection intensity image 4 delivered from the acquiring unit 21 into a plurality of regions according to a predetermined dividing algorithm based on a luminance difference between adjacent pixels. The dividing unit 24 passes information related to the divided areas to the integrating unit 25.

  The integration unit 25 acquires information about the finger candidate area in the distance image 3 from the extraction unit 23 and also acquires information about a plurality of areas of the divided reflection intensity image 4 from the division unit 24. The integration unit 25 selects, from among a plurality of regions of the reflection intensity image 4, a region whose contour is most similar to the finger candidate region as an extraction region, and integrates the finger candidate region and the extraction region, thereby Determine the finger area. Then, the integration unit 25 delivers information regarding the determined user's finger area to the determination unit 26.

  The determination unit 26 determines whether or not a touch operation has been performed on the touch operation target 98 based on the information related to the finger region delivered from the integration unit 25. When the touch operation is performed, the determination unit 26 controls the display device 30 so that an image corresponding to the touch operation is displayed on the display device 30.

  Details of the processing of the setting unit 22, the extraction unit 23, the division unit 24, the integration unit 25, and the determination unit 26 will be described later.

  The display device 30 includes a screen such as a liquid crystal screen or an organic electroluminescence screen. Various information such as an image corresponding to the touch operation is displayed on the screen of the display device 30 by display control by the determination unit 26. As an installation form of the display device 30, for example, a form like the above-described glasses-type display 91 is used in order to realize a hands-free environment. The display device 30 does not necessarily have a screen. For example, a retina display that draws an image on the retina by irradiating the user's retina while scanning with a laser may be used.

  In FIG. 3, as an example, a configuration in which the object detection device 20 is separated from the imaging device 10 and the display device 30 is shown, but the device configuration of the object detection system 1 is not limited to this. For example, the object detection device 20 may be integrated with one of the imaging device 10 and the display device 30, or the imaging device 10, the object detection device 20, and the display device 30 may be integrated as one device.

  Next, FIG. 8 shows an example of a configuration diagram when the object detection device 20 is realized by a computer.

  The computer 100 includes a CPU 102, a memory 104, and a nonvolatile storage unit 106. The computer 100 also includes an input / output (I / O) 108 that connects the computer 100 to each of the input device 112, the imaging device 10, and the display device 30 to transmit and receive data. Each of the CPU 102, the memory 104, the storage unit 106, and the I / O 108 is connected to each other via the bus 110.

  The input device 112 includes an input device for giving an instruction to the computer 100 by an operator of the computer 100 such as a keyboard and a mouse. The input device 112 includes a device that reads data recorded on the recording medium 116 such as a CD-ROM or a flash memory and writes data to the recording medium 116. Note that the input device 112 is not necessarily connected to the I / O 110.

  The storage unit 106 can be realized by an HDD (Hard Disk Drive), a flash memory, or the like.

  The storage unit 106 stores an object detection program 120 for causing the computer 100 to function as the object detection device 20 illustrated in FIG. The object detection program 120 includes an acquisition process 121, a setting process 122, an extraction process 123, a division process 124, an integration process 125, and a determination process 126.

  The CPU 102 reads the object detection program 120 from the storage unit 106 and expands it in the memory 104, and executes each process included in the object detection program 120. The CPU 102 operates as the acquisition unit 21 illustrated in FIG. 3 by executing the acquisition process 121. Further, the CPU 102 operates as the setting unit 22 illustrated in FIG. 3 by executing the setting process 122. Further, the CPU 102 operates as the extraction unit 23 illustrated in FIG. 3 by executing the extraction process 123. Further, the CPU 102 operates as the dividing unit 24 illustrated in FIG. 3 by executing the dividing process 124. Further, the CPU 102 operates as the integration unit 25 illustrated in FIG. 3 by executing the integration process 125. Further, the CPU 102 operates as the determination unit 26 illustrated in FIG. 3 by executing the determination process 126. As a result, the computer 100 that has executed the object detection program 120 operates as the object detection device 20.

  The function realized by the object detection program 120 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit).

  Next, the operation of the object detection system 1 according to the first embodiment will be described. For example, the photographing apparatus 10 notifies the object detection apparatus 20 of a photographing notification indicating that generation and output of the distance image 3 and the reflection intensity image 4 are started. When the object detection apparatus 20 acquires this shooting notification, the object detection apparatus 20 executes an object detection process shown in FIG.

  In the following, an example in which the user's finger 5 is detected from each image captured by the imaging device 10 will be described, but the object detected by the object detection device 20 is not limited to the finger 5, for example, on an object such as a pointer. It may be a specific part. Further, the distance image 3 is displayed with higher brightness of the corresponding portion as the distance from the photographing apparatus 10 to the object is longer, and the reflection intensity image 4 is higher as the reflection intensity of infrared light on the object is higher. A case where the corresponding portion is displayed with high brightness will be described as an example.

  In step S100 of the object detection process illustrated in FIG. 9, the acquisition unit 21 acquires the distance image 3 and the reflection intensity image 4 input from the imaging device 10 to the object detection device 20, and stores them in a predetermined area of the memory 104, for example. Remember each one.

  In step S200, the setting unit 22 acquires the distance image 3 from the memory 104, and performs finger candidate region luminance setting processing for setting the finger candidate region luminance (threshold value v2) used by the extraction unit 23 based on the distance image 3. Run.

  FIG. 10 is a flowchart showing an exemplary flow of the finger candidate region luminance setting process.

  In step S205, the setting unit 22 generates a luminance histogram from the acquired distance image 3.

  Next, in step S210, the setting unit 22 pays attention to the maximum luminance in the search range of the luminance (reference luminance) corresponding to the Gaussian function peak approximated to the luminance histogram of the distance image 3 in the luminance histogram of the distance image 3. Set to brightness. The reference luminance search range may be the entire range of the luminance histogram of the distance image 3 or may be a range up to a luminance corresponding to a distance (for example, 1 m) that can be touched by the user.

  Next, in step S215, the setting unit 22 determines whether or not the entire search range has been searched. If there is an unset luminance as the target luminance in the search range, the process proceeds to step S220. On the other hand, when the entire search range has been searched, the process returns to step S100 of the object detection process shown in FIG.

  In step S <b> 220, the setting unit 22 calculates the gradient change rate at the target luminance of the luminance histogram of the distance image 3 and acquires the number of pixels at the target luminance of the luminance histogram of the distance image 3. The setting unit 22 determines whether the gradient change rate in the target luminance is within a predetermined threshold value −a1 to + a1 and the number of pixels in the target luminance is equal to or greater than the predetermined threshold value a2. As a range of the thresholds −a1 to + a1, a range in which the gradient change rate can be regarded as approximately 0 is determined in advance. As shown in FIG. 11, the fact that the gradient change rate at a certain luminance is approximately 0 indicates that the luminance is a luminance having a peak in the luminance histogram. The threshold value a2 is a threshold value for removing a minute region on the distance image 3 as noise.

  If the gradient change rate in the target luminance is within the range of the threshold values −a1 to + a1 and the number of pixels is equal to or greater than the threshold value a2, the process proceeds to step S225. On the other hand, when the gradient change rate in the target luminance exceeds the range of the threshold values −a1 to + a1 or the number of pixels is less than the threshold value a2, the process proceeds to step S240.

  In step S225, the setting unit 22 sets the target luminance as the reference luminance. Then, the setting unit 22 defines a Gaussian function in which the reference luminance is the average luminance, the number of pixels in the reference luminance of the luminance histogram of the distance image 3 is the peak value, and the variance value is the variable. The setting unit 22 approximates the Gaussian function to the luminance histogram of the distance image 3 using the least square method or the like while appropriately changing the variance value.

  As shown in the upper diagram of FIG. 6, in the luminance histogram of the distance image 3, a portion having a luminance corresponding to a distance closer to the photographing apparatus 10 than a portion corresponding to the touch operation target 98 is a finger, an arm, or the like. Many pixels corresponding to are included. Therefore, when approximating the Gaussian function assuming the luminance histogram of the touch operation target 98 to the luminance histogram of the distance image 3 including the side where the distance is close to the photographing apparatus 10, that is, the side where the luminance is low, an appropriate approximation. May not be possible. Therefore, for example, as shown in FIG. 12, approximation may be performed using a range whose luminance is higher than the reference luminance as an approximation target range.

  Next, in step S230, the setting unit 22 determines whether the variance value of the Gaussian function approximated to the luminance histogram of the distance image 3 in step S225 is equal to or less than a predetermined threshold value a3. As described above, since the touch operation target 98 is assumed to be a planar object that is substantially directly opposite to the shooting direction of the shooting apparatus 10, a Gaussian function that assumes a luminance histogram of the touch operation target 98 is assumed. The variance value is considered to be a certain value or less. This value is determined in advance as a threshold value a3. That is, when the variance value of the approximated Gaussian function exceeds the threshold value a3, the luminance histogram portion of the distance image 3 corresponding to the Gaussian function is, for example, an object having an inclination with respect to the imaging direction of the imaging device 10 It is considered that this is a part representing Since such an object does not apply to the touch operation target 98 assumed in the present embodiment, the brightness histogram portion of the distance image 3 corresponding to the approximated Gaussian function is the brightness histogram of the touch operation target 98. It is determined that it is not a part to represent.

  If the variance value of the Gaussian function approximated to the luminance histogram of the distance image 3 is equal to or smaller than the threshold value a3, the process proceeds to step S235. If the variance value exceeds the threshold value a3, the process proceeds to step S240. Migrate to

  In step S235, the setting unit 22 determines whether the error between the approximated Gaussian function and the luminance histogram of the distance image 3 is equal to or less than a predetermined threshold value a4. The error can be, for example, the absolute value of the difference between the value of the Gaussian function at each luminance and the value of the luminance histogram of the distance image 3 or the sum of the squares. Further, the range for obtaining the error may be limited to the approximation target range applied when approximating the Gaussian function to the luminance histogram of the distance image 3 in step S225. If the error is equal to or smaller than the threshold value a4, the process proceeds to step S245. If the error exceeds the threshold value a4, the process proceeds to step S240.

  When the process proceeds to step S240, it indicates that the attention brightness currently set is not the reference brightness, or approximation of the Gaussian function using the attention brightness currently set as the reference brightness has failed. Therefore, the setting unit 22 sets a brightness value that is one smaller than the currently set target brightness as a new target brightness. Then, the process returns to step S215.

  On the other hand, when the process proceeds to step S245, it indicates that the approximation of the Gaussian function to the luminance histogram of the distance image 3 has succeeded when the target luminance currently set is set as the reference luminance. Therefore, the setting unit 22 sets the finger candidate region luminance (threshold value v2) based on the Gaussian function approximated in step S225. As described above, as the threshold value v <b> 2, a value corresponding to the maximum distance among shooting distances that do not reach the touch operation target 98, in other words, the nearest distance to the touch operation target 98 can be set. preferable. Specifically, as shown in FIG. 12, the setting unit 22 sets a luminance corresponding to a predetermined value on the lower side of the approximate Gaussian function, for example, 1σ (σ is the standard deviation of the Gaussian function), The threshold value v2 can be set.

  When the threshold value v2 is set, the process returns to the object detection process shown in FIG.

  Each of the threshold values a1 to a4 in the finger candidate region luminance setting process is obtained in advance based on the size of the touch operation target 98 on the distance image 3 obtained by actual measurement or computer simulation. These threshold values a1 to a4 are stored in advance in a predetermined storage area of the storage unit 106 together with the object detection program 120.

  Next, in step S300 of the object detection process shown in FIG. 9, the extraction unit 23 stores the distance image 3 acquired in step S100, the threshold value v2 set in step S200, and the threshold value stored in advance in the memory 104. Get v1.

  Then, the extraction unit 23 extracts from the distance image 3 a collection of pixels having a luminance equal to or lower than the threshold value v1 as the neighborhood area 81 (see FIG. 4D). Next, the extraction unit 23 extracts, from the distance image 3, pixels that are adjacent to the pixels included in the neighborhood area 81 and have a luminance equal to or lower than the threshold value v2. Furthermore, the extraction unit 23 repeatedly performs a process of extracting from the distance image 3 a pixel that is adjacent to the extracted pixel having a luminance equal to or lower than the threshold value v2 and has a luminance equal to or lower than the threshold value v2. Extracted from the distance image 3.

  The reason for extracting the finger candidate region 80 from the distance image 3 in this manner is that a foreign object that exists at a location away from the finger 5 and has a luminance higher than the threshold value v1 and lower than or equal to the threshold value v2 and different from the finger 5 This is to prevent erroneous extraction from the distance image 3 as 5.

  FIG. 13 is a diagram illustrating an example of a result of extracting the neighborhood area 81 and the finger candidate area 80 from the distance image 3. In addition to the finger 5, the distance image 3 includes a foreign object 83 whose luminance is higher than the threshold value v1 and lower than or equal to the threshold value v2. However, both the neighborhood region extraction result image 3A obtained by extracting the neighborhood region 81 from the distance image 3 and the extraction result image 3B obtained by extracting the finger candidate region 80 from the distance image 3 do not include the foreign matter 83 and only the finger 5 is extracted. Has been.

  Next, in step S350, the dividing unit 24 acquires the reflection intensity image 4 from the memory 104, and based on the luminance of each pixel included in the reflection intensity image 4, the reflection intensity by the reflection intensity image dividing process in step S400. The division target range of the image 4 is set.

  Here, the reason for setting the division target range will be described. For example, when the infrared light emitted from the distance sensor is a point light source, concentric irradiation unevenness occurs on the object depending on the distance and angle from the light source to the object. As a result, the intensity of the reflected light from the object is also different from the original difference in the reflection intensity based on the material of the object and the like due to irradiation unevenness. As a result, a luminance difference different from the original luminance difference also occurs in the luminance in the reflection intensity image 4, and a plurality of areas corresponding to different objects may not be divided with high accuracy.

  For example, as the infrared light irradiation unevenness increases, as shown in FIG. 14, there may be a difference in luminance between the central portion 54 and the peripheral portion 55 of the reflection intensity image 4.

  When performing a touch operation, the user tends to adjust the posture of the head or body so that the touch operation target 98 is aligned with the center of the visual field. For this reason, the infrared light emitted from the imaging device 10 mounted on the head or the like is strongest at the center of the visual field, that is, the central part 54 of the reflection intensity image 4, and is weakened as it spreads around the visual field.

  Since the central portion 54 of the reflection intensity image 4 has a high infrared light irradiation intensity, the intensity of the reflected light also increases, and accordingly, the difference in the reflection intensity between the touch operation object 98 and the finger 5 also increases.

  For example, in the example of FIG. 14, since the central portion 54 of the reflection intensity image 4 has a higher reflected light intensity than the peripheral portion 55, the touch operation target 98 and the fingers 5 are compared in the central portion 54 compared to the peripheral portion 55. The difference in the intensity of the reflected light increases. Therefore, in the range where the intensity of the reflected light of the reflection intensity image 4 is not less than a predetermined value as in the central portion 54, the detection target is detected from the region corresponding to the touch operation target 98 of the reflection intensity image 4 compared to the peripheral portion 55. A region corresponding to a certain finger 5 can be easily extracted. As a result, as shown in FIG. 14, in a divided image 6 (details will be described later) that is a result of dividing the reflection intensity image 4, a region corresponding to the fingertip 92 can be extracted with higher accuracy than the hand back portion 93.

  That is, by performing the division processing of the reflection intensity image 4 on the area where the reflection intensity of the reflection intensity image 4 is equal to or greater than a predetermined value, the extraction area corresponding to the finger 5 can be extracted with high accuracy. Therefore, an area having a reflection intensity equal to or greater than a predetermined value is set as a division target range.

  Specifically, first, the dividing unit 24 performs a smoothing process on the reflection intensity image 4 to generate a smoothed image. This is because by smoothing the reflection intensity image 4, noise components included in the reflection intensity image 4 are removed, and pixels that should be included in the division target range are lost, and the region division is not failed.

  FIG. 15 is a diagram for explaining a method for setting the division target range of the reflection intensity image 4. As illustrated in FIG. 15, the dividing unit 24 generates a luminance histogram 85 of each pixel included in the smoothed image obtained by smoothing the reflection intensity image 4 with the horizontal axis representing luminance and the vertical axis representing the number of pixels. When generating the luminance histogram 85, for example, the dividing unit 24 normalizes the luminance of each pixel from 8 bits, that is, from 0 to 255, but the number of bits is less than 8 bits or more than 8 bits. The luminance may be normalized.

  Then, the dividing unit 24 sets, for example, an area formed by pixels having a luminance equal to or higher than the threshold value ir0 as a division target range of the reflection intensity image 4.

  The threshold value ir0 is a value obtained in advance by computer simulation or the like as a value that can extract the extraction candidate region corresponding to the finger 5 from the reflection intensity image 4 with high accuracy. You may memorize it beforehand.

  FIG. 16 is a diagram illustrating an example of the flow of setting the division target range of the reflection intensity image 4. The dividing unit 24 generates the smoothed image 11 from the reflection intensity image 4, and further sets the division target range 12 from the luminance histogram 85 of the smoothed image 11.

  Next, in step S <b> 400, the dividing unit 24 acquires the reflection intensity image 4 from the memory 104, and reflects the reflection intensity image 4 into a plurality of regions based on the luminance of each pixel included in the reflection intensity image 4. Intensity image division processing is executed. The reflection intensity image division processing is executed for the division target range 12 in the reflection intensity image 4.

  FIG. 17 is a flowchart illustrating an example of the flow of the reflection intensity image division processing.

  First, in step S <b> 405, the dividing unit 24 assigns a label to each pixel of the reflection intensity image 4. Each label assigned to each pixel includes, for example, an identifier indicating an area including the pixel. In step S405, since it is unknown which region each pixel is included in, the dividing unit 24 assigns a label indicating a different region for each pixel to each pixel.

  Next, in step S410, the dividing unit 24 calculates, for each label, the number of pixels to which a label indicating the same region (hereinafter, “same label”) is assigned, and the same label is assigned. The average luminance value of the pixels is calculated for each label. Note that in the process of this step immediately after step S405, the labels assigned to the respective pixels are all different. Therefore, the number of pixels to which the same label is assigned is “1”, and the average luminance value of the pixels to which the same label is assigned is the luminance value of each pixel.

  Next, in step S415, the dividing unit 24 selects one unselected pixel from the reflection intensity image 4 as a target pixel.

  Next, in step S420, the dividing unit 24 determines whether or not the number of pixels assigned the same label as the target pixel is equal to or less than the threshold value b1. If the determination is affirmative, the process proceeds to step S425.

  In step S425, the dividing unit 24 determines whether or not the labels assigned to the target pixel and the adjacent pixels adjacent to the target pixel are different, and whether the luminance difference between the target pixel and the adjacent pixel is equal to or less than a threshold value b2. judge. If the determination is affirmative, the process proceeds to step S430.

  In addition, since the pixels of the reflection intensity image 4 are regularly arranged in a grid pattern (grid shape) in the horizontal direction and the vertical direction, the adjacent pixels are, for example, above, below, left, right, diagonally up right, There are a maximum of 8 diagonally below right, diagonally up left, and diagonally down left. Accordingly, in step S425, the dividing unit 24 refers to any one of the adjacent pixels.

  In step S430, the dividing unit 24 assigns the same label as the target pixel to the adjacent pixel referenced in step S425. The dividing unit 24 adds “1” to the number of label change pixels that is a variable for counting the number of pixels to which a new label is assigned, and stores the result in, for example, a predetermined area of the memory 104.

  As described above, the reflection intensity of infrared light may vary depending on, for example, the difference in material of the object and the orientation of the object. In general, the reflection intensity of the same object with respect to infrared light is a relatively close value. Often take. Therefore, when it is determined in step S425 that the luminance difference between the target pixel and the adjacent pixel is equal to or less than the threshold value b2, the target pixel and the adjacent pixel are highly likely to be pixels included in the same region corresponding to the same object. It can be judged.

  In other words, when the luminance values of the target pixel and the adjacent pixel are close to each other, the dividing unit 24 determines that the pixel is included in a region corresponding to the same object, and assigns the same label to the target pixel and the adjacent pixel. . By repeating this process, the area generated based on the label is expanded.

  On the other hand, if step S425 is negative, the process of step S430 is not executed, and the process proceeds to step S440.

  In step S440, the dividing unit 24 determines whether another adjacent pixel exists in the target pixel selected in step S415. In the case of an affirmative determination, the process returns to step S420, and the dividing unit 24 repeats the processes of steps S420 to S435 to combine adjacent pixels whose luminance difference is equal to or less than a predetermined threshold value b2 so that the same object We will expand the range of possible areas.

  Note that the processing of step S435 executed when step S420 is negative will be described later.

  If the determination in step S440 is negative, that is, if the pixels are combined by comparing the luminance difference between the target pixel selected in step S415 and all adjacent pixels around the target pixel, the process proceeds to step S445. To do.

  In step S445, the dividing unit 24 determines whether or not all the pixels included in the reflection intensity image 4 have been selected in step S415. If the determination is negative, the process returns to step S415, and the dividing unit 24 has not been processed. The other selected pixel is selected as a new pixel of interest. Thus, the dividing unit 24 repeats the processes of steps S415 to S445 for all the pixels included in the division target range 12 of the reflection intensity image 4, thereby further expanding the range of regions that are considered to be the same object.

  As described above, when the range of the region considered to be the same object is expanded, the luminance of the pixels in the same region may vary. As a result, there may be a case where pixels that should be included in different areas corresponding to different objects are erroneously determined as pixels included in the same area, and the different areas are erroneously combined. For this reason, in Step S425, by simply comparing the luminance difference between the target pixel and the adjacent pixel and determining whether or not the adjacent pixel is included in the same region as the target pixel, an erroneous determination regarding pixel combination is made. It can happen.

  Therefore, the dividing unit 24 compares the size (number of pixels) of the region including the target pixel with a predetermined threshold value. Specifically, when it is determined in step S420 that the number of pixels in the region has exceeded the threshold value b1 (that is, in the case of negative determination), the process proceeds to step S435, and the dividing unit 24 determines whether the pixel is between pixels. A new condition is added to determine the connection.

  Specifically, in step S435, the dividing unit 24 calculates the average brightness value of each pixel assigned the same label as the target pixel and the average brightness value of each pixel assigned the same label as the adjacent pixel. Then, it is determined whether or not these luminance average value differences are equal to or less than the threshold value b3. Even if there is a variation in luminance in the region including the target pixel and in the region including the adjacent pixel, each region is the same if the average luminance value in each region is equal to or less than the threshold value b3. This is because the possibility of representing an object is high. Therefore, in the case of an affirmative determination, the process proceeds to step S425, and the division unit 24 further determines the luminance difference between the pixels, and then assigns the same label to each pixel in step S430 to combine the regions. To do.

  On the other hand, if the determination process in step S435 is negative, the region including the target pixel and the region including the adjacent pixel are considered to represent different object regions, and thus the target pixel and the adjacent pixel are combined. Instead, the process proceeds to step S440.

  If step S445 is affirmative, the process proceeds to step S450.

  In step S450, the dividing unit 24 determines whether or not to end the division of the reflection intensity image 4 based on the ratio of the combined pixels to all the pixels included in the division target range 12 of the reflection intensity image 4. Specifically, the division unit 24 acquires the number of label change pixels changed to the new label stored in step S430 from the memory 104, and performs the process for all the numbers of pixels included in the division target range 12 of the reflection intensity image 4. It is determined whether or not the ratio of the number of label change pixels is equal to or less than the threshold value b4.

  In the case of negative determination, the process returns to step S410, and the dividing unit 24 further executes the division of the reflection intensity image 4 by repeating the processes of steps S410 to S450. Although there are many pixels whose labels are changed at the beginning of the division, the number of pixels whose labels are changed tends to decrease as the division proceeds. Accordingly, when the ratio of the pixels whose labels are updated is larger than the predetermined ratio, it is considered that the division is still in progress. Therefore, the determination in step S450 is negative, and the reflection intensity image 4 is further divided. That is, when the ratio of the pixel whose label is updated is larger than the threshold value b4, a region that approximates an object such as a finger 5 that is a detection target can be obtained by further dividing the reflection intensity image 4. .

  When step S450 is negative and the process returns to step S410, the dividing unit 24 initializes the number of label change pixels to “0” and then repeats the processes of steps S410 to S450.

  On the other hand, if the determination in step S450 is affirmative, the reflection intensity image division process ends, and the process returns to the object detection process shown in FIG.

  Thus, the reflection intensity image division processing in step S400 shown in FIG. 9 is completed, and the division target range 12 of the reflection intensity image 4 is divided into a plurality of regions based on the luminance of each image.

  FIG. 18 is a diagram illustrating a situation in which the reflected intensity image 4 is divided into a plurality of regions and the divided image 6 is generated by the reflected intensity image dividing process in step S400. 18 shows an example in which the entire reflection intensity image 4 is the division target range 12. Further, the image 41 is an enlarged view of the fingertip 92 portion of the reflection intensity image 4 before the reflection intensity image dividing process. As the reflection intensity image dividing process proceeds, the image 42 → the image 43 → the image 44 → the image 45 are sequentially displayed. It shows that the division status of the fingertip 92 changes.

  Next, in step S500 of the object detection process shown in FIG. 9, the integration unit 25 defines an area obtained by integrating the finger candidate area 80 extracted from the distance image 3 and the extraction area extracted from the reflection intensity image. An integration process for detecting the area representing 5 is executed. Details of the integration process in step S500 will be described below. Note that an image including a plurality of regions obtained from the reflection intensity image 4 as a result of the reflection intensity image division processing is referred to as a divided image 6, and is similar to the outline of the finger candidate region 80 among the plurality of regions of the divided image 6. The area to be performed is called an extraction area.

  FIG. 19 is a flowchart illustrating an example of the flow of integration processing.

  First, in step S505, the integration unit 25 has the largest number of pixels that overlap with the finger candidate region 80 extracted in step S300 from among the plurality of regions of the divided image 6 acquired in step S400 (extraction candidate region 7). ) Is selected.

  Next, in step S510, the integration unit 25 determines whether the ratio of the number of pixels overlapping the finger candidate region 80 to the number of pixels in the extraction candidate region 7 selected in step S505 is greater than or equal to a threshold value c1. To do. The threshold c1 is a value determined in advance by computer simulation or the like as a value for determining whether or not the extraction candidate region 7 is a region representing the finger 5 and is stored in a predetermined region of the memory 104, for example. ing.

  If the determination in step S510 is affirmative, it indicates that the ratio of the overlapping portion with the finger candidate region 80 in the extraction candidate region 7 is equal to or greater than the threshold c1. Therefore, the integration unit 25 determines that the extraction candidate area 7 selected in step S505 is a candidate for an area representing the finger 5, and the process proceeds to step S515.

  On the other hand, in the case of negative determination, since the ratio of the overlapping part with the finger candidate area 80 in the extraction candidate area 7 is less than the threshold c1, the extraction candidate area 7 selected in step S505 is an area representing the finger 5 It is considered that the ratio of the overlapping part with the finger candidate region 80 is small. Therefore, the integration unit 25 determines that the extraction candidate area 7 selected in step S505 is not an area representing the finger 5, and the process proceeds to step S555.

  In this case, as illustrated in FIG. 20, the division target range 12 set in step S <b> 350 in FIG. 9 is too wide, and the extraction candidate region 7 is erroneously combined with other regions other than the finger 5. It is considered that the size is larger than the finger candidate area 80 in the division target range 12.

  Therefore, in step S555, the integration unit 25 sets ON a division target range reduction flag stored in advance in a predetermined area of the memory 104, for example. The division target range reduction flag is an identifier indicating whether or not the division target range 12 of the reflection intensity image 4 is further reduced, and the integration unit 25 sets the division target range reduction flag to ON so that the division target range is turned on. 12 divisions 24 are instructed. Then, the integration process ends, and the process returns to the division process shown in FIG.

  In step S515, the integration unit 25 determines whether the ratio of the number of pixels in the extraction candidate area 7 to the number of pixels in the finger candidate area 80 is equal to or greater than the threshold c2. The threshold value c2 is a value determined in advance by computer simulation or the like as a value for determining whether or not the extraction candidate region 7 is a region representing the finger 5, and is stored in a predetermined region of the memory 104, for example. ing.

  When the determination in step S515 is affirmative, since the ratio of the extraction candidate area 7 to the finger candidate area 80 is equal to or greater than the threshold value c2, the integration unit 25 represents the finger 5 in the extraction candidate area 7 selected in step S505. It determines with it being an area | region candidate, and a process transfers to step S520.

  On the other hand, when the determination in step S515 is negative, since the ratio of the extraction candidate area 7 in the finger candidate area 80 is less than the threshold value c2, the extraction candidate area 7 selected in step S505 is an area representing the finger 5. It is considered that the ratio of the overlapping part with the finger candidate area 80 is small. Therefore, the integration unit 25 determines that the extraction candidate area 7 selected in step S505 is not an area representing the finger 5, ends the integration process, and returns to the division process illustrated in FIG.

  As described above, the cause of the negative determination in step S515 is, for example, that the extraction candidate region 7 is larger than the size of the finger candidate region 80 as shown in FIG. This is thought to be due to excessive division and reduction. In this case, for example, a message that prompts the user to adjust the number or position of the light sources that illuminate the finger 5 may be presented to the user.

  In step S520, the integration unit 25 extracts the contour lines of the finger candidate area 80 and the extraction candidate area 7, respectively. For extracting the contour line, for example, a known extraction method such as the Canny method of extracting the contour line by emphasizing the contour using a Gaussian filter can be used.

  Next, in step S525, the integration unit 25 extracts, from the contour line of the finger candidate area 80, a non-comparison part (excluded contour line) that is not used for comparison of the similarity with the contour line of the extraction candidate area 7. .

  Specifically, the integration unit 25 is adjacent to each pixel included in the contour line of the finger candidate region 80 extracted in step S520 and each pixel included in the contour line of the finger candidate region 80, and the finger candidate region 80 A luminance difference with an adjacent pixel outside the area is calculated. Then, the integration unit 25 extracts pixels whose luminance difference is smaller than a predetermined value from the pixels included in the contour line of the finger candidate region 80.

  In addition, for a pixel included in the contour line of the finger candidate region 80, when there are a plurality of adjacent pixels, for example, a pixel included in the contour line of the finger candidate region 80 and each of the plurality of adjacent pixels for the pixel The average value of the luminance differences may be compared with a predetermined value. Further, the maximum value of the luminance difference between the pixel included in the contour line of the finger candidate region 80 and each of a plurality of adjacent pixels for the pixel may be compared with a predetermined value.

  Furthermore, the integration unit 25 extracts, from the contour line of the finger candidate region 80 as an excluded contour line, a portion where a predetermined number of pixels having a luminance difference smaller than a predetermined value continue, and each of the excluded contour lines in the distance image 3. Get the coordinates of the endpoint of.

  As already described, in the distance image 3, since the distance between the touch operation target 98 and the fingertip 92 is short, the outline of the fingertip 92 is often not extracted from the distance image 3 in step S520. That is, in many cases, the contour line of the finger candidate region 80 extracted in step S520 is not the contour line of the original object. It is considered that there is a high possibility that an adjacent pixel whose luminance difference from a pixel included in the contour line of the finger candidate region 80 is smaller than a predetermined value represents the same object (in this case, the fingertip 92) as the finger candidate region 80. . Therefore, by extracting the contour line of the finger candidate region 80 that satisfies the above condition regarding the luminance difference between pixels as the excluded contour line from the contour line of the finger candidate region 80, a portion of the contour line different from the contour line of the original object Can be obtained.

  FIG. 22 is a diagram for explaining the outline of the finger candidate region 80. For example, when the finger candidate region 80 shown in FIG. 22A is extracted from the distance image 3, the excluded contour line 50 is extracted from the contour line of the finger candidate region 80 as shown in FIG. Further, the end portions of the excluded contour line 50 on the contour line of the finger candidate region 80 are extracted as end points 50A and 50B.

  Next, in step S530, the integration unit 25 acquires the position on the contour line of the extraction candidate region 7 corresponding to the end points 50A and 50B extracted from the contour line of the finger candidate region 80 in step S525. Specifically, the integration unit 25 sets the coordinates of the end points 50A and 50B of the excluded contour line 50 acquired in step S525 from the pixels included in the contour line of the extraction candidate region 7 extracted in step S520. Each of the coordinates of the nearest pixel that is the nearest pixel is acquired. The acquired coordinates of each nearest pixel represent the end points corresponding to the end points 50A and 50B of the finger candidate region 80.

  FIG. 23 is a diagram for explaining the outline of the extraction candidate region 7. For example, when the extraction candidate area 7 shown in FIG. 23A is selected from the reflection intensity image 4, as shown in FIG. 23B, the end point 50A of the finger candidate area 80 from the outline of the extraction candidate area 7 is displayed. , 50B, respectively, end points 52A, 52B are obtained.

  Next, in step S535, the integration unit 25 superimposes the division target range 12 of the reflection intensity image 4 set in step S350 of FIG. Then, the integration unit 25 intersects the coordinates of the intersection between the contour line of the finger candidate region 80 of the distance image 3 and the division target range 12 superimposed on the distance image 3 with the finger candidate region 80 that intersects the division target range 12. Acquired as the end point of the contour line.

  For example, as shown in FIG. 22A, when the division target range 12 is superimposed on the distance image 3, the contour of the finger candidate region 80 that intersects the division target range 12 as shown in FIG. Line end points 56A and 56B are acquired. A line connecting the end point 56A and the end point 56B is referred to as a divided contour line 56 in the finger candidate region 80.

  Since the divided contour line 56 is a contour line generated by the division target range 12, similarly to the excluded contour line 50, a contour line different from the contour line of the original object is shown. Therefore, of the contour lines of the finger candidate region 80 included in the division target range 12, the contour lines other than the excluded contour line 50 and the divided contour line 56 become the comparative contour line 8.

  Further, the integration unit 25 acquires the coordinates of the intersection of the contour of the extraction candidate region 7 of the reflection intensity image 4 and the division target range 12 as the end point of the contour of the extraction candidate region 7 that intersects the division target range 12. .

  For example, when the extraction candidate region 7 is selected from the reflection intensity image 4 in which the division target range 12 is set as shown in FIG. 23A, it intersects with the division target range 12 as shown in FIG. The end points 57A and 57B of the outline of the extraction candidate area 7 to be acquired are acquired. A line connecting the end point 57A and the end point 57B is referred to as a divided contour line 57 in the extraction candidate region 7.

  Similar to the divided contour line 56 in the finger candidate region 80, the divided contour line 57 is a contour line generated by the division target range 12, and thus shows a contour line different from the original object contour line. Therefore, among the contour lines of the extraction candidate region 7, the contour line between the end point 52A and the end point 57A and the contour line between the end point 52B and the end point 57B are the comparison contour line 9 in the extraction candidate region 7.

  Next, in step S540, the integration unit 25 acquires the shape feature of the comparison contour 8 in the finger candidate region 80 and the shape feature of the comparison contour 9 in the extraction candidate region 7, and each of the comparison contours 8, The difference of the shape feature of 9 is calculated. The shape features of the comparative contour lines 8 and 9 are quantified by, for example, the contour moment calculated by integrating each pixel included in the comparative contour lines 8 and 9. As a method for calculating the contour moment, a known method can be used. For example, a Hu invariant moment having properties that are invariant to contour similarity, translation, and rotation may be used.

  Therefore, the integration unit 25 calculates the difference between the Hu invariant moment value of the comparison contour 8 in the finger candidate region 80 and the Hu invariant moment value of the comparison contour 9 in the extraction candidate region 7. In addition, it represents that the shape characteristics of the comparison outlines 8 and 9 are similar, so that the difference of Hu invariant moment is small.

  In step S545, the integration unit 25 determines whether the difference between the Hu invariant moments of the comparison contour lines 8 and 9 calculated in step S540 is equal to or less than the threshold value c0. The threshold value c0 is a value determined in advance by computer simulation or the like as a value for determining whether or not the shape features of the comparison contour lines 8 and 9 are similar to each other. It is remembered.

  When the determination in step S545 is affirmative, it can be considered that the shape feature of the comparison contour 8 in the finger candidate region 80 is similar to the shape feature of the comparison contour 9 in the extraction candidate region 7. That is, the region corresponding to the finger 5 is correctly divided from the reflection intensity image 4 by the reflection intensity image division processing in step S400 of FIG. Therefore, the integration unit 25 determines that the finger candidate area 80 and the extraction candidate area 7 represent the same finger 5 and the process proceeds to step S550. In addition, the extraction candidate area | region 7 in case step S545 becomes affirmation determination becomes an extraction area | region.

  In step S550, the integration unit 25 integrates the finger candidate area 80 extracted from the distance image 3 in step S300 and the extraction area selected from the reflection intensity image 4 in step S505 to determine the user's finger area. . Here, integrating the finger candidate area 80 and the extraction area means that a logical OR is performed on the position of the pixel included in the finger candidate area 80 and the position of the pixel included in the extraction area to extract the finger candidate area 80. This means that one finger area is formed by overlapping the areas. Then, the integration process ends, and the process returns to the object detection process shown in FIG.

  On the other hand, if step S545 is negative, the integration unit 25 considers that the region corresponding to the finger 5 has not been correctly divided from the reflection intensity image 4 in the reflection intensity image division processing in step S400 of FIG. Therefore, the integration unit 25 determines that the finger candidate area 80 and the extraction candidate area 7 represent different objects, ends the integration process without integrating the finger candidate area 80 and the extraction candidate area 7, Returning to the object detection process shown in FIG.

  Next, in step S600 of the object detection process shown in FIG. 9, the dividing unit 24 determines whether or not the division target range reduction flag is set to ON. In the case of negative determination, it is considered that the division target range 12 of the reflection intensity image 4 is appropriately set in step S350, and the process proceeds to step S800.

  If the finger candidate area 80 and the extraction area are not integrated in the integration process in step S500 and the user's finger area 84 is not confirmed, the division condition of the divided image 6 is corrected and the reflection intensity in step S400 is corrected. The image division process may be executed again.

  For example, when the determination process in step S515 in FIG. 19 is negative, the user's finger area 84 is not confirmed, and the area corresponding to the finger 5 is finely divided into a plurality of areas on the divided image 6. One of the causes is considered to have been. Therefore, for example, the reflection intensity image division process may be executed again by increasing the threshold value b2 for determining the coupling condition with the adjacent pixels in the reflection intensity image division process in step S400 of FIG.

  On the other hand, if the determination process in step S600 is affirmative, the process proceeds to step S700.

  In step S700, the dividing unit 24 increases the threshold value related to the luminance used when setting the division target range 12 of the reflection intensity image 4 in step S350. For example, as illustrated in FIG. 15, if the threshold used when setting the initial division target range 12 is ir0, the dividing unit 24 newly divides ir1 with the threshold increased by ir with respect to the threshold ir0. The threshold value of the target range 12 is used. Thus, after the dividing unit 24 updates the threshold value used when setting the division target range 12, the process returns to step S350.

  In step S350, the dividing unit 24 sets an area formed by pixels having a luminance equal to or higher than the threshold (for example, the threshold ir1) updated in step S700 as the division target range 12 of the reflection intensity image 4.

  FIG. 24 is a diagram illustrating an example of the division target range 12 of the reflection intensity image 4 when the threshold value is increased to ir1. When the threshold value is increased to ir1, the division target range 12 becomes narrower than the division target range 12 (see FIG. 20) when the threshold value ir0 lower than the threshold ir1 is used. Therefore, since the possibility that the region corresponding to the finger 5 is erroneously combined with another region is lower than before the threshold update, in step S400, the dividing unit 24 reflects the shape closer to the shape of the finger 5. The intensity image 4 can be divided.

  FIG. 25 is a diagram illustrating an example of the finger region 84 integrated by the integration process illustrated in FIG. As shown in FIG. 25, the finger region 84 is represented as a region obtained by integrating the finger candidate region 80 of the distance image 3 and the extraction region of the divided image 6 obtained by dividing the reflection intensity image 4.

  In step S800 of the object detection process shown in FIG. 9, the determination unit 26 determines whether or not the user's finger 5 has touched the touch operation target 98 based on the finger region 84 determined in step S500. Execute the judgment process. Hereinafter, details of the determination process in step S800 will be described.

  FIG. 26 is a flowchart illustrating an example of the flow of determination processing.

  First, in step S805, the determination unit 26 determines whether or not the finger region 84 is confirmed in step S500. When a negative determination is made in step S515 and the process proceeds to this step, a negative determination is made because the finger area 84 has not been determined. In this case, the determination process ends and the object detection process itself shown in FIG. 9 also ends. On the other hand, if the finger area 84 is confirmed, the process proceeds to step S810.

  In step S <b> 810, the determination unit 26 estimates a fingertip position that is considered to be a part that touches the touch operation target 98 from the shape of the finger region 84. For example, the determination unit 26 can calculate the curvature for each point on the contour line of the finger region 84 and estimate the portion with the highest curvature as the fingertip position. Further, the determination unit 26 may estimate a predetermined position as a fingertip position by pattern matching or the like.

  Next, in step S815, the determination unit 26 acquires a shooting distance to the fingertip position based on the luminance value of the pixel of the distance image 3 corresponding to the estimated fingertip position. Further, the determination unit 26 estimates the shooting distance to a position on the touch operation target 98 concealed by the fingertip. The shooting distance to the position on the touch operation target 98 concealed by the fingertip is obtained by using the luminance values of the peripheral pixels of the finger region 84 in the distance image 3 using the fact that the touch operation target 98 is planar. Can be estimated. The peripheral pixels of the finger region 84 need to be pixels on the region corresponding to the touch operation target 98. Therefore, pixels that are outside the finger region 84 and are within a predetermined range from the finger region 84 are used as peripheral pixels. In addition, a pixel outside the finger region 84 and having a luminance value within a predetermined range centered on the reference luminance obtained in step S200 may be used as the peripheral pixel. This is because the reference luminance is a luminance value estimated as a luminance value corresponding to the shooting distance to the touch operation target 98.

  More specifically, the determination unit 26 obtains a straight line passing through at least two peripheral pixels around the finger region 84 and a pixel corresponding to the fingertip position in the distance image 3, and uses each luminance value of the peripheral pixels. The luminance value at the pixel position corresponding to the fingertip position is calculated by the linear interpolation. Then, the determination unit 26 can estimate the shooting distance corresponding to the luminance value calculated by linear interpolation as the shooting distance to the position on the touch operation target 98 concealed by the fingertip. Further, the determination unit 26 calculates a plane including positions corresponding to the plurality of peripheral pixels from the pixel positions and luminance values of the plurality of peripheral pixels around the finger region 84 in the distance image 3, and performs a touch operation on the plane. A plane corresponding to the object 98 is used. Then, the determination unit 26 obtains a position corresponding to the estimated fingertip position on a plane corresponding to the touch operation object 98, and the touch operation object whose distance between the position and the imaging device 10 is concealed by the fingertip. It can be estimated as a shooting distance up to a position on 98.

  Next, in step S820, the determination unit 26 determines the difference between the shooting distance to the fingertip position acquired or estimated in step S815 and the shooting distance to the position on the touch operation target 98 concealed by the fingertip. Then, it is determined whether or not it is equal to or less than a predetermined threshold value d1. As the threshold value d1, a value considering the thickness of the fingertip (for example, 1 cm) may be stored in advance in a predetermined area of the memory 104, for example. When the difference between the two shooting distances is larger than the threshold value d1, the determination unit 26 determines that the touch operation on the touch operation target 98 has not been performed, ends the determination process, and performs the object detection process illustrated in FIG. It also ends. If the difference between the two shooting distances is equal to or smaller than the threshold value d1, the process proceeds to step S825, and the determination unit 26 controls the display device 30 so that a screen corresponding to the touch operation is displayed.

  For example, the determination unit 26 uses the touch operation information such as the identification information and the installation position of the touch operation target 98 displayed on the touch operation target 98 as a character string or barcode as at least the distance image 3 and the reflection intensity image 4. One is obtained by image processing. Then, the determination unit 26 acquires presentation information associated in advance with the acquired touch operation information, for example, image data such as a manual from a predetermined storage device, and displays the acquired presentation information on the display device 30.

  The touch operation information is not limited to the case where the touch operation information is acquired by performing image processing on at least one of the distance image 3 and the reflection intensity image 4. For example, an optical camera may be attached to the user separately from the photographing apparatus 10, and the determination unit 26 may perform image processing on an image photographed by the optical camera and acquire touch operation information. In addition, a position sensor that detects the position of the user may be attached to the user, and the determination unit 26 may acquire the position information of the user when it is determined that the touch operation has been performed as touch operation information.

  When the display control on the display device 30 according to the touch operation is finished, the determination process is finished and the object detection process itself shown in FIG. 9 is also finished.

  As described above, according to the object detection device 20 according to the first embodiment, the luminance histogram of the distance image 3 is used when setting the threshold value v2 for extracting the finger candidate region 80 from the distance image 3. Then, in the luminance histogram of the distance image 3, a function similar to the shape of the luminance histogram of the touch operation target 98 is approximated, and the shooting distance of the touch operation target 98 is estimated based on the approximated function. Based on the estimated shooting distance of the touch operation target 98, a threshold value v2 for extracting the finger candidate region 80 is set. Thereby, even when the area difference between the finger region and the region corresponding to the touch operation target 98 is small, the threshold value can be set appropriately as compared with the case where the threshold value is set based on the gradient change rate of the luminance histogram. The finger region can be extracted with high accuracy.

  Further, according to the object detection device 20 according to the first embodiment, the reflection intensity image 4 is divided into a plurality of regions, and the finger candidate region 80 extracted from the distance image 3 based on the contour line from the divided regions. Get the extraction area similar to. Then, the object detection device 20 integrates the finger candidate region 80 and the extraction region, and specifies the position and shape of the object on the image of the imaging device 10. Therefore, as compared with the case where the extraction candidate region 7 and the finger candidate region 80 of the reflection intensity image 4 that overlaps with the finger candidate region 80 are simply integrated without determining the similarity of the contour lines of each region, the imaging device 10. The size, position, and shape of the object can be accurately extracted from the image.

  Further, according to the object detection device 20 according to the first embodiment, when determining the similarity of the contour lines of each region, the comparison contour line 8 among the contour lines of the finger candidate region 80 and the extraction candidate region 7 is compared. , 9 are used to determine the similarity between the finger candidate region 80 and the extraction candidate region 7. Therefore, compared to the case where the similarity between the finger candidate region 80 and the extraction candidate region 7 is determined using all the contour lines surrounding the finger candidate region 80 and the extraction candidate region 7, the size of the object from the image of the imaging device 10 is determined. The position, shape, and shape can be extracted with higher accuracy.

  In addition, according to the object detection device 20 according to the first embodiment, the reflection intensity image dividing process is executed on a region where the reflection intensity of the reflection intensity image 4 is equal to or greater than a predetermined value. Therefore, the object detection device 20 accurately extracts the size, position, and shape of the object from the image captured by the imaging device 10 even when the object to be detected has light source unevenness. Can do.

  Further, according to the object detection device 20 according to the first embodiment, when the ratio of the overlapping portion with the finger candidate region 80 in the extraction candidate region 7 selected from the reflection intensity image 4 is smaller than the threshold value, the reflection intensity image 4 The division target range 12 is reduced, and the extraction candidate area 7 is selected again. Therefore, for example, even when the object to be detected has light source irradiation unevenness, the object detection device 20 extracts the extraction region close to the actual size, position, and shape of the finger 5 from the reflection intensity image 4. can do.

  Note that the imaging device 10 is not necessarily required in the object detection system 1. For example, the acquisition unit 21 may acquire the distance image 3 and the reflection intensity image 4 that have been captured in advance and stored in an electronic device such as a computer from the electronic device via a network such as the Internet.

  In the first embodiment, the case where the division target range is set in the reflection intensity image division processing has been described. However, the entire reflection intensity image 4 may be the target of the division processing without setting the division target range. . In this case, comparison outlines 8 and 9 may be defined as shown in FIGS.

  Specifically, for example, when the finger candidate region 80 shown in FIG. 27A is extracted from the distance image 3, as shown in FIG. 27B, the contour line of the finger candidate region 80 and the distance image 3 Find the intersection with the border. And this intersection is acquired as the end points 51A and 51B of the outline of the finger candidate area | region 80 which cross | intersects a frame line. Of the contour lines of the finger candidate region 80, the contour line overlapping with the frame line of the distance image 3 (frame contour line 51) is a contour line generated by the angle of view of the distance image 3. Similarly, a contour line different from the contour line of the original object is shown. Therefore, among the contour lines of the finger candidate region 80, a contour line other than the excluded contour line 50 and the frame contour line 51 may be used as the comparative contour line 8.

  For example, when the extraction candidate area 7 shown in FIG. 28A is extracted from the reflection intensity image 4, the outline of the extraction candidate area 7 and the frame of the reflection intensity image 4 are shown in FIG. Find the intersection with the line. And this intersection is acquired as the end points 53A and 53B of the outline of the extraction candidate area | region 7 which cross | intersects a frame line. Of the contour lines of the extraction candidate region 7, the contour line (frame contour line 53) that overlaps the frame line of the reflection intensity image 4 is a contour line generated by the angle of view of the reflection intensity image 4. A contour line different from the contour line is shown. Therefore, among the contour lines of the extraction candidate region 7, the contour line between the end point 52A and the end point 53A and the contour line between the end point 52B and the end point 53B may be used as the comparative contour line 9.

  Further, when the division target area is not set, step S555 of the integration process shown in FIG. 19 and steps S600 and S700 of the object detection process shown in FIG. 9 are omitted. Also, in this case, if a negative determination is made in step S510 of the integration process shown in FIG. 19, for example, the number or position of the light source that illuminates the finger 5 is adjusted, as in the case where a negative determination is made in step S515. A message prompting the user may be presented to the user.

Second Embodiment
Next, a second embodiment will be described. In the object detection system according to the second embodiment, the same parts as those of the object detection system 1 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  As illustrated in FIG. 3, the object detection system 1A includes an imaging device 10, an object detection device 20A, and a display device 30. Functionally, the object detection apparatus 20A includes an acquisition unit 21, a setting unit 22A, an extraction unit 23, a division unit 24, an integration unit 25, and a determination unit 26.

  Here, changes in the relative distance and posture between the touch operation target 98 and the photographing apparatus 10 mounted on the user's head or the like are considered to be small during the touch operation. Therefore, it is considered that the change in the luminance histogram between the frames of the distance image 3 is small. The fact that the change in the brightness histogram of the distance image 3 is small is considered that the change in the reference brightness when approximating a Gaussian function assuming the brightness histogram of the touch operation target 98 is also small.

  Accordingly, when the setting unit 22A in the second embodiment succeeds in approximating the Gaussian function in the previous frame, the setting unit 22A performs a predetermined process centering on the reference luminance in the previous frame when searching for the reference luminance in the current frame. Limit the search range to a range. Other functions of the setting unit 22A are the same as those of the setting unit 22 in the first embodiment.

  The object detection device 20A can be realized by a computer 100 shown in FIG. The storage unit 106 of the computer 100 stores an object detection program 120A for causing the computer 100 to function as the object detection device 20A illustrated in FIG. The object detection program 120A includes an acquisition process 121, a setting process 122A, an extraction process 123, a division process 124, an integration process 125, and a determination process 126.

  The CPU 102 reads the object detection program 120A from the storage unit 106, expands it in the memory 104, and executes each process included in the object detection program 120A. The CPU 102 operates as the setting unit 22A illustrated in FIG. 3 by executing the setting process 122A. Other processes are the same as those of the object detection program 120 in the first embodiment. Thereby, the computer 100 which executed the object detection program 120A operates as the object detection apparatus 20A.

  The function realized by the object detection program 120A can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC.

  Next, the operation of the object detection system 1A according to the second embodiment will be described. In the second embodiment, the finger candidate region luminance setting process executed in step S200 of the object detection process shown in FIG. 9 is different from the first embodiment. Therefore, in the following, with reference to FIG. 29, the finger candidate region luminance setting process in the second embodiment will be described. In addition, about the process similar to the finger candidate area | region brightness setting process in 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  First, in step S205, the setting unit 22A generates a luminance histogram from the distance image 3. Then, in the next step S206, the setting unit 22A determines whether or not the approximation of the Gaussian function assuming the touch operation target 98 with respect to the luminance histogram of the distance image 3 is successful for the distance image 3 of the previous frame. To do. Specifically, the setting unit 22A checks whether or not the reference luminance is recorded in a predetermined storage area in association with the frame number of the previous frame. In this storage area, when the finger candidate area luminance setting process for the distance image 3 of the previous frame is executed, if the approximation of the Gaussian function is successful, the Gaussian function that has been successfully approximated in step S290 described later. The reference brightness is recorded. Therefore, when the reference luminance of the previous frame is recorded in this storage area, the setting unit 22A determines that approximation of the Gaussian function has succeeded in the previous frame, and acquires information on the reference luminance. The processing moves to step S207. If the reference luminance of the previous frame is not recorded, the setting unit 22A determines that approximation of the Gaussian function has failed in the previous frame, and the process proceeds to step S201.

  In step S207, the setting unit 22A determines a predetermined range centered on the reference luminance of the previous frame acquired in step S206 as a reference luminance search range in the current frame. The predetermined range is a range smaller than the entire luminance range of the luminance histogram of the distance image 3 of the current frame.

  Thereafter, in steps S250 to S285, processing similar to that in steps S210 to S245 of the finger candidate region luminance setting processing in the first embodiment is executed. When a negative determination is made in step S255, the process proceeds to step S201 instead of returning to step S100.

  In step S201, the setting unit 22A executes the finger candidate area luminance setting process in the first embodiment. Specifically, this is the same as steps S210 to S245 shown in FIG. Then, the process proceeds to step S290 after step S285.

  In step S290, setting unit 22A stores the current reference luminance in a predetermined storage area in association with the frame number of the current frame, and the process proceeds to step S300.

  As described above, according to the object detection device 20A according to the second embodiment, the search range of the reference luminance when the Gaussian function is approximated to the luminance histogram of the distance image 3 is limited. Can be reduced.

<Third Embodiment>
Next, a third embodiment will be described. Note that in the object detection system according to the third embodiment, the same parts as those of the object detection systems 1 and 1A according to the first and second embodiments are denoted by the same reference numerals, and detailed description thereof is omitted.

  As illustrated in FIG. 3, the object detection system 1B includes an imaging device 10, an object detection device 20B, and a display device 30. Functionally, the object detection apparatus 20B includes an acquisition unit 21, a setting unit 22B, an extraction unit 23, a division unit 24, an integration unit 25, and a determination unit 26.

  Here, changes in the relative distance and posture between the touch operation target 98 and the photographing apparatus 10 mounted on the user's head or the like are considered to be small during the touch operation. In addition, since the operation of bringing the finger close to the touch operation target 98 is an operation along the shooting direction of the imaging device 10, the change in the position and size of the finger within the imaging range by the imaging device 10 is considered to be small. . Therefore, it is considered that the change in the luminance histogram between the frames of the distance image 3 is small. The fact that the change in the brightness histogram of the distance image 3 is small means that the change in the parameters of the Gaussian function approximated to the brightness histogram of the distance image 3 (the reference brightness corresponding to the average, the number of pixels in the reference brightness, and the variance value) is also small. it is conceivable that.

  Accordingly, when the setting unit 22B in the third embodiment succeeds in approximating the Gaussian function in the previous frame, the setting unit 22B calculates the Gaussian function that has been approximated in the previous frame and the luminance histogram of the distance image 3 in the current frame. The error is compared with a threshold value a4. If the error is equal to or smaller than the threshold value a4, the setting unit 22B assumes that the Gaussian function that has been successfully approximated in the previous frame has also been successfully approximated in the current frame. Other functions of the setting unit 22B are the same as those of the setting units 22 and 22A in the first and second embodiments.

  The object detection device 20B can be realized by the computer 100 shown in FIG. The storage unit 106 of the computer 100 stores an object detection program 120B for causing the computer 100 to function as the object detection device 20B illustrated in FIG. The object detection program 120B includes an acquisition process 121, a setting process 122B, an extraction process 123, a division process 124, an integration process 125, and a determination process 126.

  The CPU 102 reads out the object detection program 120B from the storage unit 106, expands it in the memory 104, and executes each process included in the object detection program 120B. The CPU 102 operates as the setting unit 22B illustrated in FIG. 3 by executing the setting process 122B. Other processes are the same as those of the object detection program 120 in the first embodiment. As a result, the computer 100 that has executed the object detection program 120B operates as the object detection device 20B.

  The function realized by the object detection program 120B can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC or the like.

  Next, the operation of the object detection system 1B according to the third embodiment will be described. In the third embodiment, the finger candidate region luminance setting process executed in step S200 of the object detection process shown in FIG. 9 is different from the first embodiment. Therefore, hereinafter, the finger candidate region luminance setting process in the third embodiment will be described with reference to FIG. In addition, about the process similar to the finger candidate area | region brightness setting process in 1st or 2nd embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  First, in step S205, the setting unit 22B generates a luminance histogram from the distance image 3. Then, in the next step S208, the setting unit 22B determines whether or not the approximation of the Gaussian function assuming the touch operation target 98 with respect to the luminance histogram of the distance image 3 is successful for the distance image 3 of the previous frame. To do. Specifically, the setting unit 22B checks whether or not a Gaussian function parameter is recorded in a predetermined storage area in association with the frame number of the previous frame. In this storage area, when the finger candidate area luminance setting process for the distance image 3 of the previous frame is executed, if the approximation of the Gaussian function is successful, the Gaussian function that has been successfully approximated in step S291 described later. The parameters are recorded. Therefore, when the parameter of the Gaussian function of the previous frame is recorded in this storage area, the setting unit 22B determines that the approximation of the Gaussian function has succeeded in the previous frame, and the process proceeds to Step S209. . If the parameter of the Gaussian function of the previous frame is not recorded, the setting unit 22A determines that approximation of the Gaussian function has failed in the previous frame, and the process proceeds to step S201.

  In step S209, the setting unit 22B acquires the parameter of the Gauss function of the previous frame from a predetermined storage area. Next, in step S276, the setting unit 22B calculates an error between the Gaussian function defined by the parameter acquired in step S209 and the luminance histogram of the distance image 3 of the current frame generated in step S205. It is determined whether or not the threshold value is a4 or less. If the error is equal to or smaller than the threshold value a4, the process proceeds to step S286. If the error exceeds the threshold value a4, the process proceeds to step S201.

  In step S286, the setting unit 22B sets the threshold value v2 based on the Gaussian function defined by the parameter acquired in step S209 as in step S245 of the finger candidate region luminance setting process in the first embodiment.

  On the other hand, in step S201, the setting unit 22B executes the finger candidate region luminance setting process in the first embodiment. Specifically, this is the same as steps S210 to S245 shown in FIG.

  Next, in step S291, the setting unit 22B stores the parameters of the Gaussian function acquired in step S209 in a predetermined storage area in association with the frame number of the current frame, and the process proceeds to step S300.

  As described above, according to the object detection device 20B according to the third embodiment, the Gaussian function of the previous frame that has been successfully approximated is used for comparison with the luminance histogram of the distance image of the current frame. It is possible to reduce the calculation amount of processing and approximation of Gaussian function.

  In each of the above embodiments, the case where a Gaussian function is used as a function approximated to the luminance histogram of the distance image has been described, but the present invention is not limited to this. Any function that is similar to the shape of the luminance histogram of the touch operation object 98 may be used. Specifically, any function that has a bilaterally symmetric shape around a certain luminance may be used. When a function other than a Gaussian function is used, a parameter corresponding to the function may be used. For example, the luminance at the center of the function may be used as the reference luminance, and an index indicating the spread of the function may be used as an index similar to the variance value in the Gaussian function.

  Moreover, in each embodiment, the finger 5 was taken as an example and the object detection apparatus which can extract the size, position, and shape of the finger 5 from an image with high accuracy has been described. However, it goes without saying that the object detection apparatus according to the present invention can accurately extract the size, position, and shape of an object from an image even if it is an object other than the finger 5.

  As mentioned above, although this invention was demonstrated using each embodiment, this invention is not limited to the range as described in each embodiment. Various changes or improvements can be added to each embodiment without departing from the gist of the present invention, and embodiments to which the changes or improvements are added are also included in the technical scope of the present invention. For example, the processing order may be changed without departing from the scope of the present invention.

  Moreover, although each embodiment demonstrated the aspect by which the object detection program 120, 120A, 120B was beforehand memorize | stored (installed) in the memory | storage part, it is not limited to this. The object detection program according to the disclosed technology can also be provided in a form recorded on a computer-readable recording medium 116. For example, the object detection program according to the disclosed technology can be provided in a form recorded in a portable recording medium such as a CD-ROM, a DVD-ROM, and a USB memory. The object detection program according to the disclosed technique can also be provided in a form recorded in a semiconductor memory such as a flash memory.

  Regarding the embodiments including the above embodiments, the following additional notes are disclosed.

(Appendix 1)
On the computer,
Obtain a distance image that represents the distance to the object with the luminance according to the distance, and a reflection intensity image that represents the intensity of the reflected light from the object with the luminance,
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the object to be detected in the distance image, the brightness histogram of the operation target is estimated. ,
Using the distance to the operation target determined based on the estimated brightness histogram of the operation target, extract a candidate area representing the detection target object from the distance image;
Dividing the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
A process including detecting an extracted area whose shape is similar to the contour line of the extracted candidate area, and an area obtained by integrating the candidate areas, as an area representing the detection target object, among the plurality of areas. The object detection method which performs.

(Appendix 2)
The object detection method according to supplementary note 1, wherein the function is a left-right symmetric function having a reference luminance peaking at a luminance change rate of a predetermined value or less in the luminance histogram of the distance image.

(Appendix 3)
The object detection method according to claim 2, wherein when the function is approximated to a luminance histogram of the distance image, a luminance portion corresponding to a distance greater than or equal to a distance corresponding to the reference luminance is approximated.

(Appendix 4)
The object detection method according to supplementary note 2 or supplementary note 3, wherein the function is a Gaussian function.

(Appendix 5)
As the distance image, a plurality of frames are sequentially acquired in time series,
If an error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value in the previous frame, the reference brightness in the current frame is searched from a predetermined range centered on the reference brightness searched in the previous frame. The object detection method according to any one of appendix 2 to appendix 4.

(Appendix 6)
As the distance image, a plurality of frames are sequentially acquired in time series,
In the previous frame, if the error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value, whether or not the error between the function of the previous frame and the brightness histogram of the distance image of the current frame is less than or equal to a predetermined value The object detection method according to any one of claims 2 to 5, wherein the function of the previous frame is estimated as a luminance histogram of the current frame of the operation target object when the error is equal to or less than a predetermined value.

(Appendix 7)
Of the plurality of areas, the ratio of overlapping with the extracted candidate area is equal to or greater than a predetermined ratio, and the feature amount of the shape of the outline of the candidate area and the outline corresponding to the outline of the candidate area The object detection method according to any one of supplementary notes 1 to 6, wherein a region where a difference from a feature amount of a line shape is equal to or less than a threshold is the extraction region.

(Appendix 8)
The feature amount of the shape of the contour line of the candidate region is determined in advance from the contour pixel and the brightness of the contour pixel, out of the candidate region and out of the contour pixel among the contour pixels included in the contour line of the candidate region. The object according to appendix 7, which is obtained from a section in which pixels having a difference in luminance from neighboring pixels in a range that is smaller than a threshold are continuous and a section that overlaps a frame line of the distance image is removed from the outline of the candidate area. Detection method.

(Appendix 9)
The object detection method according to any one of appendix 1 to appendix 8, wherein a division target range of the reflection intensity image divided into the plurality of regions is reduced to a range having a predetermined luminance or more.

(Appendix 10)
When the extraction area having the largest ratio of overlapping with the candidate area is less than the predetermined ratio among the plurality of areas, until the ratio of overlapping with the candidate area becomes equal to or higher than the predetermined ratio, The object detection method according to appendix 9, wherein the predetermined luminance is increased to further reduce the division target range of the reflection intensity image divided into the plurality of regions.

(Appendix 11)
The object detection method according to claim 1, wherein the object to be detected is a human finger.

(Appendix 12)
The object detection method according to any one of appendix 1 to appendix 10, wherein the distance image and the reflection intensity image are images captured by an imaging device attached to a human body.

(Appendix 13)
An acquisition unit that acquires a distance image representing the distance to the object with luminance corresponding to the distance, and a reflection intensity image representing the intensity of reflected light from the object with luminance;
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the detection target object in the distance image, the brightness histogram of the operation target is estimated. An estimation unit;
An extraction unit that extracts a candidate area representing the detection target object from the distance image using a distance to the operation target determined based on the estimated luminance histogram of the operation target;
A dividing unit that divides the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
An integration unit that detects an extracted region similar in shape to the contour line of the extracted candidate region, and a region obtained by integrating the candidate regions, as a region representing the detection target object, among the plurality of regions;
An object detection apparatus including:

(Appendix 14)
The object detection device according to supplementary note 13, wherein the function is a left-right symmetric function having a peak at a reference luminance at which a luminance change rate is a predetermined value or less in the luminance histogram of the distance image.

(Appendix 15)
15. The object detection apparatus according to appendix 14, wherein the estimation unit approximates a luminance portion corresponding to a distance equal to or greater than a distance corresponding to the reference luminance when the function is approximated to a luminance histogram of the distance image.

(Appendix 16)
The object detection apparatus according to appendix 14 or appendix 15, wherein the function is a Gaussian function.

(Appendix 17)
The acquisition unit sequentially acquires a plurality of frames in time series as the distance image,
When the error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value in the previous frame, the estimation unit centers on the reference brightness searched in the previous frame. The object detection device according to any one of Supplementary Note 14 to Supplementary Note 16, wherein search is performed from a predetermined range.

(Appendix 18)
The acquisition unit sequentially acquires a plurality of frames in time series as the distance image,
When the error between the function and the brightness histogram of the distance image is equal to or less than a predetermined value in the previous frame, the estimation unit determines that the error between the function of the previous frame and the brightness histogram of the distance image of the current frame is predetermined. It is determined whether or not the value is equal to or less than a value, and when the error is equal to or less than a predetermined value, the function of the previous frame is estimated as a luminance histogram of the current frame of the operation target. Object detection device.

(Appendix 19)
On the computer,
Obtain a distance image that represents the distance to the object with the luminance according to the distance, and a reflection intensity image that represents the intensity of the reflected light from the object with the luminance,
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the object to be detected in the distance image, the brightness histogram of the operation target is estimated. ,
Using the distance to the operation target determined based on the estimated brightness histogram of the operation target, extract a candidate area representing the detection target object from the distance image;
Dividing the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
A process including detecting an extracted area whose shape is similar to the contour line of the extracted candidate area, and an area obtained by integrating the candidate areas, as an area representing the detection target object, among the plurality of areas. An object detection program that executes

(Appendix 20)
Item 20. The object detection program according to Item 19, wherein the function is a left-right symmetric function having a peak at a reference luminance at which a luminance change rate is a predetermined value or less in the luminance histogram of the distance image.

(Appendix 21)
The object detection program according to appendix 20, wherein when the function is approximated to a luminance histogram of the distance image, a luminance portion corresponding to a distance greater than or equal to a distance corresponding to the reference luminance is approximated.

(Appendix 22)
Item 22. The object detection program according to appendix 20 or appendix 21, wherein the function is a Gaussian function.

(Appendix 23)
As the distance image, a plurality of frames are sequentially acquired in time series,
If an error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value in the previous frame, the reference brightness in the current frame is searched from a predetermined range centered on the reference brightness searched in the previous frame. The object detection program according to any one of appendix 20 to appendix 22.

(Appendix 24)
As the distance image, a plurality of frames are sequentially acquired in time series,
In the previous frame, if the error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value, whether or not the error between the function of the previous frame and the brightness histogram of the distance image of the current frame is less than or equal to a predetermined value The object detection program according to any one of appendices 20 to 23, wherein the function of the previous frame is estimated as a luminance histogram of the current frame of the operation target.

(Appendix 25)
On the computer,
Obtain a distance image that represents the distance to the object with the luminance according to the distance, and a reflection intensity image that represents the intensity of the reflected light from the object with the luminance,
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the object to be detected in the distance image, the brightness histogram of the operation target is estimated. ,
Using the distance to the operation target determined based on the estimated brightness histogram of the operation target, extract a candidate area representing the detection target object from the distance image;
Dividing the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
A process including detecting an extracted area whose shape is similar to the contour line of the extracted candidate area, and an area obtained by integrating the candidate areas, as an area representing the detection target object, among the plurality of areas. The storage medium which memorize | stored the object detection program which performs.

1, 1A, 1B Object detection system 3 Distance image 4 Reflection intensity image 7 Extraction candidate area 10 Imaging device 20, 20A, 20B Object detection device 21 Acquisition unit 22, 22A, 22B Setting unit 23 Extraction unit 24 Division unit 25 Integration unit 26 Determination unit 30 Display device 80 Finger candidate area 84 Finger area 98 Touch operation object 100 Computer 102 CPU
104 Memory 106 Storage unit 112 Input device 116 Recording medium 120, 120A, 120B Object detection program

Claims (8)

On the computer,
Obtain a distance image that represents the distance to the object with the luminance according to the distance, and a reflection intensity image that represents the intensity of the reflected light from the object with the luminance,
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the object to be detected in the distance image, the brightness histogram of the operation target is estimated. ,
Using the distance to the operation target determined based on the estimated brightness histogram of the operation target, extract a candidate area representing the detection target object from the distance image;
Dividing the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
A process including detecting an extracted area whose shape is similar to the contour line of the extracted candidate area, and an area obtained by integrating the candidate areas, as an area representing the detection target object, among the plurality of areas. The object detection method which performs.   The object detection method according to claim 1, wherein the function is a left-right symmetric function having a peak at a reference luminance at which a luminance change rate is a predetermined value or less in a luminance histogram of the distance image.   3. The object detection method according to claim 2, wherein when approximating the function to a luminance histogram of the distance image, an object portion having a luminance corresponding to a distance equal to or greater than a distance corresponding to the reference luminance is approximated.   The object detection method according to claim 2, wherein the function is a Gaussian function. As the distance image, a plurality of frames are sequentially acquired in time series,
If an error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value in the previous frame, the reference brightness in the current frame is searched from a predetermined range centered on the reference brightness searched in the previous frame. The object detection method of any one of Claims 2-4. As the distance image, a plurality of frames are sequentially acquired in time series,
In the previous frame, if the error between the function and the brightness histogram of the distance image is less than or equal to a predetermined value, whether or not the error between the function of the previous frame and the brightness histogram of the distance image of the current frame is less than or equal to a predetermined value The object detection according to any one of claims 2 to 5, wherein when the error is equal to or less than a predetermined value, the function of the previous frame is estimated as a luminance histogram of the current frame of the operation target. Method. An acquisition unit that acquires a distance image representing the distance to the object with luminance corresponding to the distance, and a reflection intensity image representing the intensity of reflected light from the object with luminance;
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the detection target object in the distance image, the brightness histogram of the operation target is estimated. An estimation unit;
An extraction unit that extracts a candidate area representing the detection target object from the distance image using a distance to the operation target determined based on the estimated luminance histogram of the operation target;
A dividing unit that divides the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
An integration unit that detects an extracted region similar in shape to the contour line of the extracted candidate region, and a region obtained by integrating the candidate regions, as a region representing the detection target object, among the plurality of regions;
An object detection apparatus including: On the computer,
Obtain a distance image that represents the distance to the object with the luminance according to the distance, and a reflection intensity image that represents the intensity of the reflected light from the object with the luminance,
By approximating the brightness histogram of the distance image with a function similar to the shape of the brightness histogram of the operation target operated by the object to be detected in the distance image, the brightness histogram of the operation target is estimated. ,
Using the distance to the operation target determined based on the estimated brightness histogram of the operation target, extract a candidate area representing the detection target object from the distance image;
Dividing the reflection intensity image into a plurality of regions based on the luminance of each pixel of the reflection intensity image;
A process including detecting an extracted area whose shape is similar to the contour line of the extracted candidate area, and an area obtained by integrating the candidate areas, as an area representing the detection target object, among the plurality of areas. An object detection program that executes