JP2010152791A - Image processor for detecting object, method, program, and computer readable medium storing the program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for detecting an object which eliminates the need of pretreatment and can detect the position of a moving object by identifying the moving object in a moving image without being affected by a change in a surrounding environment. <P>SOLUTION: The image processor for detecting the object (1000) includes an imaging means (2000) and an irradiating means (3000) for synchronizing with a frame rate of the imaging means to flicker irradiation light to detect the object from an image caught by the imaging means. The image processor includes: an image acquiring means (10); a frame image pairing means (20) for pairing frame images; a first component image generating means (310) for generating a pair of first component frame images; a second component image generating means (320) for generating a pair of second component frame images; a first difference image generating means (410) for generating a first difference image; a second difference image generating means (420) for generating a second difference image; and an object extracting means (50) for recognizing the object. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing technique, and more particularly to an image processing apparatus, method, program, and computer readable medium for storing the program for recognizing an object from a captured image.

  Various image processing techniques have been devised for recognizing a predetermined moving object (object) in a moving image. These techniques can identify human hand movements and gestures, and can be applied to user interface devices such as computers and remote control devices such as televisions.

  A template matching technique is the most representative of the image processing techniques. Template matching is a method in which an image of an object to be recognized (template image) is prepared in advance, and the presence or absence of the object is identified by comparing the template image while moving it over the image to be processed. Since this method can discriminate even the difference in shape and color of an object, it is possible to recognize the object with high accuracy. However, the amount of processing is large, and therefore it is not suitable for an application for recognizing an object that takes a long time to process and moves at high speed. Further, the procedure for obtaining the template image described above can be a troublesome operation for the user.

  Further, template matching is easily affected by unexpected light such as disturbance light, and cannot be accurately recognized when the object changes its shape or angle.

  In order to reduce the influence of the above-mentioned disturbance light, an apparatus disclosed in Patent Document 1 may be used. In this device, the light of the lighting device is blinked at a predetermined cycle, and an image when the lighting device is turned on and an image when the lighting device is turned off are acquired by the imaging device. Next, a time difference image is obtained from these two images, and an object is detected based on the time difference image. By obtaining a time difference image from the lit image and the unlit image, it is possible to detect the moving object suitably by reducing the influence of disturbance light.

  Patent Document 2 discloses a moving object detection apparatus that accurately detects a moving object by using a stereo camera (a pair of cameras).

Further, Patent Document 3 discloses an object detection pre-processing apparatus and method having a technique for detecting an object. This apparatus and method can detect an object (a person's face) stably at a low cost and without being affected by the surrounding environment.
JP 2007-272596 A JP2008-158640 JP 2006-275988 A

  However, the devices disclosed in Patent Documents 1 to 3 have the following disadvantages.

  The time difference image used by the device of Patent Document 1 is used to identify a difference caused by a motion originally. Therefore, there is a possibility that the target cannot be accurately detected when the imaging apparatus moves due to hand shake or the like, or when another moving object exists behind the target. Therefore, this apparatus must be used in a state where the imaging apparatus is firmly fixed and no other moving object exists in the background. That is, the usage conditions are restricted.

  Since the apparatus of Patent Document 2 uses a stereo camera, the distance between the camera and the object can be calculated. Therefore, it is not affected by the moving object behind the object. However, since two cameras are used, the apparatus can be large and expensive. Furthermore, since there are two cameras, the amount of images to be processed increases, making it difficult to apply to a processing apparatus with low calculation capability, particularly home appliances.

In the apparatus of Patent Document 3, an imaging unit including a camera or the like is equipped with a specific wavelength transmission filter. This filter has a function of transmitting only light in the near-infrared region emitted from the light source of the apparatus, and serves to cut out disturbance light. However, since the light emitted from the sun, the fluorescent lamp, the incandescent lamp, or the like serving as the disturbance light source includes light in the near infrared region, the disturbance light cannot be sufficiently cut.
In addition, in the corresponding device, in order to control the amount of light, it is necessary that an area having a certain brightness exists in the image at a certain ratio and that the characteristics of the object (face) are grasped in advance. It is. That is, pre-processing is required for causing the apparatus to grasp the characteristics of the object in advance.
Furthermore, an image is picked up with different irradiation light amounts (bright and dark), and the higher quality of the images is used for recognition of the object. Therefore, the frame rate used for detection is limited to ½ of the actual frame rate of the imaging unit.

  The present invention has been made in view of the above circumstances, and the problem of the present invention is that no pre-processing is required and the surrounding environment (irradiation of ambient light, presence of other moving objects, displacement of the imaging device). Etc.) to provide an image processing apparatus, a method, a program, and a computer-readable medium for storing the program that can recognize an object in a moving image.

  Another object of the present invention is to provide an object detection image processing apparatus, method, program, and computer readable medium storing the program, which can be realized by a single imaging device and does not require advanced arithmetic processing. That is.

The invention of claim 1 devised to solve the above-described problem is an imaging means for imaging a landscape including an object at a predetermined frame rate, and light having a wavelength in any one of the three primary colors or in the infrared region. An irradiating means for irradiating the object, the irradiating means for blinking irradiation light in synchronization with a frame rate of the imaging means, and the object for detecting the object from an image captured by the imaging means An image processing apparatus for detection,
(A1) Image capturing means for receiving a plurality of frame images captured by the imaging means in time series;
From the frame image received by the image capturing unit, a frame image pairing unit that selects and pairs a frame image captured when the irradiation unit is turned on and a frame image captured when the irradiation unit is turned off. When,
(A2) first component image generation means for extracting a color component corresponding to the emission color of the illumination means from the pair of frame images and generating a pair of first component frame images;
(A3) second component image generating means for extracting any one color component different from the light emission color of the illumination means from the pair of frame images and generating a pair of second component frame images;
(A4) first difference image generation means for generating a first difference image from the pair of first component frame images;
(A5) second difference image generation means for generating a second difference image from the pair of second component frame images;
(A6) an object extraction unit that generates a third difference image from the first difference image and the second difference image, and recognizes the object based on the third difference image;
It is characterized by providing.

  Invention of Claim 2 created in order to solve the said subject is the image processing apparatus for object detection of Claim 1, Comprising: The said irradiation means is comprised from a 1 or several light emitting diode, It is characterized by the above-mentioned. And

  Invention of Claim 3 created in order to solve the said subject WHEREIN: In the image processing apparatus for object detection of Claim 1 or 2, selection of the said pair of frame image by the said frame image pairing means is the following. It is performed based on a frame rate of the imaging unit, a lighting cycle of irradiation light of the irradiation unit, and a behavior of the object.

  Invention of Claim 4 created in order to solve the said subject is the image processing apparatus for object detection as described in any one of Claim 1 thru | or 3. WHEREIN: The said imaging means is comprised from a single camera. It is characterized by that.

Invention of Claim 5 created in order to solve the said subject is the imaging means which images the scenery containing a target object with a predetermined | prescribed frame rate, and the light of the wavelength of any one area | region of three primary colors, or an infrared region. An irradiation unit that irradiates the object, and an irradiation unit that blinks irradiation light in synchronization with a frame rate of the imaging unit. This is an object detection image processing method for detecting an object.
(B1) an image capturing stage for receiving a plurality of frame images captured by the imaging unit in time series;
A frame image pairing step for selecting a pair of a frame image captured when the irradiation unit is turned on and a frame image captured when the irradiation unit is turned off from the frame image received in the image capturing step. When,
(B2) a first component image generation step of extracting a color component corresponding to the emission color of the illumination unit from the pair of frame images to generate a pair of first component frame images;
(B3) a second component image generation step of extracting any one color component different from the emission color of the illumination unit from the pair of frame images to generate a pair of second component frame images;
(B4) a first difference image generation step of generating a first difference image from the pair of first component frame images;
(B5) a second difference image generation step of generating a second difference image from the pair of second component frame images;
(B6) generating a third difference image from the first difference image and the second difference image, and recognizing the object based on the third difference image;
It is characterized by including.

  Invention of Claim 6 created in order to solve the said subject WHEREIN: The image processing method for object detection of Claim 5 WHEREIN: The said irradiation means is comprised from a 1 or several light emitting diode, It is characterized by the above-mentioned. And

  Invention of Claim 7 created in order to solve the said subject WHEREIN: In the image processing method for object detection of Claim 5 or 6, selection of the said pair of frame image in the said frame image pairing step is, It is performed based on a frame rate of the imaging unit, a lighting cycle of irradiation light of the irradiation unit, and a behavior of the object.

  Invention of Claim 8 created in order to solve the said subject is the image processing method for object detection as described in any one of Claim 5 thru | or 7. WHEREIN: The said imaging means is comprised from a single camera. It is characterized by that.

  Invention of Claim 9 created in order to solve the said subject is the software which makes a computer perform the method of any one of Claim 5 thru | or 8.

  The invention of claim 10 devised to solve the above problem is a computer readable medium storing the software of claim 9.

The apparatus of claim 1 and the method of claim 5 are capable of detecting an object in a moving image without being affected by changes in the surrounding environment.
Specifically, even if the object is irradiated with disturbance light during imaging, the detection of the object is not affected. The light source of disturbance light is usually the sun, an incandescent bulb, a fluorescent lamp, etc. Among them, the light from the sun and the incandescent bulb is almost direct current light, and thus a pair of first component frame images (or a pair of second images). Both of the component frame images are irradiated equally. Therefore, the influence is deleted when the first difference image (or the second difference image) is generated. Similarly, if a frame image pairing means selects a pair of frame images in a cycle that does not overlap with the lighting cycle of the fluorescent lamp, even if it is irradiated with light of a fluorescent lamp, particularly an inverter fluorescent lamp that lights at about several hundred Hz, The influence can be deleted when the first and second difference images are generated.
In addition, even when the imaging unit is displaced due to vibration, camera shake, or the like during imaging, the influence is removed when the first and second component images are generated. Furthermore, even if there is a moving object behind the target object, the influence is removed when the third differential image is generated.

  In addition to the above contents, the apparatus of claim 1 and the method of claim 5 do not require preprocessing such as imaging an object in advance. Therefore, it is possible to easily detect an object. Further, even if the object changes its angle or shape, it can be identified.

  In the apparatus of claim 2 and the method of claim 6, one or more LEDs can be used as the irradiation means. By using the LED, it is possible to configure an inexpensive and compact irradiation means. In addition, since the LED has a long operating life, the possibility of failure of the irradiation means is also reduced.

  In the apparatus of claim 3 and the method of claim 7, the pair of frames by the frame image pairing unit based on a frame rate of the imaging unit, a lighting cycle of irradiation light of the irradiation unit, and a behavior of the object. By selecting the frame image, it is possible to optimize the responsiveness and the amount of data to be processed.

  In the apparatus of claim 4 and the method of claim 8, since the illumination means is realized by a single camera, the illumination apparatus can be made compact and lightweight. In addition, since the amount of image data used for processing is halved compared to an apparatus using an existing stereo camera, the load on the arithmetic function of the apparatus is reduced. Therefore, the image device can be realized by a processing device that does not have high computing power, and can be applied to general household appliances such as a remote control device for a television.

  With the software according to the ninth aspect, it is possible to cause an arbitrary computer to execute the object detection image processing method.

  According to the computer readable medium of the tenth aspect, the object detection image processing method can be suitably provided to a specific user.

  According to the present invention, an image processing apparatus that does not require preprocessing and can recognize an object in a moving image without being affected by ambient environment (irradiation of ambient light, presence of other moving objects, displacement of an imaging apparatus, etc.) It is possible to provide a method, a program, and a computer-readable medium storing the program. Furthermore, it is possible to provide an object detection image processing apparatus, a method, a program, and a computer-readable medium for storing the program, which can be realized by a single imaging apparatus and do not require advanced arithmetic processing.

An embodiment of the present invention will be described with reference to the accompanying drawings. In the present embodiment, a plurality of frame images constituting a moving image are referred to as “frame images”, and detected objects in the images are referred to as “target objects”.
Further, in one embodiment of the present invention, “frame difference processing” is processing for obtaining a difference between pixels of a pair of frame images to extract a different portion between the frame images. This image is referred to as a “difference image”.

  FIG. 1 is a functional block diagram of an object detection image processing apparatus (hereinafter, “image processing apparatus”) 1000 according to an embodiment of the present invention. The image processing apparatus 1000 includes an image capturing unit 2000 and an object irradiation unit 3000 as external function elements, and further includes an image capturing unit 10, a frame image pairing unit 20, and a component image generation unit as internal function elements. 30, a component difference image generation unit 40, and a target object extraction unit 50, and has a function of identifying a target object from a moving image captured by the image capturing unit 2000 and detecting its position.

The functional elements of the image processing apparatus 1000 shown in FIG. 1 will be briefly described below.
[Image capture unit]
The image capturing unit 2000 has a function of capturing a landscape including an object at a predetermined frame rate and sending the captured landscape to the image capturing unit 10 as digital image data in a predetermined format. The image capturing unit 2000 may be a digital camera capable of capturing an existing moving image, commonly called a movie camera.

[Object irradiation unit]
The object irradiation unit 3000 has a function of irradiating the object with light like a light for a camera. However, the irradiation light is not normal incandescent light but includes only a light component having a single wavelength in one of the three primary colors (R, G, and B) or in the infrared region. Furthermore, although the details will be described later, the irradiation light blinks in synchronization with the execution / stop of the imaging of the image imaging unit 2000. Therefore, it has high-speed response on the order of microseconds.
The object irradiation unit 3000 is not particularly limited, but can be realized by a light emitting diode (LED), a laser diode and an optical component, an incandescent bulb or a fluorescent lamp, an optical filter, a shutter mechanism, and the like. However, it is most preferable to use an LED in consideration of ease of realization, cost, compactness, reliability, and the like.

[Image capture unit]
The image capturing unit 10 is connected to the image capturing unit 2000 via a cable, wireless, infrared, or the like, and has a function of receiving frame image data captured by the image capturing unit 2000 in time series. The received frame image data is sent to the frame image pairing unit 20.

[Frame image pairing part]
The frame image pairing unit 20 has a function of extracting a predetermined frame image from a plurality of frame images received from the image capturing unit 10 and combining them together. The extraction condition of the frame image will be described later, but one of the pair of frame images is a frame image captured by the image capturing unit 2000 in a state where the light of the target irradiation unit 3000 is irradiated on the target, and the other Is a frame image taken in a state in which the object is not irradiated with light.

[Component image generator]
The component image generation unit 30 includes an R component generation unit 310, a G component generation unit 320, and a B component generation unit 330, and R, G, and R respectively from a pair of frame images combined by the frame image pairing unit 20. It has a function of extracting a B component and generating a pair of R, G, and B component frame images.

[Component Difference Image Generation Unit]
The component difference image generation unit 40 includes an R component difference image generation unit 410, a G component difference image generation unit 420, and a B component difference image generation unit 430, and receives a pair of R, received from the component image generation unit 30. It has a function of generating R, G, and B component difference images based on the G and B component frame images, respectively.

[Target Extraction Unit]
The object extraction unit 50 identifies the object in the frame image captured by the image capturing unit 2000 based on the R, G, and B component difference images received from the component difference image generation unit 40, and obtains the position coordinates thereof. The function of outputting the obtained coordinates to the outside is provided. The object extraction unit 50 includes a difference image generation unit 510, a binarization processing unit 520, a labeling processing unit 530, and an area selection unit 540. Each function will be described below.
(Difference image generator)
The difference image generation unit 510 has a function of generating a difference image (hereinafter referred to as “final difference image”) based on the R, G, and B component difference images received from the component difference image generation unit 40.
(Binarization processing part)
The binarization processing unit 520 has a function of performing binarization processing on the final difference image generated by the difference image generation unit 510. For this binarization processing, a method well known in the art, for example, a fixed threshold method, a P tile method, a mode method, a discriminant analysis binarization method, or the like may be used.
(Labeling processing part)
The labeling processing unit 530 has a function of performing a labeling process on the final difference image that has been binarized by the binarization processing unit 520. The labeling process is a process of classifying a plurality of regions as a group by analyzing color information in each pixel of the image and adding the same label to the connected pixels. At the same time, it is preferable to obtain the center of gravity in each region and measure the position. This is to flexibly cope with changes in the shape and size of these areas between different frame images.
(Region selection section)
The region selection unit 540 has a function of obtaining the brightness of a plurality of regions defined by the labeling processing unit 530, comparing them, determining the region with the highest brightness as a detected object, and outputting the coordinates of the position.

  Next, referring to the processing conditions of FIG. 2 and various generated images of FIG. 3, according to the flowchart of FIG. 4, the details of image processing contents and procedures performed by the image processing apparatus 1000 having the above-described functions will be described in detail. explain. 2 and 3 are attached as reference drawings 1 and 2 to the property submission file submitted at the same time as the present application, respectively.

  FIG. 2A shows the imaging conditions of this embodiment. The image capturing unit 2000 of the image processing apparatus 1000 captures one person's face and its right palm and the other person's left palm to create a moving image, and identifies the right palm as a target object from the moving image. The position is detected. Here, the distance relationship between the image capturing unit 2000 and each part of the person is the right palm of the one person, the face, and the left palm of the other person in order of proximity to the image capturing unit 2000. In addition, it is assumed that the person does not move during imaging, and the automobile travels from the left to the right from the image capturing unit 2000 behind the person.

  First, the irradiation light of the object irradiation unit 3000 of the image processing apparatus 1000 is adjusted (step S10 in FIG. 4). First, the type of irradiation light of the object irradiation unit 3000 is determined. Here, in order to reduce the influence on the person, infrared light is used. In this case, depending on the specifications of the image capturing unit 2000, in general, information on infrared components is added to information on any of R, G, and B components and output. In this embodiment, it is added to the R component. That is, the image processing apparatus 1000 may perform the same processing as when the object irradiation unit 3000 irradiates light in the R region.

  Next, the irradiation range of the object irradiation unit 3000 is adjusted so that the entire right palm as the object is irradiated with light as uniformly as possible. In this embodiment, the right palm does not move. However, when moving, adjustment must be made so that light is irradiated to the right palm over the entire moving range. In the case where the object irradiation unit 3000 is configured by LEDs, the irradiation range of the irradiation light may be varied by increasing or decreasing the number of LEDs.

  Finally, the flashing timing of the irradiation light is adjusted. In this embodiment, as shown in FIG. 2B, adjustment is performed so that the image capturing unit 2000 repeats turning off / on each time the image capturing is performed.

  Thus, the adjustment in step S10 in FIG. 4 is completed. When the procedure of step S10 is completed, the image capturing unit 2000 next captures a landscape including the object (step S20 in FIG. 4). Each frame image constituting the captured moving image is sent from the image capturing unit 2000 to the image capturing unit 10 (step S30 in FIG. 4). An example of the frame image sent at this time is shown in FIG. The frame images are defined as first to fourth frame images in the order of image capturing. Here, the first and third frame images are captured in a state where the object (right palm) is not irradiated with the light of the object irradiation unit 3000, and the second and fourth frame images are captured on the object. It should be noted that the image was taken with light applied.

When the first to fourth frame images are sent to the image capturing unit 10, the frame image pairing unit 20 combines the first and second frame images and the third and fourth frame images in pairs (FIG. 4). Step S40).
Since the pair of frame images are subjected to the same processing, only the processing for the pair of the first and second frame images will be described below.

  Upon receipt of the first and second frame images (FIG. 3A) from the frame image pairing unit 20, the component image generation unit 30 receives the first and second R component frame images (FIG. 3B) based on them. ) Is generated (step S50 in FIG. 4). As shown in FIG. 3B, the color contrast is clear in the second R component frame image. This is because the light of the object irradiation unit 3000 is applied to each part of the person. The right palm, which is the object, is the closest to the object irradiation unit 3000 and therefore reflects the irradiation light most strongly. Therefore, it is displayed in the brightest color. Also, the face is not as bright as the right palm, but is brighter because it reflects the irradiated light. On the other hand, the car running behind the left palm and the person has a dark color. That is, the irradiation light of the object irradiation unit 3000 does not reach the left palm and the automobile. Moreover, although it is difficult to understand in the figure, the positions of the automobiles traveling behind the person are different in the first and second R component frame images.

  The first and second R component frame images are sent from the component image generation unit 30 to the component difference image generation unit 40, and the component difference image generation unit 40 generates an R component difference image that is a difference image of these images (FIG. Step S60 of 4). This R component difference image is shown on the rightmost side of FIG. In this figure, it can be seen that the right palm and the front part of the car are painted black. The right palm has a difference due to the reflection of the irradiation light, and the front portion of the automobile has a difference due to the movement of the automobile. That is, the R component difference image has a light / dark difference component and a motion difference component.

The following steps S70 and S80 are performed in parallel with the steps S50 and S60.
Similar to the procedure of step S50, the component image generation unit 30 generates the first and second G component frame images (FIG. 3C) based on the first and second frame images (FIG. 3A). (Step S70 in FIG. 4).

  Similar to the processing of step S60, when the first and second G component frame images are sent to the component difference image generation unit 40, the component difference image generation unit 40 is a G component that is a difference image of the first and second G component frame images. A difference image is generated (step S80 in FIG. 4). The G component difference image is shown on the rightmost side of FIG. In this figure, the front part of the automobile is painted black. This is the difference caused by the movement of the car as in the R component difference image. On the other hand, since the irradiation light of the object irradiation unit 3000 does not include the G component light, unlike the R component difference image, there is no difference between light and dark in the G component difference image. That is, the G component difference image has only a motion difference component.

The R and G component difference images generated by the component difference image generation unit 40 are sent to the object extraction unit 50, and a difference image of these images (final difference image (see the rightmost diagram in FIG. 3D). )) Is generated (step S90 in FIG. 4). In the final difference image of FIG. 3D, only the right palm, which is the object, is painted black. This is because the difference in motion caused by the movement of the car, which was included in the R and G component difference images, is removed, and only the difference between the right palm and the light that was included in the R component difference image is displayed. Because. That is, R component difference image (light / dark difference component + motion difference component) −G component difference image (motion difference component) = final difference image (light / dark difference component).
Thereafter, binarization, labeling, and region selection processing are performed on the final difference image to detect the position of the object (right palm) (step S100 in FIG. 4).

  The above is the contents and procedure of the object detection image processing performed by the image processing apparatus 1000 according to the embodiment of the present invention. However, the object detection image processing of the present invention is not limited to these contents and procedures.

In the first embodiment, the component image generation unit 30 and the component difference image generation unit 40 process the R component image and the G component image, but the present invention is not limited to this, and other embodiments are performed. In the example, a B component image may be handled instead of a G component image. As long as the irradiation light of the object irradiation unit 3000 is infrared light, the G component image and the B component image are the same. Therefore, the final result obtained is the same regardless of which of the G component image and the B component image is handled.
In this embodiment, the right palm, which is the object, is stationary, but even if it is moving, its position can be suitably detected as long as the irradiation light is within a reachable distance.

In the first embodiment, as shown in FIG. 2B, the irradiation light of the object irradiation unit 3000 is turned off / turned on every imaging of the image imaging unit 2000. In the present invention, the irradiation light is limited to this blinking timing. Is not to be done. Furthermore, in the embodiment, the frame image pairing unit 20 combines a pair of adjacent frame images on the time axis, but the present invention is not limited to this combination method. In another embodiment, the object irradiation unit 3000 may turn off / turn on the irradiation light every time the image capturing unit 2000 captures images twice. That is, the first and second frame images in FIG. 2C are captured when the irradiation light is turned off, and the third and fourth frame images are captured when the irradiation light is turned on. Furthermore, the frame image pairing unit 20 can combine the first frame image and the third frame image as a pair and perform no processing on the second and fourth frames. In such a case, although the detection responsiveness is reduced, the load on the image processing apparatus 1000 is reduced because the amount of data to be processed is reduced to ½.
Therefore, the timing of turning off / on the irradiation light of the image processing apparatus according to the present invention and the frame image combination method performed by the frame image pairing unit 20 include the behavior of the object, the frame rate of the image pickup object, and the image processing apparatus. Preferably, the selection is made based on various conditions such as the computing ability of

The image processing 1000 according to the embodiment of the present invention described above can provide the following advantages over the conventional image processing apparatus.
[1] Less affected by changes in the surrounding environment. Specifically, even if disturbance light (sunlight, incandescent bulb, fluorescent lamp light, etc.) irradiates the object, the component image generation unit 30 causes the component difference image (the first of FIGS. 3B and 3C). The influence of disturbance light is removed at the stage of generating the right image). Similarly, even if the image capturing unit 2000 moves somewhat during image capturing, the influence is removed at the stage of generating the component difference image. Further, even if there is a moving object behind the target object, the influence of the moving object is removed when the target object extraction unit 50 generates the final difference image (the rightmost image in FIG. 3).
[2] Unlike template matching processing and the like, no preprocessing is required. Further, even if the object moves and the size or angle of the object in the landscape image changes, it can be detected well.
[3] Since only a single imaging device is used, the amount of images to be processed is small compared to a conventional image processing device using a stereo camera. Furthermore, since no complicated arithmetic processing is performed, the image processing apparatus 1000 is not overloaded.

  Since the image recognition apparatus 1000 according to an embodiment of the present invention has the advantages described above, it can be suitably applied to various applications. For example, the present invention can be applied to a computer pointing device, which will be described later in Embodiment 2, a motion capture for analyzing a movement trajectory of an object, or a remote control device for home appliances.

  The above-described image processing method for detecting an object according to an embodiment of the present invention is preferably implemented as a program. In that case, the program is preferably downloaded from the main server to a computer executing the method or distributed in the form of a computer readable medium. Examples of the computer readable medium include a hard disk, a CD-ROM, a magnetic tape, a flexible disk, and an optical data storage device.

In Example 2, a case where the object detection image processing apparatus according to the embodiment of the present invention described above is applied to a computer user interface apparatus will be described as an example.
FIG. 5 shows a user interface device 4000 equipped with the image processing apparatus 1000. The interface device 4000 includes a light 600, a web camera 700, and a computer 800. The light 600 and the web camera 700 correspond to the object irradiation unit 3000 and the image capturing unit 2000 (see FIG. 1) of the image processing apparatus 1000, respectively. The computer 800 is implemented by a general-purpose computer or the like, and has functions of internal elements 10 to 50 (see FIG. 1) of the image processing apparatus 1000.

The user interface device 4000 of this embodiment is a non-contact type and has a function of causing the computer 800 to execute various operations according to gestures of the user's hand.
The operation procedure of the user interface device 4000 will be described below with reference to FIGS. 5 and 6 and according to the flowchart of FIG.
First, as shown in FIG. 5A, the user faces the user interface device 4000 and places the right hand in the field of view of the web camera 700. At this time, the right hand needs to be closer to the web camera 700 and the light 600 than to other parts. In this state, the user can cause the computer 800 to perform a desired operation by making a predetermined gesture with the right hand. In the present embodiment, a predetermined gesture is an act of making a ring by bringing the thumb and index finger into contact with or close to each other (hereinafter, “picking operation”).

  The user interface device 4000 always identifies an object from the landscape captured by the web camera 700. When the user's right hand appears in the imaged landscape, the user interface device 4000 identifies the hand as an object and detects its position (step S200 in FIG. 7). Note that the detailed detection procedure follows steps S10 to S100 in the flowchart of FIG. 4 described above.

  When the object is detected, next, the distance from the center of gravity of the hand to the end of the hand is obtained as shown in FIG. 5B (step S210 in FIG. 7). Specifically, the center of gravity obtained by the labeling process in step S100 in FIG. 4 is used as the center of gravity of the hand, and the center of gravity is set at a predetermined angle step (such as 5 °) within a predetermined angle range (such as 360 °). Find the edge of the hand and measure the distance between that edge and the center of gravity. Then, the five ends with the longest distance are recognized as fingertips (a, b, c, d, e).

  After recognizing the finger, it is determined whether or not the finger is bent from the shape of the finger. If the finger is bent, the bending direction is specified (step S220 in FIG. 7). In the shape of the hand shown in FIG. 5C, each finger is specified to be bent in the direction of the arrow shown at the top of the fingertip. Only the thumb of a human hand bends in a different direction from the other fingers. Therefore, if the bending direction of each finger can be specified, the thumb (a) and the index finger (b) can be specified from the fingers.

  Next, it is determined whether or not the hand is picking (step S230 in FIG. 7). If it is determined that the picking operation is being performed (“YES” in step S230), the computer 800 is caused to execute a predetermined operation (step S240 in FIG. 7). If it is determined that it has not been performed (“NO” in step S230), the procedure is repeated from step S200 again. Whether the picking operation is being performed is determined based on whether or not the thumb (a in the figure) and the index finger (b in the figure) form a closed loop, as shown in FIG.

  The above is a series of procedures performed by the user interface device 400 of this embodiment. In a conventional non-contact user interface for identifying a gesture, it is necessary to register a predetermined gesture in advance and perform a matching process between the registered image and the captured image. This prior registration work is troublesome for the user. Furthermore, it is necessary to register a hand shaped like goo, choki, par, etc. so that it can be easily identified as a gesture, but this makes the user feel fatigued when working for a long time. Moreover, there is a possibility that the user may refrain from using it because of the eyes of others. However, the user interface device 400 according to the present embodiment does not require pre-registration and can set a more natural and modest gesture such as a picking operation.

  In addition, since the user interface device 400 according to the present embodiment identifies the user's thumb and index finger based on the angle at which the finger bends, the user can use either the right hand or the left hand, or the direction of the hand can be reversed. The thumb and index finger can be accurately detected. In addition, even if the user's hand moves up and down, left and right, or in the rotation direction, it can be detected as long as it is within the field of view of the webcam 700.

  With the above features, a visual function as shown in FIG. 6 can be provided. The circular objects on the right side of (a) to (c) in this figure are icons displayed on the screen of the computer 800. This icon changes shape appropriately depending on the user's gesture.

  For example, in FIG. 6A, the user's hand is in a state where no picking operation is performed, or just before it is performed. At this time, the icon has a substantially circular shape. In FIG. 6B, the user's hand is picking. At this time, the icon has a collapsed shape. The icon is crushed at the same angle as the picking operation. Furthermore, in FIG.6 (c), the user inclines with the hand picked. At this time, the collapsed icon is similarly tilted as shown on the right side of the figure.

  As described above, the user interface device 400 according to the present embodiment can provide visual feedback linked to the user's gesture. For example, the presence / absence of the cursor picking, the picking angle, the twisting state, the moving and rotating state, etc. By visualizing, it is possible to provide operability that is more substantial and appeals to the user's intuition.

  As mentioned above, although this invention was demonstrated using embodiment shown in drawing, these are only an illustration and those skilled in this technical field can variously be within the range which does not deviate from the range and the meaning of this invention. It will be understood that modifications and variations are possible. Accordingly, the scope of the invention should not be determined by the described embodiments, but by the technical spirit described in the claims.

It is a block diagram which shows the function of the image processing apparatus for object detection which concerns on one Embodiment of this invention. It is a figure for demonstrating the image processing of the image processing apparatus for object detection which concerns on one Embodiment of this invention, (a) is a figure which shows imaging conditions, (b) is blinking of imaging and irradiation light, (C) is a frame image captured at that time. This drawing will be attached to the property submission form. It is a figure for demonstrating the image processing of the image processing apparatus for target object detection concerning one Embodiment of this invention, (a) is the imaged frame image, (b) is a frame image and difference image of R component (C) is a G component frame image and a difference image, and (d) is an R and G component difference image and a final difference image. This drawing will be attached to the property submission form. It is a flowchart of the image processing which the image processing apparatus for object detection which concerns on one Embodiment of this invention performs. It is a figure explaining the aspect of the user interface apparatus carrying the image processing apparatus for target object detection concerning one Embodiment of this invention, (a) is the said user interface apparatus, (b) thru | or (d) are object. It is a figure for demonstrating the detection method of the palm which is a thing. It is a figure for demonstrating the aspect of the cursor of the user interface apparatus carrying the image processing apparatus for target object detection concerning one Embodiment of this invention. It is a flowchart of the operation | movement procedure of the image processing apparatus for object detection which concerns on one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image capture part 20 Frame image pairing part 30 Component image generation part 40 Component difference image generation part 50 Object extraction part 310 R component generation part 320 G component generation part 330 B component generation part 410 R component difference image generation part 420 G component difference image generation unit 430 B component difference image generation unit 510 Difference image generation unit 520 Binarization processing unit 530 Labeling processing unit 540 Area selection unit 1000 Image processing device for object detection 2000 Image imaging unit 3000 Object irradiation unit

Claims (10)

An imaging unit that images a landscape including an object at a predetermined frame rate; and an irradiation unit that irradiates the object with light having a wavelength in any one of the three primary colors or an infrared region, the frame of the imaging unit In an object processing image processing apparatus for detecting an object from an image captured by the image capturing unit, the irradiation unit flashing irradiation light in synchronization with a rate,
Image capturing means for receiving a plurality of frame images captured by the imaging means in time series;
From the frame image received by the image capturing unit, a frame image pairing unit that selects and pairs a frame image captured when the irradiation unit is turned on and a frame image captured when the irradiation unit is turned off. When,
First component image generation means for extracting a color component corresponding to the emission color of the illumination means from the pair of frame images, and generating a pair of first component frame images;
Second component image generating means for extracting any one color component different from the light emission color of the illumination means from the pair of frame images and generating a pair of second component frame images;
First difference image generation means for generating a first difference image from the pair of first component frame images;
Second difference image generation means for generating a second difference image from the pair of second component frame images;
Object extraction means for generating a third difference image from the first difference image and the second difference image, and recognizing the object based on the third difference image;
An image processing apparatus for detecting an object, comprising:   The object processing image processing apparatus according to claim 1, wherein the irradiation unit includes one or a plurality of light emitting diodes.   The selection of the pair of frame images by the frame image pairing unit is performed based on a frame rate of the imaging unit, a lighting cycle of irradiation light of the irradiation unit, and a behavior of the object. The image processing apparatus for detecting an object according to claim 1 or 2.   4. The object detection image processing apparatus according to claim 1, wherein the imaging unit includes a single camera. 5. An imaging unit that images a landscape including an object at a predetermined frame rate; and an irradiation unit that irradiates the object with light having a wavelength in any one of the three primary colors or an infrared region, the frame of the imaging unit In an object detection image processing method for detecting the object from an image captured by the imaging unit, performed by an image processing apparatus including an irradiation unit that blinks irradiation light in synchronization with a rate.
An image capturing step for receiving a plurality of frame images captured by the imaging unit in time series,
A frame image pairing step for selecting a pair of a frame image captured when the irradiation unit is turned on and a frame image captured when the irradiation unit is turned off from the frame image received in the image capturing step. When,
A first component image generation step of generating a pair of first component frame images by extracting each color component corresponding to the emission color of the illumination means from the pair of frame images;
A second component image generating step of extracting any one color component different from the emission color of the illumination means from the pair of frame images to generate a pair of second component frame images;
A first difference image generation step of generating a first difference image from the pair of first component frame images;
A second difference image generation step of generating a second difference image from the pair of second component frame images;
Generating a third difference image from the first difference image and the second difference image and recognizing the object based on the third difference image; and
An image processing method for detecting an object, comprising:   The object processing image processing method according to claim 5, wherein the irradiation unit includes one or a plurality of light emitting diodes.   The selection of the pair of frame images in the frame image pairing stage is performed based on a frame rate of the imaging unit, a lighting cycle of irradiation light of the irradiation unit, and a behavior of the object. The image processing method for detecting an object according to claim 5 or 6.   The image processing method for detecting an object according to any one of claims 5 to 7, wherein the imaging unit is configured by a single camera.   Software for causing a computer to execute the method according to any one of claims 5 to 8.   A computer-readable medium storing the software according to claim 9.

Images