JP2006275988A - Apparatus and method for pretreatment of detecting object - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus for pretreatment of detecting an object that reduces cost, enables uniform irradiation over a wide area, hardly receives effect of environment such as ambient light, and enables extraction of the existence region of the object safely and speedily by a human body, and to provide a method for the same. <P>SOLUTION: The apparatus includes an irradiation means 103 for irradiating an observation object by different timing and different irradiation light quantity; an image acquisition means 106 for acquiring each image of the observation object, irradiated by the different irradiation light quantity; an ROI (region-of-interest) extraction means 108 for extracting the existence region of the observation object, on the basis of each image acquired; an irradiation instruction means 107 for generating a signal to control the irradiation light quantity of the irradiation means, when a predetermined condition is not satisfied, when extracting the existence region of the observation object; and an irradiation control means 104 for controlling the irradiation light quantity of the irradiation means, on the basis of the generated signal. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an object detection pre-processing apparatus and an object detection pre-processing method that enable detection of an observation object even in a low illumination environment.

  In recent years, specific imaging devices (hereinafter also simply referred to as devices) have been developed for detecting observation objects (hereinafter also referred to as observation objects or simply objects) under low illumination such as at night. Hereinafter, the object will be described as a person for easy understanding. Such an apparatus is applied to a driver's drowsiness detection or a robot control for interacting with a human. Here, in a method for detecting an object in a low illumination environment, an ultrasonic wave, a laser radar, a camera, or the like is used. However, the camera is more suitable considering the cost and the effect on the human body. There are two types of cameras: cameras without auxiliary lighting and cameras with auxiliary lighting. An example of a camera without auxiliary illumination is a far-infrared camera. However, there is a problem that such a camera is expensive and has a low resolution. On the other hand, in a camera with auxiliary illumination, since the auxiliary illumination to be used is visible light and near infrared light, a commercially available camera can be used. In general, such cameras are inexpensive and have high resolution. However, when the object is a person, irradiating visible light is problematic because it affects the field of view. Therefore, when the object is a person, it is suitable to use near infrared light illumination that does not affect the field of view.

  However, there is a problem with an apparatus using a camera with auxiliary illumination by near infrared light. First, when using a fixed irradiation light amount as in the prior art, when a person approaches the apparatus, the exposure amount of near-infrared light incident on the eyes increases, which may affect the eyes. Further, when a fixed amount of light is used as in the prior art, halation occurs in the person area on the captured image when the person approaches the apparatus. In addition, when the range in which the object exists is wide, it is necessary to uniformly irradiate a very wide range compared to the irradiation area. As a general problem with cameras with auxiliary lighting, it is difficult to detect the position of a person due to halation on the person's image due to the intensity of ambient light (for example, sunlight) or the forward movement of the person in the shooting environment. There is a problem of becoming. In particular, in the conventional apparatus, images are taken according to whether the illumination is on or off, and the position of the person is detected by taking the difference. However, one of the images causes halation due to ambient light or forward movement of the person. However, if one of the images is dark, it is difficult to detect the position of the person.

A conventional technique for solving these problems will be described. First, for the problem that the exposure amount of near-infrared light incident on the eyes increases when a person approaches the apparatus, the distance to the person is acquired, and the amount of irradiation light is changed based on the distance. . Such a technique is disclosed in Patent Document 1 below. In the technique disclosed in Patent Document 1, the amount of irradiation light is increased or decreased based on the distance to a person measured by a distance measuring sensor such as an ultrasonic wave. Further, for a problem that requires uniform irradiation over a wide irradiation range, a plurality of light sources are used to uniformly irradiate over a wide range. Such a technique is disclosed in Patent Document 2 below. The technique disclosed in Patent Document 2 irradiates the entire photographing range (a predetermined region in the vehicle interior) with a plurality of light sources arranged so that the irradiation ranges do not overlap each other. For the problem of halation on a person's image due to disturbance light or forward movement of the person, the amount of disturbance light is detected, and the amount of irradiation light is increased or decreased based on that. Such a technique is disclosed in Patent Document 3 below. In the technique disclosed in Patent Document 3, the amount of disturbance light (visible light) is detected by a light amount detection unit including an illuminance sensor, and the amount of irradiation light is increased or decreased according to the result.
Japanese Patent Laying-Open No. 2004-16483 (paragraph 0030) JP 2004-144512 A (paragraph 0017) JP-A-7-32907 (paragraph 0008)

  However, the technique disclosed in Patent Document 1 has a problem that the cost increases due to a distance measuring sensor such as an ultrasonic wave. In addition, the technique disclosed in Patent Document 2 has a problem that the cost increases by using a plurality of light sources, and the installation of the plurality of light sources is complicated. In addition, the technique disclosed in Patent Document 3 has a problem that the illuminance sensor increases the cost, and it is unrealistic to install the illuminance sensor around the faces of an unspecified number of people.

  The present invention is intended to solve the above-mentioned problems, can reduce the cost, can perform uniform irradiation over a wide range, is not easily influenced by the surrounding environment such as ambient light, and is safe and fast for the human body. It is an object of the present invention to provide an object detection pre-processing apparatus and an object detection pre-processing method that enable extraction of an existing region of the object and estimation of an object state.

  In order to achieve the above object, according to the present invention, an irradiation unit that irradiates an observation object at different timings and different irradiation light amounts, and images of the observation object irradiated with the different irradiation light amounts are acquired. When the image acquisition means, the attention area extraction means for extracting the observation object existing area based on each acquired image, and the observation object existing area are not satisfied when a predetermined condition is not satisfied The object detection unit includes: an irradiation command unit that generates a signal for controlling the irradiation light amount of the irradiation unit; and an irradiation control unit that controls the irradiation light amount of the irradiation unit based on the generated signal. A pre-processing device is provided. With this configuration, it is possible to reduce costs, perform uniform irradiation over a wide range, and extract a region where an observation target exists at high speed.

  Further, in the present invention, it is a preferable aspect of the present invention that the irradiating means irradiates near infrared light so as to enable photographing for 24 hours. With this configuration, the object detection pretreatment apparatus can be operated for 24 hours.

  In the present invention, the image acquisition means transmits only the reflected light of the light irradiated by the irradiation means in order to reduce the influence of disturbance light including visible light and enable stable image acquisition. Providing the specific wavelength transmission filter is a preferred aspect of the present invention. With this configuration, the influence of disturbance light can be reduced, and stable quality image capturing can be performed against changes in the illumination environment.

  Further, in the present invention, the irradiation control means synchronizes the timing of irradiating the observation object with different irradiation light amounts by the irradiation means with the timing of acquiring the image of the observation object by the image acquisition means. Controlling the irradiation by the irradiation means is a preferred embodiment of the present invention. With this configuration, it is possible to reduce the cost without affecting the observation object.

  Further, in the present invention, the irradiation command means generates the signal for controlling the irradiation light amount of the irradiation means according to the distance of the observation object with respect to the image acquisition means. This is a preferred embodiment. With this configuration, it is possible to capture an image of an observation object with stable quality.

  Moreover, in this invention, it is a preferable aspect of this invention that the said attention area extraction means extracts the presence area | region of the said observation target object by the difference process of the acquired said image. With this configuration, it is possible to increase the speed of the observation target object detection process.

  In the present invention, it is a preferred aspect of the present invention that the observation object is a human face. With this configuration, the characteristics of a person's face can be used.

  Moreover, in this invention, it is a preferable aspect of this invention to provide the diffusion filter for diffusing the light which the said irradiation means irradiates uniformly over a wide range. With this configuration, a wide imaging space can be uniformly irradiated.

  Further, according to the present invention, the step of irradiating the observation object with different timing and different irradiation light amount, the step of acquiring the respective images of the observation object irradiated with the different irradiation light amount, and the acquired each And extracting a region for observing the object, and generating a signal for controlling the amount of irradiation light when a predetermined condition is not satisfied when extracting the region for observing the object. There is provided an object detection pre-processing method comprising the steps of: and controlling the amount of irradiation light based on the generated signal. With this configuration, it is possible to reduce costs, perform uniform irradiation over a wide range, and extract a region where an observation target exists at high speed.

  Further, in the present invention, in the step of irradiating the observation object with different irradiation light amounts, it is a preferable aspect of the present invention to irradiate near infrared light so as to enable photographing for 24 hours. With this configuration, the apparatus can be operated for 24 hours.

  Further, in the present invention, in the step of acquiring the image of the observation object, the observation object is irradiated with a different amount of irradiation in order to reduce the influence of disturbance light including visible light and to enable stable image acquisition. It is a preferred aspect of the present invention to use a specific wavelength transmission filter that transmits only the reflected light of the light irradiated in the irradiating step. With this configuration, the influence of disturbance light can be reduced, and stable quality image capturing can be performed against changes in the illumination environment.

  Further, in the present invention, in order to synchronize the timing of irradiating the observation object with a different irradiation light amount with the timing of acquiring the image of the observation object, the irradiation of the step of irradiating the observation object with a different irradiation light amount is performed. It is a preferred aspect of the present invention to further have a controlling step. With this configuration, it is possible to reduce the cost without affecting the observation object.

  Further, in the present invention, in the step of generating a signal for controlling the amount of irradiation light, for controlling the amount of irradiation light according to the distance of the observation object with respect to the means for acquiring the image of the observation object. Generating the signal is a preferred aspect of the present invention. With this configuration, it is possible to capture an image of an observation object with stable quality.

  In the present invention, in the step of extracting the existence area of the observation object, it is a preferable aspect of the invention that the existence area of the observation object is extracted by the difference processing of the acquired image. With this configuration, it is possible to increase the speed of the observation target object detection process.

  In the present invention, it is a preferred aspect of the present invention that the observation object is a human face. With this configuration, the characteristics of a person's face can be used.

  In the present invention, it is a preferable aspect of the present invention to use a diffusion filter for uniformly diffusing irradiated light over a wide range in the step of irradiating the observation object with different irradiation light amounts. With this configuration, a wide imaging space can be uniformly irradiated.

  The object detection pre-processing apparatus and the object detection pre-processing method of the present invention have the above-described configuration, can reduce the cost, can perform uniform irradiation over a wide range, are not easily influenced by the surrounding environment such as ambient light, and are applied to the human body. On the other hand, it is possible to extract a region where an object exists and estimate the state of the object safely and at high speed.

  Hereinafter, an object detection preprocessing apparatus and an object detection preprocessing method according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a configuration diagram showing a configuration of an object state estimation apparatus including an object detection preprocessing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram showing the spectral sensitivity characteristics of the imaging unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram showing the transmission wavelength characteristics of the specific wavelength transmission filter in the object detection pretreatment apparatus according to the embodiment of the present invention. FIG. 4 is a view showing a timing chart of irradiation of the auxiliary illuminator in the object detection pretreatment apparatus according to the embodiment of the present invention. FIG. 5A is a diagram showing an irradiation pattern in a time-series continuous image of the auxiliary illuminator in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 5B is a diagram showing two time-series continuous images photographed by the photographing unit in the object detection preprocessing apparatus according to the embodiment of the present invention.

  FIG. 6 is a block diagram showing the processing configuration of the attention area extraction unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 7 is a configuration diagram showing a processing configuration of the difference image validity determination step of the attention area extraction unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 8 is a block diagram showing the processing configuration of the attention area image creation step of the attention area extraction unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 9 is a diagram showing the attention area image created in the attention area image creation step of the attention area extraction unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 10 is a configuration diagram showing the processing configuration of the attention area image selection step of the attention area extraction unit in the object detection preprocessing apparatus according to the embodiment of the present invention.

  FIG. 11 is a configuration diagram showing a configuration of an irradiation command unit in the object detection pre-processing apparatus according to the embodiment of the present invention. FIG. 12 is a configuration diagram showing the processing configuration of the initial search unit of the irradiation command unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 13 is a configuration diagram showing a processing configuration of the continuous search unit of the irradiation command unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 14 is a configuration diagram showing a processing configuration of the halation countermeasure unit of the irradiation command unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 15 is a configuration diagram showing a processing configuration of the control unit according to the distance of the irradiation command unit in the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 16 is a diagram showing the relationship between the distance between the object detection preprocessing apparatus and the observation object (person) according to the embodiment of the present invention.

  FIG. 17 is a configuration diagram showing a processing configuration of the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 18A is a diagram showing an image before inversion in the inversion / smoothing process step of the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 18B is a diagram showing an image after inversion in the inversion / smoothing process step of the face detection unit in the object state estimation device including the object detection preprocessing device according to the embodiment of the present invention. FIG. 18C is a diagram showing a smoothed image in the inversion / smoothing process step of the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention.

  FIG. 19A is a diagram showing a luminance value distribution of an input image in a density information normalization step of the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 19B is a diagram showing a luminance value distribution of the output image in the density information normalization step of the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 20 is a diagram for explaining a mask applied to the background portion of the input image of the face detection processing unit of the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention. is there. FIG. 21 is a diagram for explaining position information output from the post-processing unit of the face detection unit in the object state estimation device including the object detection pre-processing device according to the embodiment of the present invention. FIG. 22 is a diagram showing past face detection results that can be used by the face detection unit in the object state estimation apparatus including the object detection preprocessing apparatus according to the embodiment of the present invention. FIG. 23 is a flowchart for explaining a human state estimation flow in the object state estimation apparatus including the object detection preprocessing apparatus of the present invention.

  First, the configuration of an object state estimation apparatus including an object detection preprocessing apparatus according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, the object state estimation device 100 includes an object detection preprocessing device (hereinafter also referred to as a person detection preprocessing device) 101, a face detection unit 109, a history information storage unit 110, and a state estimation unit 111. It is configured. Further, the object detection pre-processing apparatus 101 includes a diffusion filter 102, an auxiliary illuminator (corresponding to the above-described irradiation means) 103, an auxiliary illumination control unit (corresponding to the above-described irradiation control means) 104, a specific wavelength transmission filter 105, and an imaging. A part 106 (corresponding to the above-mentioned image acquisition means) 106, an irradiation command part (corresponding to the above-mentioned irradiation command means) 107, and an attention area extraction part (corresponding to the above-mentioned attention area extraction means) 108.

  Here, in the embodiment described below, the object state estimation device 100 will be described as a person state estimation device 100 that particularly estimates the state of a person. The human state estimation device 100 including the object detection preprocessing device 101 according to the present invention can be used in various fields. Here, the explanation will be made for air bag deployment control, for driver's snooze / side-side driving prevention, for robot control, and for game operation control. First, the case where it is used for airbag deployment control will be described. The person state estimation device 100 transmits occupant face position information detected by a face detection unit 109 described later to an airbag deployment control device (not shown). Based on the transmitted position information, the airbag deployment control device deploys only the airbag related to the position where the occupant is present when the airbag needs to be deployed due to an accident or the like.

  Next, a description will be given of a case where it is used to prevent a driver from falling asleep or driving aside. The person state estimation device 100 transmits the driver state (sleeping, looking aside, etc.) information estimated by a state estimation unit 111 described later to a driving support system (not shown). Based on the transmitted status information of the driver, the driving support system performs a warning to the driver and driving support. Next, the case where it is used for robot control will be described. The person state estimation device 100 transmits the person state (such as laughing) information estimated by the state estimation unit 111 based on facial expressions or the like to an autonomous behavior system (not shown). Based on the transmitted state information, the autonomous behavior system changes the face of the robot into a facial expression such as happy or sad. Next, the case where it is used for game operation control will be described. The person state estimation device 100 transmits the facial expression (laughing) information of the person estimated by the state estimation unit 111 to an operation control device (not shown). Based on the transmitted state information, the operation control device operates the contents of the game, the displayed color, and the like.

  Hereinafter, each component of the object state estimation apparatus 100 will be described. The diffusion filter 102 is for spreading the light irradiated by the auxiliary illuminator 103 uniformly and over a wide range. The auxiliary illuminator 103 is composed of a plurality of near-infrared light sources (for example, LED light sources). In general, the light sources are preferably arranged around the photographing unit 106. The auxiliary illumination control unit 104 generally controls the lighting of the light source and the length of the lighting time. The imaging unit 106 is configured by a camera whose spectral sensitivity range corresponds to the wavelength of near-infrared light, and the peak of the relative sensitivity of the spectral sensitivity characteristic is preferably coincident with the center wavelength of the auxiliary illumination as shown in FIG.

  The imaging unit 106 includes a specific wavelength transmission filter 105. By attaching the specific wavelength transmission filter 105 to the imaging unit 106, the imaging unit 106 can receive only the reflected light of the light emitted from the auxiliary illuminator 103, thereby reducing the influence of disturbance light including sunlight. be able to. Further, it is desirable that the transmission wavelength characteristic of the specific wavelength transmission filter 105 transmits a narrow wavelength band near the center wavelength of the irradiation light of the auxiliary illuminator 103 as shown in FIG.

  The timing of irradiation of the auxiliary illuminator 103 is synchronized with the timing of opening and closing the shutter of the photographing unit 106 as shown in FIG. Synchronizing the irradiation timing of the auxiliary illuminator 103 with the opening / closing timing of the shutter of the imaging unit 106 allows the auxiliary illuminator 103 to be irradiated only during the period in which the imaging unit 106 receives light, thereby realizing pulsed irradiation. Pulsed irradiation reduces the power and power consumption of light irradiated to humans, compared to continuous lighting (always on). Therefore, the irradiation of the auxiliary illuminator 103 is safer for humans and can be realized at lower cost than the constant lighting.

  The auxiliary illuminator 103 irradiates the time-series continuous images with different light amounts. As shown in FIG. 5A, two images captured in succession are acquired by irradiation with different light amounts (A% and B%). When an NTSC camera is used for the photographing unit 106, the two time-series continuous images correspond to field images. As shown in FIG. 5B, two field images captured in this manner are brighter in one image (first captured image, also referred to as First_image), and one image (an image captured later). And also called Second_image) is dark and two images with luminance differences are obtained. Therefore, even when the brightness of one image is poor (too bright or too dark), the other image can be obtained and used for face detection. It is also possible to select an image with good brightness from the two images and use the image to detect an observation target by the face detection unit 109 described later.

Further, the auxiliary illuminator 103 performs irradiation with different amounts of light to obtain two images having different luminances, and these images are obtained in the difference processing step (step S201) of the attention area extraction unit 108 illustrated in FIG. The existence area of the observation object (here, a human face) can be obtained by obtaining the difference between the two. The difference process between the two images can be obtained by the following equation (1). Here, I _diff (x, y) is the difference between the luminance values of the two images, I _{First_image} (x, y) is the luminance value of _{First_image} , and I _{Second_image} (x, y) is the luminance value of Second_image.

  As described above, in First_image and Second_image, one image is taken with a high light amount, and the other image is taken with a low light amount. For example, as shown in FIG. 5B, it is assumed that the amount of light (A%) irradiated to acquire First_image is always larger than the amount of light (B%) irradiated to acquire Second_image. However, it is necessary to change the amount of light of the auxiliary illuminator 103 in accordance with the distance between the observation object and the person detection preprocessing apparatus 101 (particularly, the auxiliary illuminator 103). If the amount of light is not controlled according to the distance to the observation object, the luminance on the observed observation object may be too bright or too dark. Therefore, in order to obtain an appropriate amount of light, difference information between two consecutive images is used in the difference image validity determination step (S202) shown in FIG. In FIG. 7 showing the difference image validity determination step (S202), in the step (S2021) of calculating the ratio (C) of the pixels having a certain luminance or higher, first, C is calculated by examining the luminance of the difference image. As an example of calculating C, C can be obtained by the following equation (2).

Here, I _{graysacle> = T} (x, y) is a pixel having a luminance value equal to or higher than T, and T is a predetermined parameter value. H and W are the height and width (unit: pixel) of the image, respectively. Next, the calculated C is compared with Cmin, which is a determination condition for the luminance difference of the difference image (comparative step S2022). If C is smaller than Cmin, it is determined that the current luminance is too bright or too dark, An instruction is issued to the irradiation command unit 107 (continuous search unit 302 described later) in order to change the irradiation light amount of the auxiliary illuminator 103. On the other hand, if C is equal to or greater than Cmin, the next determination is made. Cmin is a predetermined parameter.

The next determination is a halation determination. First, in the step of calculating the average luminance value (Tavg) of pixels (S2023), an average luminance value Tabg of I _{graysacle> = T} (x, y) is obtained. The obtained Tavg is compared with Tmax which is a halation determination condition (comparison step S2024). If Tavg is larger than Tmax, halation occurs on the surface of the image, and it is determined that the current light amount is inappropriate. Then, an instruction is issued to the irradiation command unit 107 (halation countermeasure unit 303 described later) in order to reduce the amount of light emitted from the auxiliary illuminator 103. On the other hand, if Tabg is equal to or smaller than Tmax, it is determined that the current light amount is appropriate, and the difference image and the two time-series continuous images are shifted to the attention area image creation step (S203). Tmax is a predetermined parameter.

  In the attention area image creation step (S203), information such as the position and size of the attention area is acquired from the obtained difference image, and the attention area image is generated based on the information. In FIG. 8 showing the attention area image creation step (S203), first, in the difference image binarization step (S2031), the input difference image is binarized. There are various binarization processing methods, and here, a combination of the P tile method and the fixed threshold method is used. The luminance value Ti to be binarized is calculated from the P tile method. This Ti is a luminance value that obtains the sum of the appearance frequencies of pixels from the higher luminance value and becomes the ratio of Pt to the total number of pixels. Here, Pt is a predetermined parameter. The calculated Ti is compared with a fixed threshold Ts, and a threshold Tbin used for binarization is selected. Selection of Tbin is performed by the following formula (3). Ts is a predetermined parameter.

  Next, binarization processing is performed using Tbin. The binarization process is performed based on the following equation (4). Here, I (x, y) is a luminance value of a certain pixel on the image.

  Then, in the binary image grouping step (S2032), the pixels having the luminance value A are grouped. Here, a block of pixels having a luminance value A is obtained by using a labeling process or the like, and the area (number of pixels) of each block is calculated. Next, noise or the like is removed using the area information of the lump. Specifically, it is determined whether or not there is a group (lumb) having an area larger than AreaMin (determination step S2033). If the area of the chunk is larger than AreaMin, the attention area image creation step (S2034) is performed. Transition. A block having an area larger than AreaMin is regarded as an observation target, and in the step of creating a region of interest image using the block (S2034), as shown in FIG. 9, position information (x1, y1, x2, y2) of the region of interest ) And the region-of-interest image is created using these pieces of position information. Here, two attention area images are created for the two time-series continuous images transferred to the attention area image creation step (S203). Then, the two attention area images thus created are shifted to the attention area image selection step (S204). If there is no block having an area larger than AreaMin, the difference image is determined to be invalid, and an instruction is issued to the irradiation command unit 107 in order to change the irradiation light amount in the auxiliary illuminator 103. AreaMin is a predetermined parameter value.

  In the attention area image selection step (S <b> 204), the two consecutive attention area images that have been transferred are narrowed down to one attention area image, and the narrowed attention area image is input to the face detection unit 109. Hereinafter, a method of narrowing down two attention area images will be described. In FIG. 10 showing the attention area image selection step (S204), first, in the luminance average value (Iave1, Iave2) calculation step (S2041), the average luminance values Iave1 and Iave2 of each attention area image are obtained. . Note that Iave1 is the average luminance value of the attention area image generated from First_image, and Iave2 is the average luminance value of the attention area image generated from Second_image.

  Next, the validity determination (halation determination) of the attention area image generated from First_image is performed using Iave1. Here, Iave1 is compared with the maximum allowable average value Gmax (comparison step S2042), and when Iave1 exceeds Gmax, it is determined that the image causes halation. In this case, the attention area image generated from the First_image is invalidated, and the attention area image generated from the Second_image is selected as the attention area image input to the face detection unit 109 (selection step S2044). At the same time, an instruction is issued to the irradiation command unit 107 in order to change the irradiation light amount of the auxiliary illuminator 103. On the other hand, when Iave1 is equal to or less than Gmax, the attention area image generated from First_image is selected as the attention area image input to the face detection unit 109 (selection step S2043). Here, Gmax is a predetermined parameter value.

  Here, as shown in FIG. 11, the irradiation command unit 107 includes four functions (an initial search unit 301, a continuous search unit 302, a halation countermeasure unit 303, and a control unit 304 corresponding to a distance). First, the initial search unit 301 will be described with reference to FIG. When the person detection preprocessing apparatus 101 is activated, the system initialization unit 112 initializes the system. When the initialization is completed, an irradiation start command signal is transmitted to the irradiation light amount setting unit 3011. The irradiation light amount setting unit 3011 that has received the irradiation start command signal sets the irradiation light amount First_image_power (%) and Second_image_power (%), and transmits these information to the auxiliary illumination control unit 104 and the history information storage unit 110. There are various methods for setting First_image_power (%) and Second_image_power (%). Here, as an example, the irradiation light amount First_image_power (%) and Second_image_power (%) are set by the following equation (5). Pini (%) is an initial value of the light amount and is a predetermined parameter. Pdiff (%) is a difference in the amount of irradiation light and is a predetermined parameter.

  Next, the continuous search unit 302 will be described with reference to FIG. In the continuous search unit 302, when the instruction signal for changing the irradiation light amount is received from the attention area extraction unit 108, First_image_power (%) at the time t-1 that is the previous time, which is input from the history information storage unit 110, is the maximum. It is determined whether the amount of irradiation light is present (determination step S3021). When First_image_power (%) at time t-1 is the maximum irradiation light amount, the irradiation light amount is changed to the minimum value in the irradiation light amount minimizing step (S3022). As an example of the method for changing the light amount, the irradiation light amount can be minimized by the following equation (6). Note that Pmin (%) is the minimum value of the light amount and is a predetermined parameter. On the other hand, if First_image_power (%) at time t-1 is not the maximum irradiation light amount, the irradiation light amount is increased in the irradiation light amount increase step (S3023). As an example of the method of changing the light quantity in this case, the irradiation light quantity can be increased by the following equation (7). Pstep (%) is the amount of increase in the amount of light, and is a predetermined parameter. Then, the changed First_image_power (%) and Second_image_power (%) are transmitted to the auxiliary lighting control unit 104 and the history information storage unit 110.

  Next, the halation countermeasure unit 303 will be described with reference to FIG. When the halation countermeasure unit 303 receives an irradiation light amount change instruction signal from the attention area extraction unit 108, whether First_image_power (%) at time t-1 input from the history information storage unit 110 is smaller than the minimum irradiation light amount. Is determined (determination step S3031).

  When First_image_power (%) at time t-1 is equal to or less than the minimum irradiation light amount, the current irradiation light amount is maintained without changing the irradiation light amount. On the other hand, if First_image_power (%) at time t−1 is larger than the minimum irradiation light amount, the irradiation light amount is changed to a value lower than the current value in the irradiation light amount reduction step (S3032). As an example of changing the irradiation light amount, the irradiation light amount can be reduced by the following equation (8).

Next, the control unit 304 corresponding to the distance will be described with reference to FIG. The control unit 304 corresponding to the distance receives the face detection result at the time t from the face detection unit 109 and the face detection result at the time t−1 from the history information storage unit 110. Here, it is determined whether or not the irradiation light amount needs to be changed using both of the received face detection results (determination step S3041). As shown in FIG. 16, an increase in the size of the face on the image means a decrease in the distance to the person to be observed. On the other hand, the smaller face size on the image means that the distance to the person to be observed becomes longer. The following formula (9) can be used as an example of determining whether or not to change the irradiation light quantity. Here, Face_size _t , Face_size _t−1 , and dFace_size respectively indicate the face size at time t, the face size at time t−1, and the allowable values of the face size at the same distance. dFace_size is a predetermined parameter.

  If the determination result is “Yes”, then a change in distance (distance increase) is determined (determination step S3042). Here, it is determined whether the distance of the observation object at time t is closer to or farther from the distance at time t−1. As an example of the determination method, a change in the distance of the observation object can be obtained using the following equation (10). From Expression (10), if the determination result is “Yes”, the irradiation light quantity at time t + 1 is increased using Expression (7) in the irradiation light quantity increase step (S3043). On the other hand, if the determination result is “No”, the irradiation light amount at time t + 1 is reduced using Expression (8) in the irradiation light amount reduction step (S3044). When the irradiation light amount at time t + 1 is determined, this information is transmitted to the auxiliary illumination control unit 104 and the history information storage unit 110.

  The auxiliary illumination control unit 104 controls the amount of power supplied to the auxiliary illuminator 103 based on the information on the amount of irradiation light received from the irradiation command unit 107.

  As shown in FIG. 17, the face detection unit 109 includes a pre-processing unit 41, face detection processing units 42, 43, 44, and a post-processing unit 45. The processing in the pre-processing unit 41 includes a noise removal step S411, an image reduction step S412, an image area acquisition step S413, an inversion / smoothing processing step S414, and density information normalization steps S415 and S416. In the noise removal step, a low-pass filter (not shown) is applied to the image input from the attention area extraction unit 108 to remove noise. In the image reduction step, the input image is converted into a plurality of sizes in order to cope with a change in the size of the face due to a change in the distance to the face of the person to be photographed.

  In the image area acquisition step, a plurality of images converted in the image reduction step are cut out with a predetermined size, for example, h (vertical) × w (width) pixels. Here, the clipped image is referred to as a clipped image. In the inversion / smoothing process step, all the cut-out images cut out in the image region acquisition step are subjected to a process for reducing luminance unevenness generated on the face surface due to biased illumination or the like. Here, the process of reducing luminance unevenness is performed by the following equations (11) and (12). Here, I ′ (x, y) is the luminance value after inversion, I (x, y) is the luminance value before inversion, and w is the width of the above-described clipped image. I ″ (x, y) is a smoothed luminance value. However, the domains of x and y are 0 to w−1 and 0 to h−1, respectively.

  In order to make the explanation of the process in the inversion / smoothing process easy to understand, an explanation will be given using an image in which the entire face is projected as a cut-out image. The image before inversion is shown in FIG. 18A, the image after inversion is shown in FIG. 18B, and the image after smoothing is shown in FIG. 18C. FIG. 18A shows an image before inversion, and there is a dark (high) portion 1800. By reversing the image of FIG. 18A, the dark portion 1800 is reversed, and a dark portion 1801 as shown in FIG. 18B is obtained. By smoothing FIGS. 18A and 18B, a smoothed image as shown in FIG. 18C is obtained, and luminance unevenness of the entire cutout image is reduced.

  In the gradation information normalization steps S415 and S416, the entire cutout image is obtained by using the following equation (13) for the image processed in the inversion / smoothing processing step and the cutout image cut out in the image region acquisition step. In order to deal with fluctuations in brightness, for example, nighttime and daytime outdoors, the shading information of the cut-out image is normalized. h (x) is the relative appearance frequency of the luminance value x, and f (g) is the luminance value after smoothing the histogram. Round [] means rounding off the first decimal place. By this conversion, a new luminance value is assigned so as to fall within the luminance value from 0 to 255 in proportion to the cumulative appearance ratio of the luminance value. FIG. 19A shows the luminance value distribution of the input image, and FIG. 19B shows the luminance value distribution of the output image. By substituting the relative appearance frequency of the luminance value of the input image shown in FIG. 19A into Expression (13), the shade of the image is normalized as shown in FIG. 19B.

  The output from the shading information normalization step S415 is input to the face detection processing unit 42. Further, the output from the shading information normalization step S416 is input to the face detection processing units 43 and 44. Note that face detection processing units 43 and 44 are provided to enable detection of a face that rotates around the optical axis of the imaging unit (detection of the tilt of the face). Details of the face detection processing units 43 and 44 will be described later. First, the face detection processing unit 42 will be described. The face detection processing unit 42 is a detection processing unit that detects a face facing the front.

  The face detection processing unit 42 has a hierarchical neural network 421. The hierarchical neural network 421 includes an input layer, a hidden layer, and an output layer. The output value from the output layer is a numerical value from 0 to 1, and the closer the output value is to 1, the more the input image looks like a face, and the closer the output value is to 0, the less the face looks like the face. Show. Since the hierarchical neural network is a known technique, detailed description thereof is omitted.

  Since both the face detection processing units 43 and 44 have the same configuration, only the face detection processing unit 43 will be described. The face detection processing unit 43 includes a hierarchical neural network 431, a mask 432, and a left / right reversing unit 433. Here, since the hierarchical neural network 431 is the same as the above-described hierarchical neural network 421, the description thereof is omitted. The left / right reversing unit 433 enables the detection of a face when the person to be photographed is tilted left and right toward the front, so that the input image has the front (the person is not tilted left and right) as the axis of symmetry. Invert. That is, the left / right reversing unit 433 is input when, for example, an image in which a person to be photographed is inclined to the right by α ° (rotating with respect to the optical axis of the camera) is input to itself. Invert the image to obtain an image tilted α ° to the left. The same applies to the left / right reversing unit 443 of the face detection processing unit 44. For example, when an image in which a person to be photographed is inclined β ° to the right toward the front is input to itself, the input image is inverted, Acquire an image tilted β ° to the left.

  Here, the background portion is included in the image in which the person to be photographed is tilted to the left and right, and the face portion cannot be detected with high accuracy. It is the mask 432 that excludes the background portion. The mask 432 will be described with reference to FIG. As shown in FIG. 20, the mask 432 masks the background portion of the input image. By applying the mask, the background portion can be eliminated, and more accurate face detection can be performed.

  As described above, the face detection processing units 42 to 44 obtain the degree of similarity of a face (representing how much the input image is similar to the learned face image) with respect to the input image. The result is output to the post-processing unit 45. The post-processing unit 45 selects face candidates using a predetermined threshold or the like for the input face similarity, checks whether the plurality of selected face candidates are the same face, and narrows down the candidates. The post-processing unit 45 outputs face position information on the image, for example, center coordinates of the detected face, face size, etc., for the narrowed face. The position information will be described with reference to FIG. A face 2100 on the narrowed-down image is subjected to a detection window 2101 of 2Hc × 2Wc with a center coordinate of the face obtained by enlarging the cut-out image cut out by h × w and Oc.

  Note that in order to speed up the face detection process, it is possible to limit the processing range in the face detection processing units 42 to 44 using the past face detection results shown in FIG. This past face detection result may be stored in the history information storage unit 110, or may be stored in a predetermined storage area (not shown). Here, FIG. 22 will be described. For example, in the past time, when the person to be imaged is tilted by α ° to the left (when the face detection processing unit 43 in the second row in FIG. 22 (left direction) is identified), the next time ( At the present time), there are two possible states: the person to be imaged further tilts α ° or more to the left or returns to the front direction. This is because it is difficult to think that it is inclined more than α ° to the right from a state where it is inclined α ° to the left in a short time. Therefore, if the person to be imaged is tilted by α ° to the left at the past time (when the face is identified in the face detection processing unit 43 (left direction) in FIG. 22), the next time ( Only the face detection processing unit 44 (left direction), the face detection processing unit 43 (left direction), and the face detection processing unit 42 (front) assumed at the current time may be processed by each face identification unit. As a result, the processing speed can be increased.

  The state estimation unit 111 cuts out a face image using the face position information that is the result output from the face detection unit 109, extracts facial expressions, eye open / closed states, and the like from the cut out face image, and observes the human being to be observed. Is estimated.

  Next, the human state estimation flow in the human state estimation apparatus including the object (person) detection preprocessing apparatus of the present invention will be described with reference to FIG. The auxiliary illumination control unit 104 supplies power to the auxiliary illuminator 103 based on the synchronization signal input from the imaging unit 106, and the auxiliary illuminator 103 irradiates the observation object (here, a person) by the supply of electric power. (Step S2301). The photographing unit 106 photographs the person being irradiated (step S2302). The attention area extraction unit 108 performs difference processing and binarization processing on the image input from the imaging unit 106, and extracts a human presence area on the image (step S2303).

  The face detection unit 109 outputs face position information in the image region based on the information on the human presence region input from the attention region extraction unit 108 (step S2304). At this time, the irradiation command unit 107 determines the irradiation light amount of the auxiliary illuminator 103 at the next time based on information such as the result of the attention area extraction unit 108 and the face detection unit 109, and uses the information as the auxiliary illumination control unit. It outputs to 104 (step S2305). In addition, the state estimation unit 111 analyzes facial expressions, eye opening / closing, and the like from the face image based on the face position information input from the face detection unit 109, and estimates a human state (step S2306).

  The object detection pre-processing apparatus and the object detection pre-processing method according to the present invention can reduce costs, perform uniform irradiation over a wide range, are not easily affected by the surrounding environment such as ambient light, and are safe for the human body. Object detection pre-processing device and object detection pre-processing method that enable detection of an observation object even in a low-light environment to enable extraction of the existence area of the object and estimation of the state of the object at high speed Useful for.

It is a block diagram which shows the structure of the target object state estimation apparatus containing the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the spectral sensitivity characteristic of the imaging | photography part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the transmission wavelength characteristic of the specific wavelength transmission filter in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the timing chart of irradiation of the auxiliary | assistant illuminator in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the irradiation pattern in the time series continuous image of the auxiliary | assistant illuminator in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows two time series continuous images image | photographed by the imaging | photography part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the attention area extraction part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the difference image validity determination step of the attention area extraction part in the target object detection preprocessing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the attention area image creation step of the attention area extraction part in the target object detection preprocessing apparatus which concerns on embodiment of this invention. It is a figure which shows the attention area image produced in the attention area image creation step of the attention area extraction part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the selection step of the attention area image of the attention area extraction part in the target object detection preprocessing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the irradiation command part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the initial stage search part of the irradiation command part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the continuous search part of the irradiation command part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the halation countermeasure part of the irradiation command part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the process structure of the control part according to the distance of the irradiation command part in the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the relationship of the distance of the target object detection pre-processing apparatus which concerns on embodiment of this invention, and an observation target object (person). It is a block diagram which shows the process structure of the face detection part in the target object state estimation apparatus containing the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the image before inversion in the inversion and smoothing process step of the face detection part in the target object state estimation apparatus containing the target object detection preprocessing apparatus which concerns on embodiment of this invention. It is a figure which shows the image after inversion in the inversion and smoothing process step of the face detection part in the object state estimation apparatus containing the object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the smoothed image in the inversion and smoothing process step of the face detection part in the target object state estimation apparatus containing the target object detection preprocessing apparatus which concerns on embodiment of this invention. It is a figure which shows the luminance value distribution of the input image in the density | concentration information normalization step of the face detection part in the target object state estimation apparatus containing the target object detection preprocessing apparatus which concerns on embodiment of this invention. It is a figure which shows the luminance value distribution of the output image in the shading information normalization step of the face detection part in the target object state estimation apparatus including the target object detection preprocessing apparatus according to the embodiment of the present invention. It is a figure for demonstrating the mask applied to the background part of the input image of the face detection process part of the face detection part in the target object state estimation apparatus containing the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the positional information output from the post-processing part of the face detection part in the target object state estimation apparatus containing the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a figure which shows the past face detection result which can be used by the face detection part in the target object state estimation apparatus containing the target object detection pre-processing apparatus which concerns on embodiment of this invention. It is a flowchart for demonstrating the estimation flow of a person's state in the target object state estimation apparatus containing the target object detection pre-processing apparatus of this invention.

Explanation of symbols

41 Pre-processing unit 42, 43, 44 Face detection processing unit 45 Post-processing unit 100 Object state estimation device (person state estimation device)
101 Object detection pre-processing device (person detection pre-processing device)
102 Diffusion filter 103 Auxiliary illuminator (irradiation means)
104 Auxiliary illumination control unit (irradiation control means)
105 Specific wavelength transmission filter 106 Imaging unit (image acquisition means)
107 Irradiation command section (irradiation command means)
108 attention area extraction unit (attention area extraction means)
109 face detection unit 110 history information storage unit 111 state estimation unit 112 system initialization unit 301 initial search unit 302 continuous search unit 303 halation countermeasure unit 304 control unit 421, 431, 441 hierarchical neural network 432, 442 mask 433 , 443 Left-right reversing part 1800, 1801 Bright (high) part 2100 Face on image 2101 Detection window 3011 Irradiation light quantity setting part

Claims (16)

Irradiating means for irradiating the observation object with different timing and different irradiation light quantity;
Image acquisition means for acquiring respective images of the observation object irradiated with the different irradiation light amounts;
Attention area extraction means for extracting an existence area of the observation object based on each acquired image;
An irradiation command means for generating a signal for controlling the irradiation light amount of the irradiation means when a predetermined condition is not satisfied when extracting the existence region of the observation object;
An irradiation control means for controlling the amount of irradiation light of the irradiation means based on the generated signal;
An object detection pre-processing apparatus provided.   The object detection pretreatment apparatus according to claim 1, wherein the irradiation unit irradiates near infrared light so as to enable photographing for 24 hours.   The image acquisition unit includes a specific wavelength transmission filter that transmits only reflected light of the light irradiated by the irradiation unit in order to reduce the influence of disturbance light including visible light and enable stable image acquisition. The object detection pre-processing device according to claim 1 or 2.   The irradiation control unit is configured to perform irradiation by the irradiation unit in order to synchronize a timing at which the observation target is irradiated with a different amount of irradiation light by the irradiation unit with a timing at which the image acquisition unit acquires an image of the observation target. The object detection pretreatment device according to any one of claims 1 to 3, which is controlled.   The said irradiation command means produces | generates the said signal for controlling the said irradiation light quantity of the said irradiation means according to the distance of the said observation target object with respect to the said image acquisition means. Object detection pretreatment device.   6. The target object detection pre-processing device according to claim 1, wherein the attention area extraction unit extracts an existence area of the observation target object by performing a difference process on the acquired image.   The object detection pre-processing apparatus according to claim 1, wherein the observation object is a human face.   The object detection pre-processing apparatus according to claim 1, wherein the irradiation unit includes a diffusion filter for uniformly diffusing light to be irradiated over a wide range. Irradiating the observation object with different timing and different irradiation light quantity;
Acquiring each image of the observation object irradiated with the different irradiation light amount;
Extracting an existing region of the observation object based on each acquired image;
Generating a signal for controlling the amount of irradiation light when a predetermined condition is not satisfied when extracting the existence region of the observation object;
Controlling the amount of irradiation light based on the generated signal,
An object detection pre-processing method.   The object detection pre-processing method according to claim 9, wherein in the step of irradiating the observation object with different amounts of irradiation light, near-infrared light is irradiated in order to enable imaging for 24 hours.   In the step of acquiring the image of the observation object, the object is irradiated by the step of irradiating the observation object with a different irradiation light quantity in order to reduce the influence of disturbance light including visible light and enable stable image acquisition. The object detection pre-processing method according to claim 9 or 10, wherein a specific wavelength transmission filter that transmits only reflected light of the reflected light is used.   In order to synchronize the timing of irradiating the observation object with different irradiation light amounts with the timing of acquiring the image of the observation object, the method further includes the step of controlling the irradiation in the step of irradiating the observation object with different irradiation light amounts. The object detection pre-processing method according to any one of claims 9 to 11.   The step of generating a signal for controlling the amount of irradiation light generates the signal for controlling the amount of irradiation light in accordance with a distance of the observation object relative to a means for acquiring an image of the observation object. Item 13. The object detection pretreatment method according to any one of Items 9 to 12.   The object detection pre-processing according to any one of claims 9 to 13, wherein in the step of extracting the existence area of the observation object, the existence area of the observation object is extracted by difference processing of the acquired image. Method.   The object detection preprocessing method according to claim 9, wherein the observation object is a human face. The object detection preprocessing method according to any one of claims 9 to 15, wherein a diffusion filter for uniformly diffusing irradiated light over a wide range is used in the step of irradiating the observation object with different amounts of irradiation light.

Images