JP2006259902A - Image processor and processing method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily and accurately recognize a face image. <P>SOLUTION: An imaging device 12 images the face of a user 11, and outputs a picture element value substantially proportional to a logarithm of incident luminous energy. A difference image calculation part 21 calculates a difference image comprising a difference between the first image that is an image of the face of the user 11 imaged by the imaging device 12 under the condition where a lighting system 13 emits no light toward the face of the user 11, and the second image that is an image of the face of the user 11 imaged by the imaging device 12 under the condition where the lighting system 13 emits the light toward the face of the user 11. A face area extracting part 22 extracts an area of the face of the user 11 within the difference image, a face direction determination part 23 determines a direction of the face of the user 11, and a face organ extracting part 24 extracts an organ of the face of the user 11. The present invention is applicable, for example, to an image processor for processing the image of the face of the human being. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and in particular, an image processing apparatus and method, a recording medium, and a program that make it possible to easily and accurately perform facial image recognition processing. About.

  Conventionally, a user's face area or an organ area such as an eye or mouth included in an image captured by an imaging device is recognized, and the position of the recognized area is output. There is a face image recognition device that extracts an image (see, for example, Patent Document 1).

Japanese Patent Laid-Open No. 05-91406

  However, in the conventional face image recognition device, the user's face and face are accurately detected due to the influence of an object other than the face included in the captured image and the difference in the light condition of the subject when the image is captured. The organ could not be recognized. For example, an image of a bright subject that is captured in a bright outdoors and an image of a subject that is captured in a dark room and that is captured in a dark room can be accurately recognized unless a different condition is set. There were things that could not be done.

  The present invention has been made in view of such a situation, and makes it possible to easily and accurately perform facial image recognition processing.

  In the image processing apparatus according to the present invention, the illumination unit irradiates the face with light, and the first image that is an image of the face captured by the imaging unit in a state where the illumination unit does not irradiate the face with light. In this state, a face image acquisition unit that acquires a face image that is a difference image including a difference in pixel value from a second image that is an image of a face captured by the imaging unit, and a face acquired by the face image acquisition unit Using a pixel value addition means for calculating a pixel addition value by adding a pixel value for each column in the horizontal or vertical direction of the image and a pixel addition value for each column in the horizontal or vertical direction of the difference image, Separating the eye image including the eye region, the nose image including the nose region, and the mouth image including the mouth region, and processing for extracting the eye region from the eye image separated by the separating device By eye area processing means to perform and separation means A nose region processing unit that performs a process of extracting the nose region from the separated nose image; and a mouth region processing unit that performs a process of extracting the mouth region from the mouth image separated by the separating unit. Features.

  In the image processing apparatus of the present invention, the illumination unit irradiates the face with light, and the illumination unit irradiates the face with light in a state where the illumination unit does not irradiate the face with light. In this state, a face image that is a difference image made up of a difference in pixel value from the second image that is an image of the face captured by the imaging means is acquired. Then, a pixel addition value is calculated by adding the pixel value for each column in the horizontal or vertical direction of the acquired face image, and the face image is calculated using the pixel addition value for each column in the horizontal or vertical direction of the difference image. Are separated into an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region. Furthermore, the process of extracting the eye area from the separated eye image is performed, the process of extracting the nose area from the separated nose image is performed, and the mouth area is extracted from the separated mouth image. Processing is performed.

  Therefore, it is possible to accurately extract the eye, nose, and mouth regions of the face image including the face to be recognized from the difference image. As a result, the face image recognition process can be performed easily and accurately.

  The image acquisition means, the pixel value addition means, the separation means, the eye area processing means, the nose area processing means, and the mouth area processing means are performed by an arithmetic device such as a CPU (Central Processing Unit) or DSP (Digital Signal Processor). Composed.

  The eye area processing means is obtained by an eye image pixel value addition means for calculating a pixel addition value obtained by adding pixel values for each column in the horizontal or vertical direction to the eye image, and an eye image pixel value addition means. The eye area extraction means for extracting the eye area using the pixel addition value for each column in the horizontal or vertical direction of the eye image, and the eye position from the eye area extracted by the eye area extraction means Eye representative point determining means for determining representative points to be determined.

  In the eye region processing means, a pixel addition value is calculated by adding pixel values for each column in the horizontal or vertical direction to the eye image, and the calculated pixels for each column in the horizontal or vertical direction of the calculated eye image. An eye region is extracted using the added value, and a representative point representing the position of the eye is determined from the extracted eye region.

  Therefore, the eye region and the representative point representing the eye can be accurately extracted.

  The eye image pixel value adding means, the eye area extracting means, and the eye representative point determining means are configured by arithmetic units such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor).

  The nose region processing means is obtained by a nose image pixel value addition means for calculating a pixel addition value obtained by adding pixel values for each column in the horizontal or vertical direction to the nose image, and a nose image pixel value addition means. The nose area extraction means for extracting the nose area using the pixel addition value for each column in the horizontal or vertical direction of the nose image, and the nose position from the nose area extracted by the nose area extraction means And a nose representative point determining means for determining a representative point to be determined.

  In the nose region processing means, a pixel addition value is calculated by adding the pixel value for each column in the horizontal or vertical direction to the nose image, and the pixel addition value for each column in the horizontal or vertical direction of the nose image is calculated. Using the extracted nose region, a representative point representing the position of the nose is determined from the extracted nose region.

  Therefore, the nose region and the representative point representing the nose can be accurately extracted.

  The nose image pixel value adding means, the nose region extracting means, and the nose representative point determining means are configured by arithmetic units such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor).

  The mouth area processing means is obtained by mouth image pixel value adding means for calculating a pixel added value obtained by adding pixel values for each column in the horizontal or vertical direction to the mouth image, and mouth image pixel value adding means. The mouth area extraction means for extracting the mouth area using the pixel addition value for each column in the horizontal or vertical direction of the mouth image, and the mouth position from the mouth area extracted by the mouth area extraction means It is possible to have mouth representative point determining means for determining a representative point to be determined.

  In the mouth area processing means, a pixel addition value is calculated by adding pixel values for each column in the horizontal or vertical direction to the mouth image, and the pixel addition value for each column in the horizontal or vertical direction of the mouth image is calculated. Using the extracted mouth area, a representative point representing the mouth position is determined from the extracted mouth area.

  Accordingly, it is possible to accurately extract the mouth region and the representative point representing the mouth.

  The mouth image pixel value adding means, the mouth area extracting means, and the mouth representative point determining means are constituted by arithmetic units such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor).

  The first and second images can be imaged by an imaging means having a logarithmic conversion type imaging device that outputs a pixel value substantially proportional to the logarithm of the incident light quantity by utilizing the sub-threshold characteristic of the semiconductor.

  The image sensor can be an HDRC (High Dynamic Range CMOS (Complementary Metal Oxide Semiconductor)).

  Thereby, even when intense light is irradiated to the subject, the luminance distribution of the subject can be faithfully represented without saturation of the pixel value. In addition, there is no need to adjust the aperture, shutter speed, etc., and a difference image consisting of the difference in pixel value between an image obtained by imaging a subject illuminated only by natural light and an image obtained by imaging an object illuminated by strong light Can be easily obtained.

  The image processing method, the program, and the program recorded in the recording medium of the present invention include a first image that is an image of the face captured by the imaging unit in a state where the illumination unit does not irradiate the face with light. Face image acquisition that acquires a face image that is a difference image composed of a difference in pixel value from a second image that is an image of the face captured by the imaging means while the illumination means irradiates the face with light A pixel value addition step for calculating a pixel addition value obtained by adding a pixel value for each column in the horizontal or vertical direction of the face image acquired by the step, the face image acquisition step, and a horizontal or vertical direction of the difference image, respectively Separation step for separating a face image into an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region, using the pixel addition value for each column, and processing of the separation step In An eye region processing step for performing processing for extracting the eye region from the separated eye image, and a nose region processing step for performing processing for extracting the nose region from the nose image separated by the processing of the separation step; A mouth region processing step for performing processing for extracting a mouth region from the mouth image separated by the processing of the separation step.

  In the image processing method, the program, and the program recorded on the recording medium of the present invention, a first image that is an image of the face captured by the imaging unit in a state where the illumination unit does not irradiate the face with light. Then, in a state where the illumination unit irradiates the face with light, a face image that is a difference image made up of a difference in pixel value from the second image that is an image of the face captured by the imaging unit is acquired. Then, a pixel addition value is calculated by adding the pixel value for each column in the horizontal or vertical direction of the acquired face image, and the face image is calculated using the pixel addition value for each column in the horizontal or vertical direction of the difference image. Are separated into an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region. Furthermore, the process of extracting the eye area from the separated eye image is performed, the process of extracting the nose area from the separated nose image is performed, and the mouth area is extracted from the separated mouth image. Processing is performed.

  Therefore, it is possible to accurately extract the eye, nose, and mouth regions of the face image including the face to be recognized from the difference image. As a result, the face image recognition process can be performed easily and accurately.

  According to the present invention, face image recognition processing can be performed easily and accurately.

  FIG. 1 shows a configuration example of an embodiment of an image processing system to which the present invention is applied.

  The image processing system 1 in FIG. 1 captures (shoots) or irradiates an imaging device 12 that captures an image of a user 11 as a subject, an illumination device 13 that irradiates the user 11 with light, and the imaging device 12 and the illumination device 13. Are configured by a timing control device 14 that commands (controls) the timing and an image processing device 15 that processes an image captured by the imaging device 12.

  In accordance with the timing control signal supplied from the timing control device 14, the imaging device 12 captures an image of the user 11 as a subject and supplies the image (image signal) obtained as a result to the image processing device 15. As will be described later, the imaging device 12 is an imaging device having an imaging element that outputs a pixel value proportional to the logarithm of the amount of incident light. Moreover, in the imaging device 12, the pixel values of all the pixels (imaging elements) are read at the same timing.

  The illumination device 13 irradiates the user 11 with light according to the timing control signal supplied from the timing control device 14.

  The timing control device 14 supplies a timing control signal that is an imaging cycle to the imaging device 12. For example, the timing control device 14 supplies the timing control signal to the imaging device 12 at a cycle (frame cycle) of capturing 30 images per second. In addition, the period (field period) which images 60 images per second, or other periods may be used. In addition, the captured image may be either a progressive (non-interlaced) image or an interlaced image.

  Further, the timing control device 14 is configured so that an image when the user 11 is not irradiated by the lighting device 13 and an image when the user 11 is irradiated are alternately captured by the imaging device 12. A timing control signal for commanding (controlling) the timing of irradiating light is supplied to the lighting device 13. For example, when the timing control device 14 supplies a timing control signal to the imaging device 12 at a frame period, the user 11 is irradiated by the illumination device 13 in an odd frame among images captured by the imaging device 12. The timing control signal is supplied to the illumination device 13 so that the image becomes a non-image and the image irradiated by the user 11 in the even-numbered frame.

  Further, the timing control device 14 captures an image (hereinafter referred to as a captured image) supplied from the imaging device 12 to the image processing device 15 in a state where the user 11 is not irradiated with light by the illumination device 13. In order to determine whether the image is an image (hereinafter referred to as a reference image) or an image (hereinafter referred to as an illumination image) captured by the illumination device 13 while the user 11 is irradiated with light. The determination signal is supplied to the image processing device 15.

  The image processing apparatus 15 includes a difference image calculation unit 21, a face area extraction unit 22, a face orientation determination unit 23, and a face organ extraction unit 24.

  The image processing device 15 selects an image suitable for performing face recognition processing (face authentication) on the captured image supplied from the imaging device 12, and includes a face or a face included in the selected captured image. A captured image is output to a subsequent apparatus (not shown) together with information for specifying the region of the organ. Here, facial organs represent eyes, nose and mouth.

  The captured image (corresponding image signal) from the imaging device 12 is supplied to the difference image calculation unit 21 and the facial organ extraction unit 24. Further, the above-described determination signal is also supplied to the difference image calculation unit 21 from the timing control device 14.

  The difference image calculation unit 21 determines whether the captured image supplied from the imaging device 12 is a reference image or an illumination image according to the determination signal, and also determines the reference image and the imaging device as a subsequent frame. A difference image is generated using the illumination image (supplied from the image 12) and supplied to the face area extraction unit 22. Here, the difference image is an image in which the pixel value of each pixel constituting the image is a difference in pixel value between corresponding pixels between the reference image and the illumination image.

  The face area extraction unit 22 extracts a face area of the user 11 from the difference image supplied from the difference image calculation unit 21, and includes face area information for specifying the extracted area together with the difference image as a face orientation determination unit. 23.

  The face orientation determination unit 23 determines the face of the user 11 in the captured image from the image of the face area of the difference image specified by the face area information from the face area extraction unit 22 (hereinafter referred to as a face image as appropriate). Determine the direction. There are three types of determination results determined by the face direction determination unit 23: front, right, and left. In the extraction (identification) of the facial organ region performed by the facial organ extraction unit 24 in the subsequent stage, or the face recognition processing performed in the output destination device from which the image processing device 15 outputs the processing result, the face is faced forward. When the recognition process is performed using an image that is not suitable, the recognition accuracy decreases. Therefore, the face orientation determination unit 23 determines whether the difference image (captured image) is an image facing the front suitable for recognition. The face orientation determination unit 23 supplies a determination result indicating the orientation of the face of the user 11 to the face organ extraction unit 24 together with the difference image and face region information supplied from the face region extraction unit 22.

  When the determination result supplied from the face orientation determination unit 23 (face orientation) indicates that the face organ extraction unit 24 represents the front, the face organ extraction unit 24 is based on the face image from the face orientation determination unit 23. Extract (specify) regions of facial organs (eyes, nose, and mouth). Then, the facial organ extraction unit 24 outputs the eye region information, the nose region information, and the mouth region information obtained as a result of extracting the facial organ region, together with the above-described facial region information, to the subsequent apparatus. The facial organ extraction unit 24 also outputs a captured image supplied from the imaging device 12 to a subsequent device.

  In the image processing system 1 configured as described above, the difference image calculation unit 21 of the image processing apparatus 15 captures the face of the user 11 while the illumination apparatus 13 is not irradiating the user 11. The imaging device 12 acquires a reference image and an illumination image obtained by imaging the face of the user 11 while the illumination device 13 is irradiating the face of the user 11.

  The difference image calculation unit 21 calculates a difference image between the reference image and the illumination image. The face area extraction unit 22 extracts a face area of the user 11 from the difference image, and supplies face area information for specifying the extracted area to the face direction determination unit 23. The face organ extraction unit 24 extracts the face eye, nose, and mouth regions of the face image determined by the face orientation determination unit 23 that the face of the user 11 is facing the front, and the extracted regions. Eye region information, nose region information, and mouth region information that identify The face organ extraction unit 24 outputs the obtained eye region information, nose region information, and mouth region information, the face region information obtained by the face region extraction unit 22, and the captured image supplied from the imaging device 12. To do.

  With reference to FIG. 2 and FIG. 3, the outline of the processing performed by the image processing device 15 will be described.

  FIG. 2 shows an example of an image (captured image) captured by the imaging device 12 and input to the image processing device 15.

  The image 31 in FIG. 2 shows the user 11 positioned in front of a predetermined background, and the image 31 includes at least the face of the user 11.

  In the image processing apparatus 15, the upper left corner of the image is the origin, and as shown in FIG. 2, the right direction (horizontal direction) of the drawing is the (positive) X direction, and the downward direction (vertical direction) of the drawing is (positive). It is assumed that a coordinate system in the Y direction is used.

  For example, when the image 31 illustrated in FIG. 2 is input, the image processing device 15 extracts (specifies) the face area of the user 11 included in the image 31.

  Therefore, FIG. 3 shows an image (face image) FR including a face area extracted from the image 31 of FIG.

  Further, the image processing device 15 extracts the left eye region IL, the representative point ILp representing the region IL, the right eye region IR, and the representative point IRp representing the region IR in the face image FR ( Identify.

  Further, the image processing device 15 extracts a nose region NR, a representative point NRp representing the region NR, a mouth region MR, and a representative point MRp representing the region MR in the face image FR ( Identify.

  In the following, the imaging device 12 that captures the image 31 shown in FIG. 2 and supplies the image processing device 15 to the image processing device 15, and the face image FR, the left eye region IL and its representative point ILp, and the right eye region from the supplied image 31. Detailed configurations of the image processing device 15 that extracts (specifies) the IR and its representative point IRp, the nose region IR and its representative point NRp, and the mouth region MR and its representative point MRp will be described.

  FIG. 4 is a block diagram illustrating a detailed configuration example of the imaging device 12 of FIG.

  The imaging device 12 includes a lens 41 and a logarithmic conversion type imaging device 42.

  The logarithmic conversion type image pickup device 42 is a logarithmic conversion type image pickup device such as HDRC (High Dynamic Range CMOS (Complementary Metal Oxide Semiconductor)), and includes a light detection unit 51, a logarithmic conversion unit 52, and an A / D conversion unit 53. , And an imaging timing control unit 54.

  Light emitted from the subject (user 11) imaged by the imaging device 12 (or light reflected by the subject) enters the lens 41 and is not shown in the light detection unit 51 of the logarithmic conversion type image sensor 42. An image is formed on the light detection surface.

  The light detection unit 51 is configured by, for example, a light receiving element including a plurality of photodiodes. The light detection unit 51 converts the light of the subject imaged by the lens 41 into a charge corresponding to the brightness (illuminance) of the incident light, and accumulates the converted charge. The light detection unit 51 supplies the accumulated charge to the logarithmic conversion unit 52 in synchronization with the control signal supplied from the imaging timing control unit 54.

  The logarithmic conversion unit 52 includes, for example, a plurality of MOSFETs (Metal Oxide Semiconductor Field Effect Transistors). The logarithmic conversion unit 52 uses the sub-threshold characteristic of the MOSFET to convert the charge supplied from the light detection unit 51 into the logarithm of the number of charges (the intensity of current) for each pixel (the logarithm of the amount of light of the subject). An analog electrical signal converted into a voltage value approximately proportional to The logarithmic converter 52 supplies the generated analog electrical signal to the A / D converter 53.

The A / D conversion unit 53 A / D converts an analog electric signal into digital image data in synchronization with the control signal supplied from the imaging timing control unit 54. For example, in the case of conversion to 14-bit unsigned binary digital image data, the pixel value of the image data takes a value in the range of 0 to the brightest 2 ¹⁴ −1. The A / D conversion unit 53 supplies the pixel value of the converted digital image data to the image processing device 112.

  Thus, the imaging device 12 outputs a pixel value proportional to the logarithm of the brightness (incident light amount) of the light of the subject incident on the light detection unit 51. The details of the logarithmic conversion type image pickup device are disclosed in, for example, JP-T-7-506932.

  FIG. 5 is a graph showing sensitivity characteristics of the logarithmic conversion type image pickup element 42, the CCD image pickup element, the silver salt film, and the human eye.

  The horizontal axis in FIG. 5 indicates the logarithmic value of the illuminance (unit is lux) of the incident light, and the vertical axis indicates the sensitivity to the illuminance of the incident light. A curve (straight line) L1 indicates the sensitivity characteristic of the logarithmic conversion type image pickup element 42, a curve L2 indicates the sensitivity characteristic of the CCD image pickup element, a curve L3 indicates the sensitivity characteristic of the silver salt film, and the curve (straight line) L4 indicates the human characteristic. It shows the sensitivity characteristics of the eyes.

  As described above, the logarithmic conversion type image pickup element 42 outputs a pixel value that is substantially proportional to the logarithm of the incident light quantity, so that the capacitance of the photodiode, MOSFET, etc. constituting the logarithmic conversion type image pickup element 42 is not saturated. A CCD imaging device, a silver salt film, and a subject can be imaged with a dynamic range of about 170 dB, which is wider than the human eye, from about 1 millilux to about 500 kilolux higher than the brightness of sunlight.

  Therefore, the imaging device 12 using the logarithmic conversion type imaging element 42 does not generate luminance clipping in a luminance range that can be visually recognized by humans, and therefore, it is not necessary to adjust the aperture and shutter speed to adjust the amount of incident light. That is, the imaging device 12 can faithfully capture the detailed luminance distribution of the subject without adjusting the amount of incident light.

  For example, when imaging the front of a vehicle from inside the vehicle during the daytime, even if the sun is within the angle of view, the image captured by the imaging device 12 can be obtained from the sun and the road ahead without adjusting the amount of incident light. The image is a faithful reproduction of the luminance distribution. In addition, when imaging the front of the vehicle from inside the vehicle at night, even if the headlight of the oncoming vehicle is illuminated from the front, the image captured by the imaging device 12 does not have to adjust the incident light amount. This is an image that faithfully reproduces the luminance distribution from the light of the headlight to the area not illuminated by the headlight of the host vehicle.

  On the other hand, the CCD imaging device has a narrower dynamic range than the human eye, as shown in FIG. Therefore, in an image pickup apparatus using a CCD image pickup device, it is necessary to adjust an aperture, a shutter speed, and the like so that the illuminance of incident light is within the dynamic range of the CCD image pickup device.

  However, when the illuminance range of the subject's light exceeds the dynamic range of the CCD image sensor, the pixel value of the pixel in the bright area of the subject is limited to the maximum pixel value that can be output by the CCD image sensor, Luminance clipping occurs in which the pixel value of a pixel in a dark region is limited to the minimum pixel value that can be output by the CCD image sensor. In addition, when the incident light amount is adjusted, for example, the incident light amount corresponding to a region where the luminance of the subject does not fluctuate fluctuates, and the pixel value of the region fluctuates. . That is, in an imaging apparatus using a conventional CCD imaging device, the pixel value varies due to factors other than the variation in luminance of the subject and the movement of the subject.

  Further, in the CCD image pickup device and the silver salt film, as shown by the curves L2 and L3, the logarithmic conversion type image pickup device is compared with the case where the sensitivity characteristic is not proportional to the logarithm of the illuminance of incident light due to factors such as the gamma characteristic. At 42, the sensitivity characteristic is approximately proportional to the logarithm of the illuminance of the incident light.

  Thus, since the imaging device 12 using the logarithmic conversion type imaging device 42 is not affected by the occurrence of luminance clipping, adjustment of the incident light amount, and gamma characteristics, the pixel value of the image captured by the imaging device 12 is It fluctuates so as to reflect the change in luminance of the subject and the movement of the subject almost faithfully. That is, the pixel value (difference value) of each pixel of the difference image obtained by taking the image difference between frames is a value in which the luminance variation of the subject and the motion of the subject are reflected almost faithfully.

Further, since the pixel value of the image output from the imaging device 12 is a value that is substantially proportional to the logarithm of the incident light quantity, the image obtained by capturing the subject regardless of the brightness (illuminance) of the light irradiated to the subject. The distribution of the pixel values at is a reflection of the reflectance distribution of the subject in substantially the same manner. For example, when a subject whose ratio between the maximum value and the minimum value of reflectance is 10: 1 is imaged by irradiating light having an illuminance difference of about 100 times between the first time and the second time, the first image and The width of the histogram showing the distribution of pixel values with the second image is almost the same value (1 = log ₁₀ 10). On the other hand, when the pixel value of the image is proportional to the amount of incident light, the difference in the width of the histogram indicating the distribution of the pixel values of the first image and the second image is about 100 times.

Furthermore, regardless of the distribution of the luminance (reflectance) of the subject, when the luminance of the subject fluctuates at substantially the same ratio, the variation value of the pixel value of the image obtained by imaging the subject is substantially the same. For example, if there are two areas in the subject where the luminance ratio is 100: 1, the illuminance of the light radiated to the subject changes almost uniformly, and the subject's luminance changes by + 5%, which is almost the same ratio The variation values of the pixel values corresponding to the two areas are almost the same value (log ₁₀ 1.05). On the other hand, when the pixel value is proportional to the amount of incident light, the difference between the fluctuation values of the pixel values corresponding to the two regions described above is about 100 times.

  FIG. 6 is a block diagram illustrating a detailed configuration example of the difference image calculation unit 21 in FIG. 1.

  The difference image calculation unit 21 includes an image acquisition unit 61, memories 62 and 63, a difference processing unit 64, and a filter processing unit 65.

  The image acquisition unit 61 acquires a captured image supplied from the imaging device 12. Here, as described above, the image acquisition unit 61 receives an image (reference image) captured in a state where the user 11 is not irradiated with light by the illumination device 13 and the user 11 emits light by the illumination device 13. Images (illumination images) picked up in this state are alternately input (supplied).

  The image acquisition unit 61 determines whether the captured image from the imaging device 12 is a reference image or an illumination image according to the determination signal from the timing control device 14. Then, when the captured image from the imaging device 12 is a reference image, the image acquisition unit 61 supplies the acquired captured image (reference image) to the memory 62. When the captured image from the imaging device 12 is an illumination image, the image acquisition unit 61 supplies the acquired captured image (illumination image) to the memory 63.

  Each of the memories 62 and 63 stores the reference image or the illumination image supplied from the image acquisition unit 61 and supplies the reference image or the illumination image to the difference processing unit 64 as necessary.

  The difference processing unit 64 calculates a difference image from the reference image stored in the memory 62 and the illumination image stored in the memory 63 and supplies the difference image to the filter processing unit 65. Specifically, the difference processing unit 64 calculates the difference image by subtracting the pixel values of the pixels constituting the reference image from the corresponding pixels from the pixel values of the pixels constituting the illumination image.

  The filter processing unit 65 performs filtering (filter processing) on the difference image from the difference processing unit 64, and supplies the processed difference image to the face area extraction unit 22 (FIG. 1). The filtering process performed here, for example, divides the difference image into a plurality of blocks having a predetermined size, and sets the pixel values of all the pixels in the block as the average value of the pixel values of each pixel in the block. It can be a mosaic process.

  Further, as the filter processing, when the pixel at the center in the block is the target pixel, the pixel value of the target pixel is corrected so as to be close to the pixel value of the surrounding (in the block) pixel. May be employed for all pixels of the difference image (smoothing process for smoothing the difference image) or the like.

  That is, the filter processing performed by the filter processing unit 65 is performed by a pixel value unique to surrounding pixels when processing is performed by the face area extraction unit 22, the face orientation determination unit 23, and the face organ extraction unit 24 in the subsequent stage. Any material may be used as long as it has an effect of eliminating (reducing) singular points. In other words, when the face processing unit 22 or the like calculates a curve (such as a column pixel value addition curve or a row pixel value addition curve, which will be described later) from the pixel value of each pixel, the filtering process smoothes the curve. It can function as a low-pass filter.

  Also, when mosaic processing is performed as filter processing, the average value of the pixel values of each pixel in the block is assigned to the pixel value of one pixel instead of all the pixels in the block. It is also possible to reduce (compress) the total number of pixels of the difference image and reduce the information processing amount of the difference image. For example, when assigning an average value of 4 pixels of 2 pixels each in the horizontal direction and the vertical direction to one pixel, the number of pixels after the filter processing can be ¼ of the number of pixels before the processing.

  FIG. 7 is a block diagram illustrating a detailed configuration example of the face area extraction unit 22 of FIG.

  The face area extracting unit 22 includes a pixel value adding unit 71, a threshold setting unit 72, and a face area data extracting unit 73.

  The difference image supplied from the difference image calculation unit 21 (the filter processing unit 65) (after the filter processing) is supplied to the pixel value addition unit 71 and the face area data extraction unit 73.

The pixel value addition unit 71 adds pixel values for each column in the horizontal or vertical direction of the difference image, and obtains pixel addition values V _i or W _j for each column in the vertical or horizontal direction of the difference image.

That is, the number of pixels in the horizontal direction (X direction) of the difference image is N, the number of pixels in the vertical direction (Y direction) is M, and the pixel values of the i-th pixel in the horizontal direction and the j-th pixel in the vertical direction are When G _ij (i = 1 to N, j = 1 to M), the pixel addition value V _i for each column in the vertical direction of the difference image can be obtained by the following equation (1).

・・・（１）

... (1)

Similarly, the pixel addition value W _j for each column (row) in the horizontal direction of the difference image can be obtained by the following equation (2).

・・・（２）

... (2)

The pixel value addition unit 71 sets a pixel addition value V _i or W _j for each column in the vertical or horizontal direction of the difference image (hereinafter referred to as a pixel addition value V _i or W _{j of the} difference image as appropriate) as a threshold value. To the unit 72 and the face area data extraction unit 73.

The threshold value setting unit 72 uses the pixel addition value V _i or W _j of the difference image supplied from the pixel value addition unit 71 to determine the face area and other areas of the user 11 in the horizontal direction or the vertical direction of the difference image. Calculate and set (determine) the threshold TH _X or TH _Y to be distinguished.

That is, the threshold setting unit 72 calculates the average value of the pixel addition values V _i of the difference image as the threshold TH _X for distinguishing the face area of the user 11 from other areas in the horizontal direction of the difference image. The threshold TH _X can be obtained by the following equation (3).

・・・（３）

... (3)

The threshold setting unit 72 calculates an average value of the pixel addition values W _j of the difference image as a threshold TH _Y for distinguishing the face area of the user 11 from other areas in the vertical direction of the difference image. The threshold TH _Y can be obtained by the following equation (4).

・・・（４）

... (4)

Then, the threshold setting unit 72 supplies the calculated threshold TH _X or TH _Y to the face area data extraction unit 73.

The face area data extraction unit 73 calculates the difference based on the pixel addition values V _i and W _{j of the} difference image supplied from the pixel value addition unit 71 and the threshold values TH _X and TH _Y supplied from the threshold setting unit 72. The face area of the user 11 in the image is specified.

Specifically, the face area data extraction unit 73 sequentially plots the pixel addition value V _i for each column in the vertical direction of the difference image in the horizontal direction (in order of i = 1 to N), thereby obtaining the column pixel value. An addition curve (a column pixel value addition curve 221Y in FIG. 11D described later) is obtained. In addition, the face area data extraction unit 73 sequentially plots the pixel addition value W _j for each column in the horizontal direction of the difference image in the vertical direction (in the order of j = 1 to M), thereby obtaining a row pixel value addition curve ( A later-described row pixel value addition curve 221X) of FIG. 11D is obtained.

Then, the face area data extracting unit 73, the above preliminary TH _Y supplied from the threshold setting unit 72, the horizontal range of the row pixel value total curve is equal to or greater than the threshold value TH _X, and the column pixel value total curve However, the range in the vertical direction that is equal to or greater than the threshold value TH _Y is specified as the face area of the user 11.

  Then, the face area data extraction unit 73 supplies the face direction information for specifying the face area of the user 11 together with the difference image supplied from the difference image calculation unit 21 (FIG. 1) to the face orientation determination unit 23 (FIG. 1). To do.

  FIG. 8 is a block diagram illustrating a detailed configuration example of the face orientation determination unit 23 of FIG.

  The face orientation determination unit 23 includes a pixel value addition unit 81, a center position detection unit 82, and a determination unit 83.

  The difference image and face area information from the face area extracting unit 22 (FIG. 7) are supplied to the pixel value adding unit 81 and the determining unit 83.

The pixel value adding unit 81 specifies the pixel addition value V ′ _{i ′} for each column in the vertical direction for the face image in the difference image specified by the face area information (hereinafter, appropriately referred to as the pixel addition value V ′ _{i for the} face image _{). Is} calculated) and supplied to the center position detector 82. Here, the subscript variable i ′ represents a horizontal position extracted as a face image of i = 1 to N.

The center position detection unit 82 obtains a column pixel value addition curve (for example, a column pixel value addition curve 240B in FIG. 13A described later) obtained by sequentially plotting the pixel addition values V ′ _{i ′} of the face image in the horizontal direction. The center position (center of gravity position) X ′ of the user 11 in the horizontal direction is determined and supplied to the determination unit 83.

  The determination unit 83 obtains the center position X ′ of the face of the user 11 in the horizontal direction obtained by the center position detection unit 82 within a predetermined distance ± XR from the horizontal midpoint Xp of the face image, or The orientation of the face of the user 11 is determined depending on whether it is located on the left side or the right side of the range.

  That is, the determination unit 83 has a range on the left side (smaller side) than the position (Xp−XR) in which the horizontal center position X ′ of the face of the user 11 is smaller than the horizontal center point Xp of the face image by the distance XR. If it is located in (second range), it is determined that the orientation of the face of the user 11 is leftward (a state in which the user 11 is facing leftward when the captured image is viewed from the front).

  The determination unit 83 also determines that the horizontal center position X ′ of the face of the user 11 is smaller by a distance XR than the horizontal midpoint Xp of the face image (Xp−XR) and larger than the midpoint Xp by a distance XR. When it is located in the range (first range) between the position (Xp + XR), the face direction of the user 11 is determined to be the front.

  Further, the determination unit 83 determines a range (first side) on the right side (larger side) than the position (Xp + XR) where the center position X ′ in the horizontal direction of the face of the user 11 is greater than the horizontal center point Xp of the face image by the distance XR. 3), it is determined that the face direction of the user 11 is rightward (the user 11 is facing the right side when the captured image is viewed from the front).

  Then, the determination unit 83 supplies the determination result of the orientation of the face of the user 11 together with the difference image (supplied from the face region extraction unit 22) and the face region information to the face organ extraction unit 24 (FIG. 1).

  FIG. 9 is a block diagram illustrating a detailed configuration example of the facial organ extraction unit 24 of FIG.

  The face organ extraction unit 24 includes a pixel value addition unit 101, a memory 102, an eye / nose / mouth image separation unit 103, an eye image processing unit 104, a nose image processing unit 105, a mouth image processing unit 106, a memory 107, and an image output. The unit 108 is configured.

  The eye image processing unit 104 includes a pixel value addition unit 111, an eye region extraction unit 112, and a representative point determination unit 113. The nose image processing unit 105 includes a pixel value addition unit 121, a nose region extraction unit 122, The mouth image processing unit 106 includes a pixel value adding unit 131, a mouth region extracting unit 132, and a representative point determining unit 133.

  The facial organ extraction unit 24 is supplied with a difference image, face area information, and a determination result from the face orientation determination unit 23 (FIG. 8). The difference image and the face area information among the difference image, the face area information, and the determination result are supplied to the pixel value adding unit 101 and the memory 102. Further, the face area information and the determination result are supplied to the image output unit 108.

For the face image in the difference image specified by the face area information, the pixel value adding unit 101 applies the pixel addition value W ′ _{j ′} for each column in the horizontal direction (hereinafter, appropriately referred to as pixel addition value W ′ _{j ′ for the} face image). Is calculated and supplied to the eye / nose / mouth image separation unit 103. Here, the subscript variable j ′ represents the position in the vertical direction extracted as a face image from j = 1 to M.

  The memory 102 stores the difference image and face area information, and supplies them to the eye / nose / mouth image separation unit 103 as necessary.

The eye / nose / mouth image separation unit 103 (hereinafter simply referred to as the image separation unit 103) is stored in the memory 102 using the pixel addition value W ′ _{j ′} of the face image from the pixel value addition unit 101. The face image specified by the face area information for the difference image is an image including an eye area (hereinafter referred to as an eye image), an image including a nose area (hereinafter referred to as a nose image), and a mouth area. Are divided into three (vertical direction) images (hereinafter referred to as mouth images). Then, the image separation unit 103 supplies the separated eye image, nose image, and mouth image to the eye image processing unit 104, the nose image processing unit 105, and the mouth image processing unit 106, respectively.

  Based on the eye image supplied from the image separation unit 103, the eye image processing unit 104, as shown in FIG. 3, the region IL of the left eye of the user 11, the representative point ILp representing the region IL, and the right eye Region IR and a representative point IRp representing the region IR are identified.

Specifically, for the eye image, the pixel value addition unit 111 performs the pixel addition value V ′ _{ip ′} for each vertical column and the pixel addition value W ′ _{jp ′} for each column in the horizontal direction (hereinafter, appropriately, Pixel addition values V ′ _{ip ′} and W ′ _{jp ′} ) are calculated and supplied to the eye region extraction unit 112. Here, the subscript variable ip ′ represents the horizontal position extracted as the eye image of i = 1 to N, and the variable jp ′ is extracted as the eye image of j = 1 to M. Represents the vertical position.

The eye region extraction unit 112 generates the column pixel value addition curve obtained by sequentially plotting the pixel addition value V ′ _{ip ′} of the eye image in the horizontal direction and the pixel addition value _{W′jp ′} of the eye image in the vertical direction. The eye region of the user 11 is specified (extracted) using a row pixel value addition curve (hereinafter simply referred to as a column pixel value addition curve and a row pixel value addition curve of the eye image) obtained by sequentially plotting. . Then, the eye area extraction unit 112 supplies information specifying the eye area of the user 11 to the representative point determination unit 113 together with the eye image.

  The representative point determination unit 113 specifies the eye region in the eye image from the information for specifying the eye region of the user 11, and the column pixel value addition curve and the row pixel value addition curve for the specified eye region. Is recalculated. Then, the representative point determination unit 113 specifies (determines) a representative point representing the eye position of the user 11 using the column pixel value addition curve and the row pixel value addition curve recalculated for the eye region. To do. The representative point of the eye does not necessarily represent the center of the eye (iris) or the position of the black eye, but is almost the same position for each user.

  The representative point determination unit 113 then supplies information specifying the eye region supplied from the eye region extraction unit 112 and information specifying the eye representative point (collectively referred to as eye region information) to the image output unit 108. Supply.

  The eye image processing unit 104 performs the above-described processing for each eye.

  As shown in FIG. 3, the nose image processing unit 105 identifies the nose region NR of the user 11 and the representative point NRp representing the region NR based on the nose image supplied from the image separation unit 103. .

Specifically, for the nose image, the pixel value addition unit 121 sets the pixel addition value V ′ _{iq ′} for each column in the vertical direction and the pixel addition value W ′ _{jq ′} for each column in the horizontal direction (hereinafter, the nose image as appropriate). Pixel addition values V ′ _{iq ′} and W ′ _{jq ′} ) are calculated and supplied to the nose region extraction unit 122. Here, the subscript variable iq ′ represents the horizontal position extracted as a nose image of i = 1 to N, and the variable jq ′ is extracted as a nose image of j = 1 to M. Represents the vertical position.

The nose region extraction unit 122 generates the column pixel value addition curve obtained by sequentially plotting the pixel addition value V ′ _{iq ′} of the nose image in the horizontal direction and the pixel addition value W ′ _{jq ′} of the nose image in the vertical direction. A nose region of the user 11 is specified (extracted) using a row pixel value addition curve (hereinafter simply referred to as a column pixel value addition curve and a row pixel value addition curve of a nose image) obtained by sequentially plotting. . Then, the nose region extraction unit 122 supplies information specifying the nose region of the user 11 to the representative point determination unit 123 together with the nose image.

  The representative point determination unit 123 specifies the nose region in the nose image from the information for specifying the nose region of the user 11, and the column pixel value addition curve and the row pixel value addition curve for the specified nose region. Is recalculated. Then, the representative point determination unit 123 specifies (determines) a representative point representing the nose position of the user 11 using the column pixel value addition curve and the row pixel value addition curve recalculated for the nose region. To do. The representative point of the nose does not necessarily represent the center position of the nose, but is almost the same position for each user.

  Then, the representative point determination unit 123 sends the information specifying the nose region supplied from the nose region extraction unit 122 and the information specifying the representative point of the nose (also referred to as nose region information) to the image output unit 108. Supply.

  Based on the mouth image supplied from the image separation unit 103, the mouth image processing unit 106 identifies the mouth region MR of the user 11 and the representative point MRp representing the region MR, as shown in FIG. .

Specifically, for the mouth image, the pixel value addition unit 131 sets the pixel addition value V ′ _{ir ′} for each column in the vertical direction and the pixel addition value W ′ _{jr ′} for each column in the horizontal direction. Pixel addition values _{V′ir ′} and _{W′jr ′} ) are calculated and supplied to the mouth region extraction unit 132. Here, the subscript variable jr ′ represents each horizontal position extracted as a mouth image of i = 1 to N, and the variable jr ′ is extracted as a mouth image of j = 1 to M. Represents the vertical position.

The mouth area extraction unit 132 sequentially _{applies the} column pixel value addition curve obtained by sequentially plotting the pixel addition value V ′ _{i3 ′} of the mouth image in the horizontal direction and the pixel addition value W _{j3 ′} of the mouth image in the vertical direction. The region of the mouth of the user 11 is specified (extracted) using a row pixel value addition curve (hereinafter simply referred to as a column pixel value addition curve and a row pixel value addition curve of the mouth image) obtained by plotting. Then, the mouth area extraction unit 132 supplies information specifying the mouth area of the user 11 to the representative point determination unit 133 together with the mouth image.

  The representative point determination unit 133 specifies the mouth area in the mouth image from the information for specifying the mouth area of the user 11, and the column pixel value addition curve and the row pixel value addition curve for the specified mouth area. Is recalculated. Then, the representative point determination unit 133 specifies (determines) a representative point representing the position of the mouth of the user 11 using the column pixel value addition curve and the row pixel value addition curve recalculated for the mouth area. To do. The representative point of the mouth does not necessarily represent the center position of the mouth, but is almost the same position for each user.

  Then, the representative point determination unit 133 sends information specifying the mouth region supplied from the mouth region extraction unit 132 and information specifying the mouth representative point (also referred to as mouth region information) to the image output unit 108. Supply.

  The memory 107 stores the captured image (reference image or illumination image) supplied from the imaging device 12 (FIG. 1) and supplies it to the image output unit 108 as necessary.

  As described above, the face area information and the determination result are supplied to the image output unit 108 from the face orientation determination unit 23 (FIG. 8). The image output unit 108 is also supplied with eye region information, nose region information, and mouth region information from each of the eye image processing unit 104, the nose image processing unit 105, and the mouth image processing unit 106.

  The image output unit 108 determines whether to output the captured image stored in the memory 107 according to the determination result supplied from the face orientation determination unit 23. That is, when the determination result indicates rightward or leftward, the image output unit 108 determines that the captured image stored in the memory 107 is an image that is inappropriate for the face image recognition processing, and that (captured image) ) Is not output. On the other hand, if the determination result represents the front, the image output unit 109 determines that the captured image stored in the memory 107 is an image suitable for the face image recognition process, and uses the image as the face area information, Output together with eye area information, nose area information, and mouth area information.

  Therefore, in the output destination device to which the image processing device 15 (the image output unit 108) supplies (outputs) the captured image, only an image suitable for the recognition processing of the face image can be obtained. Since the accurate face area, eye area nose area, and mouth area in the captured image can be specified by the eye area information, the nose area information, and the mouth area information, the face image can be accurately recognized. it can.

  Hereinafter, the details of the processing performed by the image processing device 15 will be further described with reference to the drawings.

  FIG. 10 is a diagram for explaining the irradiation direction of light to the user 11 by the lighting device 13.

  As illustrated in FIG. 10, the illumination device 13 is arranged so that, for example, strong light is emitted only to the face of the user 11 from a direction obliquely downward to the front of the user 11. Accordingly, the background other than the face of the user 11 has a constant brightness regardless of whether the illumination device 13 is irradiated with light or not.

  It is assumed that the image 201A and the image 201B of FIG. 11 are sequentially supplied (input) to the image processing device 15 as images captured by the imaging device 12.

  An image 201A in FIG. 11A shows a reference image captured in a state where the lighting device 13 is not irradiating the user 11. Moreover, the image 201B of FIG. 11B has shown the illumination image imaged in the state in which the illuminating device 13 arrange | positioned as shown in FIG.

  Note that the pixel values of each pixel in the image 201A in FIG. 11A and the image 201B in FIG. 11B are actually various pixel values due to the difference in the light reflected by the subject. Since it is difficult to represent the difference in pixel value, it is simplified.

  In the image 201A, the user 11 is illuminated by only natural light (ambient light) without being illuminated by the lighting device 13, and is slightly dark.

  On the other hand, in the image 201B, the light from the lighting device 13 is reflected by the lighting device 13 on the face of the user 11, so that the light from the lighting device 13 is reflected by the face of the user 11, and the face of the user 11 is It is very bright (white on the drawing). Of the face of the user 11, the facial organ indicated by the dotted line represents the brightest (white) part due to the reflection of light from the illumination device 13.

  Assuming that an image pickup device such as a normal CCD image pickup device is picked up under the same illumination conditions, the image corresponding to the image 201B saturates the pixel value of the area of the face of the user 11, and the light from the lighting device 13 is irradiated. The image 201 </ b> A captured in a state where the user 11 is not performed and the image 201 </ b> B captured in a state where the user 11 is irradiated with strong light cannot be directly compared (under the same conditions). On the other hand, in the imaging device 12 having the logarithmic conversion type imaging device, as described above, luminance clipping does not occur in the luminance range that can be visually recognized by humans. Therefore, as shown in FIG. The detailed luminance distribution of the user 11 can be faithfully represented even in the image 201B that is captured in a state where only intense light is irradiated. That is, from the image 201A imaged in a state in which the illumination device 13 is not irradiated with light without adjusting the amount of incident light, and the image 201B imaged in a state in which strong light is irradiated to the user 11 A difference image can be obtained.

  Therefore, the difference image calculation unit 21 determines the difference between the pixel values of the corresponding pixels between the image 201A (hereinafter referred to as the reference image 201A) and the image 201B (hereinafter referred to as the illumination image 201B). By calculating (value), a difference image 201C shown in FIG. 11C is generated.

  In the illumination image 201B, as described above, the eyebrows, the eye contour, the base of the nose (the periphery of the nostril), and the convex part of the lips, which are indicated by dotted lines, are remarkably bright. Since it is an image (with a large pixel value), the difference image 201C is similarly shown by a dotted line in the eyebrow, eye contour, base of the nose (periphery of the nostril) of the user 11, And the convex part of a lip becomes a remarkably bright image (a pixel value is large).

  Further, the background area of the reference image 201A and the illumination image 201B and the hair area of the user 11 do not change in brightness (pixel value) depending on whether or not the illumination device 13 emits light. The pixel value is close to 0. In addition, in the difference image 201C in FIG. 11C, pixels (regions) whose pixel values are close to 0 are shown in gray.

  The difference image calculation unit 21 supplies the generated difference image 201C to the face area extraction unit 22.

The pixel value addition unit 71 of the face area extraction unit 22 plots the pixel addition values V _i for each column in the vertical direction of the difference image 201C sequentially in the horizontal direction (in the order of i = 1 to N), thereby FIG. The column pixel value addition curve 221Y shown in FIG. In addition, the pixel value addition unit 71 sequentially plots the pixel addition values W _j for each column in the horizontal direction of the difference image 201C in the vertical direction (in the order of j = 1 to M), thereby performing the row pixels illustrated in FIG. 11D. A value addition curve 221X is obtained.

In FIG. 11D, the scale for the column pixel value addition curve 221Y uses the lower horizontal axis and the left vertical axis, and the lower horizontal axis represents the position in the horizontal direction (X direction), and the left vertical axis. The axis represents the pixel addition value V _i when pixel values are added for each column in the vertical direction (Y direction). The scale for the row pixel value addition curve 221X uses the upper horizontal axis and the right vertical axis, the right vertical axis represents the position in the vertical direction (Y direction), and the upper horizontal axis represents the horizontal direction ( This represents a pixel addition value W _j when the pixel value is added for each column in the (X direction). In FIG. 11D, N = 256 and M = 128.

When the difference image 201C shown in FIG. 11C is compared with the row pixel value addition curve 221X and the column pixel value addition curve 221Y shown in FIG. 11D, a range Xb in which the face area exists in the horizontal direction (X direction) of the difference image 201C. In addition, the column pixel value addition curve 221Y (pixel addition value V _i ) becomes large, and in the vertical direction (Y direction) of the difference image 201C, the row pixel value addition curve 221X (pixel addition value) falls within a range Yb where the face area exists. W _j ) is increasing.

Therefore, the threshold value setting unit 72 of the face area extracting unit 22 calculates the threshold value TH _X for distinguishing the face area of the user 11 from other areas in the horizontal direction of the difference image by the above-described equations (3) and (4). To do. Further, the threshold setting unit 72 calculates a threshold TH _Y for distinguishing the face area of the user 11 from other areas in the vertical direction of the difference image. Then, the calculated threshold value TH _X or TH _Y is supplied to the face area data extraction unit 73. Note that the threshold TH _X or TH _Y may be obtained by experimental data (statistical data) in addition to the method of obtaining by the above formulas (3) and (4).

The face area data extraction unit 73 specifies the position of the face area of the user 11 in the difference image based on the threshold values TH _X and TH _Y supplied from the threshold setting unit 72.

That is, for the horizontal direction of the difference image 201C, facial area data extracting unit 73, as shown in FIG. 12A, calculates the position X _ma and X _mb row pixel value total curve 221Y intersects with the threshold TH _X.

As for the vertical direction of the difference image 201C, the face area data extracting unit 73 calculates the position Y _ma and Y _mb column pixel value total curve 221X intersects with the threshold TH _Y.

Then, the face area data extraction unit 73 includes a range of X _ma ≦ i ≦ X _{mb in which} the column pixel value addition curve 221Y is greater than or equal to the threshold value TH _X and a row pixel value addition curve 221X, which are indicated by diagonal lines in FIG. 12B. A region 231 surrounded by a range of Y _ma ≦ j ≦ Y _mb in which is _{equal to} or greater than the threshold TH _Y is specified as the face region of the user 11.

The face area data extraction unit 73 specifies the area X _ma and X _mb in the horizontal direction and the positions Y _ma and Y _{mb in} the vertical direction that specify the face area of the user 11 as face area information together with the difference image 201C. This is supplied to the face orientation determination unit 23.

The pixel value addition unit 81 of the face orientation determination unit 23 calculates a pixel addition value V ′ _{i ′} for each column in the vertical direction for the face image in the difference image specified by the face area information. Then, the center position detection unit 82 uses the column pixel value addition curve obtained by sequentially plotting the pixel addition values V ′ _{i ′} of the face image in the horizontal direction, so that the center position of the face of the user 11 in the horizontal direction ( The center of gravity position) X ′ is determined and supplied to the determination unit 83.

  FIG. 13 shows the relationship between the horizontal center position (center of gravity position) X ′ of the face of the user 11 and the direction of the face of the user 11 (front, right, or left).

  In FIG. 13, a markedly bright (large pixel value) portion shown by a dotted line in FIG. 11C is shown in gray (the same applies to FIG. 15 described later).

It is assumed that a column pixel value addition curve 240A shown on the upper side of FIG. 13A is obtained by sequentially plotting the pixel addition values V ′ _{i ′} of the face image calculated by the pixel value addition unit 81 in the horizontal direction.

  The center position detection unit 82 sets the position where the maximum value 241A of the column pixel value addition curve 240A is the center position (center of gravity position) X ′ in the horizontal direction of the face of the user 11. In the difference image, as shown in the lower side of FIG. 13A, the pixel value (the portion shown in gray) corresponding to the eyebrow, eye, nose, and mouth regions of the face image has a large (brighter) pixel value. Among these regions, the region (area) where the nose and mouth are located is wider than the region corresponding to the eyebrows and eyes, so if the column pixel value addition curve is at the maximum value, This is because it can be considered.

  Accordingly, the determination unit 83 determines that the center position X ′ regarded as the center of the face of the user 11 (the position of the nose and mouth) is within a predetermined distance ± XR from the horizontal center point Xp of the face image. The orientation of the face of the user 11 is determined depending on whether it is on the left or right side of the user 11.

  In FIG. 13A, since the center position X ′ of the face of the user 11 is within a range (second range) of a predetermined distance ± XR from the horizontal midpoint Xp of the face image, the determination unit 83 The direction of the face of the user 11 is determined to be the front.

  When the column pixel value addition curve 240B illustrated in FIG. 13B is obtained, the center position detection unit 82 determines the position of the maximum value 241B of the column pixel value addition curve 240B in the horizontal direction of the face of the user 11. The center position is X ′.

  Then, the determination unit 83 has a range (third side) on the right side (larger side) than the position (Xp + XR) where the center position X ′ of the face of the user 11 is larger by the predetermined distance XR than the midpoint Xp in the horizontal direction of the face image. Therefore, it is determined that the direction of the face of the user 11 is the right direction.

  Furthermore, when the column pixel value addition curve 240C illustrated in FIG. 13C is obtained, the center position detection unit 82 determines the position where the maximum value 241C of the column pixel value addition curve 240C is in the horizontal direction of the face of the user 11. The center position is X ′.

  Then, the determination unit 83 has a range (on the smaller side) to the left (smaller side) than the position (Xp−XR) where the center position X ′ of the face of the user 11 is smaller than the horizontal center point Xp of the face image by a predetermined distance XR. Therefore, it is determined that the direction of the face of the user 11 is the left direction.

  The determination unit 83 supplies the determination result of the face orientation of the user 11 determined as described above to the face organ extraction unit 24 together with the difference image (supplied from the face region extraction unit 22) and the face region information. .

The pixel value addition unit 101 of the facial organ extraction unit 24 performs pixel addition value W ′ _{j ′} (pixel addition value W ′ of the face image) for each horizontal column for the face image in the difference image specified by the face area information. _{j ′} ) is calculated and supplied to the image separation unit 103.

The image separation unit 103 uses the pixel addition value W ′ _{j ′} of the face image supplied from the pixel value addition unit 101 to convert the face image into an image including an eye region (eye image) and an image including a nose region. (Nose image) and image including mouth area (mouth image) are separated.

Specifically, the image separation unit 103 obtains the row pixel value addition curve 260 shown in FIG. 14 by sequentially plotting the pixel addition values W ′ _{j ′} of the face image in the vertical direction.

  Then, the image separation unit 103 detects a minimum point generated in the row pixel value addition curve 260. In the example illustrated in FIG. 14, the image separation unit 103 detects local minimum points 261A to 261E from the row pixel value addition curve 260.

Next, the image separation unit 103 moves the eye image and the nose image up and down (vertical direction) at the position j where the pixel addition value W ′ _{j ′} of the face image is minimum among the detected minimum points 261A to 261E. B) Detect as a boundary (position) to be separated.

  Further, the image separation unit 103 sets the position j of the minimum point detected next downward from the minimum point detected as the boundary separating the eye image and the nose image vertically, and the nose image and the mouth image vertically ( Detect as a boundary (position) to separate (in the vertical direction).

In the example illustrated in FIG. 14, the position Y _g having the minimum point 261C where the pixel addition value W ′ _{j ′} of the face image is the smallest among the detected minimum points 261A to 261E is the eye image and the nose image. Is detected as a boundary (position) separating the top and bottom.

Further, the position Y _h of the minimum point 261D detected next downward from the minimum point 261C detected as the boundary that separates the eye image and the nose image vertically is the boundary that separates the nose image and the mouth image vertically ( Position).

  Thereby, as shown in FIG. 15, the image separation unit 103 converts a face area (face image) 231 in the difference image into an image (eye image) 271 including an eye area, and an image including a nose area ( The image is separated into a nose image 272 and an image (mouth image) 273 including a mouth region. Then, the image separation unit 203 supplies the eye image 271 to the eye image processing unit 104, supplies the nose image 272 to the nose image processing unit 105, and supplies the mouth image 273 to the mouth image processing unit 106.

  Note that in the image separation unit 103, when detecting the minimum point 261C as a boundary for separating the eye image and the nose image vertically, the boundary for separating the eye image and the nose image vertically is a face region (face image). Since it is assumed that the pixel exists in the vicinity of the center in the vertical direction (vertical direction), the row pixel only in the vicinity of the center in the vertical direction of the face region (for example, a range that is half the vertical length). The minimum point that is the minimum value of the value addition curve 260 may be detected. Alternatively, the local minimum point closest to the center in the vertical direction of the face region may be detected.

Further, when detecting a minimum point, in order not to detect a point different from the minimum point to be originally detected, such as a minimum point caused by a minute change in pixel value, the row pixel value addition curve 260 is detected. The above-described processing can be executed after performing a filter (low-pass filter) processing for removing high-frequency components. Furthermore, a restriction condition such that the difference in the pixel addition value W ′ _{j ′} of the face image from the maximum point located before and after (up and down) the detected minimum point is a certain value or more is added, which is inappropriate. Local minimum points (due to noise) may be excluded.

Next, the pixel value addition unit 111 of the eye image processing unit 104 performs the pixel addition value V ′ _{ip ′} or W ′ _{jp ′} for each column in the vertical or horizontal direction with respect to the eye image supplied from the image separation unit 103. Is calculated and supplied to the eye region extraction unit 112. The eye area extraction unit 112 sequentially plots the pixel addition values V ′ _{ip ′} or W ′ _{jp ′} of the eye image in the horizontal direction or the vertical direction, thereby respectively, the column pixel value addition curve 271Y of the eye image shown in FIG. 16A. Alternatively, a row pixel value addition curve 271X is obtained.

Further, the pixel value addition unit 121 of the nose image processing unit 105 calculates the pixel addition value V ′ _{iq ′} or W ′ _{jq ′} for each column in the vertical or horizontal direction with respect to the nose image supplied from the image separation unit 103. Calculate and supply to the nose region extraction unit 122. The nose region extraction unit 122 sequentially plots the pixel addition values V ′ _{iq ′} or W ′ _{jq ′} of the nose image in the horizontal direction or the vertical direction, so that the column pixel value addition curve 272Y of the nose image shown in FIG. 16B respectively. Alternatively, a row pixel value addition curve 272X is obtained.

Further, the pixel value adding unit 131 of the mouth image processing unit 106 subjects the supplied mouth image from the image separating unit 103, a vertical or a horizontal pixel total values V of each column of _'ir' or W _'jr' Calculate and supply to the mouth region extraction unit 132. The mouth area extraction unit 132 sequentially plots the pixel addition value V ′ _{i3 ′} or W _{j3 ′} of the mouth image in the horizontal direction or the vertical direction, thereby respectively, the column pixel value addition curve 273Y of the mouth image shown in FIG. A row pixel value addition curve 273X is obtained.

  Next, processing performed by the eye region extraction unit 112 and the representative point determination unit 113 of the eye image processing unit 104 will be described with reference to FIG.

  The eye region extraction unit 112 specifies the eye region of the user 11 using the column pixel value addition curve 271Y and the row pixel value addition curve 271X of the eye image.

  Specifically, the eye area extraction unit 112 is centered on the middle point (intermediate position) in the horizontal direction of the eye image, and the width is, for example, in a range Xe that is half the width Xd of the eye image (Xd / 2). The minimum value of the column pixel value addition curve 271Y is detected.

  In FIG. 17A, the minimum point 281 is detected as the minimum value in the range Xe. The horizontal position Xdp of the minimum point 281 is a boundary that separates the eye image into a left-eye image and a right-eye image each including a left eye and a right eye.

Next, the eye area extraction unit 112 pays attention to the column pixel value addition curve 271Y in the area (left eye image) on the left side of the position Xdp in the horizontal direction of the face image. The eye region extraction unit 112 detects a local maximum point for the column pixel value addition curve 271Y in the region on the left side of the position Xdp (left eye image), and from the detected local maximum points, as shown in FIG. Two local maximum points 291A and 291B are selected in descending order of value. The horizontal positions XIL ₁ and XIL ₂ corresponding to the two maximum points 291A and 291B are the horizontal positions that specify the left eye region. That is, in the horizontal direction, it is specified that the left eye exists in the range of XIL ₁ ≦ i ≦ XIL ₂ .

Similarly, the eye region extraction unit 112 detects a local maximum value for the column pixel value addition curve 271Y for the region on the right side of the position Xdp (right-eye image), and from among the detected local maximum points, FIG. As shown in 17A, the two maximum points 291C and 291D are selected in descending order. Then, the horizontal positions XIR ₁ and XIR ₂ corresponding to the two maximum points 291C and 291D are the horizontal positions that specify the region of the right eye. That is, in the horizontal direction, it is specified that the right eye exists in the range of XIR ₁ ≦ i ≦ XIR ₂ .

On the other hand, with respect to the vertical direction, the eye region extraction unit 112 detects a minimum point in the row pixel value addition curve 271X (pixel addition value W ′ _{jp ′ of the} eye image), and from among the detected minimum points. As shown in FIG. 17A, the minimum point 301 having the smallest value is selected.

  Then, the eye region extraction unit 112 detects a local maximum point located on both sides of the local minimum point 301 having the smallest value in the row pixel value addition curve 271X. That is, the eye region extraction unit 112 detects the first maximum points 311A and 311B by searching in the vertical direction from the minimum point 301 having the smallest value in the row pixel value addition curve 271X.

Then, the vertical positions YI ₁ and YI ₂ corresponding to the local maximum points 311A and 311B are positions for specifying the left eye area and the right eye area in the vertical direction. That is, in the vertical direction, it is specified that the left eye and the right eye are present in the range of YI ₁ ≦ j ≦ YI ₂ . Note that the position in the vertical direction corresponding to the minimum point 301 is the position Ydp.

From the above, the eye region extraction unit 112 determines the region surrounded by the range of XIL ₁ ≦ i ≦ XIL _{2 and} the range of YI ₁ ≦ j ≦ YI ₂ , which is indicated by hatching in FIG. Is identified. Further, the eye region extraction unit 112 identifies the region surrounded by the range of XIR ₁ ≦ i ≦ XIR _{2 and} the range of YI ₁ ≦ j ≦ YI ₂ as the right eye region IR. The eye area extraction unit 112 uses the horizontal positions XIL ₁ , XIL ₂ , XIR ₁ , and XIR ₂ and the vertical positions YI ₁ and YI ₂ as representative information for identifying the eye area of the user 11. This is supplied to the point determination unit 113.

  The representative point determination unit 113 specifies a representative point that represents the position of the eye within the eye area of the user 11.

  FIG. 17B shows the row pixel value addition curve 271X ′ and the column pixel value addition curve 271Y ′ recalculated in the region IL of the left eye. FIG. 17C shows the row pixel value addition curve 271X ″ and the column pixel value addition curve 271Y ″ recalculated within the region IR of the right eye.

In the left eye region IL shown in FIG. 17B, the representative point determination unit 113 detects the minimum value 321 of the column pixel value addition curve 271Y ′, and obtains the horizontal position IL _X corresponding to the minimum value 321. The representative point determining unit 113 detects the minimum value 322 of the row pixel value total curve 271X ', obtain the location IL _Y in the vertical direction corresponding to the minimum value 322. The positions IL _X and IL _Y represent a representative point ILp = (IL _X , IL _Y ) that represents the position of the left eye of the user 11.

Similarly, in the region IR of the right eye shown in FIG. 17C, the representative point determination unit 113 detects the minimum value 331 of the column pixel value addition curve 271Y ″ and determines the horizontal position IR _X corresponding to the minimum value 331. obtained. the representative point determining unit 113 detects the minimum value 332 of the row pixel value total curve 271X ", obtaining a position IR _Y in the vertical direction corresponding to the minimum value 332. The positions IR _X and IR _Y represent a representative point IRp = (IR _X , IR _Y ) representing the position of the right eye of the user 11.

The representative point determination unit 113 uses the position (IL _X , IL _Y ) and the position (IR _X , IR _Y ) as information for specifying the representative points ILp and IRp of the user 11 and specifies the above-described eye region. The information is supplied to the image output unit 108 together with the information.

  The representative point of the eye does not necessarily represent the center of the eye (iris) or the position of the black eye, but is almost the same position for each user. Therefore, by examining (comparing) the positional relationship between the left and right eyes as representative points, the accuracy of individual (person) authentication can be improved.

  In addition, by detecting the trajectories (moving directions) of the positions of the left and right eyes as representative points for a plurality of images captured continuously, the moving direction of the user's line of sight and the orientation of the face are detected. Is also possible.

  Next, processing performed by the nose region extraction unit 122 and the representative point determination unit 123 of the nose image processing unit 105 will be described with reference to FIG.

  The nose region extraction unit 122 specifies the nose region of the user 11 using the column pixel value addition curve 272Y and the row pixel value addition curve 272X of the nose image.

Specifically, the nose region extraction unit 122 is centered on the midpoint (intermediate position) in the horizontal direction of the nose image, and the width is within, for example, a range Xe that is half the width Xd of the nose image (Xd / 2). In FIG. 18A, the local minimum point of the column pixel value addition curve 272Y is detected, and two local minimum points 341 and 342 are selected in ascending order of value, as shown in FIG. 18A. The horizontal positions XN ₁ and XN ₂ corresponding to the two local minimum points 341 and 342 are the horizontal positions that specify the nose region. That is, in the horizontal direction, it is specified that the nose exists in the range of XN ₁ ≦ i ≦ XN ₂ .

  On the other hand, for the vertical direction, the nose region extraction unit 122 detects a local maximum point in the row pixel value addition curve 272X, and selects the local maximum point 351 having the largest value from the detected local maximum points. When only one local maximum point 351 exists as in the row pixel value addition curve 272X in FIG. 18A, the local maximum point 351 is selected.

The nose area extracting unit 122, from the position YN ₁ in the vertical direction corresponding to the maximum point 351, by a distance Y _ND same distance and to a lower limit position YN ₃ in the vertical direction of the nose image, upwardly from the position YN ₁ The position YN ₂ that has been advanced to is determined. The vertical positions YN ₂ and YN ₃ determined in this way are positions that specify the nose region in the vertical direction. That is, in the vertical direction, it is specified that the nose exists in the range of YN ₂ ≦ j ≦ YN ₃ .

From the above, the nose region extraction unit 122 determines the region surrounded by the range of XN ₁ ≦ i ≦ XN _{2 and} the range of YN ₂ ≦ j ≦ YN ₃ , which is indicated by hatching in FIG. 18A, as the nose region NR. Is identified. Also, the nose region extraction unit 122 uses the horizontal positions XN ₁ and XN ₂ and the vertical positions YN ₂ and YN ₃ as information for specifying the nose region of the user 11 to the representative point determination unit 123. Supply.

  The representative point determination unit 123 specifies a representative point representing the nose position in the nose region of the user 11.

  FIG. 18B shows the row pixel value addition curve 272X ′ and the column pixel value addition curve 272Y ′ recalculated in the nose region NR.

In the nose region NR shown in FIG. 18B, the representative point determination unit 123 detects the maximum value 361 of the column pixel value addition curve 272Y ′ and obtains the horizontal position NR _X corresponding to the maximum value 361. The representative point determining unit 123 detects the maximum value 371 of the row pixel value total curve 272X ', obtaining position NR _Y in the vertical direction corresponding to the minimum value 371. The positions NR _X and NR _Y represent a representative point NRp = (NR _X , NR _Y ) representing the position of the user 11 nose.

The representative point determination unit 123 uses the position (NR _X , NR _Y ) as information for specifying the representative point NRp of the nose of the user 11, and supplies the information to the image output unit 108 together with the information for specifying the nose region described above.

  Next, processing performed by the mouth area extraction unit 132 and the representative point determination unit 133 of the mouth image processing unit 106 will be described with reference to FIG.

  The mouth area extraction unit 132 specifies the mouth area of the user 11 using the column pixel value addition curve 273Y and the row pixel value addition curve 273X of the mouth image.

Specifically, the mouth area extraction unit 132 is centered on the middle point (intermediate position) in the horizontal direction of the mouth image, and the width is, for example, within the range Xe that is half the width Xd (Xd / 2) of the mouth image. , Three minimum points 381 to 383 of the column pixel value addition curve 273Y are detected. Then, the mouth area extraction unit 132 selects the minimum and maximum (ie, both sides) minimum points 381 and 383 in the horizontal direction from the detected minimum points 381 to 383. The horizontal positions XM ₁ and XM ₃ corresponding to the two local minimum points 381 and 383 are horizontal positions that specify the mouth area. That is, in the horizontal direction, it is specified that the mouth exists in the range of XM ₁ ≦ i ≦ XM ₃ .

  On the other hand, with respect to the vertical direction, the mouth area extraction unit 132 detects a minimum point in the row pixel value addition curve 273X, and selects the minimum point 384 having the smallest value from the detected minimum points. When only one local minimum point 384 exists like the row pixel value addition curve 273X in FIG. 19A, the local minimum point 384 is selected.

The mouth area extracting unit 132, from the position YM ₁ in the vertical direction corresponding to the minimum point 384, by the same distance and the distance Y _MD of up to the upper limit position YM ₂ in the vertical direction of the mouth image, downward from the position YM ₁ The position YM ₃ that has been advanced to is determined. The vertical positions YM ₂ and YM ₃ determined in this way are positions for specifying the vertical mouth area. That is, in the vertical direction, it is specified that the mouth exists in the range of YM ₂ ≦ j ≦ YM ₃ .

From the above, the mouth area extraction unit 132 converts the area surrounded by the range of XM ₁ ≦ i ≦ XM _{3 and} the range of YM ₂ ≦ j ≦ YM ₃ , which is indicated by hatching in FIG. Is identified. In addition, the mouth area extraction unit 132 uses the horizontal positions XM ₁ and XM ₃ and the vertical positions YM ₂ and YM ₃ as information for specifying the mouth area of the user 11 to the representative point determination unit 133. Supply.

  The representative point determination unit 133 specifies a representative point that represents the position of the mouth in the mouth area of the user 11.

  FIG. 19B shows the row pixel value addition curve 273X ′ and the column pixel value addition curve 273Y ′ recalculated in the mouth region MR.

In the region MR of the mouth shown in FIG. 19B, the representative point determining unit 133 detects the minimum value 391 of the row pixel value total curve 273Y ', to obtain the position MR _X in the horizontal direction corresponding to the minimum value 391. The representative point determining unit 133 detects the minimum value 392 of the row pixel value total curve 273X ', to obtain the position MR _Y in the vertical direction corresponding to the minimum value 392. The positions MR _X and MR _Y represent a representative point MRp = (MR _X , MR _Y ) that represents the position of the mouth of the user 11.

The representative point determination unit 133 uses the position (MR _X , MR _Y ) as information for specifying the representative point MRp of the mouth of the user 11, and supplies the information to the image output unit 108 together with the information for specifying the mouth region.

  Next, processing of the image processing apparatus 15 will be described with reference to the flowcharts of FIGS. This process is started when a captured image is supplied from the imaging device 12 to the image processing device 15.

  First, in step S1, the image acquisition unit 61 of the difference image calculation unit 21 acquires a reference image and an illumination image. More specifically, the image acquisition unit 61 acquires a captured image supplied from the imaging device 12, and according to a determination signal from the timing control device 14, whether the captured image supplied from the imaging device 12 is a reference image, Or it is determined whether it is an illumination image. Then, the image acquisition unit 61 supplies the captured image to the memory 62 when the supplied captured image is a reference image, and supplies the captured image to the memory 63 when the supplied captured image is an illumination image.

  In step S <b> 2, the difference processing unit 64 of the difference image calculating unit 21 calculates a difference image from the reference image stored in the memory 62 and the illumination image stored in the memory 63.

  In step S <b> 3, the filter processing unit 65 of the difference image calculation unit 21 filters (filters) the difference image from the difference processing unit 64 and supplies the processed difference image to the face area extraction unit 22.

In step S4, the pixel value addition unit 71 of the face area extraction unit 22 adds pixel values for each column in the horizontal or vertical direction of the difference image, and adds pixel values for each column in the vertical or horizontal direction of the difference image. V _i or W _j is calculated.

In step S5, the threshold value setting unit 72 of the face area extracting unit 22 calculates and sets a threshold value TH _X or TH _Y for distinguishing the face area of the user 11 from other areas in the horizontal direction or the vertical direction of the difference image. .

In step S6, the face area data extracting unit 73 of the face area extracting unit 22 extracts a face image from the difference image. That is, the face area data extraction unit 73 is based on the pixel addition values V _i and W _{j of the} difference image supplied from the pixel value addition unit 71 and the threshold values TH _X and TH _Y supplied from the threshold setting unit 72. The position of the face area of the user 11 in the difference image is specified. Then, the face area data extraction unit 73 supplies face area information for specifying the face area of the user 11 to the face orientation determination unit 23 together with the difference image supplied from the difference image calculation unit 21.

In step S <b> 7, the pixel value addition unit 81 of the face orientation determination unit 23 calculates the pixel addition value V ′ _{i ′} of the face image and supplies it to the center position detection unit 82.

  In step S <b> 8, the center position detecting unit 82 of the face orientation determining unit 23 determines the center position (center of gravity position) X ′ of the user 11 in the horizontal direction and supplies the determined position to the determining unit 83.

  In step S9, the determination unit 83 of the face orientation determination unit 23 determines that the center position X ′ of the face of the user 11 in the horizontal direction is within a predetermined distance ± XR from the horizontal midpoint Xp of the face image. The direction of the face of the user 11 is determined depending on whether it exists on the left side or the right side. Then, the determination result is supplied to the face organ extraction unit 24 together with the difference image and the face area information.

  In step S <b> 10, the memory 107 of the facial organ extraction unit 24 stores the captured image (reference image or illumination image) supplied from the imaging device 12.

In step S _< b> 11, the pixel value addition unit 101 of the face organ extraction unit 24 calculates the pixel addition value W ′ _{j ′} of the face image and supplies it to the image separation unit 103.

In step S12, the image separation unit 103 of the facial organ extraction unit 24 uses the pixel image addition value W ′ _{j ′} of the face image from the pixel value addition unit 101 to _convert the face image into an eye image, a nose image, and a mouth image. To separate. Then, the image separation unit 103 supplies the separated eye image, nose image, and mouth image to the eye image processing unit 104, the nose image processing unit 105, and the mouth image processing unit 106, respectively.

In step S <b> 13, the pixel value addition unit 111 of the eye image processing unit 104 calculates a pixel addition value V ′ _{ip ′} or W ′ _{jp ′} of the eye image and supplies it to the eye region extraction unit 112.

  In step S <b> 14, the eye region extraction unit 112 of the eye image processing unit 104 specifies the eye region of the user 11 using the column pixel value addition curve and the row pixel value addition curve of the eye image. Then, the eye area extraction unit 112 supplies information specifying the eye area of the user 11 to the representative point determination unit 113 together with the eye image.

  In step S15, the representative point determination unit 113 of the eye image processing unit 104 represents the eye position of the user 11 using the column pixel value addition curve and the row pixel value addition curve recalculated for the eye region. The representative point to be identified is specified. Then, the representative point determination unit 113 supplies eye region information (information specifying the eye region and information specifying the eye representative point) to the image output unit 108.

In step _{S <b>} 16, the pixel value addition unit 121 of the nose region processing unit 105 calculates the pixel addition value V ′ _{iq ′} or W ′ _{jq ′} of the nose image and supplies it to the nose region extraction unit 122.

  In step S <b> 17, the nose region extraction unit 122 of the nose region processing unit 105 specifies the nose region of the user 11 using the column pixel value addition curve and the row pixel value addition curve of the nose image. Then, the nose region extraction unit 122 supplies information specifying the nose region of the user 11 to the representative point determination unit 123 together with the nose image.

  In step S <b> 18, the representative point determination unit 123 of the nose region processing unit 105 represents the position of the nose of the user 11 using the column pixel value addition curve and the row pixel value addition curve recalculated for the nose region. Identify (determine) the representative points to be used. Then, the representative point determination unit 123 supplies the nose region information (information specifying the nose region and information specifying the nose representative point) to the image output unit 108.

In step S19, the pixel value adding unit 131 of the mouth image processing unit 106 calculates the pixel total values of the mouth image V _'ir' and W _'jr', and supplies the mouth area extracting unit 132.

  In step S20, the mouth area extraction unit 132 of the mouth image processing unit 106 specifies (extracts) the mouth area of the user 11 using the column pixel value addition curve and the row pixel value addition curve of the mouth image. Then, the mouth area extraction unit 132 supplies information specifying the mouth area of the user 11 to the representative point determination unit 133 together with the column pixel value addition curve and the row pixel value addition curve of the mouth image.

  In step S21, the representative point determination unit 133 of the mouth image processing unit 106 represents the position of the mouth of the user 11 using the column pixel value addition curve and the row pixel value addition curve recalculated for the mouth region. Identify (determine) the representative points to be used. Then, the representative point determination unit 133 supplies the mouth area information (information specifying the mouth region and information specifying the mouth representative point) to the image output unit 108.

  In step S <b> 22, the image output unit 108 of the mouth image processing unit 106 determines whether to output the captured image stored in the memory 107 in step S <b> 10 described above according to the determination result supplied from the face orientation determination unit 23. Determine. That is, when it is determined from the determination result that the orientation of the face of the user 11 is rightward or leftward, the image output unit 108 skips step S23 (does not output the captured image stored in the memory 107). To finish the process.

  On the other hand, if it is determined from the determination result that the orientation of the face of the user 11 is front, the process proceeds to step S23, and the image output unit 108 together with the face area information, eye area information, nose area information, and mouth area information. Then, the captured image stored in the memory 107 is output, and the process ends.

  As described above, the image processing device 15 outputs only the captured image that is optimal for the face image recognition process, and outputs the face region, the eye region, the nose region, the mouth region, and the eye of the output captured image. Output representative points of nose and mouth. As a result, the device that receives the output from the image processing device 15 can easily and accurately perform the facial image recognition processing.

  It should be noted that any of the processes in steps S13 to S15, steps S16 to S18, or steps S19 to S21 described above may be performed first, or may be performed simultaneously (in parallel). is there.

  Further, in the processing of step S23 described above, the image output unit 108 determines whether the captured image stored in the memory 107 is the reference image or not when the face direction of the user 11 is determined to be the front. Regardless of whether the captured image stored in the memory 107 is output or not, it may be output only for the reference image.

  FIG. 22 is a block diagram showing another embodiment of the face area extraction unit 22 of FIG.

  In FIG. 22, the face area extracting unit 22 includes a histogram creating unit 401, a threshold setting unit 402, a face area data extracting unit 403, and a memory 404.

  The difference image (after the filter processing) supplied from the difference image calculation unit 21 (the filter processing unit 65) is supplied to the histogram creation unit 401 and the face area data extraction unit 403.

  The histogram creation unit 401 creates a histogram of the pixel values of the difference image by summing up the number of pixels for each pixel having the same pixel value for the difference image, and supplies the histogram to the threshold setting unit 402. Further, the histogram creation unit 401 generates a table in which pixel positions and pixel values are associated with each pixel of the difference image, and supplies the table to the memory 404. By referring to this table, when a predetermined pixel value is specified, the position of the pixel having the pixel value in the difference image can be specified.

The threshold setting unit 402 determines (sets) a pixel value that distinguishes the face area of the user 11 from other areas as a threshold Q ₂ (FIG. 23) based on the pixel value histogram of the difference image from the histogram creation unit 401. And is supplied to the face area data extraction unit 403.

The face area data extraction unit 403 specifies the face area of the user 11 in the difference image based on the threshold value Q ₂ from the threshold value setting unit 402 and the table stored in the memory 404. Then, the face area data extraction unit 403 supplies the face direction information for specifying the face area of the user 11 together with the difference image supplied from the difference image calculation unit 21 (FIG. 1) to the face orientation determination unit 23 (FIG. 1). To do.

  With reference to FIG. 23 and FIG. 24, the process which the face area extraction part 22 of FIG. 22 performs is demonstrated.

  FIG. 23 shows a histogram of pixel values of the difference image created by the histogram creation unit 401.

  In FIG. 23, the horizontal axis represents the pixel value, and the vertical axis represents the number of pixels (frequency).

First, the threshold setting unit 402 detects the maximum value of the histogram of the pixel values of the difference image. In FIG. 23, a maximum value 411 is detected at the position of the pixel value Q ₁ . When a plurality of maximum values are detected, the threshold setting unit 402 employs the maximum value (maximum maximum value) having the largest number of pixels (frequency) among the detected plurality of maximum values. To do.

  In the difference image, as shown in FIG. 11C, the pixel value of the background area (area other than the face area) is 0 or a value close to 0, and the pixel value of the face area is a value away from 0. As shown in FIG. 23, the pixel value histogram has a distribution in which the pixel value is concentrated in the vicinity of 0 and the other two pixel values (the number of pixels increases).

  Therefore, when detecting the maximum value of the histogram of the pixel values of the difference image, the maximum value near the pixel value of 0 is clearly the background region, and therefore the detected maximum value is equal to or greater than the predetermined pixel value near 0. It is also possible to detect the maximum value by providing a constraint condition that there is a certain number or a constraint condition that the number of pixels (frequency) is not less than a predetermined value.

The threshold value setting unit 402 is an intermediate pixel value between the pixel value Q ₁ having the detected maximum value 411 and the pixel value 0, that is, a pixel value Q _{2 that is} half of the pixel value Q ₁ having the detected maximum value 411. Is determined as a threshold for distinguishing the face area of the user 11 from other areas, and is supplied to the face area data extraction unit 403.

The threshold value Q ₂ is supplied from the threshold value setting unit 402 to the face area data extraction unit 403. Facial area data extracting unit 403 is stored in the memory 404, the position of the pixel in the difference image, by referring to the table and the pixel value is associated differential image with a threshold Q ₂ or more pixel values Identify the pixels.

FIG. 24 shows pixels having a pixel value equal to or higher than the threshold value Q ₂ supplied from the threshold value setting unit 402 in a certain difference image 421.

In Figure 24, the difference image 421 "○" or "×" represents the pixels constituting the difference image 421, the pixels indicated by "○", have a threshold Q ₂ or more pixel values to indicate that is, the pixels indicated by "×", indicating that it has a threshold Q ₂ is smaller than the pixel value.

The face area data extraction unit 403 has an upper limit value (maximum value) of pixels (pixels indicated by “◯”) having a pixel value equal to or greater than the threshold value Q ₂ for each of the horizontal direction and the vertical direction of the difference image. Position) and lower limit (minimum position).

In the difference image 421 shown in FIG. 24, the face region extraction unit 403 determines the position X _mb as the upper limit value of pixels having a pixel value equal to or greater than the threshold value Q _{2 in} the horizontal direction of the difference image. as the lower limit value of the pixel having a threshold Q ₂ or more pixel values, the position X _ma is determined.

Further, the differential image with respect to the vertical direction, the upper limit value of the pixel having a threshold Q ₂ or more pixel values, the position Y _mb is determined, the pixels having a threshold Q ₂ or more pixel values The position Y _ma is determined as the lower limit value.

The face area data extraction unit 403 performs an area 422 surrounded by a range of X _ma ≤ i ≤ X _{mb and} a range of Y _ma ≤ j ≤ Y _mb , which are indicated by bold lines in FIG. Identify the region.

The face area data extraction unit 403 identifies the face area of the user 11 and uses the horizontal positions X _ma and X _mb and the vertical positions Y _ma and Y _mb as face area information together with the difference image 421. This is supplied to the face orientation determination unit 23.

  In the image processing apparatus 15 of FIG. 1, when the face area extraction unit 22 is configured as shown in FIG. 22 instead of being configured as shown in FIG. 7, the processing of FIG. 20 and FIG. In step S4, the histogram creation unit 401 creates a histogram of pixel values of the difference image and supplies the histogram to the threshold setting unit 402, and the pixel position and the pixel value are associated with each pixel of the difference image. The generated table is generated and supplied to the memory 404.

In step S <b> 5, the threshold setting unit 402 determines a threshold Q ₂ for distinguishing the face area of the user 11 from the other areas based on the pixel value histogram of the difference image, and supplies the threshold value Q ₂ to the face area data extraction unit 403. .

In step S6, the face area data extracting unit 403, based on the table stored in the threshold Q ₂ and memory 404 from the threshold setting unit 402 specifies the position of the face region of the user 11 within the difference image. Then, the face area data extraction unit 403 supplies the face direction information for specifying the face area of the user 11 together with the difference image supplied from the difference image calculation unit 21 (FIG. 1) to the face orientation determination unit 23 (FIG. 1). To do.

  Since the same processes are performed for the processes of steps S1 to S3 and S7 to S23, excluding the processes of steps S4 to S6 described above, the description thereof is omitted.

  As described above, even when the face area extraction unit 22 of the image processing device 15 is configured as shown in FIG. 22, the image processing device 15 outputs and outputs only the captured image that is optimal for recognition of the face image. The face area, eye area, nose area, mouth area, and representative points of the eyes, nose, and mouth of the captured image are output. As a result, the device that receives the output from the image processing device 15 can easily and accurately perform the facial image recognition processing.

  In the above-described embodiment, as shown in FIG. 10, the illumination device 13 is arranged so as to irradiate light from the front obliquely lower direction with respect to the face of the user 11. As shown, the illuminating device 13 may be arranged so as to irradiate the face of the user 11 from a diagonally upper front direction. Moreover, you may arrange | position the illuminating device 13 so that light may be irradiated from both diagonal left front and diagonal right front with the user's 11 face and horizontal height.

  In the above-described embodiment, an example of recognizing a human (user) face area as a recognition target has been described. However, an animal face or the like can also be set as a recognition target.

  The processing shown in FIGS. 20 and 21 can be executed by dedicated hardware or can be executed by software. When this processing is performed by software, for example, a series of processing can be realized by causing a (personal) computer as shown in FIG. 26 to execute a program.

  In FIG. 26, a CPU (Central Processing Unit) 501 executes various processes in accordance with a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 508 into a RAM (Random Access Memory) 503. To do. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output interface 505 is also connected to the bus 504.

  The input / output interface 505 includes an input unit 506 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal display), an output unit 507 including a speaker, and a hard disk. A communication unit 509 including a storage unit 508, a terminal adapter, an ADSL (Asymmetric Digital Subscriber Line) modem, a LAN (Local Area Network) card, and the like is connected. A communication unit 509 performs communication processing via various networks such as the Internet.

  A drive 510 is connected to the input / output interface 505 as necessary, and a magnetic disk (including a floppy disk), an optical disk (including a CD-ROM (Compact Disk-Read Only Memory) and a DVD (Digital Versatile Disk)). In addition, a removable medium (recording medium) 521 such as a magneto-optical disk (including MD (Mini-Disk)) or a semiconductor memory is appropriately mounted, and a computer program read from these is stored in the storage unit 508 as necessary. Installed.

  In the present specification, the steps described in the flowcharts are executed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the described order. It also includes processing.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

It is a block diagram which shows the structural example of one Embodiment of the image processing system to which this invention is applied. It is a figure explaining the outline | summary of the process which the image processing apparatus 15 performs. It is a figure explaining the outline | summary of the process which the image processing apparatus 15 performs. It is a block diagram which shows the detailed structural example of the imaging device 12 of FIG. It is a figure explaining the sensitivity characteristic of a logarithm conversion type image sensor. It is a block diagram which shows the detailed structural example of the difference image calculation part 21 of FIG. It is a block diagram which shows the detailed structural example of the face area extraction part 22 of FIG. It is a block diagram which shows the detailed structural example of the face direction determination part 23 of FIG. It is a block diagram which shows the detailed structural example of the face organ extraction part 24 of FIG. It is a figure explaining the irradiation direction of the light with respect to the user. It is a figure explaining the process of the difference image calculation part. It is a figure explaining the process of the face area extraction part. It is a figure explaining the process of the face orientation determination part. It is a figure explaining the process of the image separation part 103 of the facial organ extraction part 24. FIG. It is a figure explaining the process of the image separation part 103 of the facial organ extraction part 24. FIG. It is a figure explaining the process of the pixel value addition part 111, the pixel value addition part 121, and the pixel value addition part 131. FIG. It is a figure explaining the process of the eye image process part. It is a figure explaining the process of the nose image process part. It is a figure explaining the process of the mouth image process part. 4 is a flowchart for explaining processing of the image processing apparatus 15. 4 is a flowchart for explaining processing of the image processing apparatus 15. It is a block diagram which shows other embodiment of the face area extraction part 22 of FIG. It is a figure explaining the process which the face area extraction part 22 of FIG. 22 performs. It is a figure explaining the process which the face area extraction part 22 of FIG. 22 performs. It is a figure explaining the irradiation direction of the light with respect to the user. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image processing system, 12 Imaging device, 13 Illumination device, 15 Image processing device, 21 Difference image calculation part, 22 Face area extraction part, 23 Face direction determination part, 24 Face organ extraction part, 61 Image acquisition part, 64 Difference process Unit, 65 filter processing unit, 71 pixel value addition unit, 72 threshold value setting unit, 73 face area data extraction unit, 81 pixel value addition unit, 82 center position detection unit, 83 determination unit, 101 pixel value addition unit, 103th Nose / mouth image separation unit 103, 104 eye image processing unit, 105 nose image processing unit, 106 mouth image processing unit, 111 pixel value adding unit, 112 eye region extracting unit, 113 representative point determining unit, 121 pixel value adding unit, 122 nose region extracting unit, 123 representative point determining unit, 131 pixel value adding unit, 132 mouth region extracting unit, 133 representative Determining unit

Claims (9)

In an image processing apparatus that processes a face image including a face to be recognized,
In a state where the illumination unit is not irradiating light on the face, the first image which is an image obtained by imaging the face by the imaging unit, and the illumination unit irradiating light on the face, A face image acquisition means for acquiring the face image, which is a difference image consisting of a difference in pixel value from a second image that is an image obtained by imaging the face by the imaging means;
Pixel value addition means for calculating a pixel addition value by adding pixel values for each horizontal or vertical column of the face image acquired by the face image acquisition means;
Using the pixel addition value for each column in the horizontal or vertical direction of the difference image, the face image is an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region. Separating means for separating into
Eye region processing means for performing processing for extracting the eye region from the eye image separated by the separation means;
Nose region processing means for performing processing for extracting the nose region from the nose image separated by the separation means;
An image processing apparatus comprising: mouth area processing means for performing processing for extracting the mouth area from the mouth image separated by the separating means. The eye area processing means
Eye image pixel value addition means for calculating a pixel addition value by adding a pixel value for each column in the horizontal or vertical direction with respect to the eye image;
Eye area extraction means for extracting the eye area using the pixel addition value for each column in the horizontal or vertical direction of the eye image obtained by the eye image pixel value addition means;
The image processing apparatus according to claim 1, further comprising: an eye representative point determining unit that determines a representative point representing the position of the eye from the eye region extracted by the eye region extracting unit. The nose region processing means includes
Nose image pixel value addition means for calculating a pixel addition value by adding a pixel value for each column in the horizontal or vertical direction with respect to the nose image;
A nose region extraction means for extracting the nose region using the pixel addition value for each column in the horizontal or vertical direction of the nose image obtained by the nose image pixel value addition means;
The image processing apparatus according to claim 1, further comprising: a nose representative point determining unit that determines a representative point representing the position of the nose from the nose region extracted by the nose region extracting unit. The mouth area processing means includes
Mouth image pixel value addition means for calculating a pixel addition value obtained by adding a pixel value for each column in the horizontal or vertical direction with respect to the mouth image;
Mouth area extracting means for extracting the mouth area using the pixel added value for each column in the horizontal or vertical direction of the mouth image obtained by the mouth image pixel value adding means;
The image processing apparatus according to claim 1, further comprising: a mouth representative point determining unit that determines a representative point representing the position of the mouth from the mouth region extracted by the mouth region extracting unit. The first and second images are picked up by the image pickup means having a logarithmic conversion type image pickup device that outputs a pixel value substantially proportional to the logarithm of the incident light amount by utilizing a sub-threshold characteristic of a semiconductor. The image processing apparatus according to claim 1, wherein: The image processing apparatus according to claim 5, wherein the imaging element is an HDRC (High Dynamic Range CMOS (Complementary Metal Oxide Semiconductor)). In an image processing method of an image processing apparatus that processes a face image including a face to be recognized,
In a state where the illumination unit is not irradiating light on the face, the first image which is an image obtained by imaging the face by the imaging unit, and the illumination unit irradiating light on the face, A face image acquisition step of acquiring the face image that is a difference image composed of a difference in pixel value from a second image that is an image in which the face is imaged by an imaging unit;
A pixel value addition step of calculating a pixel addition value obtained by adding a pixel value for each column in the horizontal or vertical direction of the face image acquired by the processing of the face image acquisition step;
Using the pixel addition value for each column in the horizontal or vertical direction of the difference image, the face image is an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region. A separation step of separating into,
An eye region processing step for performing processing for extracting the eye region from the eye image separated by the processing of the separation step;
A nose region processing step for performing processing for extracting the nose region from the nose image separated by the processing of the separation step;
A mouth region processing step for performing processing for extracting the mouth region from the mouth image separated by the processing in the separation step. In a program for causing a computer to execute processing on a face image including a face to be recognized,
In a state where the illumination unit is not irradiating light on the face, the first image which is an image obtained by imaging the face by the imaging unit, and the illumination unit irradiating light on the face, A face image acquisition step of acquiring the face image that is a difference image composed of a difference in pixel value from a second image that is an image in which the face is imaged by an imaging unit;
A pixel value addition step of calculating a pixel addition value obtained by adding a pixel value for each column in the horizontal or vertical direction of the face image acquired by the processing of the face image acquisition step;
Using the pixel addition value for each column in the horizontal or vertical direction of the difference image, the face image is an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region. A separation step of separating into,
An eye region processing step for performing processing for extracting the eye region from the eye image separated by the processing of the separation step;
A nose region processing step for performing processing for extracting the nose region from the nose image separated by the processing of the separation step;
A mouth area processing step for performing a process of extracting the mouth area from the mouth image separated by the process of the separation step. In a recording medium on which a program for causing a computer to execute processing on a face image including a face to be recognized is recorded,
In a state where the illumination unit is not irradiating light on the face, the first image which is an image obtained by imaging the face by the imaging unit, and the illumination unit irradiating light on the face, A face image acquisition step of acquiring the face image that is a difference image composed of a difference in pixel value from a second image that is an image in which the face is imaged by an imaging unit;
A pixel value addition step of calculating a pixel addition value obtained by adding a pixel value for each column in the horizontal or vertical direction of the face image acquired by the processing of the face image acquisition step;
Using the pixel addition value for each column in the horizontal or vertical direction of the difference image, the face image is an eye image including an eye region, a nose image including a nose region, and a mouth image including a mouth region. A separation step of separating into,
An eye region processing step for performing processing for extracting the eye region from the eye image separated by the processing of the separation step;
A nose region processing step for performing processing for extracting the nose region from the nose image separated by the processing of the separation step;
A mouth area processing step for performing processing for extracting the mouth area from the mouth image separated by the processing of the separation step. A recording medium on which a program is recorded.

Images