JP2007025901A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique for accurately detecting a photographic subject even in when the photographic subject is dark or partially shaded according to a photographing environment, especially in an illumination condition in imaging the photographic subject. <P>SOLUTION: When a rectangle of a predetermined size is arranged in each location on a reduced image, this image processor extracts a pattern in this rectangle (S104), corrects a luminance value of each pixel which constitutes the pattern using low frequency components of each pixel (S106), determines whether or not the corrected pattern is a facial pattern (S107), and outputs a pattern which has been determined to be the facial pattern (S108). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for detecting a subject in an image.

  An image processing method for automatically detecting a specific subject pattern from an image is very useful, and can be used for, for example, determination of a human face. Such methods can be used in many areas such as teleconferencing, man-machine interfaces, security, monitor systems for tracking human faces, image compression, and the like. As a technique for detecting a face from such an image, for example, Non-Patent Document 1 discloses various methods. Among them, use some prominent features (two eyes, mouth, nose, etc.) and the unique geometric positional relationship between those features, or symmetric features of human face, human face color A method for detecting a human face by using features, template matching, neural network, etc. is shown.

  On the other hand, the method proposed in Non-Patent Document 2 is a method of detecting a face pattern in an image using a neural network. The face detection method according to Non-Patent Document 2 will be briefly described below.

  First, image data including a face is read into a memory, and a predetermined area to be compared with the face is cut out from the image. Then, one output is obtained by calculation using a neural network with the distribution of pixel values in the cut out region as an input. At this time, the weight and threshold of the neural network are learned in advance using a face image pattern and a non-face image pattern, and for example, a face is determined if the output of the neural network is 0 or more, and a non-face is determined otherwise. To do.

  Then, the face is detected from the image by scanning the cutout position of the image pattern to be collated with the face which is an input of the neural network, for example, vertically and horizontally from the entire image as shown in FIG. Further, in order to cope with the detection of faces of various sizes, as shown in FIG. 3, the read images are sequentially reduced at a predetermined rate, and the above-described face detection scanning is performed on the images.

  In addition, when determining whether a cut-out partial area is a face or not, if the face is dark due to the subject environment such as lighting conditions at the time of image capture, or if the shadow is partially covered by the face, Since the distribution of the pixel values in the region is greatly changed, the performance is remarkably deteriorated.

  In order to deal with such a situation, in Non-Patent Document 2, linear correction of pixel values and correction by histogram smoothing are performed as preprocessing for the cut out partial area. Linear correction of pixel values is to correct the change in linear pixel values by approximating the distribution of pixel values in the cut out partial area on the primary plane and subtracting the primary component from the original distribution of pixel values. It is. Histogram smoothing is an image processing method well known to those skilled in the art that enhances contrast by converting pixel values so that the distribution (histogram) of pixel values in the extracted partial region is uniform.

However, linear correction of pixel values can cope with some changes when the lighting conditions that affect the face change globally. For example, the subject may be wearing a hat or standing in the shade of a tree. In the situation where the face is partially shaded, it cannot be dealt with, and there is a problem that the detection accuracy of the subject pattern is deteriorated.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.24, NO.1, JANUARY 2002, “Detecting Faces in Images: A Survey” IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.20, NO.1, JANUARY 1998, “Neural network-based face detection”

  The present invention has been made in view of the above problems, and when the subject is photographed, particularly when the subject is dark or partially shaded depending on the illumination conditions. It is an object of the present invention to provide a technique for accurately detecting a subject.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, acquisition means for acquiring an image including a predetermined subject;
Generating means for generating a luminance image composed of luminance components of the image;
Reduction means for generating a plurality of reduced images by recursively reducing the luminance image;
A first extracting means for extracting, when a rectangle of a predetermined size is arranged at each position on the reduced image, extracting a luminance value of each pixel constituting the pattern in the rectangle;
Calculating means for obtaining a low-frequency component of each pixel;
Correction means for correcting the luminance value of each of the pixels using a low-frequency component of each of the pixels;
Determining means for determining whether the pattern whose luminance value has been corrected by the correcting means is a face pattern;
Output means for outputting a pattern determined as a face pattern by the determining means.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, an acquisition step of acquiring an image including a predetermined subject;
A generation step of generating a luminance image composed of luminance components of the image;
A reduction process for generating a plurality of reduced images by recursively reducing the luminance image;
A first extraction step of extracting, when a rectangle of a predetermined size is arranged at each position on the reduced image, extracting a luminance value of each pixel constituting the pattern in the rectangle;
A calculation step for obtaining a low frequency component of each of the pixels;
A correction step of correcting the luminance value of each of the pixels using a low frequency component of each of the pixels;
A determination step of determining whether the pattern whose luminance value has been corrected in the correction step is a face pattern;
An output step of outputting the pattern determined as the face pattern in the determination step.

  According to the configuration of the present invention, it is possible to detect a subject with high accuracy even when the subject is dark or is partially shaded depending on the photographing environment for photographing the subject, particularly lighting conditions. it can.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

  The image processing apparatus according to the present embodiment is configured by a computer such as a PC (personal computer) or WS (workstation), and an external device via an image input from an imaging apparatus such as a digital camera or a network such as the Internet. A predetermined subject included in an image input in various input forms such as an image downloaded from, an image input by reading from a storage medium such as a CD-ROM or a DVD-ROM is detected. In this embodiment, a human face is used as a subject, but other subjects may be used.

  First, an image processing apparatus according to the present embodiment that performs such processing will be described. FIG. 7 is a diagram illustrating a hardware configuration of a computer applicable to the image processing apparatus according to the present embodiment.

  A CPU 201 controls the entire computer using programs and data stored in the RAM 202 and the ROM 203, and executes each process described later performed by the computer.

  Reference numeral 202 denotes a RAM, an area for temporarily storing programs and data read from the external storage device 207 and the storage medium drive device 208, and for temporarily storing data received from the outside via the I / F 209. Various areas such as an area and a work area used by the CPU 201 to execute various processes can be provided as appropriate.

  A ROM 203 stores a boot program, setting data of the computer, and the like.

  Reference numerals 204 and 205 denote a keyboard and a mouse, respectively, and various instructions can be input to the CPU 201 when operated by a computer operator.

  A display unit 206 includes a CRT, a liquid crystal screen, and the like, and displays the processing result by the CPU 201 using characters, images, and the like.

  Reference numeral 207 denotes an external storage device, which is a large-capacity information storage device such as a hard disk drive device, and stores an OS (Operating System) and programs and data for causing the CPU 201 to execute each process described later performed by the computer. These are read out to the RAM 202 as appropriate under the control of the CPU 201.

  A storage medium drive device 208 reads out programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and outputs them to the RAM 202, the external storage device 207, or the like. A part of the program or data stored in the external storage device 207 may be recorded on the storage medium. In that case, when using the stored program or data, The storage medium drive device 208 reads out programs and data recorded on the storage medium and outputs them to the RAM 202.

  Reference numeral 209 denotes an I / F (interface), to which a digital camera, the Internet, a LAN network line, or the like can be connected.

  A bus 210 connects the above-described units.

  In addition, the input form of the image to the computer is not particularly limited, and various forms are conceivable.

  FIG. 1 is a block diagram showing a functional configuration of a computer applicable to the image processing apparatus according to the present embodiment. As shown in the figure, the image processing apparatus according to this embodiment includes an image input unit 10, an image memory 20, an image reduction unit 30, a matching pattern extraction unit 40, a low frequency component extraction unit 50, a luminance correction unit 60, a face discrimination. Part 70 and face area output part 80.

  The image input unit 10 receives image data output from a device such as a digital still camera or a film scanner, and outputs the image data to the subsequent image memory 20. As described above, the image input form is not particularly limited.

  The image memory 20 is a memory for storing image data output from the image input unit 10.

  The image reduction unit 30 first generates a luminance image composed of luminance components of image data received from the image memory 20. Then, a plurality of reduced images are generated by recursively reducing the generated luminance image. Each generated reduced image (if the original luminance image generated based on the image data received from the image memory 20 is also interpreted as a 1/1 reduced image, this original can also be included in the reduced image). The data is sequentially output to the subsequent matching pattern extraction unit 40.

  When the collation pattern extraction unit 40 receives the reduced image from the image reduction unit 30, the pixel group included in the rectangle is sequentially extracted as a “collation target pattern” while moving a rectangle of a predetermined size on the reduced image, This is output to the low-frequency component extraction unit 50 and the luminance correction unit 60 in the subsequent stage. Such processing is performed for each reduced image received from the image reduction unit 20.

  When receiving the verification target pattern from the verification pattern extraction unit 40, the low frequency component extraction unit 50 extracts the low frequency component of each pixel constituting the verification target pattern. The extracted low frequency component data is output to the luminance correction unit 60 in the subsequent stage.

  When the luminance correction unit 60 receives the verification target pattern from the verification pattern extraction unit 40 and the low frequency component of the verification target pattern from the low frequency component extraction unit 50, the luminance correction unit 60 uses this low frequency component to calculate the luminance distribution of the verification target pattern. The process which correct | amends is performed.

  The face discriminating unit 70 discriminates whether or not the verification target pattern whose luminance distribution has been corrected by the luminance correcting unit 60 is a face pattern.

  The face area output unit 70 outputs a verification target pattern that the face determination unit 70 has determined to be a face pattern.

  Each of the above units operates as one function of the CPU 201, for example.

  Next, a process performed by the CPU 201 operating as each unit illustrated in FIG. 1, that is, a process for detecting a subject included in an image will be described below with reference to FIG. 2 showing a flowchart of the process. explain. Note that a program and data for causing the CPU 201 to execute the processing according to the flowchart of FIG. 10 are stored in the external storage device 207 (or a storage medium readable by the storage medium drive device 208). Accordingly, the computer executes each process described below by loading the data into the RAM 202 as needed and the CPU 201 executing the process using this.

  When image data is input from the outside via the external storage device 207 or the I / F 209, the CPU 201 temporarily stores it in an area corresponding to the image memory 20 in the RAM 202 (step S101). If an image input to the computer is compressed, the image is decompressed and temporarily stored in the RAM 202.

  In the present embodiment, it is assumed that each pixel constituting the input image data is represented by R, G, and B. Accordingly, the CPU 201 determines, based on the image data stored in the RAM 202 in step S101, an image (luminance image) composed of the luminance components of this image, that is, the value of each pixel constituting this image as the luminance value of this pixel. The image converted into is generated (step S102). However, if each pixel constituting the image data stored in the RAM 202 in step S101 is expressed by YCrCb, a luminance image is generated using only the Y component in step S102.

  Next, the CPU 201 generates a plurality of reduced images by recursively reducing the generated luminance image (step S103). For example, a reduced image 2 obtained by multiplying the vertical and horizontal sizes of the luminance image generated in step S102 (hereinafter referred to as reduced image 1) by 1 / 1.2 is generated, and then the vertical and horizontal sizes of the reduced image 2 are generated. A plurality of reduced images are generated, such as generating a reduced image 3 obtained by multiplying 1 / 1.2. This is because when detecting a face in the subsequent processing, detection is sequentially performed on image data of a plurality of sizes in order to support detection of faces of various sizes. The number of reduced images to be generated is not particularly limited.

  In step S104 and subsequent steps, each generated reduced image is processed. That is, the processing after step S104 is repeatedly performed for the number of generated reduced images.

  In the following description, the generated reduced images are referred to as reduced image 1, reduced image 2,... Reduced image M in descending order of size, and the subsequent processing is performed on reduced image 1 first. Note that the order of selection as processing targets is not particularly limited.

  First, the CPU 201 arranges a rectangle of a predetermined size on the reduced image 1, and extracts a pixel group in the rectangle as a verification target pattern (step S104). When this rectangle is arranged at each position on the reduced image 1, it is for obtaining the luminance distribution within the rectangle at each position. For example, this rectangle is initially arranged at the upper left corner of the image.

  Next, a low frequency component of each pixel constituting the verification target pattern extracted in step S104 is obtained (step S105). Various processes can be considered in step S105, and an example is shown below.

  For example, when obtaining a low frequency component of a pixel i (i = 1,..., N) constituting the verification target pattern, a block having a predetermined size centered on the position of the pixel i is set on the verification target pattern. An average value of luminance values of each pixel in the block is obtained, and this average value is set as a low frequency component of the pixel i.

  Here, as shown in FIG. 6, when the verification target pattern is a region F that substantially covers the face to be detected, this block may be a region P that is large enough to cover the eyes and nose. More preferred. This is due to the high frequency of partial shadows on the face due to depressions and protrusions in the eyes and nose depending on the lighting conditions and the surrounding environment, and low frequency components are used to deal with such situations. It is because it is more preferable to extract. For example, when the size of the verification target pattern is about 20 pixels × 20 pixels, the block size is preferably about 5 × 5 pixels to 9 × 9 pixels. Note that when calculating the average luminance value in a block, only a partial block can be set in the peripheral area of the pattern to be matched. In this case, the average luminance value is calculated using the luminance value in the block of the settable portion. To ask. In this embodiment, the average luminance value in the block is obtained as a specific method for extracting the low frequency component. However, by performing a product-sum operation on the luminance distribution in the block and the Gaussian function, May be obtained.

  Therefore, by performing such processing for i = 1 to N, it is possible to obtain the low-frequency component (in other words, the luminance distribution of the verification target pattern) of each pixel constituting the verification target pattern. When priority is given to high-speed processing, a block is set for each group of pixels (for example, 2 pixels × 2 pixels), and the average luminance value of the pixels in the block is set to a low-frequency component for the grouped pixels. It is also good.

  Next, the CPU 201 performs a process of correcting the luminance value of each pixel constituting the verification target pattern extracted in step S104 with the low frequency component obtained in step S105 for each pixel (step S106).

  That is, if the luminance value of the pixel i (i = 1,, N) constituting the verification target pattern is Yi and the low frequency component of the pixel i is Li, the corrected luminance value Y′i of the pixel i is as follows: It can obtain | require according to the formula of.

Y'i = Yi / Li
Here, Y′i, Yi, and Li are normalized to a value of 0 or more and 1 or less, and when Li is 0, 0 is output as Y′i. When Y′i exceeds 1, 1 is output as Y′i. Alternatively, Y′i may be obtained according to the following equation using a predetermined parameter γ.

Y′i = Yi / Li ^γ
By correcting the luminance value by any of the above formulas, the luminance value of the pixel to be corrected does not change much when the luminance around this pixel is bright, but the luminance around this pixel does not change. In the case of darkness, the brightness is corrected so as to be bright, so that the brightness value of each pixel constituting a partially dark portion in the image can be corrected in view of surrounding brightness values.

  In the present embodiment, the luminance correction based on the low frequency component has been described, but it goes without saying that it may be combined with the histogram smoothing process proposed in Non-Patent Document 2.

  Therefore, by obtaining Y′i for i = 1 to N using any one of the above equations, the luminance value of each pixel constituting the verification target pattern extracted in step S104 can be corrected. What corrected the luminance value of each pixel which comprises the collation target pattern extracted by step S104 is obtained as Y'i data.

  Next, the face discriminating unit 70 discriminates whether or not the collation target pattern (luminance pattern) whose luminance distribution has been corrected in step S106 is a face pattern (face and ghost pattern) (step S107).

  FIG. 4 is a diagram showing the operation of the neural network for identifying the pattern in the predetermined area. In the figure, R represents an area to be identified on the image, for example. In this embodiment, as shown in the figure, this area R is further divided into three areas by each of the neurons (indicated by N). ) As a receptive field. Then, the luminance distribution of the divided area is input to each neuron, and an output in the intermediate layer is obtained. Then, the final output is obtained by using the output of each neuron as the input of the neuron in the output layer.

  Here, in each neuron, the product-sum operation of the weight and the luminance distribution obtained by learning in advance and the operation by the sigmoid function as a result are performed. In this embodiment, the output value of the neuron in the output layer is compared with a predetermined threshold value, and an output pattern exceeding the threshold value is set as a face and the other pattern is not a face (non-face) (details and learning of neural network) (Refer to the above-mentioned Non-Patent Document 2).

  Note that the process of determining whether the pattern to be verified whose luminance distribution has been corrected in step S106 is a face pattern is not limited to this. The subject is determined for a predetermined input pattern, and the face pattern and Any device that outputs a value that can be distinguished from a non-face pattern may be used. For example, the receptive field may be set so as to divide the input partial area based on the positional relationship of the face outline, eyes, nose, mouth, and cheeks as shown in FIG. FIG. 5 shows each part of the face.

  In addition, it is not limited to neural network discrimination. For example, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2001 was proposed in a report by Viola and Jones entitled “Rapid Object Detection using a Boosted Cascade of Simple Features”. A method based on AdaBoost may be used.

  FIG. 3 illustrates a process for determining whether a verification target pattern is a face pattern for reduced images of various sizes (reduced image 1, reduced image 2,... Reduced image M in this embodiment). It is a figure to do. When a rectangle of the same size is arranged at each position on each reduced image, in order to determine whether or not the area within the rectangle at each position is a face pattern, first, as shown on the left side of FIG. A rectangle is arranged at the upper left corner of the reduced image, and the position of the rectangle is moved from top to bottom from the right to the right. Each time the pixel group is moved, the pixel group in the rectangle is used as a collation target pattern for discrimination of the face pattern.

  If it is determined in step S107 that the verification target pattern whose luminance value has been corrected in step S106 is a face pattern, the process proceeds to step S108, and image data of the verification target pattern is output (step S108). S108). The collation target pattern to be output may be a pattern before correction in step S106 or a pattern after correction.

  The output destination is not particularly limited, but may be a predetermined area in the RAM 202 or an external device capable of data communication via the external storage device 207 or the I / F 209. .

  On the other hand, if it is determined in step S107 that the verification target pattern whose luminance value has been corrected in step S106 is not a face pattern, or after the process in step S108, the process proceeds to step S114. It is checked whether or not there is a rectangular destination on 1 (step S114). That is, when the position of the rectangle on the reduced image 1 is moved and the pixel group in the rectangle at the next position is extracted as a verification target pattern, and there is no destination, for example, the current rectangle position is If the position is already at the lower right corner of the reduced image 1, the rectangle cannot be moved anymore. On the other hand, if the position of the current rectangle is not already the position of the lower right corner of the reduced image 1, the rectangle can be moved.

  Accordingly, if there is a destination, the process proceeds from step S114 to step S115, and the position of the rectangle on the reduced image 1 is moved (step S115). When the movement of the rectangle is completed, the process proceeds to step S104, the pattern to be collated in the moved rectangle is extracted, and the subsequent processes are performed.

  On the other hand, if there is no rectangular movement destination, the process proceeds to step S116 to determine whether or not all the reduced images have been processed (step S116), and there is a reduced image that has not yet been processed. In this case, the process proceeds to step S117, the position of the rectangle to be arranged on the reduced image is initialized (for example, returned to the position of the upper left corner of the reduced image) (step S117), and the process after step S104 is performed for the next reduced image. I do.

  In the present embodiment, since processing is currently performed on the reduced image 1, processing is performed on the reduced image 2 next. Therefore, in this case, a rectangle is arranged at the position of the upper left corner on the reduced image 2 (step S117), and the processing after step S104 is performed on the reduced image 2.

  Therefore, as described above, for each of the reduced images having different reduction ratios, a matching target pattern of the same size is extracted, and the face detection process is performed using the extracted matching target pattern. Even if it occupies such a ratio, the face in the image can be detected. For example, the size of a reduced image with a high reduction ratio is as close as possible to the size of the pattern to be collated, so even if the face occupies a large face in the image, this face may fit in the pattern to be collated The probability of detecting this face is high.

  As described above, according to the present embodiment, it is possible to accurately extract a face region from an image even in a situation where the shooting environment of a subject is bad such as illumination conditions.

  Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the computer applicable to the image processing apparatus which concerns on embodiment of this invention. It is a flowchart of the process for detecting the to-be-photographed object contained in the image. It is a figure explaining the process which discriminate | determines whether the collation target pattern is a face pattern about the reduced image of various sizes. It is the figure shown about operation | movement of the neural network for identifying the pattern in a predetermined area | region. It is a figure which shows each part of a face. FIG. 6 is a diagram illustrating blocks. It is a figure which shows the hardware constitutions of the computer applicable to the image processing apparatus which concerns on embodiment of this invention.

Claims (8)

Acquisition means for acquiring an image including a predetermined subject;
Generating means for generating a luminance image composed of luminance components of the image;
Reduction means for generating a plurality of reduced images by recursively reducing the luminance image;
A first extracting means for extracting, when a rectangle of a predetermined size is arranged at each position on the reduced image, extracting a luminance value of each pixel constituting the pattern in the rectangle;
Calculating means for obtaining a low-frequency component of each pixel;
Correction means for correcting the luminance value of each of the pixels using a low-frequency component of each of the pixels;
Determining means for determining whether the pattern whose luminance value has been corrected by the correcting means is a face pattern;
An image processing apparatus comprising: output means for outputting a pattern determined as a face pattern by the determination means.   The calculation means sets a block of a predetermined size centered on the position of the pixel i (i = 1,, N) constituting the pattern on the pattern, and averages the luminance values of the pixels in the block The image processing apparatus according to claim 1, wherein a process for obtaining a value as a low frequency component of the pixel i is performed for i = 1 to N.   The image processing apparatus according to claim 2, wherein the size of the block is a size that substantially covers an eye and a nose when the pattern is a size that substantially covers the predetermined subject. When the luminance value of the pixel i (i = 1,, N) constituting the pattern is Yi and the low frequency component of the pixel i is Li, the correcting unit corrects the luminance value Y ′ of the pixel i after correction. i is expressed by the following formula: Y′i = Yi / Li
The image processing apparatus according to claim 1, wherein the process for obtaining i = 1 to N is performed in accordance with When the luminance value of the pixel i (i = 1,, N) constituting the pattern is Yi and the low frequency component of the pixel i is Li, the correcting unit corrects the luminance value Y ′ of the pixel i after correction. i is represented by the following formula: Y′i = Yi / Li ^γ
The image processing apparatus according to claim 1, wherein the process for obtaining i = 1 to N is performed in accordance with An acquisition step of acquiring an image including a predetermined subject;
A generation step of generating a luminance image composed of luminance components of the image;
A reduction process for generating a plurality of reduced images by recursively reducing the luminance image;
A first extraction step of extracting, when a rectangle of a predetermined size is arranged at each position on the reduced image, extracting a luminance value of each pixel constituting the pattern in the rectangle;
A calculation step for obtaining a low frequency component of each of the pixels;
A correction step of correcting the luminance value of each of the pixels using a low frequency component of each of the pixels;
A determination step of determining whether the pattern whose luminance value has been corrected in the correction step is a face pattern;
An image processing method comprising: an output step of outputting the pattern determined as the face pattern in the determination step.   A program causing a computer to execute the image processing method according to claim 6.   A computer-readable storage medium storing the program according to claim 7.

Images