JP2005346474A - Image processing method and image processor and program and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To discriminate with good accuracy a prescribed object from a picture even when an environment such illumination conditions involving the object is change. <P>SOLUTION: An image processor detects the prescribed object from the picture and comprises: conversion sections 20, 30 converting the picture into a color space picture based on a human visual perception amount; a feature amount extract section 40 extracting a feature amount from the picture converted by the conversion sections; and object detection sections 50, 60 detecting the prescribed object based on the feature amount extracted by the feature amount extraction section. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for discriminating a predetermined subject from an image.

  Conventionally, several methods for detecting an area where a predetermined subject exists in an image by image processing are known.

  For example, according to Japanese Patent Laid-Open No. 6-44365 (Patent Document 1), color information necessary for processing is extracted from target image data, and portions that are characteristic of a face such as a human skin region, lips, and eyes are extracted. The face position information is obtained from the extracted image. Further, according to Japanese Patent Laid-Open No. 11-1579 (Patent Document 2), a region where the brightness is somewhat dark and a skin color region are detected from the image, and it is determined whether or not the predetermined region in the image is a human face. It is carried out.

However, in any of the methods described above, brightness and color characteristics peculiar to the subject are important points in subject discrimination, but changes in the subject's brightness and color appearance are not taken into consideration. There has been a problem that it is extremely difficult to discriminate a subject when the environment in which the light is applied changes significantly.
JP-A-6-44365 Japanese Patent Application Laid-Open No. 11-15979 Japanese Patent Laid-Open No. 11-53547

  Considering the above points, for example, in Japanese Patent Laid-Open No. 11-53547 (Patent Document 3), a subject image is acquired in advance by changing the environment such as the illumination condition, and the distribution of the feature amount of the acquired subject image is used as a basis. A reference dictionary is created, the subject is collated, and the subject is extracted from the target image.

  However, in comparison with a reference dictionary created by photographing a subject under different illumination conditions, as a result, texture information is more important than color information. It has not been effective in discriminating subjects for which color information is important. Also, since it is necessary to acquire subject images by changing the environment such as lighting conditions, it is not easy to create a reference dictionary.

  Therefore, the present invention has been made in view of the above-described problems, and an object of the present invention is to make it possible to accurately determine a predetermined subject from an image even when an environment such as an illumination condition surrounding the subject changes. That is.

  In order to solve the above-described problems and achieve the object, an image processing method according to the present invention is an image processing method for detecting a predetermined subject from an image, and the image is processed in a color space based on a human perception amount. A conversion step for converting to an image; a feature amount extraction step for extracting a feature amount from the image converted in the conversion step; and a subject for detecting the predetermined subject based on the feature amount extracted in the feature amount extraction step And a detection step.

  In the image processing method according to the present invention, in the conversion step, adaptation processing is performed on an image based on a shooting environment when the image is shot.

  In the image processing method according to the present invention, in the subject determination step, the conversion step is performed on an image including a subject to be detected, which has been captured in a different environment in advance, to convert the image into a color space image based on a human perception amount. The predetermined subject is discriminated based on the distribution of the feature amount extracted in the feature amount extraction step.

  A program according to the present invention causes a computer to execute the above image processing method.

  A storage medium according to the present invention stores the above-mentioned program so as to be readable by a computer.

  An image processing apparatus according to the present invention is an image processing apparatus that detects a predetermined subject from an image, a conversion unit that converts the image into an image in a color space based on a human perception amount, and the conversion unit. And a feature amount extraction unit that extracts a feature amount from the converted image, and a subject detection unit that detects the predetermined subject based on the feature amount extracted by the feature amount extraction unit. .

  According to the present invention, it is possible to accurately determine a predetermined subject from an image even when an environment such as an illumination condition surrounding the subject changes.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings.

  The image processing apparatus according to the present embodiment is configured by a computer such as a PC (personal computer) or WS (workstation), and is input from a digital camera, downloaded from the Internet, a CD-ROM, a DVD-ROM, or the like. An object of the present invention is to determine a predetermined subject in an image input by reading from a storage medium or the like and detect the area.

  Hereinafter, the image processing apparatus according to the present embodiment that performs such processing will be described in more detail. In particular, in the present embodiment, an example in which a human face area is detected from an image captured by a camera will be described.

  FIG. 2 is a diagram illustrating a basic configuration of the image processing apparatus according to the present embodiment.

  A CPU 201 controls the entire apparatus using programs and data stored in the RAM 202 and the ROM 203, and performs each process described below.

  A RAM 202 includes an area for temporarily storing programs and data read from the external storage device 207 and the storage medium drive device 208, and also includes a work area used by the CPU 201 to execute various processes.

  A ROM 203 stores a boot program, setting data of the apparatus, and the like.

  Reference numerals 204 and 205 denote a keyboard and a mouse, respectively, for inputting various instructions to the CPU 201.

  A display unit 206 includes a CRT, a liquid crystal screen, and the like, and displays a processing result by the CPU 201 using characters, images, or the like.

  Reference numeral 207 denotes an external storage device, which is a large-capacity information storage device such as a hard disk drive device, for example, and stores an OS (Operating System) and programs and data for causing the CPU 201 to execute each process described later. These are read to the RAM 202 as required by the control of the CPU 201.

  A storage medium drive device 208 reads out programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and outputs them to the RAM 202 or the external storage device 207. A part of the program or data stored in the external storage device 207 may be recorded on the storage medium. In that case, when using the stored program or data, The storage medium drive device 208 reads out programs and data recorded on the storage medium and outputs them to the RAM 202.

  Reference numeral 209 denotes an I / F (interface), to which a digital camera, the Internet, a LAN network line, or the like can be connected.

  A bus 210 connects the above-described units.

  FIG. 1 is a block diagram illustrating a functional configuration of a computer that functions as an image processing apparatus according to the present embodiment.

  In FIG. 1, reference numeral 10 denotes an image input unit, and the following image data to be processed is input to the image processing apparatus 100 via the image input unit 10. In this image (input image), a human face is shown as a subject. An appearance parameter setting unit 20 sets a parameter for converting the input image into appearance color space data. Here, the environment, the brightness of the subject, the type of the light source, and the like are set as parameters representing the shooting environment when the input image is shot. A color appearance conversion unit 30 converts the input image into appearance color space image data based on the amount of human perception according to the appearance parameter set by the appearance parameter setting unit 20. Reference numeral 40 denotes a subject feature amount calculation unit that calculates a feature amount for detecting a face from image data in the appearance color space converted by the color appearance conversion unit 30. That is, a predetermined region of interest is cut out from the image as a partial image, and a feature amount calculated from brightness and color information is obtained. Reference numeral 50 is a reference dictionary storage unit for storing in advance, as dictionary data, a distribution of feature amounts for determining a subject to be detected. Reference numeral 60 denotes a subject determination unit that compares the feature amount calculated by the subject feature amount calculation unit 40 with the dictionary data stored in the reference dictionary storage unit 50, and whether the target region belongs to the category of the subject to be detected. Is determined. A subject area output unit 70 outputs the detected subject area as a detection result.

  1 may be configured by dedicated hardware, but in the present embodiment, each unit is realized as a program for causing the CPU 201 to execute each process to be described later that each unit is to perform.

  FIG. 3 is a flowchart of processing performed by each unit shown in FIG. 3 is loaded into the RAM 202 by the external storage device 207 or the storage medium drive device 208, and is executed by the CPU 201, the image processing apparatus according to the present embodiment performs each process described below. Will do.

  First, data of a processing target image (processing target image) is loaded into the RAM 202 (step S101). The form of loading is not particularly limited. For example, the processing target image data is stored in the external storage device 207, and the file name of the processing target image data is input using the keyboard 204 or the mouse 205. Then, the CPU 201 may be instructed to load the RAM 202, or a digital camera (not shown) is connected to the I / F 209, and this digital camera (actually, the digital camera loaded in the RAM 202 is connected). By sending an instruction to transfer image data captured by the digital camera and stored in its own internal memory to the RAM 202 of the image processing apparatus, the image data to be processed is stored in the RAM 202. You may make it forward.

  In this embodiment, this processing target image is an RGB image in which one pixel is expressed by 8 bits, and the size of the processing target image is M pixels × N pixels (where M is the number of horizontal pixels and N is a vertical pixel) Number). In addition, the processing target image includes only a plane composed of only R (red) pixel values, a plane composed only of G (green) pixel values, and a B (blue) pixel value. The plane is composed of a total of three planes, but is not limited to this, and may be a single image.

  When the processing target image data is compressed by JPEG or the like, the CPU 201 decompresses the image data and generates an image in which each pixel is expressed by R, G, and B components.

  Next, an appearance parameter corresponding to the processing target image loaded in the RAM 202 in step S101 is set (step S102). Here, the appearance parameter setting process in step S102 will be described in more detail.

  The appearance parameters and setting methods to be set are listed below.

  Camera profile: A table in which image data (RGB) output from a photographed camera is associated with color data (color tristimulus values XYZ) in a standard photographing environment. It is created from RGB data obtained by photographing a plurality of color charts in the environment where the image data was photographed and data obtained by measuring the spectral distribution characteristics of the color charts in a standard photographing environment and converting them into XYZ. The scenes to be photographed may be roughly classified for each light source such as daylight, cloudy, fluorescent light, tungsten, strobe, etc., and a camera profile may be created for each scene in advance, and the camera profile may be represented from these. In that case, the user selects a camera profile corresponding to the scene shot according to the image.

  White point: White XYZ data serving as a reference of image data. The user designates a white level area from the image. The image data of the designated area is converted from RGB to XYZ by the camera profile, and the average value is used as white point data. A white spot may be obtained by automatically extracting a relatively high luminance portion from the image.

  Adaptation luminance: represents the absolute luminance in the adaptation region. Input subject brightness at the time of shooting as adaptation brightness. Since the subject brightness can be calculated from the sensitivity, aperture value, and exposure time during shooting, the sensitivity, aperture value, and exposure time when shooting with the camera may be input.

  Background luminance: The relative luminance of the background area. A region up to a viewing angle of 10 degrees is set as a background region. Accordingly, the size of the partial area corresponding to the viewing angle of 10 degrees of the image data is input, and the average luminance in the partial area is set as the background luminance. Here, the image data is converted from RGB to XYZ by the camera profile, and the average value of the Y values is output. The calculation is performed for each pixel, and in this case, the partial area is set each time around the target pixel. The size of the partial area corresponding to the viewing angle of 10 degrees can be obtained from the number of pixels of the photographed image, the pixel size, and the focal length of the photographing lens. Therefore, instead of inputting the image size, the number of pixels, the pixel size, the photographing lens The focal length may be input.

  Ambient conditions: The user selects and designates from among an average shooting environment, a dim environment, and a dark environment. Images taken in a dark environment such as a night view are taken as a dark environment, images taken in the morning or dusk, a dimly lit indoor environment, etc. are dim environments, and other normal bright environments are average environments.

  That is, the user inputs the above conditions to be set in the appearance parameter setting unit 20 using the keyboard 204 and the mouse 205, but when the white balance mode for each light source is set as incidental information in the image data. Can be used to select camera profiles. Further, when sensitivity, aperture value, and exposure time are set in the incidental information as a shooting mode, it can be used for calculation of adaptation luminance. In addition, the subject brightness can be calculated from the sensitivity, aperture value, and exposure time, and used for automatic selection of ambient conditions. Further, when the number of pixels, the pixel size, and the focal length of the photographing lens are set in the incidental information, it can be used for calculating the background luminance. Further, when a scene mode capable of specifying an environment such as night view shooting or sunset is set in the incidental information, it can be used for automatic selection of ambient conditions.

  Next, the image data is converted into appearance color space data in accordance with the set appearance parameter (step S103).

  The conversion to the appearance color space described below is performed by the color appearance conversion unit 30 in FIG. 1, and the function of the color appearance conversion unit 30 can be divided into four blocks as shown in FIG. Reference numeral 31 denotes a color matching unit, which converts image data captured by a camera into colors in a standard shooting environment. A color adaptation unit 32 performs color adaptation processing based on white spots. A cone response unit 33 performs nonlinear compression processing that simulates the response of a visual cone based on the adaptation luminance. Reference numeral 34 denotes a perceptual attribute extracting unit that obtains perceptual attributes of brightness, saturation, and hue from the cone response output based on the background luminance and the ambient conditions. That is, conversion to appearance color space includes color matching, chromatic adaptation, luminance adaptation, and perceptual attributes (brightness, saturation, hue) for different environments.

  First, the RGB data acquired by the image input unit 10 is converted into tristimulus value XYZ data in a standard imaging environment by a camera profile which is an appearance parameter set in advance by the color matching unit 31. In the camera profile, XYZ values corresponding to typical RGB values are stored as a table in a grid pattern with a predetermined interval, and calculation of XYZ values corresponding to desired RGB values is performed by, for example, existing tetrahedral interpolation processing. It is assumed that interpolation calculation is performed by

  The chromatic adaptation unit 32 performs the following chromatic adaptation processing based on the white point.

Ra = {D × Rs / Rw + (1−D)} × R
Ga = {D × Gs / Gw + (1−D)} × G
Ba = {D × Bs / Bw + (1−D)} × B (1)
However, (R, G, B) is a value obtained by converting XYZ converted by the color matching unit 31 into RGB in accordance with a predetermined conversion formula (for example, 3 × 3 matrix calculation, hereinafter referred to as XYZ-RGB conversion), (Rs, Gs, Bs) is a value obtained by converting XYZ data representing a white point in a standard photographing environment into RGB by XYZ-RGB conversion, and (Rw, Gw, Bw) is an XYZ set as a white point from image data A value obtained by converting data into RGB by XYZ-RGB conversion, (Ra, Ga, Ba) is RGB data after chromatic adaptation conversion, D is a parameter indicating the degree of adaptation, and a predetermined value such as 0.1 is set in advance. . D may be calculated from the adaptation brightness and the ambient condition, or may be set by the user in the appearance parameter setting unit 20. Then, the RGB data after the chromatic adaptation conversion is converted into XYZ according to a predetermined conversion formula (inverse conversion of XYZ-RGB conversion, for example, matrix operation of 3 rows and 3 columns, hereinafter referred to as RGB-XYZ conversion), and is output.

  Next, the cone response unit 33 performs nonlinear compression processing that simulates the response of the visual cone based on the adaptation luminance. First, XYZ data is converted into color data LMS in three frequency bands corresponding to a cone in human vision according to a predetermined conversion formula (for example, a matrix operation of 3 rows and 3 columns, hereinafter referred to as XYZ-LMS conversion). Then, non-linear compression of the LMS data is performed in accordance with a predetermined compression formula using the adaptation luminance as a parameter, and output.

  Next, the perceptual attribute extraction unit 34 obtains brightness J, saturation C, and hue h, which are perceptual attributes, from the cone response output according to a predetermined conversion formula based on background luminance and ambient conditions. First, the hue h is obtained from LMS data that is a cone response output. Next, luminance adaptation is performed on the LMS data, which is a cone response output, from the constant parameter determined by the ambient conditions and the background luminance, and the brightness J is obtained. Then, the saturation C is obtained from the constant parameter determined by the obtained hue h, brightness J and ambient conditions, and the background luminance. These processes can be realized by a conversion process based on a color appearance model such as CIE CAM97s or CIE CAM02.

  Next, a partial region is cut out from the image in order to detect a desired subject (step S104).

  In step S104, a partial area having a plurality of predetermined image sizes is moved so as to scan the image for each predetermined pixel, and the partial area is cut out. When detecting the face of a person from a person image, for example, a partial region having an area of 5% to 25% of the image size is scanned in 5 steps every 5% from a large size.

  Next, the feature amount calculation unit 40 calculates a feature amount for detecting a face from image data (JCh) in the appearance color space for each partial region (step S105).

  First, the cut out partial area is normalized to a predetermined size. Then, the following five feature amounts are obtained from the normalized image data.

  Average value of brightness: Average value of J component of image data.

  Brightness of skin color portion: Pixels in a predetermined range from the hue h component of the image data are skin color pixels, and an average value of J components of the skin color pixels is obtained.

  Brightness of eyes, mouth, and nose parts: As shown in FIG. 5, a predetermined area of image data extracted and normalized as a partial area is defined as eyes (A in FIG. 5), mouth (B in FIG. 5), nose ( The average value of the J components of the pixels belonging to each region is obtained as part C) of FIG.

  In the case of a face image, these features are determined based on the assumption that the skin color and brightness of the nose are brighter than the brightness of the entire face, and the brightness of the eyes and mouth is darker than the brightness of the entire face. It is decided to do. At this time, even if the hue of the skin color changes slightly due to the difference in the environment in which the image is captured, the skin color pixel is determined based on the perceptual attribute of the appearance color space, so that it is possible to cope with the accuracy deterioration due to the change in the color of the light source. In addition, even when shadows are generated in the face image due to illumination, components depending on shadows are removed to some extent by luminance adaptation, so that feature quantities can be extracted with high accuracy.

  In addition, when only the brightness and hue perceptual attributes are required for the feature amount calculation as in the present embodiment, it is only necessary to calculate what is necessary. In this example, it is particularly necessary to calculate the saturation. It goes without saying that it is not.

  Here, five brightness feature values have been described as feature values representing the face, but feature values based on color, edge strength, and direction may be calculated and used for discrimination.

  Next, based on the calculated feature value, the subject determination unit 50 determines the facialness of the partial area (step S106).

  In order to determine the facial appearance, dictionary data stored in the reference dictionary storage unit 50 is used. The reference dictionary storage unit 50 stores feature amount data obtained by calculating image data including faces photographed in various environments in advance by the same processing as steps S101 to S105 of the present embodiment. However, the extraction of the partial region corresponding to step S104 when creating the reference dictionary is to input the position of the face portion in the image while confirming the image. The calculated feature value for each face is statistically processed, and the upper limit value, lower limit value and center value of each feature value, and the upper limit value and lower limit value of the evaluation function combining each feature value, to determine the likelihood of a face. The value and the center value are set so as to optimally discriminate between a face image and an image other than a face, and are stored as a reference dictionary. As an evaluation function combining each feature amount, for example, the ratio of the brightness of the skin color part, eyes, mouth, and nose to the brightness of the entire face, the difference between the brightness of the skin color part and the brightness of the eyes, mouth, etc. . In this embodiment, even when creating the reference dictionary in this way, the image data is converted into an appearance color space image, and normalization based on the perceptual amount is performed. Even if the photographing environment at the time of photographing the target image is different, accurate detection can be performed.

  The determination of facial appearance is performed by comparing the feature values extracted from the image data of the partial area with each data of the reference dictionary. First, the value of the evaluation function combining each feature value and each feature value is an upper limit value and a lower limit value. It is determined whether it is within the range. Then, all the values within the range between the upper limit value and the lower limit value are selected as face candidates. Further, the reciprocal of the sum of the distances between the feature values and the evaluation function value combining the feature values and the corresponding central value is obtained as an index of facialness.

  In the present embodiment, each feature amount and an evaluation function that combines each feature amount are individually determined, and an index of facialness is obtained by the sum of distances from the center value. For example, each feature amount and An evaluation function combining each feature amount is used as a feature vector, and the reference dictionary stores the average value of the feature vectors of the face image to be referred to, a covariance matrix, and a determination threshold value based on the Mahalanobis distance, and image data of the partial region. The Mahalanobis distance is calculated from the feature vector extracted from the average value of the reference dictionary and the covariance matrix, and the sub-region below the threshold is determined as a face candidate, and the reciprocal of the calculated Mahalanobis distance is obtained as an index of facialness. Also good.

  Next, it is determined whether or not face discrimination scanning has been completed for all partial regions (step S107). When face discrimination scanning has not been completed for all partial areas, the process returns to step S104 so that face discrimination is performed for the next partial area. If the face discrimination scan is completed for all the partial areas, the process proceeds to the next step S108.

  Next, the face area overlap is determined for all partial areas determined as face candidates by the subject area output unit 70 (step S108).

  First, it is determined whether sets of partial areas determined as face candidates are sequentially overlapped. If a pair of overlapping partial areas is detected, a partial area having a low facial likelihood index is removed from the face candidates based on the facial likelihood index obtained in step S106. The above processing is repeated until there is no overlapping of partial areas, and the final face candidate is obtained.

  Finally, a process of outputting all partial areas determined as face candidates by the subject area output unit 70 to the display unit 206 is performed (step S109). For example, a rectangular frame indicating the partial area of the detected face portion is superimposed and displayed on the image data acquired by the image input unit 10.

  As described above, according to the present embodiment, it is possible to accurately determine the face portion of a person from the image even when the environment such as the illumination condition surrounding the subject changes.

  In the present embodiment, the case of detecting a face has been described. However, the subject feature amount extraction unit 40 extracts a feature amount suitable for the subject to be detected, and the subject determination unit 50 is suitable for the subject to be detected. If the determination is made based on the reference dictionary, it can be widely applied to detection of a desired subject. In particular, in this implementation failure, the human visual mechanism is referenced, and the image input in the previous stage of subject recognition processing is converted to color / luminance adaptation and perceptual amount. The robustness to the environment is achieved.

(Other embodiments)
An object of the present invention is to supply a recording medium (or storage medium) that records software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the computer of the system or apparatus (or CPU or MPU). Needless to say, this can also be achieved by reading and executing the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

1 is a block diagram illustrating a functional configuration of a computer that functions as an image processing apparatus according to an embodiment of the present invention. 1 is a diagram illustrating a basic configuration of an image processing apparatus according to an embodiment of the present invention. It is a flowchart of the process performed by each part shown in FIG. It is a block diagram which shows the detail of a color appearance conversion part. It is a figure explaining the feature-value of a face image.

Claims (6)

An image processing method for detecting a predetermined subject from an image,
A conversion step of converting the image into an image in a color space based on the amount of human perception;
A feature amount extraction step of extracting a feature amount from the image converted in the conversion step;
A subject detection step of detecting the predetermined subject based on the feature amount extracted in the feature amount extraction step;
An image processing method comprising:   The image processing method according to claim 1, wherein in the conversion step, an adaptation process is performed on the image based on a shooting environment when the image is shot.   In the subject determination step, the feature amount extracted in the feature amount extraction step is performed by performing the conversion step on an image including a subject to be detected that has been captured in a different environment in advance and converting the image into a color space image based on a human perception amount. The image processing method according to claim 1, wherein the predetermined subject is discriminated based on the distribution of the image.   A program for causing a computer to execute the image processing method according to any one of claims 1 to 3.   A storage medium storing the program according to claim 4 in a computer-readable manner. An image processing apparatus for detecting a predetermined subject from an image,
Conversion means for converting an image into an image in a color space based on a human perception amount;
Feature quantity extraction means for extracting feature quantities from the image converted by the conversion means;
Subject detection means for detecting the predetermined subject based on the feature amount extracted by the feature amount extraction means;
An image processing apparatus comprising:

Images