JP2019096073A - Image processor, method for processing image, and program - Google Patents

Abstract

To recognize an attribute more accurately.SOLUTION: The image processor includes: detection means for detecting the position of an object in an image; extraction means for extracting the amount of feature from an image according to the position of an object; attribute identifying means for outputting an attribute score for each attribute of the object, using the feature amount; and correction means for correcting the attribute score for at least one of the attributes of the object on the basis of the attribute score of at least another attribute of the object output by the attribute identifying means.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

Recently, in addition to identifying facial expressions and individuals, there has been active development of technology for recognizing attributes related to faces such as race, age, sex, and beard, and further attributes related to human bodies such as clothes.
For example, Non-Patent Document 1 detects positions of eyes, mouth, and nose from an image, and extracts various feature amounts such as color features of RGB and HSV, and features of an edge and a histogram system based on these positions. Keep it. Then, there is a technology for recognizing attributes related to as many as 65 types of faces such as glasses, gender, age, hair, hair and the like by inputting these feature quantities into a classifier called Support Vector Machine (hereinafter referred to as SVM).
Also, as in Non-Patent Document 2, there is a technology for recognizing attributes related to the human body such as long pants, jeans, T-shirts and the like.

N. Kumer, "Attribute and Simile Classifier for Face Verification", IEEE ICCV, 2009 L. Bourdev, "Describing People: A Poselet-Based Approach to Attribute Classification", IEEE ICCV, 2011 P. Viola, M. Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", in Proc. Of CVPR, vol. 1, pp. 511-518, December, 2001 Xudong Cao, Yichen Wei, Fang Wen, Jian Sun, "Face Alignment by Explicit Shape Regression", CVPR, pp. 2887-2894, 2012

However, there are two problems with the technology that recognizes various attributes.
First, since the number of attributes related to the face and the human body is very large, it is necessary to prepare a positive sample and a negative sample for each attribute and learn the classifier as the number of attributes is increased. is there. In other words, it takes a great deal of effort.
The second point is that the difficulty level of identification differs greatly for each attribute. For example, for eyeglasses that are attributes related to the face, an eyeglass frame or the like can be easily detected as an edge. Further, by using the face detection technology of Non-Patent Document 3 and the face feature point detection technology of Non-Patent Document 4, it is possible to accurately set an area where glasses may exist. Therefore, it can be identified with high accuracy.
On the other hand, with respect to the color of hair which is an attribute relating to the face, the color largely changes due to the influence of the white balance of the imaging device and the ambient light, so it is very difficult to identify with high accuracy.
Also, with regard to attributes related to the human body, such as long pants and T-shirts, unlike the face, upper and lower body regions can not be accurately identified, so it is very difficult to identify with high accuracy.

  The image processing apparatus according to the present invention comprises a detection unit that detects the position of an object included in an image, an extraction unit that extracts a feature quantity from the image based on the position of the object, and a plurality of the objects using the feature quantity. Attribute identification means for outputting an attribute score for each of the at least one of the plurality of attributes based on the attribute score for at least one of the plurality of attributes of the object output by the attribute identification means; And correction means for correcting the attribute score for.

  According to the present invention, the attribute of an object can be recognized with higher precision.

It is a figure showing an example of the hardware constitutions of an image processing device. It is a figure showing an example of the software composition of an image processing device. It is a figure which shows an example of a structure of a position detection part. It is a figure which shows an example of a structure of a feature extraction part. It is a figure which shows an example of a structure of an attribute identification result correction | amendment part. It is a flowchart which shows an example of the information processing in an image processing apparatus. It is a figure explaining a face feature point. It is a figure explaining a local field. It is a figure which shows the example which produces _| generates feature vector Vn from various feature quantities. It is a figure which shows an example of the score output from each attribute discriminator. It is a figure which shows an example which correct | amends the attribute score of the mustache which conflicts with the attribute of a woman. It is a flowchart which shows an example of an attribute identification result correction process. It is a figure which shows an example of the feature vector which makes an attribute score an element. It is a diagram illustrating an example of a sample S _n in all attribute space V before the correction. It is a diagram illustrating an example of a sample S _n in all attribute space V after correction. It is a figure which shows the modification of the software configuration of an image processing apparatus. It is a flowchart which shows the modification of attribute identification result correction processing.

  Hereinafter, embodiments of the present invention will be described based on the drawings.

First Embodiment
In the first embodiment, the process of correcting the attribute identification result by using the correlation between the attributes or the correlation between the imaging condition (date and time and location) and the attribute that can be acquired from other sensors such as an imaging apparatus explain.
FIG. 1 is a diagram showing an example of the hardware configuration of the image processing apparatus 10. As shown in FIG. The image processing apparatus 10 includes a CPU (Central Processing Unit) 11, a memory 12, and a communication I / F 13 as a hardware configuration. The CPU 11 controls the entire image processing apparatus 10. The memory 12 is a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD) or the like. The memory 12 stores programs, images, data used when the CPU 11 executes processing based on the programs, setting values, and the like. The communication I / F 13 is an interface for connecting the image processing apparatus 10 to a network. By executing processing based on the program stored in the memory 12 by the CPU 11, software configurations such as FIG. 2 to FIG. 5 and FIG. 17 to be described later and processing of flowcharts such as FIG. . The image processing apparatus may be an imaging apparatus or may be a PC or the like. In the case of an imaging device, in addition to the hardware configuration shown in FIG. 1, an imaging element, an AD converter, etc. are included as a hardware configuration. When the image processing device is an imaging device, processing is performed on the image taken with confidence, and when the image processing device is a PC etc., an image captured by the imaging device via a network is acquired and acquired Perform processing on the image. Further, the CPU 11 may store data, setting values, etc. in the memory 12 based on a user operation via an input device, a display device, etc., and a command transmitted from an external device etc. via the communication I / F 13 Data, setting values and the like may be stored in the memory 12 based on the like.

FIG. 2 is a view showing an example of the software configuration of the image processing apparatus 10. As shown in FIG. The image processing apparatus 10 includes an image acquisition unit 100, a position detection unit 101, a feature extraction unit 102, an attribute identification unit 103, and an attribute identification result correction unit 104 as a software configuration. The processing of each unit will be described using a flowchart and the like described later.
FIG. 3 is a diagram showing an example of the configuration of the position detection unit 101. As shown in FIG. The position detection unit 101 includes a face position detection unit 10100 and a face feature point detection unit 10101. In the first embodiment, description will be made using an example for the face, and by introducing the human body position detection technology, the face and the human body may be taken as the target. The processing of each unit will be described using a flowchart and the like described later. A face or a human body is an example of an object included in an image.
FIG. 4 is a diagram showing an example of the configuration of the feature extraction unit 102. As shown in FIG. The feature extraction unit 102 includes a normalization unit 10200, an area setting unit 10201, and a feature extraction unit 10202. The processing of each unit will be described using a flowchart and the like described later.
FIG. 5 is a diagram showing an example of the configuration of the attribute identification result correction unit 104. As shown in FIG. The attribute identification result correction unit 104 includes an attribute selection unit 10400, a partial space selection unit 10401, and an attribute score correction unit 10402. The processing of each unit will be described using a flowchart and the like described later.

FIG. 6 is a flowchart showing an example of information processing in the image processing apparatus 10.
In S10000, the image acquisition unit 100 is obtained by passing through a condensing element such as a lens, an imaging element such as a CMOS or CCD that converts light into an electric signal, and an AD converter that converts an analog signal into a digital signal. Digital image (hereinafter referred to as an image). Further, the image acquiring unit 100 may acquire an image converted to, for example, VGA (640 × 480 [pixel]) or QVGA (320 × 240 [pixel]) by performing thinning processing or the like.
In S10100, the face position detection unit 10100 detects the position where the face is present from the image acquired in S10000. If there are multiple faces in the image, the position of each face is detected. For example, as described in Non-Patent Document 3, the face position detection unit 10100 can detect a face in an image with high accuracy and at high speed by using an integral image and a cascade discriminator. It can not be
In S10101, the face position detection unit 10100 determines whether a face is detected in S10100. If the face position detection unit 10100 determines that a face is detected, the process advances to step S10102; if it is determined that a face is not detected, the process returns to step S10000. In S10000, the image acquisition unit 100 acquires the next image.
In S10102, the face position detection unit 10100 selects one face from the faces detected in S10100.

In S10103, the face feature point detection unit 10101 detects the face feature point position for the face selected in S10102. The face feature points are the corners of the back of the eye, the head of the eye, the top of the nose, the corners of the left and right mouth ends, and the parts of the nose and the mouth as in 101030 of FIG. For example, the face feature point detection unit 10101 extracts a feature amount based on the face feature point position after setting an average face feature point position as an initial arrangement in advance as in Non-Patent Document 4. Then, the face feature point detection unit 10101 has a method of moving / deforming the face feature point position based on a table indicating the relationship between the feature amount and the movement / deformation amount of the face feature point prepared in advance. However, detection of face feature points is not limited to this method.
In S10200, based on the face feature points detected in S10103, the normalization unit 10200 performs affine transformation processing so that the size of the face size is set and the orientation of the face is erected. For example, the normalization unit 10200 obtains the center of gravity of four feature points related to the eyes, eyes, upper eyelid and lower eyelid, and the distance connecting the center of gravity of the left and right eyes is 75 [pixel], and the inclination with respect to the horizontal direction is 0 Perform affine transformation to be [degree].
Next, the region setting unit 10201 performs setting of the local region 102010 based on the face feature points detected in S10103 as shown in FIG. Then, the feature extraction unit 10202 extracts various feature amounts necessary for attribute recognition, such as color features such as color histograms, edges, and histogram features such as Histogram Oriented Gradient (hereinafter referred to as HOG).

In S10300, as shown in FIG. 9, the attribute identifying unit 103 generates a feature vector V _n from the various feature amounts extracted in S 10200, and uses the feature vector V _n as an input to set frame glasses, men, women, children Identify attributes such as adults, knit hats, mustaches, beards, blondes, skin heads, wave hair, makeup, etc. The attribute discriminator calculates a score for the attribute. In addition, when the identification target is a human body, the attribute identification unit 103 may identify clothes such as outerwear and pants. The types of these identifying attributes are determined in advance manually.
The feature vector V _n is a feature vector for the _n- th face, and the elements a _nm constituting the feature vector V _n are elements of each bin of the color histogram in a certain area m, pixel values constituting the edge image, and HOG It is the concatenation of the elements of the bin.
The attribute classifier learns for each attribute. For example, n samples of male positive samples and female negative samples are prepared. The attribute discriminator extracts feature vectors V _n corresponding to these samples. It is learned by giving teacher data to the classifier so that the output value is 1 for male positive samples and the output value is -1 for female negative samples. Although a Support Vector Machine (hereinafter referred to as SVM) is used as a classifier, it is not limited thereto.

In S10400, the attribute identification result correction unit 104 corrects the score for each attribute identified in S10300. FIG. 10 is a diagram showing an example of the score output from each attribute discriminator for a certain person. Since each attribute classifier learns independently, as shown in FIG. 10, there is a contradictory case where the female score is 0.9, that is, although the possibility of being a female is high, the mustache score is 0.7. Can occur. As shown in FIG. 10 and the like, the image processing apparatus 10 has a plurality of attribute discriminators.
So, in this embodiment, the attribute score of the mustache contradictory to the female attribute as shown in FIG. 11 is corrected. That is, the attribute score correction unit 10402 selects one or more attribute identification results, trusts the selected attribute identification result to be a correct result, and corrects the other attribute scores. For example, since the attribute identification score of a woman is 0.9, which is close to 1.0, the attribute score correction unit 10402 determines that the face selected in S10103 is a woman and contradicts with a woman, such as a mustache or the like. To correct the score. The attribute to be trusted is checked beforehand by a human to evaluate the attribute with high accuracy by setting the attribute etc., and the attribute score correction unit 10402 preferentially selects this set attribute with high accuracy. You may At least one attribute score is selected from the attribute scores output from each of the plurality of attribute discriminators, and at least one attribute other than the selected attribute score among the plurality of attribute scores of the object based on the selected attribute score It is an example of the process which corrects a score.

FIG. 12 is a flowchart showing an example of the attribute identification result correction process of S10400.
In S10401, the attribute identification result correction unit 104 selects one or more attributes from the attribute identifier in S10300. As the attribute selection method, as described above, a human examines the attribute with high accuracy in advance to evaluate and set the attribute with high accuracy, and the attribute identification result correction unit 104 determines the attribute with high accuracy based on this setting. It may be selected preferentially.
The method of selecting the attribute may be a method of statistically analyzing the score itself output from the classifier and using this statistical analysis result, in addition to the method of using the evaluation result for each attribute. For example, when a certain attribute discriminator has high precision, it outputs an output value close to 1.0 for positive samples and an output value close to -1.0 for negative samples. Therefore, the attribute identification result correction unit 104 classifies the score output from each attribute device into two classes, and selects an attribute with which the degree of separation between the two classes (inter-class variance / in-class variance) is maximum. You may As described above, in the method of statistically analyzing the score output from the classifier and determining the attribute to select, it is not necessary to know what attribute each attribute identifies.

In S10402, the attribute identification result correction unit 104 determines whether the attribute score selected in S10401 is equal to or greater than the set threshold. The attribute identification result correction unit 104 proceeds to S10403 if the attribute score is equal to or greater than the set threshold, and returns to S10401 if the attribute score is less than the set threshold, and selects another attribute.
In S10403, the attribute identification result correction unit 104 selects one attribute partial space P _i from the plurality of attribute partial spaces P prepared in advance, based on the one or more attribute identification results selected in S10401.
Here, the attribute subspace P will be described.
When the scores output from the first to i-th attribute classifiers for one person are S ₁ , S ₂ ,..., S _ni and each score is considered as one element of the feature vector, FIG. Thus, for the n-th person, S _n = [S _n1 , S _n2 ,..., S _ni ]. Here, the S _n, the space to define vectors each attribute (hereinafter, referred to as total attribute space V) taken as one point.
If a part of the attribute extending the entire attribute space V is considered to be an attribute correlated with the attribute of a woman, it is possible to define a hyperplane in which only a woman can exist as shown in FIG. Women score 0.9 described above, the mustache score to the sample and S _n of 0.7, the sample S _n is considered to have deviated from the true scope of which only women can exist as shown in Figure 14. Therefore, in the present embodiment, as shown in FIG. 15, the attribute identification result correction unit 104 determines that the sample S _n which is out of the true range where only women can exist is true that only women can exist on the entire attribute space V. Correct the attribute score by moving to the range of. As a method of moving the sample S _n on the entire attribute space V, the attribute identification result correction unit 104 uses the attribute subspace P. The attribute identification result correction unit 104 obtains a partial attribute space P having a dimension lower than that of the entire attribute space V from the sample distribution scattered in the entire attribute space V, and projects the input sample S _n onto this partial attribute space P. . Assuming that the input sample projected to the partial attribute space P is S _n ^′ , the attribute identification result correction unit 104 backprojects the input sample S _n ^′ projected to the partial attribute space P back to the original entire attribute space V. , Move the sample S _n on the entire attribute space V. Here, there is principal component analysis (hereinafter referred to as PCA) as one method of obtaining a basis vector of the partial attribute space P. For example, a large sample only for women is prepared, and PCA is performed on the large sample to obtain an eigenvector U = [u1, u2,..., Uk] using the sample variation as an index. The attribute identification result correction unit 104 determines that the eigenvector U = [u1, u2,..., Ui] having a small eigen value (variation) is an eigenvector specific to a female sample, and the eigenvector having a large eigen value Do. Then, the attribute identification result correction unit 104 configures the partial attribute space P with eigenvectors U = [u1, u2,..., Ui] having small eigenvalues (variations).
Assuming that an input sample on the partial attribute space P is S _n ^{′ ′} , it can be expressed as S _n ^{′ ′} = [u 1, u 2,..., Uk] S _n ^′ . Further, since the transformation from the total attribute space V to the partial attribute space P is a linear projection, the input sample S _n ^{′ ′} in the partial attribute space P can be backprojected onto the original total attribute space V. The method of obtaining the subspace P may not be PCA. The attribute identification result correction unit 104 may use another linear projection such as a locality preserving projection (LPP) that brings closer samples closer to each other on the entire attribute space V.
In S10403, the attribute identification result correction unit 104 projects the input sample S _n ^′ onto the partial attribute space P selected in S10402, as described above, and then performs reverse projection on the entire attribute space V.
In S10403, the attribute identification result correction unit 104 projects the input sample S _n ^′ onto the partial attribute space P selected in S10402, as described above, and then performs reverse projection on the entire attribute space V.

In the example described above, the method of selecting the partial space P based on the score of the attribute discriminator has been described, but the attribute discrimination result correction unit 104 is not an attribute, but other sensors such as date and time when the image was taken You may make it utilize the imaging condition information obtained as information. In other words, there are no people wearing knit hats in the summer season, no children in the late-night hours, no people wearing helmets in elementary schools, etc. The attribute identification result correction unit 104 You may use correlation with this. FIG. 16 is a view showing a modified example of the software configuration of the image processing apparatus 10. As shown in FIG. In the software configuration of FIG. 16, an information acquisition unit 105 is added to the software configuration of FIG. 2. FIG. 17 is a flowchart showing a modification of the attribute identification result correction process of S10400.
In S10411, the information acquisition unit 105 acquires date and time information and location information that can be acquired from an imaging device that can communicate via a network or the like. Date and time information is an example of date and time information related to an image. The place information is an example of the place information on the image.
In S10412, the attribute identification result correction unit 104 performs partial space selection based on the information on the date and time and the place acquired in S10411. For example, a partial space specific to a time zone such as midsummer or midnight, a partial space specific to a place such as an elementary school, etc. are prepared, and the attribute identification result correction unit 104 is acquired from these plural partial spaces in S10411. Subspace selection based on date and time and location information.

Thus, in the first embodiment, the image processing apparatus 10
1. At least one or more attribute discriminators are selected from the attribute discriminators to refer to the score. 2. Select a partial space P prepared in advance based on the score of the selected attribute discriminator. After projecting from the entire attribute space V to the selected subspace P, the process of correcting the attribute score by back-projecting the subspace P to the entire attribute space V is performed. This makes it possible to realize a more accurate attribute discriminator.

<Other Embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium. And, it is also possible to realize the processing in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

As mentioned above, although an example of an embodiment of the present invention was explained in full detail, the present invention is not limited to such a specific embodiment.
For example, as a hardware configuration of the image processing apparatus 10, a plurality of CPUs may exist, and the plurality of CPUs may execute processing based on a program stored in a memory or the like. Further, as a hardware configuration of the image processing apparatus 10, a GPU (Graphics Processing Unit) may be used instead of the CPU.
Further, part of the software configuration of the image processing apparatus 10 may be implemented as a hardware configuration in the image processing apparatus 10.
Further, the image processing apparatus 10 corrects the attribute identification result group by selecting a subspace based on the result of one or more attribute discriminators and the date / time information and / or the location information, and performing projection and back projection. You may do so.

  According to each embodiment described above, the attribute can be recognized with higher accuracy.

10 image processing device 11 CPU
12 Memory 13 Communication I / F

Claims (14)

A detection unit that detects a position of an object included in an image; and an extraction unit that extracts a feature amount from the image based on the position of the object.
Attribute identification means for outputting an attribute score for each of a plurality of attributes of the object using the feature amount;
Correction means for correcting an attribute score for at least one of the plurality of attributes based on the attribute score for at least one of the plurality of attributes of the object output by the attribute identification means;
An image processing apparatus having: Having a plurality of the attribute identification means,
The correction means selects at least one attribute score from the attribute scores output from each of the plurality of attribute identification means, and the selected one of the plurality of attribute scores of the object is selected based on the selected attribute score. The image processing apparatus according to claim 1, wherein at least one attribute score other than the attribute score is corrected.   The image processing apparatus according to claim 2, wherein the correction unit selects at least one attribute score of the attribute scores output by the plurality of attribute identification units based on settings.   The correction means according to claim 2, wherein the correction means statistically analyzes the attribute scores outputted by the plurality of attribute identification means, and selects at least one attribute score of the attribute scores outputted by the plurality of attribute identification means. Image processing device. Acquisition means for acquiring shooting status information regarding the image;
A detection unit that detects a position of an object included in an image; and an extraction unit that extracts a feature amount from the image based on the position of the object.
Attribute identification means for outputting an attribute score for at least one attribute of the object using the feature amount;
A correction unit that corrects the attribute score on the basis of the shooting status information acquired by the acquisition unit;
An image processing apparatus having:   The image processing apparatus according to claim 5, wherein the acquisition unit acquires date and time information indicating a date and time when the image is captured as the shooting status information.   The image processing apparatus according to claim 5, wherein the acquisition unit acquires location information representing a location at which the image is captured as the capturing status information.   The image processing apparatus according to any one of claims 5 to 7, comprising a plurality of the attribute identification means.   The image processing apparatus according to any one of claims 1 to 8, wherein the object is a face included in an image.   The image processing apparatus according to any one of claims 1 to 8, wherein the object is a human body included in an image.   The image processing apparatus according to any one of claims 1 to 9, wherein the image processing apparatus is an imaging apparatus. A detection step of detecting a position of an object included in an image; and an extraction step of extracting a feature amount from the image based on the position of the object.
An attribute identification step of outputting an attribute score for each of a plurality of attributes of the object using the feature amount;
Correcting the attribute score for at least one of the plurality of attributes based on the attribute score for at least one of the plurality of attributes of the object output by the attribute identification step;
Image processing method including: An acquisition step of acquiring shooting status information regarding an image;
A detection step of detecting a position of an object included in an image; and an extraction step of extracting a feature amount from the image based on the position of the object.
An attribute identification step of outputting an attribute score for at least one attribute of the object using the feature amount;
A correction step of correcting the attribute score based on the photographing situation information acquired by the acquisition step;
Image processing method including:   A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 10.

Images