JP2012053813A - Person attribute estimation device, person attribute estimation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a person attribute estimation device and the like capable of accurately estimating person attributes from a face image.SOLUTION: A wrinkle feature amount, a spot feature amount and a lip feature amount in a face area are calculated as feature amounts for estimating person attributes. The spot feature amount is calculated for each block using a difference between an image after mask processing with a skin mask and a blurred image obtained by further performing smoothing processing thereof. As the wrinkle feature amount, the intensity and direction of a wrinkle are calculated for each block from the image after mask processing with the skin mask. The lip feature amount is calculated for a lip area extracted using the skin mask as a color comparison value with respect to a reference color in the original face area. By using the spot feature amount, the wrinkle feature amount and the lip feature amount calculated in this way, person attributes such as age and sex can be accurately estimated, and robustness against environmental variation can be improved.

Description

  The present invention relates to a person attribute estimation device and the like for estimating attributes such as sex and age of a person.

In recent years, information distribution using digital signage has attracted attention as an advertising method in addition to conventional media such as outdoor signboards, televisions, newspapers, and magazines.
Digital signage is a medium that distributes information using a display device such as a display in places such as outdoors, stores, and transportation facilities. By utilizing digital signage, it is possible to provide expressive content such as video and audio, as well as high information distribution so that flexible program creation, timely information presentation through networking, and narrowing down of targets are possible. Efficiency is expected.
In addition, attributes (age and gender) of the person browsing the advertisement are frequently used as an index for measuring the effectiveness of the advertisement. For example, it is possible to apply the advertisement viewer's personal attributes to marketing indicators, or to switch the content to be broadcast on digital signage in a timely manner according to the acquired personal attributes of the advertisement viewer. Content delivery with high merit for both parties.

  By the way, as a conventional person attribute estimation method, a method shown in Non-Patent Document 1 has been proposed. In the method shown in Non-Patent Document 1, since the spots and wrinkles appearing on the facial skin are important features representing the age after adolescence, the gender and age are estimated from the face image. Skin texture features and hue features are taken into account. That is, large wrinkles are detected using a Sobel filter on a face image, small wrinkles are detected using a Gabor jet, and the average and deviation of edge strengths of these wrinkles are defined as wrinkle features. As for stains, an average value filter is used for the original image to generate a blurred image, and contours and wrinkles such as facial organs are removed from the difference image between the original image and the blurred image by threshold processing, and the stain is extracted. Yes. Further, as hue characteristics, color information in the lip area and cheek area is characterized by the average and deviation of the hue and saturation and the redness component a * of the L * a * b color system among the stimulation values of the modified HSV color system. It is defined as a quantity.

Hironori Takimoto and three others, “Gender age estimation from face images not affected by posture changes”, Denki Theory C, Vol. 127, No. 7, 2007, p.1022-p.1029

  However, in the method of Non-Patent Document 1, since a stain is extracted from the difference between the original image and the blurred image, the contours of the face, eyes, nose, and mouth may also be extracted as skin texture (stains and wrinkles). There was room for improvement. Further, the above-described hue feature is extracted only in the lip region and the cheek region, and the average for each color space (L * a * b, HSV) is used as a feature amount. There is concern about robustness to

  The present invention has been made in view of such problems, and the object of the present invention is to provide a person attribute estimation device capable of accurately estimating a person attribute from a face image even if there is some environmental fluctuation. It is to provide a person attribute estimation method and program.

  In order to solve the above-described problem, the first invention is a face detection unit that detects a human face region from input image data, and a skin mask generation unit that generates a skin mask indicating a skin region of the face region. , Using the difference between the masked image obtained by extracting the skin part of the original face region using the skin mask and the blurred image obtained by further smoothing the image after masking, and extracting the spot on the skin part. A wrinkle feature amount calculating means for calculating an amount, and a wrinkle feature amount calculating means for calculating the wrinkle intensity and direction as a wrinkle feature amount from an image after mask processing in which the skin portion of the original face region is extracted using the skin mask. Lip feature amount calculating means for calculating, as lip feature amounts, a color comparison value with respect to a reference color in the original face region for the lip region extracted using the skin mask, and a plurality of input learning human images Calculated from Wrinkle feature value, the stain feature value, the lip feature value, attribute learning means for creating person attribute learning data based on the attribute information about the person image, and the stain feature value for the input person image And a estimator for calculating the wrinkle feature amount and the lip feature amount, and estimating a person attribute of the person image based on the calculated feature amount and the person attribute learning data. It is an attribute estimation device.

  According to the first aspect of the present invention, a face area is detected from a plurality of person images for learning, and a post-mask processing image obtained by performing mask processing using a skin mask is used. The lip feature amount calculating means calculates the wrinkle feature amount, the stain feature amount, and the lip feature amount, respectively, and creates person attribute learning data. Further, the stain feature amount, the wrinkle feature amount, and the lip feature amount are calculated for the input person image, and the person attribute of the person image is estimated based on the calculated feature amount and the person attribute learning data. . Since the spot feature amount is calculated using the difference between the image after masking with the skin mask and the blurred image obtained by further smoothing the spot feature, the spot feature is less affected by the face contour, eyes, nose, mouth, glasses, etc. The amount can be determined. Further, since the wrinkle feature value is obtained in consideration of the strength and direction of the wrinkle, the wrinkle feature value indicating an age difference or a sex difference can be calculated with high accuracy. In addition, since the lip feature value is calculated as a color comparison value with respect to the reference color in the original face region for the lip region extracted using the skin mask, the lip feature amount is a feature that is highly robust to environmental fluctuations in the image. Extraction is possible. By using the spot feature amount, the wrinkle feature amount, and the lip feature amount calculated in this way, it is possible to accurately estimate a person attribute such as age and sex.

  In the first invention, the wrinkle feature amount calculating unit includes a gradient calculating unit that calculates a gradient intensity and a gradient direction for each pixel of the image after mask processing, and a gradient intensity and gradient calculated by the gradient calculating unit. A gradient histogram generating unit that generates a gradient histogram indicating a cumulative gradient intensity with respect to the gradient direction for each block obtained by dividing the face area, and a gradient gradient maximum value of the gradient histogram for each block. Is preferably a wrinkle feature amount.

In the present invention, paying attention to the fact that the direction in which wrinkles appear tends to vary depending on the age, the wrinkle feature value is obtained from the strength and direction of the wrinkles as described above. Specifically, the wrinkle feature amount is defined as the maximum value of the gradient strength (wrinkle strength) of the gradient histogram. The gradient histogram is obtained from the gradient intensity (wrinkle intensity) and gradient direction (wrinkle direction) of each pixel of the image after mask processing. That is, in the present invention, the largest wrinkle intensity (gradient intensity) accumulated in each wrinkle direction (gradient direction) for each block is the wrinkle feature amount in that block. Since the wrinkle feature amount obtained in this way is used for age estimation or the like, it can greatly contribute to improvement in accuracy of age estimation as compared with the conventional case.
On the other hand, in the conventional method, for example, according to the method described in Non-Patent Document 1 described above, the wrinkle feature amount is determined according to ARSM (feature point determination method proposed by Non-Patent Document 1). It is defined as the average and deviation of the edge strength in the wrinkle feature amount extraction area (forehead, eye corner, age line, both ends of the lip area, chin tip). Therefore, the conventional method is different from the present invention in that the direction of wrinkles is not taken into consideration and the gradient histogram is not used.
Further, the conventional method uses the color information in the lip region and cheek region for the hue feature, and includes the hue and saturation of the stimulation value of the modified HSV color system and the redness component a * of the L * a * b color system. Mean and deviation are defined as feature quantities. On the other hand, in the present invention, since the lip feature amount is calculated as a color comparison value with respect to the reference color in the original face area determined for each image captured under various shooting environments, And robust feature extraction. By using the spot feature amount, the wrinkle feature amount, and the lip feature amount calculated in this way, it is possible to accurately estimate the person attributes such as age and sex as compared with the conventional case.

Also, in the first invention, the spot feature value calculating means generates the blurred image using an MTM filter for the image after mask processing, and the image after mask processing for each block obtained by dividing the face area. It is desirable to calculate an average value of the difference between the image and the blurred image and use it as a spot feature amount.
By using the MTM filter, it is possible to exclude large spots, moles, and the like that have a large difference in pixel values from the surroundings from the subject of blurring processing, and to exclude them from the subject of spot feature amount consideration. For this reason, the feature amount can be extracted only for a stain having a high correlation with age and the like.

In the first invention, the lip feature value calculating means extracts a lip region from the face region using a reversal mask obtained by reversing the skin mask, and a reference in the lip region and the original face region. It is desirable to generate a histogram with the hue H, saturation S, and redness R as the target color space for each region, and calculate the distance between these histograms as the lip feature amount.
Since the lip feature amount is obtained from the difference (histogram distance) between the original face region and the reference region, feature extraction that is robust against environmental changes such as illumination intensity can be performed.

  According to a second aspect of the present invention, there is provided face detection means for detecting a human face area from input image data, skin mask generation means for generating a skin mask indicating a skin area of the face area, and the skin mask. A spot feature that extracts the spot of the skin using the difference between the masked image obtained by extracting the skin of the original face area and the blurred image obtained by further smoothing the image after the mask processing, and calculates the spot feature A calculating means; a wrinkle feature amount calculating means for calculating the wrinkle intensity and direction as a wrinkle feature amount from the mask-processed image obtained by extracting the skin portion of the original face area using the skin mask; and the skin mask. For the extracted lip region, lip feature amount calculating means for calculating a color comparison value for the reference color in the original face region as a lip feature amount, the wrinkle feature amount calculated from a plurality of learning person images, Stain feature, and Memory means for storing person attribute learning data created by a predetermined learning algorithm based on the lip feature quantity and attribute information given for the person image, the stain feature quantity for the input person image, Human attribute estimation, comprising: an estimation means for calculating a wrinkle feature amount and the lip feature amount, and estimating a person attribute of the person image based on the calculated feature amount and the person attribute learning data Device.

  According to the second invention, the wrinkle feature amount, the stain feature amount, and the lip feature amount of the plurality of learning person images and the attribute information given to the person image are created by a predetermined learning algorithm. Person attribute learning data is stored in the memory means, the spot feature amount, the wrinkle feature amount, and the lip feature amount are calculated for the input person image, and the calculated feature amount and the person attribute learning data are used. Based on this, the person attribute of the person image is estimated. Since the spot feature amount is calculated using the difference between the image after masking with the skin mask and the blurred image obtained by further smoothing the spot feature, the spot feature is less affected by the face contour, eyes, nose, mouth, glasses, etc. The amount can be determined. Further, since the wrinkle feature value is obtained in consideration of the strength and direction of the wrinkle, the wrinkle feature value indicating an age difference or a sex difference can be calculated with high accuracy. In addition, since the lip feature value is calculated as a color comparison value with respect to the reference color in the original face region for the lip region extracted using the skin mask, the lip feature amount is a feature that is highly robust to environmental fluctuations in the image. Extraction is possible. Based on the person attribute learning data created in advance using the spot feature amount, the wrinkle feature amount, and the lip feature amount calculated in this way, the person attributes such as age and gender are set for the input person image. It can be estimated accurately.

In the second invention, it is desirable to further include an imaging unit that captures an image, and a person image input unit that inputs an image captured by the imaging unit as a person attribute estimation target in the estimation unit.
As a result, an image captured by the imaging unit can be input as a person attribute estimation target. For example, if a viewer of digital signage is imaged by an imaging means, and an image of a person included in the image is input as a person attribute estimation target, the person attribute estimation result of the present invention is used to measure the effect of digital signage. It can be used as a device.

  A third invention uses a face detection step for detecting a human face area from input image data, a skin mask generation step for generating a skin mask indicating a skin area of the face area, and the skin mask. A spot feature that extracts the spot of the skin using the difference between the masked image obtained by extracting the skin of the original face area and the blurred image obtained by further smoothing the image after the mask processing, and calculates the spot feature A wrinkle feature amount calculating step for calculating a wrinkle intensity and direction as a wrinkle feature amount from a mask-processed image obtained by extracting a skin portion from an original face area using the skin mask; and using the skin mask. For the extracted lip region, a lip feature amount calculating step for calculating a color comparison value with respect to the reference color in the original face region as a lip feature amount, and the lip feature amount calculated from a plurality of input learning human images An attribute learning step for creating person attribute learning data based on the feature value, the spot feature quantity, the lip feature quantity, and attribute information about the person image, and the spot feature quantity about the input person image, The computer executes a process including calculating the wrinkle feature value and the lip feature value, and estimating a person attribute of the person image based on the calculated feature value and the person attribute learning data. Is a person attribute estimation method characterized by

  According to the third invention, a wrinkle feature amount, a stain feature amount, and a lip feature amount are respectively detected using a mask-processed image obtained by detecting a face region of a plurality of learning person images and performing a mask process using a skin mask. And person attribute learning data is created. Further, the stain feature amount, the wrinkle feature amount, and the lip feature amount are calculated for the input person image, and the person attribute of the person image is estimated based on the calculated feature amount and the person attribute learning data. . Since the spot feature amount is calculated using the difference between the image after masking with the skin mask and the blurred image obtained by further smoothing the spot feature, the spot feature is less affected by the face contour, eyes, nose, mouth, glasses, etc. The amount can be determined. Further, since the wrinkle feature value is obtained in consideration of the strength and direction of the wrinkle, the wrinkle feature value indicating an age difference or a sex difference can be calculated with high accuracy. In addition, since the lip feature value is calculated as a color comparison value with respect to the reference color in the original face region for the lip region extracted using the skin mask, the lip feature amount is a feature that is highly robust to environmental fluctuations in the image. Extraction is possible. By using the spot feature amount, the wrinkle feature amount, and the lip feature amount calculated in this way, it is possible to accurately estimate a person attribute such as age and sex.

In the third invention, the wrinkle feature amount calculating step includes a gradient calculating step for calculating a gradient strength and a gradient direction for each pixel of the image after mask processing, and a gradient based on the calculated gradient strength and gradient direction. A gradient histogram generation step for generating a gradient histogram indicating a cumulative gradient intensity with respect to a direction for each block obtained by dividing the face area, and the maximum gradient intensity of the gradient histogram for each block is a wrinkle feature amount. It is desirable to do.
Thus, in the person attribute estimation method of the present invention, focusing on the fact that the direction of wrinkles tends to vary depending on the age, the wrinkle feature amount is defined as the maximum value of the gradient strength of the gradient histogram. The gradient histogram is obtained from the gradient intensity and gradient direction of each pixel of the image after mask processing. That is, in the present invention, the largest wrinkle intensity (gradient intensity) accumulated in each wrinkle direction (gradient direction) for each block is the wrinkle feature amount in that block. Since the wrinkle feature amount obtained in this way is used for age estimation or the like, it can greatly contribute to improvement in accuracy of age estimation as compared with the conventional case.

  A fourth aspect of the invention is a program described in a computer-readable format, a face detecting step for detecting a human face area from input image data, and a skin mask indicating a skin area of the face area A skin mask using a difference between a skin mask generation step that generates a skin mask, an image after mask processing in which the skin portion of the original face area is extracted using the skin mask, and a blurred image obtained by further smoothing the image after mask processing A wrinkle feature amount calculating step for extracting a wrinkle feature amount, and a wrinkle intensity and direction as a wrinkle feature amount from an image after mask processing in which the skin portion is extracted from the original face region using the skin mask. A wrinkle feature amount calculating step for calculating and a lip feature for calculating a lip feature value by comparing a color comparison value with respect to a reference color in the original face region for the lip region extracted using the skin mask. Personal attribute learning data based on a quantity calculating step, the wrinkle feature quantity, the spot feature quantity, the lip feature quantity calculated from a plurality of input learning person images, and attribute information about the person image An attribute learning step for generating the image, calculating the spot feature amount, the wrinkle feature amount, and the lip feature amount for the input person image, and the person image based on the calculated feature amount and the person attribute learning data A program for causing a computer to execute a process including an estimation step of estimating a person attribute of the computer.

  In the fourth invention, the wrinkle feature amount calculating step includes a gradient calculating step for calculating a gradient strength and a gradient direction for each pixel of the image after mask processing, and a gradient based on the calculated gradient strength and gradient direction. A gradient histogram generation step for generating a gradient histogram indicating a cumulative gradient intensity with respect to a direction for each block obtained by dividing the face area, and the maximum gradient intensity of the gradient histogram for each block is a wrinkle feature amount. It is desirable to do.

  According to the fourth invention, it is possible to cause a computer to function as the person attribute estimation device of the first invention.

  According to the present invention, it is possible to provide a person attribute estimation device, a person attribute estimation method, and a program capable of accurately estimating a person attribute from a face image even if there is some environmental variation.

1 is an overall configuration diagram of a person attribute estimation system 1 according to the present invention. Functional block diagram of person attribute estimation system 1 The block diagram which shows the internal structure of the person attribute estimation apparatus (computer) 2 Flow chart explaining the flow of person attribute estimation processing Diagram explaining detection of face area and creation of skin mask The figure explaining the extraction process of a stain feature-value Example of spot image 65 An example of facial area division The figure explaining the extraction process of a wrinkle feature-value The figure explaining the gradient histogram 69 The figure explaining the extraction process of lip feature-value An example of the feature amount table 81

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.
First, the configuration of the person attribute estimation device according to the present invention will be described with reference to FIGS.

  FIG. 1 is a diagram showing a hardware configuration of a person attribute estimation system 1 using a person attribute estimation apparatus according to the present invention, FIG. 2 is a functional block diagram of the person attribute estimation system 1, and FIG. 3 is a person attribute estimation. 3 is a block diagram showing an internal configuration of a computer that functions as the device 2. FIG.

  As shown in FIG. 1, in the present embodiment, a person attribute estimation system 1 includes a computer that functions as a person attribute estimation device 2, an imaging device 3, and a display device 4.

  The display device 4 is a display device used for digital signage (electronic signage) or the like, and may be a projection device such as a display device or a projector, or other display medium.

  The imaging device 3 is a camera or the like that images the persons 10a, 10b, 10c, 10d,... Browsing the display device 4, and outputs the captured person image to the person attribute estimation device 2.

The person attribute estimation device 2 is configured by a computer, for example, and has an attribute learning function shown in FIG. 2A and an attribute estimation function shown in FIG.
The attribute learning function is a function for generating the person attribute learning data 10 based on the person image for learning and the attribute information (age, gender, etc.) of the person. As shown in FIG. An extraction unit 8 and an attribute learning unit 9 are provided.
The attribute estimation function determines the attribute of the input processing target person image from the person image to be processed and the person attribute learning data 10 obtained by the attribute learning function of FIG. This is a function for outputting as a determination result, and includes a feature extraction unit 8 and an attribute estimation unit 11.

  In the following embodiments, the human attribute estimation device 1 having both the attribute learning function and the attribute estimation function described above will be described. However, the personal attribute learning apparatus created by another computer or the like does not have its own attribute learning function. A person attribute estimation device including a memory (RAM, storage unit) that captures and stores data and the above-described attribute estimation function (feature extraction unit 8 and attribute estimation unit 11) is also included in the present invention.

  The feature extraction unit 8 detects a face region from the input person image, creates a skin mask, and extracts a stain feature value, a wrinkle feature value, and a lip feature value. Details of extraction of each feature amount in the feature extraction unit 8 will be described later.

  Based on the wrinkle feature amount, the stain feature amount, and the lip feature amount of each person image extracted by the feature extraction unit 8 and the attributes (age, sex, etc.) of those persons, the attribute learning unit 9 Learning data 10 is generated.

The attribute estimation unit 11 is based on the wrinkle feature amount, the stain feature amount, the lip feature amount extracted from the person image input from the imaging device 3 by the feature extraction unit 8, and the person attribute learning data 10. Estimate image attributes (such as age and gender).
In this embodiment, it is preferable to use SVM (Support Vector Machine) for attribute learning and attribute estimation. By using SVM, it is possible to set a non-linear boundary for identifying an attribute from a high-dimensional feature vector (support vector) having each of the above-described feature amounts as a component. Since SVM is a known technique, detailed description thereof is omitted, but for example, detailed in “Koji Tsuda: What is a support vector machine, Journal of IEICE, 83, pp. 460-466 (2000)”. . Note that the learning algorithm is not limited to the above-described SVM, and other algorithms may be applied.

  FIG. 3 is a block diagram showing an internal configuration of the person attribute estimation device 2. As shown in FIG. 3, the person attribute estimation device 2 includes a control unit 21, a storage unit 22, a media input / output unit 23, an input unit 24, a display unit 25, a communication unit 26, a peripheral device I / F unit 27, and the like. 28 is connected and configured. The imaging device 3 is connected to the peripheral device I / F unit 27.

The control unit 21 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.
The CPU calls and executes a program stored in the storage unit 22, ROM, recording medium or the like to a work memory area on the RAM, and drives and controls each unit connected via the bus 28. The ROM permanently holds a computer boot program, a program such as BIOS, data, and the like. The RAM temporarily holds the loaded program and data, and includes a work area used by the control unit 21 for performing various processes.

  In the present embodiment, the control unit 21 (CPU) calls and executes an attribute estimation processing program stored in the storage unit 22. The attribute estimation process will be described later (see FIG. 4).

  The storage unit 22 is an HDD (hard disk drive), and stores a program executed by the control unit 21, data necessary for program execution, an OS (operating system), and the like. These program codes are read by the control unit 21 as necessary, transferred to the RAM, and read and executed by the CPU.

  The media input / output unit 23 is a media input / output device such as an HD (hard disk) drive, a floppy (registered trademark) disk drive, a memory card drive, a PD drive, a CD drive, a DVD drive, or an MO drive. Output.

The input unit 24 is an input device such as a keyboard, a pointing device such as a mouse, or a numeric keypad, and outputs input data to the control unit 21.
The display unit 25 includes, for example, a display device such as a liquid crystal panel or a CRT monitor, and a logic circuit (video adapter or the like) for executing display processing in cooperation with the display device, and is input under the control of the control unit 21. Display information is displayed on a display device.
The communication unit 26 includes a communication control device, a communication port, and the like, and is a communication interface that mediates communication with the network, and performs communication control.

The peripheral device I / F (interface) unit 27 is a port for connecting a peripheral device to the person attribute estimation device 2, and the person attribute estimation device 2 communicates data with the peripheral device via the peripheral device I / F unit 27. Send and receive. The peripheral device I / F unit 27 is configured by USB, IEEE 1394, RS-232C, or the like. The imaging device 3 is connected to the peripheral device I / F unit 27. An image captured by the imaging device 3 is input to the control unit 21 as a person attribute estimation target. That is, the peripheral device I / F unit 27 functions as a person image input unit.
The bus 28 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

Next, the operation of the person attribute estimation device 2 will be described with reference to FIGS.
The control unit 21 of the person attribute estimation device 2 reads the attribute estimation processing program from the storage unit 22 and executes processing according to the following procedure.

First, the attribute learning function will be described.
As shown in FIGS. 4 and 5, image data for learning is input to the person attribute estimation device 2 (step S1). The input image data is preferably an image in which the entire face is photographed from the front, such as an ID photo.

  The control unit 21 detects a human face area 61 from the input image 6 (step S2). When the image data is moving image data, the face area 61 is detected for each frame, and the following processing is performed on each detected face area 61 as a processing target.

  There are various methods for face detection. For example, a method using Haar-like features may be used. The object detection method using the Haar-like feature is characterized by performing high-speed identification based on a light / dark pattern of an object to be detected, and is used for detecting a face in a still image or a moving image. In particular, high detection accuracy is exhibited for a frontal, hatless person image such as an ID photo. A face detection result using the Haar-like feature is represented as an area including local information of the face such as eyes, nose, and mouth, and information such as the center and size of the face. Note that the principle of object detection based on the Haar-like feature, an application example to face detection, and the like are introduced in Non-Patent Documents 2 and 3 below.

<Non-Patent Document 2> Paul Viola and Michel
J. Jones: "Rapid Object Detection Using a Boosted Cascade of Simple Features",
Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, pp.511-518, (2001)
<Non-Patent Document 3> Rainer Lienhart and
Jochen Maydt: "An Extended Set of Haar-likeFeatures for Rapid Object
Detection ", Proceedings of the 2002 IEEE International Conference on Image
Processing, Vol.1, pp.900-903, (2002)

  In addition, a method for identifying a face from an image using a Haar-like feature and acquiring the size and position of the face has already been provided as an image processing library (OpenCV library developed and released by Intel). These can be used to guide the face area 61.

  When the face area 61 is specified in this way, the control unit 21 generates a skin mask 61 (step S3). In generating the skin mask, the control unit 21 uses, for example, the average color of each pixel in the nose part (the frame area 61a of the face area 61) in the face area 61 as a reference color, and the color of each pixel in the face area 61 and the above reference. The Mahalanobis distance from the color is calculated, and a portion above a certain threshold is extracted as a skin region. A binarized image in which the extracted skin region pixels are represented by “1” and the other pixels are represented by “0” is defined as a skin mask 62. An image obtained by masking the original face area 61 detected in step S2 using the skin mask 62 is defined as an image 63 after mask processing. Since the method for extracting a person's skin region and the method for creating a skin mask are disclosed in Japanese Patent Application Laid-Open No. 2009-59209 by the same applicant as the present application, detailed description thereof will be omitted.

Next, the control unit 21 extracts a spot feature amount by using the mask-processed image 63 (step S4). The post-mask processing image 63 to be processed is an image expressed in gray scale.
In the process of calculating the spot feature amount in step S4, the control unit 21 further smoothes the masked image 63 obtained by extracting the skin part of the original face region 61 using the skin mask 62 and the masked image 63. The spot image 65 of the skin part is extracted using the difference from the blurred image 64, and the spot feature amount is calculated.

First, the control unit 21 performs filter processing for smoothing the post-mask processing image 63. For this filtering process, it is preferable to use an MTM filter (step S41).
The MTM filter is a kind of averaging filter, but when the difference between the target pixel and the neighboring pixel is small, the correlation between the target pixel and the neighboring pixel is assumed to be strong, and this neighboring pixel is taken into the calculation target, and conversely the difference Is large, the filter effect is focused on the small amplitude signal by excluding it from the calculation target. Since the detailed explanation of the principle of the MTM filter is described in Japanese Patent Application Laid-Open No. 2010-66943 by the same applicant as the present application, the explanation is omitted here.
By using the MTM filter, it is possible to exclude large spots, moles, and the like that have a large difference in pixel values from the surroundings from the subject of blurring processing, and to exclude them from the subject of spot feature amount consideration.

  FIG. 6 shows an example of the image 63 after mask processing and the blurred image 64 after MTM filter processing. FIG. 6 illustrates only a part of the face area. As shown in FIG. 6, the moles around the nose are not blurred, but the darkening of the chin from the cheeks is blurred and smoothed.

  After the MTM filter processing, the control unit 21 takes the difference between the image 63 after mask processing and the image 64 after MTM filter processing (step S42). When the difference between the image 63 after the mask processing and the image 64 after the MTM filter processing is taken, only a stain (a darkened portion of the skin, fine unevenness, etc.) can be extracted from the face region as shown in FIG. Mole that is not related to aging is not extracted as a spot. Hereinafter, the image from which the stain is extracted is referred to as a stain image 65. In FIG. 7, the part where the pixel value is large and displayed in white indicates that the spot is strong.

  For example, as shown in FIG. 8, the control unit 21 divides the stain image 65 obtained in step S42 into 5 blocks vertically and horizontally, obtains an average value of pixel values for each divided block, and uses it as a stain feature amount ( Step S43). In this way, the stain feature amount for 25 blocks is obtained. The control unit 21 stores the spot feature amount extracted from each block in the RAM. Note that the number of divided blocks is not limited to 5 × 5, and may be changed to an arbitrary number of blocks. However, spots where spots appear are characteristics representing sex and age, so it is desirable to divide them into divisions that do not subdivide and that can identify facial parts such as cheeks, forehead, and nose.

Further, the control unit 21 extracts a wrinkle feature amount considering the strength and direction of the wrinkle using the image 63 after the mask process (step S5).
In the process of calculating the wrinkle feature amount in step S5, first, as shown in FIG. 9, the control unit 21 calculates the gradient strength and gradient direction for each pixel using the image 63 after mask processing. The gradient strength indicates the strength of wrinkles, and the gradient direction indicates the direction of wrinkles. The post-mask processing image 63 to be processed is an image expressed in gray scale.
The control unit 21 multiplies each pixel of the post-mask processing image 63 by a gradient extraction filter (step S51). A Sobel filter is suitable as the gradient extraction filter.

  The Sobel filter is a filter used for edge detection in image processing. The Sobel filter multiplies each pixel value of the Sobel filter shown in FIG. By summing the multiplication results in the X direction (horizontal direction) or the Y direction (vertical direction), the gradient in the X direction (value of Δx) and the gradient in the Y direction (value of Δy) are obtained. The gradient strength can be calculated using the following equation (1), and the gradient direction can be calculated using the following equation (2).

  Based on the gradient intensity and gradient direction of each pixel, the control unit 21 creates a gradient histogram 69 (see FIG. 10) with the X axis as the gradient direction and the Y axis as the gradient intensity. The gradient histogram 69 is created for each divided block obtained by dividing the face area into 5 blocks vertically and horizontally, similarly to the case of extracting the spot feature amount (step S52).

In FIG. 10, the horizontal axis of the gradient histogram 69 indicates the gradient direction. The bin width of the gradient histogram is 10 degrees, and it is desirable to accumulate the gradient intensities for a total of 36 bins (360 degrees). As a result, the direction of the wrinkle that is the strongest or most marked in each block of the face area is extracted. The gradient strength also represents the wrinkle amount itself, which is suitable for age estimation. In the younger age group, the intensity value of the gradient histogram 69 tends to be smaller overall, and in the older age group, the intensity value of the gradient histogram 69 tends to become larger overall.
The control unit 21 calculates the maximum intensity value of the gradient histogram 69 from each block and sets it as a wrinkle feature amount (step S53). The control unit 21 stores the calculated wrinkle feature amount in the RAM.

  In the wrinkle feature amount extraction, the number of divided blocks is not limited to 5 × 5, and may be changed to an arbitrary number of blocks. However, since the wrinkle-appearing part is a feature that represents gender and age, it is desirable to divide the face into parts that are not subdivided and that can identify facial parts such as the cheek, forehead, and nose.

  Further, the control unit 21 extracts a lip feature amount from the detected face area (step S6). Since the lip feature value is a color feature, the process of calculating the lip feature value in step S6 is performed based on the color image. As shown in FIG. 11, the control unit 21 first generates a reversed skin mask 70 obtained by inverting the skin mask 62 (binary image) (step S61), and the original face area image (color) 61 is converted into the inverted skin mask 70. A reverse skin mask processing result 71 obtained by masking is obtained. A lip region 71a is extracted from the inverted skin mask processing result 71 (step S62). The control unit 21 determines redness of the lip region 71a (step S63).

In the redness determination in step S63, the control unit 21 obtains a histogram in the skin reference region 61a in the original face region image 61 and obtains a histogram in the lip region 71a extracted in step S62. The target color space of the histogram is assumed to be H (hue), S (saturation), and R (RGB R (red)).
The skin reference area 61a is preferably the nose of the original face area image 61, for example. And the control part 21 compares each histogram regarding a color, and extracts the redness of the lip | rip area | region 71a on the basis of the color in the reference | standard area | region 61a as a feature-value. For comparison of histograms, it is preferable to use, for example, a Bachtalia distance, but a comparison may be made with other distances (for example, Mahalanobis distance).

  As described above, in the lip feature extraction of the present embodiment, in order to obtain redness of the lip region with respect to the reference color in the original face region determined for each image captured under various photographing environments, Lip feature values (color-related feature values) that are less dependent on changes in the shooting environment can be obtained.

  The control unit 21 stores the lip feature amount extracted by the process of step S6 in the RAM.

FIG. 12 shows an example of the feature amount table 81 in which the calculated feature amounts are collected in the above processing.
As shown in FIG. 12, a plurality of feature amounts are extracted from each person image for learning (No. 001, No. 002,...) And are collected in a feature amount table 81. The feature amount includes a stain feature amount, a wrinkle feature amount, and a lip feature amount.

  For example, in the above-described spot feature amount extraction (step S4), when the intensity level of the MTM filter is changed to three levels and filtering is performed, and feature amounts are obtained for each divided region (5 × 5), 5 × 5 × 3 = 75 spot feature values are extracted. Further, in the above-described wrinkle feature amount extraction (step S5), 5 × 5 = 25 spot feature amounts are extracted in order to obtain feature amounts for each divided region (5 × 5). Further, in the above-described lip feature value extraction (step S6), three lip feature values are extracted in order to obtain feature values for H (hue), S (saturation), and R (red), respectively. In this case, since 103 types of feature amounts are extracted, 103-dimensional feature vectors are obtained. About these learning person images, attributes (age, gender) are known, and a feature amount table 81 is aggregated for each attribute.

  That is, the control unit 21 aggregates the feature amounts extracted from the person image for each person attribute category, and obtains the person attribute learning data 10 by SVM (Support Vector Machine) (step S7). In SVM, pattern recognition based on statistical learning theory is possible, and in the feature amount space obtained by learning, the Euclidean distance is the most on the basis of the feature amount vector (support vector) existing closest to other classes. An identification boundary is set at a position where it becomes large (linear identification is performed using a separation hyperplane). Since SVM is a known method, detailed description thereof is omitted.

  The control unit 21 generates the person attribute learning data 10 and stores it in the storage unit 22. As the number of person images learned increases, more accurate learning data 10 can be obtained.

When the person attribute learning data 10 is generated, it is possible to perform attribute estimation for the processing target image.
In the attribute estimation function, as shown in FIG. 2B, when an arbitrary person image is input to the person image estimation device 2, the control unit 21 detects the face area 61 from the input person image 6. Then, as described above, the spot feature amount, the wrinkle feature amount, and the lip feature amount are calculated from the detected face region 61, and the processing target input based on the calculated feature amounts and the person attribute learning data 10 The person attribute of the person image is estimated, and the estimation result is output.

  For example, if an image obtained by capturing a digital signage viewer with the imaging device 3 is used as a person image to be processed, the above-described estimation result of the person attribute is aggregated and output as the signage viewer attribute, It can be used as a measurement result. Further, using the estimation result, the control unit 21 of the computer 2 may switch and control the content to be displayed on the digital signage display medium (display device 4) in real time.

  When the image input from the imaging device 3 is a moving image, the person attribute is estimated for each frame, and for example, the mode value of the person attribute estimation result of each frame may be used as the person attribute estimation result. (However, in the case where only one person is the same person in the moving image) The classification of the person attribute is, for example, four age categories (0-19 years old, (20 years old-34 years old, 35 years old-59 years old, over 60 years old) may be set as an attribute classification of 8 classifications, or may be other appropriate classifications according to uses.

  As described above, in the human attribute estimation system 1 according to the present embodiment, the human attribute estimation device 2 detects the human face area 61 from the input learning person image, and the face area 61 detects the spot feature amount. Wrinkle feature values and lip feature values are calculated, and person attribute learning data 10 is created based on these feature values and the known attributes (age, gender) of the person. In addition, when the person attribute learning data 10 is created, the person attribute estimation system 1 calculates a spot feature amount, a wrinkle feature amount, and a lip feature amount for the person image input from the imaging device 3, and the calculated feature amount. And the person attribute learning data 10 are used to estimate and output the person attribute of the person image.

  In this method, since the spot feature amount is calculated using a difference between the image 63 after mask processing using the skin mask 62 and the blurred image 64 obtained by further smoothing the image, the contour of the face, eyes, nose, mouth, glasses It is possible to obtain a spot feature amount that is less affected by the above. Further, since the wrinkle feature value is obtained in consideration of the strength and direction of the wrinkle, the wrinkle feature value indicating an age difference or a sex difference can be calculated with high accuracy. Further, since the lip feature amount is calculated as a color comparison value with respect to the reference color in the original face region for the lip region 71a extracted using the skin mask 62, the lip feature amount is robust against environmental variations of the image. High feature extraction can be performed. By using the spot feature amount, the wrinkle feature amount, and the lip feature amount calculated in this way, it is possible to accurately estimate a person attribute such as age and sex.

  Further, when calculating the wrinkle feature amount, a gradient histogram 69 indicating the cumulative gradient intensity with respect to the gradient direction is generated for each block obtained by dividing the face area, and the maximum gradient strength value of the gradient histogram 69 of each block is determined as the wrinkle feature amount. Therefore, the wrinkle state (direction and strength) and the amount of wrinkles in each block can be acquired, which can greatly contribute to the improvement of the accuracy of age estimation. Also, because wrinkle features are extracted for each block, for example, wrinkles appearing more or less, or relatively large wrinkles such as normal lines and forehead wrinkles, and relatively large wrinkles such as the corners of the eyes. Wrinkle features can be extracted without confusing fine wrinkles.

  Further, when calculating the spot feature amount, a blurred image 64 is generated for the image 63 after mask processing using an MTM filter, and the difference between the image 63 after mask processing and the blurred image 64 for each block into which the face area is divided. The average value is calculated as a spot feature amount. By using the MTM filter, it is possible to exclude large spots, moles, and the like having a small correlation with age from the subject of blurring processing, and it is possible to exclude them from the subject of consideration of the spot feature amount. For this reason, the feature amount can be extracted only for a stain having a high correlation with age and the like.

  Further, when calculating the lip feature amount, the lip region is extracted using the reversal mask 70 obtained by reversing the skin mask 62, and the hue H, the saturation S, and the reference region in the lip region and the original face region, respectively. Since histograms with redness R as the target color space are generated and the distance between these histograms is calculated as the lip feature amount, feature extraction with high robustness to environmental fluctuations such as illumination intensity can be performed.

  In addition, if the person attribute estimated from such a feature amount is used for acquiring the person attribute of the advertisement viewer, it is possible to accurately measure the advertisement effect.

In the conventional technique, in the present invention, since the spot feature amount is calculated using the difference between the image after mask processing using the skin mask and the blurred image obtained by further smoothing the image, the contour of the face, eyes, nose, mouth, glasses It is possible to obtain a spot feature amount that is less affected by the above.
In the conventional method, the wrinkle feature amount is detected by detecting a large wrinkle using a Sobel filter on a face image, detecting a small wrinkle using a Gabor jet, and calculating the average and deviation of the edge strength of these wrinkles. Although defined as wrinkle features, in the present invention, the maximum value of the gradient strength of the gradient histogram of each block is defined as the wrinkle feature amount. The gradient histogram is obtained from the gradient intensity and gradient direction of each pixel of the image after mask processing. That is, according to the present invention, the largest wrinkle intensity (gradient intensity) accumulated in each gradient direction (wrinkle direction) for each block is set as the wrinkle feature amount in the block. Focusing on the fact that the direction in which wrinkles appear tends to differ depending on the age, the present invention also takes into account the direction in which wrinkles appear and is used for age estimation. Therefore, the accuracy of age estimation is improved compared to conventional methods. Can greatly contribute.
In addition, as a hue feature, the conventional method uses the color information in the lip region and cheek region as the hue and saturation of the stimulation value of the modified HSV color system and the redness component a * of the L * a * b color system. Mean and deviation are defined as feature quantities. On the other hand, in the present invention, since the lip feature amount is calculated as a color comparison value with respect to the reference color in the original face area determined for each image captured under various shooting environments, And robust feature extraction. By using the spot feature amount, the wrinkle feature amount, and the lip feature amount calculated in this way, it is possible to accurately estimate the person attributes such as age and sex as compared with the conventional method.

In the present embodiment, SVM is used for attribute learning and attribute estimation, but other learning methods may be used. Also, the detection method of the face area is arbitrary.
Moreover, although the example which utilizes the person attribute estimation system of this invention for the advertisement effect measurement of digital signage was shown in the above-mentioned embodiment, you may utilize for purposes other than such a use. In addition, it is obvious that those skilled in the art can come up with various changes and modifications within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. It is understood.

1 ... Person attribute estimation system 2 ... Computer (person attribute estimation device)
3 ... Imaging device 4 ... Display devices 10a, 10b, 10c, 10d ... Person 6 ... Input image 61 ... Face region 61a ... Reference region 62 ... Skin mask 63 ... Image after mask processing 64 ... Blurred image 65 ... Spot image 67 ... Image after Sobel filter processing 68 ... Image after Sobel filter processing 69 divided into blocks ... Gradient histogram 70 ... Reversal mask image 71 ... Reverse mask processing result 71a ... Lip region 81 ... Feature amount table 8 ... Feature extraction unit 9 ... Attribute learning unit 10 ... Person attribute learning data 11 ... attribute estimation part

Claims (10)

Face detection means for detecting a human face area from input image data;
A skin mask generating means for generating a skin mask indicating a skin area of the face area;
Using the difference between the image after mask processing in which the skin portion of the original face area is extracted using the skin mask and the blurred image obtained by further smoothing the image after mask processing, a spot in the skin portion is extracted, and the spot feature amount A spot feature amount calculating means for calculating
A wrinkle feature amount calculating means for calculating a wrinkle intensity and direction as a wrinkle feature amount from an image after mask processing in which the skin portion of the original face region is extracted using the skin mask;
Lip feature amount calculating means for calculating, as a lip feature amount, a color comparison value with respect to a reference color in the original face region for the lip region extracted using the skin mask;
Attribute learning that creates personal attribute learning data based on the wrinkle feature amount, the stain feature amount, the lip feature amount calculated from a plurality of input learning person images, and attribute information about the person image Means,
Estimating means for calculating the spot feature amount, the wrinkle feature amount, and the lip feature amount for an input person image, and estimating the person attribute of the person image based on the calculated feature amount and the person attribute learning data When,
A person attribute estimation device comprising: The wrinkle feature amount calculating means includes:
Gradient calculating means for calculating gradient intensity and gradient direction for each pixel of the image after masking;
Gradient histogram generation means for generating a gradient histogram indicating the cumulative gradient intensity with respect to the gradient direction based on the gradient intensity and gradient direction calculated by the gradient calculation means for each block obtained by dividing the face area;
The human attribute estimation apparatus according to claim 1, wherein a maximum value of the gradient strength of the gradient histogram for each block is used as a wrinkle feature amount. The spot feature amount calculating means includes:
The blurred image is generated using an MTM filter for the image after mask processing, an average value of the difference between the image after mask processing and the blurred image is calculated for each block obtained by dividing the face region, The person attribute estimation apparatus according to claim 1, wherein the person attribute estimation apparatus is an amount. The lip feature quantity calculating means includes:
Using a reversal mask obtained by reversing the skin mask, a lip region is extracted from the face region, and the hue H, saturation S, and redness R of the reference region in the lip region and the original face region are set as the target color space. The human attribute estimation apparatus according to claim 1, wherein a histogram is generated and a distance between the histograms is calculated as a lip feature amount. Face detection means for detecting a human face area from input image data;
A skin mask generating means for generating a skin mask indicating a skin area of the face area;
Using the difference between the image after mask processing in which the skin portion of the original face area is extracted using the skin mask and the blurred image obtained by further smoothing the image after mask processing, a spot in the skin portion is extracted, and the spot feature amount A spot feature amount calculating means for calculating
A wrinkle feature amount calculating means for calculating a wrinkle intensity and direction as a wrinkle feature amount from an image after mask processing in which the skin portion of the original face region is extracted using the skin mask;
Lip feature amount calculating means for calculating, as a lip feature amount, a color comparison value with respect to a reference color in the original face region for the lip region extracted using the skin mask;
A person attribute created by a predetermined learning algorithm based on the wrinkle feature quantity, the spot feature quantity, and the lip feature quantity calculated from a plurality of learning person images, and attribute information given to the person image Memory means for storing learning data;
Estimating means for calculating the spot feature amount, the wrinkle feature amount, and the lip feature amount for an input person image, and estimating the person attribute of the person image based on the calculated feature amount and the person attribute learning data When,
A person attribute estimation device comprising: An imaging means for capturing an image;
A person image input means for inputting an image picked up by the image pickup means as a person attribute estimation target in the estimation means;
The person attribute estimation device according to claim 1, further comprising: A face detection step of detecting a human face area from the input image data;
A skin mask generating step for generating a skin mask indicating a skin region of the face region;
Using the difference between the image after mask processing in which the skin portion of the original face area is extracted using the skin mask and the blurred image obtained by further smoothing the image after mask processing, a spot in the skin portion is extracted, and the spot feature amount A spot feature amount calculating step for calculating
A wrinkle feature amount calculating step of calculating the strength and direction of wrinkles as a wrinkle feature amount from an image after mask processing in which a skin portion is extracted from an original face region using the skin mask;
A lip feature amount calculating step for calculating, as a lip feature amount, a color comparison value with respect to a reference color in the original face region for the lip region extracted using the skin mask;
Attribute learning that creates personal attribute learning data based on the wrinkle feature amount, the stain feature amount, the lip feature amount calculated from a plurality of input learning person images, and attribute information about the person image Steps,
An estimation step of calculating the spot feature amount, the wrinkle feature amount, and the lip feature amount for the input person image, and estimating a person attribute of the person image based on the calculated feature amount and the person attribute learning data. When,
A person attribute estimation method, wherein a computer executes a process including: The wrinkle feature amount calculating step includes:
A gradient calculating step of calculating gradient intensity and gradient direction for each pixel of the image after masking;
A gradient histogram generating step for generating, for each block obtained by dividing the face area, a gradient histogram indicating a cumulative gradient intensity with respect to the gradient direction based on the calculated gradient intensity and gradient direction,
The person attribute estimation method according to claim 7, wherein the maximum value of the gradient strength of the gradient histogram for each block is used as a wrinkle feature amount. A program written in a computer-readable format,
A face detection step of detecting a human face area from the input image data;
A skin mask generating step for generating a skin mask indicating a skin region of the face region;
Using the difference between the image after mask processing in which the skin portion of the original face area is extracted using the skin mask and the blurred image obtained by further smoothing the image after mask processing, a spot in the skin portion is extracted, and the spot feature amount A spot feature amount calculating step for calculating
A wrinkle feature amount calculating step of calculating the strength and direction of wrinkles as a wrinkle feature amount from an image after mask processing in which a skin portion is extracted from an original face region using the skin mask;
A lip feature amount calculating step for calculating, as a lip feature amount, a color comparison value with respect to a reference color in the original face region for the lip region extracted using the skin mask;
Attribute learning that creates personal attribute learning data based on the wrinkle feature amount, the stain feature amount, the lip feature amount calculated from a plurality of input learning person images, and attribute information about the person image Steps,
An estimation step of calculating the spot feature amount, the wrinkle feature amount, and the lip feature amount for the input person image, and estimating a person attribute of the person image based on the calculated feature amount and the person attribute learning data. When,
A program for causing a computer to execute processing including The wrinkle feature amount calculating step includes:
A gradient calculating step of calculating gradient intensity and gradient direction for each pixel of the image after masking;
A gradient histogram generating step for generating, for each block obtained by dividing the face area, a gradient histogram indicating a cumulative gradient intensity with respect to the gradient direction based on the calculated gradient intensity and gradient direction,
The program according to claim 9, wherein a maximum value of the gradient strength of the gradient histogram for each block is set as a wrinkle feature amount.