JP2020135369A - Information processing device, system, information processing method, and program - Google Patents

Abstract

To allow for outputting more appropriate attribute information.SOLUTION: An information processing method is provided, comprising: extracting feature quantities of an object in an input image; identifying a class of the object based on the extracted feature quantities; determining reliability of an identification result; and outputting preset attribute information associated with the class of the object provided that the reliability meets a predefined condition.SELECTED DRAWING: Figure 7

Description

The present invention relates to information processing devices, systems, information processing methods and programs.

There is an attribute recognition technology that determines attribute information such as the age and gender of a person in an image. Application destinations of attribute recognition technology include marketing applications for grasping the purchasing age group of products and security applications. In these applications, the attribute recognition technology is used in a wide variety of situations, and images under various shooting conditions are taken depending on the usage scene. Here, the shooting condition indicates the state of the object such as the face orientation and facial expression of the object, and the shooting state of the object such as the lighting condition and occlusion. Under shooting conditions where occlusion, overexposure, and underexposure may occur, there is a high possibility that the attribute recognition result will deteriorate compared to under shooting conditions where occlusion, overexposure, and underexposure do not occur.
Therefore, in order to compensate for the performance deterioration of the attribute recognition technology, an attribute recognition technology that combines an object authentication technology such as a face has been proposed. Patent Document 1 describes a technique for determining whether a person is subject to age restriction by performing face recognition processing when it is determined that the age is difficult to determine whether the person is subject to age restriction after performing age estimation based on an input image. Is disclosed. Further, in Patent Document 2, the feature amount indicating the individual difference of the face extracted from the input image is compared with the reference feature amount of each age group stored in advance, and the age group having the highest degree of similarity is selected. A technique for determining the age of a subject in an input image is disclosed.

Japanese Patent No. 4910717 Japanese Patent No. 4521086

However, in the prior art, the accuracy of attribute estimation and object authentication for the captured image of the object deteriorates depending on the imaging conditions of the image, and inappropriate attribute information may be output.
An object of the present invention is to output more appropriate attribute information.

The information processing apparatus of the present invention includes an extraction means for extracting a feature amount of an object included in an input image, a specific means for specifying a class of the object based on the feature amount extracted by the extraction means, and the identification. When the determination means for determining the reliability of the specific result by the means and the reliability determined by the determination means satisfy the predetermined conditions, the determination means for the class of the object specified by the specific means is predetermined. It has an output means for outputting attribute information.

According to the present invention, more appropriate attribute information can be output.

It is a figure which shows an example of the system configuration of an information processing system. It is a figure which shows an example of the hardware composition of an image processing apparatus. It is a figure which shows an example of the functional structure of an image processing apparatus. It is a flowchart which shows an example of a registration process. It is a figure which shows an example of the registration information. It is a flowchart which shows an example of a collation process. It is a flowchart which shows an example of output processing. It is a figure which shows an example of the functional structure of an image processing apparatus. It is a flowchart which shows an example of output processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Embodiment 1>
The information processing system according to the present embodiment is a system that outputs attribute information of an object in an image. Attribute information is information about the corresponding attribute.
In the present embodiment, the information processing system estimates the age of a person in front of the product shelf based on an image taken by a camera installed near the product shelf of the store, and analyzes the age information of the person in the purchasing age group. An example of outputting to the device will be described. The age information is attribute information about the age attribute.
FIG. 1 is a diagram showing an example of a system configuration of the information processing system of the present embodiment. In the present embodiment, the information processing system estimates the age as an attribute of the person who approaches the product shelf, and outputs the information of the estimated age. The information processing system includes an image processing device 1, a lighting distribution acquisition device 3, a photographing device 4, and an analysis processing device 5. The elements are communicably connected to each other via the network 2. FIG. 1 shows how a person to be age-estimated progresses in front of a product shelf.
The image processing device 1 is an information processing device such as a personal computer (PC), a server device, or a tablet device that performs face recognition processing on a person whose age is to be estimated. The image processing device 1 outputs age information to the analysis processing device 5 of the purchasing age group based on the result of the face recognition processing.
The network 2 is a local area network and is used for communication between each device. However, as another example, the network 2 may be another communication network such as the Internet.

The illumination distribution acquisition device 3 is a camera equipped with a fisheye lens and acquires the illumination distribution in the surrounding three-dimensional space, and outputs the acquired illumination distribution to the image processing apparatus 1 via the network 2. The illumination distribution is data showing the distribution of illumination around the illumination distribution acquisition device 3 in a three-dimensional space, that is, the direction of illumination, the intensity of illumination, and the distribution of color temperature of illumination. More specifically, the illumination distribution is a spherical or dome-shaped two-dimensional image showing the ambient illumination distribution. In places affected by external light, it is preferable to use a camera capable of HDR (High Dynamic Range) photography in order to more accurately photograph the illumination distribution of illumination including sunlight, which has a relatively wide range of intensity. .. Therefore, the illumination distribution acquisition device 3 is a camera capable of HDR photographing. In the present embodiment, the illumination distribution acquisition device 3 is a camera equipped with a fisheye lens. However, the illumination distribution acquisition device 3 may be another device as long as it can acquire the ambient illumination distribution. For example, the illumination distribution acquisition device 3 may be an apparatus that acquires a plurality of images using a normal camera and an automatic pan head, and acquires the ambient illumination distribution by adding stitch processing.
Further, the illumination distribution acquisition device 3 may be a device that acquires the ambient illumination distribution by combining a normal camera with a mirror ball, a reflector, or the like. Further, when the color temperature is not used in the subsequent processing, the illumination distribution acquisition device 3 only needs to be able to acquire at least the distribution of the illumination direction and the intensity.

The photographing device 4 is a network camera including an optical lens, an image sensor, and a communication unit, and photographs a person whose age is to be estimated. Then, the photographing device 4 outputs the photographed image to the image processing device 1.
The analysis processing device 5 performs a process of analyzing the age information output from the image processing device 1.
In the present embodiment, the number of imaging devices 4 is one, but there may be a plurality of imaging devices 4.
Further, in the present embodiment, the image processing device 1, the illumination distribution acquisition device 3, the photographing device 4, and the analysis processing device 5 are each individual devices, but a plurality of devices among these devices are combined into one. It may be configured as a device. For example, the illumination distribution acquisition device 3 and the photographing device 4 may be configured as one device having both functions. Further, the image processing device 1 and the analysis processing device 5 may be configured as one device having both functions.

<Hardware configuration>
FIG. 2 shows an example of the hardware configuration of the image processing device 1. The image processing device 1 includes a CPU 11, a ROM 12, a RAM 13, a secondary storage device 14, a communication device 15, a video output device 16, an operation input device 17, and a connection bus 18. The elements are communicably connected to each other via the connection bus 18.
The CPU 11 is a Central Processing Unit that controls the image processing device 1, and executes processing according to a program stored in the ROM 12 or the secondary storage device 14. The ROM 12 is a non-volatile memory, and is a Read Only Memory that stores a control program and various parameter data. The RAM 13 is a volatile memory, and is a Random Access Memory that temporarily stores images, control programs, processing execution results, and the like. The secondary storage device 14 is a rewritable secondary storage device such as a hard disk or a flash memory, and stores various programs, images, lighting distribution, age information, and the like.

The communication device 15 is a wired communication unit and is used for communication with other devices via a network. The communication device 15 may be a wireless communication unit. The video output device 16 is a monitor such as a CRT or a TFT liquid crystal. The CPU 11 displays an image acquired from the RAM 13, an execution result of the control program, and the like on the video output device 16.
The operation input device 17 is an input device that accepts operations from users such as a keyboard and a mouse. The connection bus 18 is a bus used for connecting each element of the image processing device 1.
When the CPU 11 executes the process according to the program stored in the ROM 12, the secondary storage device 14, and the like, the function of the image processing device 1 described later in FIGS. 3 and 8 and the processing of the flowchart of FIGS. 4, 6, 7, and 9. Etc. are realized.

<Functional configuration>
FIG. 3 shows an example of the functional configuration of the image processing device 1. The image processing device 1 includes an acquisition unit 101, a detection unit 102, a face feature extraction unit 103, a registration unit 104, an age information acquisition unit 105, a collation unit 106, a lighting feature extraction unit 107, a discrimination unit 108, and a reliability calculation unit 109. The reliability determination unit 110, the generation unit 111, and the output unit 112 are included.
The acquisition unit 101 acquires a two-dimensional image captured by the photographing device 4 and outputs it to the detection unit 102. Further, in the present embodiment, the acquisition unit 101 captures a two-dimensional image for discriminating the illumination distribution via the illumination distribution acquisition device 3, and outputs the captured two-dimensional image to the illumination feature extraction unit 107.
The detection unit 102 detects an object from the image acquired from the acquisition unit 101. In the present embodiment, the detection unit 102 detects a human face as an object and outputs an image of the detected face region to the face feature extraction unit 103.
The face feature extraction unit 103 extracts the feature amount for face recognition from the image of the face area acquired from the detection unit 102, and outputs the extracted feature amount to the registration unit 104 or the collation unit 106.

The registration unit 104 stores the feature amount for face authentication acquired from the face feature extraction unit 103 as registration data in the secondary storage device 14, and outputs the feature amount to the collation unit 106. The registration unit 104 stores the person information indicating which face the feature amount corresponds to as the registration data in the secondary storage device 14 after associating the person information with the feature amount. The registration unit 104 acquires personal information based on the user's operation using the operation input device 17.
The age information acquisition unit 105 displays the person registered in the video output device 16, and accepts the input of the age information of the registered person based on the user's operation via the operation input device 17. Then, the age information acquisition unit 105 associates the received age information with the feature amount and the person information stored as the registration data by the registration unit 104, and stores the received age information in the secondary storage device 14.

The collation unit 106 performs collation based on the feature amount for face authentication acquired from the face feature extraction unit 103 and the registration data registered by the registration unit 104, and outputs the collation result to the generation unit 111.
The illumination feature extraction unit 107 extracts the feature amount used for determining the lighting condition from the image captured by the illumination distribution acquisition device 3 acquired from the acquisition unit 101, and outputs the feature amount to the determination unit 108. Here, the illumination feature extraction unit 107 associates the two-dimensional image captured by the photographing device 4 with the feature amount for determining the illumination condition. More specifically, the illumination feature extraction unit 107 links the two-dimensional image captured by the photographing device 4 with the correspondence information of the feature amount for determining the illumination condition by storing it in the RAM 13 or the like.

The determination unit 108 determines the lighting condition based on the feature amount for determining the lighting condition acquired from the lighting feature extraction unit 107, and outputs the determined lighting condition to the reliability calculation unit 109 and the reliability determination unit 110. ..
The reliability calculation unit 109 calculates the reliability of the result of face authentication (class determination) based on the lighting condition determination result acquired from the determination unit 108, and outputs the reliability to the reliability determination unit 110.
The reliability determination unit 110 determines whether or not the reliability of the class determination acquired from the reliability calculation unit 109 satisfies a predetermined condition, and outputs the determination result to the generation unit 111.
The generation unit 111 acquires the age information associated with the registration data of the person of the collation result based on the collation result acquired from the collation unit 106. Then, the generation unit 111 generates output information indicating the age information based on the determination result acquired from the reliability determination unit 110, and outputs the output information to the output unit 112. The output unit 112 outputs the output information acquired from the generation unit 111 to the analysis processing device 5 via the communication device 15.

In the present embodiment, the information processing system includes a registration process for registering a person who may use the store in advance, a collation process for collating the person to be collated, and an output process for outputting age information. I do. The details of each process will be described below.
The registration process will be described with reference to FIG.
In S1001, the acquisition unit 101 acquires a two-dimensional image captured by the photographing device 4.
In the present embodiment, the acquisition unit 101 is to perform the process of acquiring the image captured by the photographing device 4 in S1001. However, as another example, the acquisition unit 101 may perform a process of acquiring an image prepared in advance, or may perform a process of acquiring an image of a person pseudo-generated by an image processing program. Alternatively, an image of a person may be acquired by performing these processes in combination.

In S1002, the detection unit 102 detects the face of a person from the image acquired in S1001. The detection unit 102 detects the face by using a known technique. For the known technique, for example, the following Reference 1 can be used.
Reference 1: Rapid object detection using a boosted cascade of simple objects: P.I. Viola, M.D. Jones: 2001
The detection unit 102 acquires the face detection result as the coordinates of the rectangular area representing the face area in the image. The detection unit 102 performs a normalization process on the detected face region so that the face size and the inclination of the face become constant. As another example, the detection unit 102 may extract feature points such as eyes and nose of the face and output a partial region of the face. In that case, the detection unit 102 extracts facial feature points using a known technique. This known technique includes, for example, Reference 2.
Reference 2: Japanese Patent Application Laid-Open No. 2009-21177

In S1003, the face feature extraction unit 103 extracts the feature amount for face recognition from the face image of the person detected in S1002. The face feature extraction unit 103 extracts a known feature amount as the feature amount of the face image. The known features include, for example, LBP (Local Binary Pattern) features and HOG (Histogram of Oriented Gradient) features. Further, the known feature amount includes SIFT (Scale-Invariant Feature Transfer Transfer) feature amount.
As another example, the face feature extraction unit 103 may extract a feature amount obtained by mixing two or more known feature amounts such as an LBP feature amount, a HOG feature amount, and a SIFT feature amount. Further, the face feature extraction unit 103 may extract the feature amount by the neural network. Further, the face feature extraction unit 103 may dimensionally compress the extracted feature amount by using a method such as PCA (Principal Component Analysis sis).

In S1004, the registration unit 104 accepts the input of the person information detected in S1002 based on the operation by the user via the operation input device 17. Then, the registration unit 104 registers the set of the feature amount extracted in S1003 and the information of the person corresponding to the received feature amount in the secondary storage device 14 as registration data.
In S1005, the age information acquisition unit 105 registers the age information input by the user via the operation input device 17 in the secondary storage device 14 in association with the registration data registered in S1004. In the present embodiment, the age information acquisition unit 105 receives information on the date of birth as age information. However, as another example, the age information acquisition unit 105 may receive information on the actual age, information on the designated age, and the like as the age information.
FIG. 5 shows an example of a management table that manages the registration data registered in S1004 and S1005 and the age information. The management table is a table stored in the secondary storage device, and includes items of personal information, features, and date of birth. In the item of person information, information indicating which person is stored is stored. In the feature item, the feature amount extracted from the face of the corresponding person is stored. The age information registered in S1005 is stored in the date of birth item.
The above is the details of the registration process.

The details of the collation process performed by the information processing system will be described with reference to FIG.
In S1101, the acquisition unit 101 acquires an image captured by the photographing device 4. The image acquired in S1101 is an example of an input image.
In S1102, the detection unit 102 detects the face of a person from the image acquired in S1101. In the present embodiment, if there is a person who came in front of the product shelf shown in FIG. 1, the face of that person is detected by the detection unit 102 in S1102 and collated in a later process. In the present embodiment, the detection unit 102 does not detect the face from the entire area of the image, but detects the face from a predetermined range in the image. By doing so, the detection unit 102 can prevent a passerby who just passes in front of the product shelf from being mistakenly subject to age authentication as a person looking at the product shelf.
In S1103, the detection unit 102 determines whether or not one or more faces have been detected in S1102. If the detection unit 102 detects one or more faces in S1102, the process proceeds to S1104, and if no face is detected in S1102, the detection unit 102 advances the process to S1101.

In S1104, the face feature extraction unit 103 extracts the feature amount from the face image detected in S1102. The face feature extraction unit 103 extracts the feature amount by the same method as the method described in S1003. When there are a plurality of faces detected in S1102, the face feature extraction unit 103 extracts the feature amount for each of the plurality of faces. In that case, the information processing system executes the processing of S1105 in order from the feature amount of the face closest to the product shelf.
Further, as another example, the face feature extraction unit 103 sets a coordinate range in a predetermined image as a priority processing target, and based on the face coordinates and the set coordinate range. , You may select the face to extract the features. For example, when the coordinate range of the area where the person looking at the product shelf can exist is set, the information processing system can execute the process for the person who is more interested in the product shelf. Further, for example, when a coordinate range of a region in which a person viewing a certain product can exist is set, the information processing system determines which product the person is interested in, and the purchase age group of the product. Can be analyzed appropriately.
Further, the photographing device 4 may be arranged so that the angle of view is adjusted so that the product shelf to be analyzed most and the related product shelf are reflected. The information processing system may extract features from all faces detected from the image captured by the photographing device 4. This makes it possible for the information processing system to analyze the product shelves and the purchasing age groups of the related product shelves together. In addition, the information processing system can further analyze the movement route at the time of purchase by tracking the face.

In S1105, the collation unit 106 performs collation processing between the feature amount of each person registered as the registration data and the feature amount extracted in S1104. In the present embodiment, it is assumed that the feature amount extracted from the image of the face region is the feature amount in the vector format. The collation unit 106 obtains the collation degree (similarity) between the feature quantities (vectors). The collation unit 106 calculates the degree of collation between the feature quantities by using a known method. For example, the collation unit 106 obtains the cos similarity as the collation degree between the feature quantities. With the interpersonal collation degree as S, the collation unit 106 obtains the cos similarity between the feature amount A and the feature amount B to be compared using the following formula 1.
S = (A / B) / (| A || B |) ... (Equation 1)
A ∈ R ^D , B ∈ R ^D. D represents the number of dimensions of the feature quantities A and B.

The collation unit 106 obtains the collation degree S by setting one of the feature amounts extracted in S1104 as A and each of the feature amounts registered as the registration data as B in sequence. Then, it is assumed that the collation unit 106 collates A with B corresponding to the largest S. That is, the collation unit 106 authenticates that the person corresponding to B collated with A is the person corresponding to A (the person whose face is detected in S1102). As a result, the collating unit 106 identifies the person whose face is detected in S1102 as the person corresponding to B. In other words, the collating unit 106 identifies the class of the person whose face is detected in S1102 as the class of the person corresponding to B.
However, as another example, the collation unit 106 may use another index different from the cos similarity as the collation degree between the feature quantities. For example, the collating unit 106 may use the normalized cross-correlation between the features, or may obtain an index showing the degree of similarity between the features by using a metric learning method in which the distance function is calculated by learning. Good.
Further, the collation unit 106 may normalize the obtained collation degree to a value within a predetermined range.

In S1106, the collating unit 106 determines whether or not the processing of S1105 is completed for all the facial features extracted in S1104. When the collating unit 106 determines that the processing of S1105 has been completed for all the facial features extracted in S1104, the processing proceeds to S1107, and when it is determined that there is an uncompleted feature amount, the processing is performed in S1105. Proceed to.
In S1107, the collating unit 106 outputs the result of the processing of S1105 (information indicating which person the face detected in S1102 is) to the generation unit 111.
The above is the collation processing flow.

The details of the output processing executed by the information processing system will be described with reference to FIG. 7.
In S1201, the acquisition unit 101 acquires the image taken by the illumination distribution acquisition device 3 when the image acquired in S1101 is taken.
In S1202, the illumination feature extraction unit 107 extracts the feature amount for determining the illumination condition from the image acquired in S1201. In the present embodiment, the illumination feature extraction unit 107 extracts the brightness as a feature amount for determining the illumination condition.

In S1203, the discriminating unit 108 discriminates the lighting conditions in the shooting environment of the image acquired in S1101 based on the feature amount extracted in S1202. In the present embodiment, the discriminating unit 108 discriminates the lighting conditions of the space where the face of a person can exist. The illumination condition is a condition indicating how the illumination (light) is irradiated in the image shooting environment. In the present embodiment, the illumination conditions include the direction of illumination (either forward light, oblique light, or backlight) and the average illuminance of the illumination. The normal light indicates a state in which the person who is the subject is illuminated from the front (the side of the photographing device 4). Oblique light indicates a state in which a person who is a subject is illuminated from the side. Backlight refers to a state in which illumination is emitted from behind the person who is the subject.
The discriminating unit 108 discriminates the lighting conditions by using a known technique. Such known techniques include, for example, Reference 3 below.
Reference 3: Japanese Patent No. 5726792

In Reference 3, an illuminance estimation model that defines the relationship between the brightness acquired from the captured image and the illuminance estimated from the brightness is generated. In the present embodiment, the discrimination unit 108 generates such an illuminance estimation model, and when discriminating the lighting conditions, uses the generated illuminance estimation model-based illuminance estimator to determine the illuminance distribution of the image shooting environment. Based on this, it is determined whether the direction of the illumination is forward light, back light, or oblique light.
The discrimination unit 108 converts the two-dimensional illuminance distribution in the rectangular image acquired in S1201 into a one-dimensional illuminance vector, and prepares in advance a lighting condition discriminator that discriminates the lighting condition based on this illuminance vector. , Illumination conditions are discriminated using a lighting condition discriminator. Further, the discrimination unit 108 obtains the average illuminance based on this illuminance vector.

In S1204, the reliability calculation unit 109 obtains the reliability of the face authentication result (class identification result) in S1105 based on the lighting conditions determined in S1203, and transmits the obtained reliability to the reliability determination unit 110. Output.
More specifically, the reliability calculation unit 109 sets the reliability of face recognition to 100% if the direction of lighting indicated by the determined lighting condition is normal light, and 80% if the direction of lighting is oblique light. If it is backlit, the reliability of face recognition is set to 50%. Then, the reliability calculation unit 109 reduces the face recognition reliability by 20% when the average illuminance indicated by the determined lighting condition is outside the range of 100 lux to 1500 lux. For example, when the lighting condition is determined to be normal light and the average illuminance is 67 lux, the reliability calculation unit 109 sets the face recognition reliability to 100% -20% = 80%.

In S1205, the reliability determination unit 110 determines whether or not the reliability of face authentication obtained in S1204 satisfies a predetermined condition. In the present embodiment, the predetermined condition is that the reliability of face recognition is equal to or higher than a predetermined threshold value. In this embodiment, this threshold is set to 70%.
The reliability determination unit 110 advances the process to S1206 when the reliability of face authentication obtained in S1204 is equal to or higher than a predetermined threshold value (70%), and when it is less than the threshold value (70%), FIG. Ends the processing of.
For example, when the direction of the illumination indicated by the determined illumination condition is oblique light and the average illuminance indicated by the illumination condition is 50 lux, the reliability of face recognition is 60%. Therefore, the reliability determination unit 110 ends the process of FIG. 7, assuming that the reliability is less than a predetermined threshold value (70%).
However, as another example, the reliability determination unit 110 of the face-authenticated face is based on the average and standard deviation of the reliability of the face recognition for the face tracked by the tracking process for the face to be authenticated for a certain period of time. It may be determined whether or not to output the output information indicating the age information.

In S1206, the generation unit 111 generates output information indicating the age information of the person indicated by the authentication result based on the result of the face authentication in S1105. In the present embodiment, the generation unit 111 acquires the registration data of the person indicated by the result of the face authentication in S1105 from the registration data registered by the registration unit 104. Then, the generation unit 111 acquires the age information corresponding to the acquired registration data, and generates information indicating the current age of the person as output information.
In the present embodiment, since the age information is the date of birth, the generation unit 111 specifies the shooting time of the image based on the EXIF information of the image acquired in S1101, and the specified shooting time and the age information indicate the birth year. Identify the actual age of the person from the date. Then, the generation unit 111 generates output information indicating the specified actual age. For example, when the result of face recognition in S1105 is "Mr. A" in the registration database of FIG. 5, and the shooting time of the image acquired in S1101 is "March 1, 2017", the generation unit 111 Perform the following processing. That is, the generation unit 111 has a shooting time of the image "March 1, 2017", a date of birth of Mr. A "January 1, 1980", and Mr. A at the time of shooting the image. Is specified as "37 years old", and output information indicating "37 years old" is generated.

In S1207, the output unit 112 outputs the output information generated in S1206 to the analysis processing device 5. In the present embodiment, the output unit 112 outputs only the output information indicating the age generated in S1206. However, as another example, the output unit 112 may output the reliability of face recognition in S1105 as reference information indicating the reliability of the output information together with the output information. In addition, the generation unit 111 may output information on shooting conditions such as lighting conditions in addition to the output information.
In the present embodiment, the output unit 112 outputs the output information to the analysis processing device 5. However, the output unit 112 may output by displaying the output information on the video output device 16.
Further, when the output information is output, the output unit 112 controls so as not to output information indicating who is the person corresponding to the output information (information as a result of face authentication in S1105). As a result, the information processing system can reduce the possibility of leakage of personal information. For example, an information processing system may be applied to a use case that displays whether or not a person is a minor on the cashier screen of a cashier in a store that sells alcoholic beverages. In this case, the purchaser's face recognition process is performed internally and the result is not displayed on the screen, so that the personal information is not inadvertently disclosed to the cashier.

As described above, in the present embodiment, the information processing system obtains the result of face authentication and the reliability of face authentication for the person to be processed, and outputs an output indicating the age attribute of the person based on the obtained reliability of face authentication. Information will be generated and output. In this way, the information processing system generates output information indicating the age of the person to be face-recognized, taking into consideration the reliability of face recognition. As a result, the information processing system can reduce the possibility of outputting inappropriate age information.
In addition, the information processing system can prevent inadvertent leakage of personal information by controlling so as not to output the result of face authentication.

(Modification example 1)
In the present embodiment, the acquisition unit 101 acquires the two-dimensional image photographed by the photographing apparatus 4 in S1001 and S1101. However, as another example, the acquisition unit 101 may acquire a two-dimensional image rendered based on the three-dimensional data photographed by the photographing apparatus 4. Further, the acquisition unit 101 may acquire an image taken by another device via a network.
(Modification 2)
In the present embodiment, the information processing system seeks and outputs age information as personal attribute information. However, the information processing system may obtain and output information on other attributes such as gender and race as the attribute information of the person.

(Modification 3)
Further, in the present embodiment, the information processing system determines the lighting conditions of the shooting environment of the shooting device 4 based on the images taken by the lighting distribution acquisition device 3 which is a single camera. However, as another example, the information processing system may use the plurality of illumination distribution acquisition devices 3 as follows. That is, the information processing system discriminates the lighting conditions around each of the plurality of lighting distribution acquisition devices 3 based on each of the plurality of images taken by the plurality of lighting distribution acquisition devices 3, and sets the determined lighting conditions. Based on this, the lighting conditions of the shooting environment of the shooting device 4 may be determined.
Further, the information processing system is based on the signals of a sensing device such as a lighting sensor arranged around the photographing device 4 instead of the image photographed by the lighting distribution acquisition device 3, and the lighting conditions of the photographing environment of the photographing device 4. May be determined. Further, the information processing system discriminates a plurality of lighting conditions from each of the image captured by the illumination distribution acquisition device 3 and the signal of the sensing device arranged around the imaging device 4, and a plurality of discriminated lighting conditions. The final lighting condition may be obtained based on the lighting condition.

(Modification example 4)
In the present embodiment, the reliability calculation unit 109 determines the reliability of face recognition in S1105 based on the lighting conditions determined in S1204. However, as another example, the reliability calculation unit 109 may obtain the reliability of face recognition based on shooting conditions other than the lighting conditions.
For example, the reliability calculation unit 109 may obtain the reliability of face recognition based on the posture of the person in the image acquired in S1101. For example, the reliability calculation unit 109 sets the reliability of face recognition to 100% when the posture of the person in the image acquired in S1101 is upright, and sets the reliability of face recognition to 50% when crouching. May be good.
Further, for example, the reliability calculation unit 109 may obtain the reliability of face recognition based on the orientation of the person in the image acquired in S1101. For example, the reliability calculation unit 109 sets the reliability of face recognition to 100% when the orientation of the person in the image acquired in S1101 is frontward, and sets the reliability of face recognition to 50% when the orientation is landscape. May be good.
Further, for example, the reliability calculation unit 109 may obtain the reliability of face recognition based on the position of a person in the image acquired in S1101. For example, the reliability calculation unit 109 sets the reliability of face recognition to 100% when the position of the person in the image acquired in S1101 is within a predetermined area, and sets the reliability of face recognition to 100% when the position is within the other area. The reliability of face recognition may be set to 50%.

Further, for example, the reliability calculation unit 109 may obtain the reliability of face recognition based on the degree of occlusion with respect to a person in the image acquired in S1101. For example, the reliability calculation unit 109 may use the ratio obtained by subtracting the value corresponding to the degree of occlusion with respect to the person in the image acquired in S1101 from 100% as the reliability of face recognition.
Further, the reliability calculation unit 109 is based on two or more of the posture, orientation, position of the person in the image acquired in S1101, the degree of occlusion with respect to the person, and the lighting conditions determined in S1204. You may ask for the reliability of the authentication.

Further, as another example, the reliability calculation unit 109 may obtain the reliability of face recognition based on the stability of the tracking process performed on the face to be authenticated. The stability of the tracking process is an index showing how stable the tracking process is, and is, for example, the ratio of frames in which the face of the person to be tracked can be detected in a plurality of consecutive frames. .. For example, the reliability calculation unit 109 determines the reliability of face authentication to a higher value as the stability of the tracking process performed on the face to be authenticated is higher.
Further, as another example, the reliability calculation unit 109 may obtain the reliability of face authentication based on the collation degree obtained at the time of the face authentication process in S1105. For example, the reliability calculation unit 109 determines the reliability of face recognition to a higher value as the degree of verification is higher.

<Embodiment 2>
In the present embodiment, the information processing system changes the information to be output according to the reliability of face recognition.
The system configuration of the information processing system of the present embodiment is the same as that of the first embodiment. The hardware configuration of each element of the information processing system is the same as that of the first embodiment.
FIG. 8 is a diagram showing an example of the functional configuration of the image processing device 1 of the present embodiment. In the present embodiment, the image processing device 1 includes an attribute feature extraction unit 201 and an attribute recognition unit 202 in addition to the functions shown in FIG.
The attribute feature extraction unit 201 extracts the feature amount related to the attribute of the person with the face from the image of the face taken by the photographing device 4. In this embodiment, this attribute is age.
The attribute recognition unit 202 recognizes the attribute of the person whose face is detected by the detection unit 102 based on the feature amount extracted by the attribute feature extraction unit 201.

The information processing system of the present embodiment performs registration processing, collation processing, and output processing as in the first embodiment. Of these processes, the registration process and the collation process are the same processes as in the first embodiment. Further, among the processes of the information processing system of the present embodiment, the output process is different from that of the first embodiment.
The details of the output processing of the information processing system of the present embodiment will be described with reference to FIG. Of the processes of FIG. 9, the processes of S1201 to S1204 and S1206 are the same as those of FIG. 7.
In S2201, the reliability determination unit 110 determines whether or not the reliability of face authentication obtained in S1204 is equal to or higher than a predetermined first threshold value. When the reliability determination unit 110 determines that the reliability of the face authentication obtained in S1204 is equal to or higher than the first threshold value, the process proceeds to S1206, and when it is determined that the reliability is less than the first threshold value, the process is performed. Proceed to S2202. In this embodiment, the first threshold value is 85% (0.85).

In S2202, the reliability determination unit 110 determines whether or not the reliability of face recognition obtained in S1204 is smaller than the first threshold value and equal to or higher than a predetermined second threshold value. When the reliability determination unit 110 determines that the reliability of the face authentication obtained in S1204 is equal to or higher than the second threshold value, the process proceeds to S2203, and when it is determined that the reliability is less than the second threshold value, the process is performed. Proceed to S2204. In the present embodiment, the second threshold value is set to 60% (0.60).
For example, when the lighting direction indicated by the lighting conditions obtained in S1203 is normal light and the average illuminance is 300 lux, the reliability of face recognition obtained in S1204 is 80%. Therefore, the reliability determination unit 110 advances the process to S2203 through the processes of S2201 and S2202.

In S2203, the generation unit 111 obtains the age information of the person whose face has been authenticated in S1105 by the same process as in S1206. In addition, the generation unit 111 obtains the error of the obtained age as follows. That is, the generation unit 111 identifies a person corresponding to three collation degrees, which is a predetermined number from the largest collation degree obtained by the collation unit 106 in S1105. The generation unit 111 acquires the date of birth information corresponding to the registration data of each of the specified persons. Then, the generation unit 111 obtains the ages of these three persons based on the acquired date of birth. Then, the generation unit 111 obtains the range between the maximum value and the minimum value of the obtained three ages as the information of the age error.
For example, if the ages of the persons corresponding to the top three collations are "28", "22", and "32" from the top of the collation, the age error is "22 to 32". ..
Then, the generation unit 111 generates age information including the obtained age information and the age error information as output information.

In S2204, the attribute feature extraction unit 201 extracts the feature amount for attribute recognition from the image acquired in S1101. In the present embodiment, the attribute feature extraction unit 201 extracts a luminance gradient histogram as this feature amount. The attribute feature extraction unit 201 extracts a luminance gradient histogram using the technique of Reference 4 below.
Reference 4: A Pedestrian Detector Using Histograms of Oriented Gradients and a Support Vector Machine Classifier: M.D. Bertozi, A. Broggi, M.D. Del Rose, M.D. Felisa, A. Rakotomamonjy and F. Sound, IEEE Intelligent Transport Systems Conference: 2007

In S2205, the attribute recognition unit 202 recognizes the age attribute of the person in the image acquired in S1101 based on the feature amount for attribute recognition extracted in S2204. In the present embodiment, the attribute recognition unit 202 inputs the luminance gradient histogram extracted in S2204 into the age estimator corresponding to each age prepared in advance, and indicates the certainty that the target person belongs to each age. Age recognition is performed by finding the likelihood. The attribute recognition unit 202 uses the Support Vector Machine (hereinafter referred to as SVM) disclosed in Reference 4 to obtain the likelihood of belonging to each age group. The attribute recognition unit 202 determines the age information corresponding to the highest likelihood as the age information of the person in the image acquired in S1101.
For example, when constructing an estimator for determining the likelihood that the target person belongs to a teenager, this estimator has +1 as a learning label for the feature amount of the image of the teenager, and the image of a person of another age. The feature quantity of is learned by giving -1 as a learning label.

Here, the attribute recognition unit 202 may obtain the error range or the reliability of the attribute recognition result together with the attribute recognition result. For example, the attribute recognition unit 202 obtains the difference between the likelihoods of the two age groups having the highest likelihood, and when the difference is equal to or greater than a predetermined threshold value, the information of the second highest age group is used as the information of the error range. It may be output. For example, the likelihood for each age group is set to a minimum value of 0 and a maximum value of 1, and the difference threshold is set to 0.15. When the likelihood in the 30s is the highest at 0.8 and the likelihood in the 40s is the next highest at 0.6, the difference is 0.2, which exceeds the threshold of the difference. In this case, the attribute recognition unit 202 may request the information of "40s" as the information of the error range of the age information.
Further, in the present embodiment, the attribute recognition unit 202 determines the age recognition in consideration of the possibility that the face recognition reliability is low and the accuracy of the age recognition deteriorates, but as another example, the age recognition is performed. May be good.
Further, the information processing system may obtain age information based on the collation result in S1105 and use the obtained age information for attribute recognition processing in S2205 or correction of the attribute recognition result.

In S2206, the generation unit 111 generates output information indicating the age determined in S2205. Further, the generation unit 111 may include the age error information obtained in S2205 in the output information.
In S2207, the output unit 112 outputs the output information generated by any one of S1206, S2203, and S2206 to the analysis processing device 5. However, as another example, the output unit 112 may output by displaying the output information on the video output device 16.
Further, when the output information is output, the output unit 112 controls so as not to output information indicating who is the person corresponding to the output information (information as a result of face authentication in S1105).

Further, as another example, the output unit 112 may output the age and the information of the average error obtained by averaging the differences up to the upper limit and the lower limit of the error of the age. In that case, the generation unit 111 obtains the information of the average error by using the following equation 2 in any of S1206, S2203, and S2206, and includes the obtained information in the output information.
Err _Ave = ・ {(Err _max -Age) + (Age-Err _min )} / 2 ... (Equation 2)
Err _Ave indicates the average error to be obtained. Age indicates the age information acquired by any one of S1206, S2203, and S2206. Err _max indicates the upper limit of the age error indicated by the age error information. Err _min indicates the lower limit of the age error indicated by the age error information.
For example, when the age error is 22 to 32 years old, Err _max is 32 and Err _min is 22. Then, assuming that the age information indicates 28 years old, the average error of the age is {(32-28) + (28-22)} / 2 = 5 using Equation 3. The generation unit 111 generates information indicating "28 years old (error range: ± 5)" as output information.

As described above, in the present embodiment, the information processing system adjusts the information to be output according to the reliability of face authentication. As a result, the information processing system can output more appropriate information.
In the present embodiment, when the information processing system determines in S2201 that the reliability of the face authentication obtained in S1204 is less than a predetermined first threshold value, the processing proceeds to S2202. However, as another example, when the information processing system determines that the reliability of the face authentication obtained in S1204 is less than a predetermined first threshold value, the processing may proceed to S2204.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

For example, a part or all of the functional configuration of the above-mentioned information processing system may be mounted on the image processing device 1 as hardware. Although an example of the embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment. For example, the above-described embodiments may be arbitrarily combined.

1 Image processing device 4 Imaging device 11 CPU

Claims (14)

Extraction means for extracting the features of the object included in the input image,
A specific means for specifying the class of the object based on the feature amount extracted by the extraction means, and
A determination means for determining the reliability of a specific result by the specific means, and
When the reliability determined by the determination means satisfies a predetermined condition, an output means for outputting predetermined attribute information regarding the class of the object specified by the specific means and an output means.
Information processing device with. The information processing device according to claim 1, wherein the determination means determines the reliability of a specific result by the specific means based on the shooting conditions of the input image. The information processing apparatus according to claim 2, wherein the shooting conditions include at least one of a posture, orientation, position, degree of occlusion, and lighting conditions in the shooting environment of the input image. The information processing device according to claim 1, wherein the determination means determines the reliability of a specific result by the specific means based on the stability of the tracking process of the object. The determination means determines the reliability of the specific result by the specific means based on the degree of collation between the feature amount corresponding to the class specified by the specific means and the feature amount extracted by the extraction means. The information processing device according to claim 1. When the reliability determined by the determination means does not satisfy the condition, the output means includes the attribute information regarding the class of the object specified by the specific means, information indicating an error of the attribute information, and information indicating an error of the attribute information. The information processing apparatus according to any one of claims 1 to 5. When the reliability determined by the determination means does not satisfy the condition, the generation that generates the attribute information regarding the class of the object specified by the specific means based on the feature amount extracted from the input image. Have more means,
The information processing apparatus according to any one of claims 1 to 5, wherein the output means outputs the attribute information generated by the generation means when the reliability determined by the determination means does not satisfy the condition. .. When the reliability determined by the determination means does not satisfy the above conditions, the highest degree of matching between each feature amount corresponding to each of a plurality of predetermined classes and the feature amount extracted by the extraction means. Further having a generation means for generating the attribute information regarding the class of the object specified by the specific means based on the attribute information about each class corresponding to each of a predetermined number of collation degrees from the object.
The information processing apparatus according to any one of claims 1 to 5, wherein the output means outputs the attribute information generated by the generation means when the reliability determined by the determination means does not satisfy the condition. .. The information processing device according to any one of claims 1 to 8, wherein the attribute information includes at least one of age, gender, and race. The information processing device according to any one of claims 1 to 9, wherein the object is a face of a person. The information processing apparatus according to any one of claims 1 to 10, further comprising a control means for controlling the output means so as not to output information indicating a class of the object specified by the specific means. An imaging device that captures the input image and
The information processing device according to any one of claims 1 to 11.
System with. It is an information processing method executed by an information processing device.
An extraction step that extracts the features of the object included in the input image,
A specific step for identifying the class of the object based on the feature amount extracted in the extraction step, and
A determination step that determines the reliability of a specific result in the specific step, and
When the reliability determined in the determination step satisfies a predetermined condition, an output step for outputting predetermined attribute information regarding the class of the object specified in the specific step, and an output step.
Information processing methods including. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 11.

Images