JP2022088146A - Learning data generation device, person identifying device, learning data generation method, and learning data generation program - Google Patents

Abstract

To provide a person identifying device which no longer requires users to perform face registration in advance, and is able to highly-accurately identify persons even when users having similar faces exist, a person identifying method, and a person identifying program.SOLUTION: A person identifying device 1 comprises: a face detection section 11 for detecting areas for face images of persons from an image group photographed by a camera; a correspondence relation estimation section 13 for identifying an identical person based on coordinates of the face images from two images photographed within a time less than a threshold, assigning an identical person ID to the face image of the identical person, and assigning a separate person ID to the other face image; a learning data recording section 14 for associating and recording the face images with the person IDs, and recording a pair of the person IDs obtained from the image group photographed at an identical time; and a learning data set generation section 15 for generating learning data in a Triplet format composed of two face images having the identical person ID and one face image of the person ID configuring the pair with the person ID.SELECTED DRAWING: Figure 1

Description

The present invention relates to devices, methods and programs for identifying a person in an image.

Techniques for identifying a person in an image acquired from a camera or the like have been studied for many years, and are used in various situations such as communication robots and IoT devices used at home.

In order to identify a person, it is generally necessary to perform two processes, a face detection process and a face identification process.
The face detection process is a process of detecting a rectangular area (face area) in which a person's face is reflected from an input image. As the face detection process, a method using a cascade detector such as Non-Patent Document 1 and an object detection method by deep learning such as Non-Patent Document 2 and Non-Patent Document 3 have been proposed. Further, when the object detection method by deep learning is used, Non-Patent Document 4 can be used as a data set specialized for face detection of a person.

On the other hand, the face recognition process is a process of estimating that a given face image is one of known persons or an unknown person. The input face image is an image in which only the face area of one person is shown, or an image in which the face area detected by the above-mentioned face detection process is specified. The face identification process is composed of two steps: a learning step of registering a person to be identified and an estimation step of identifying a person in an input face image.

In the learning step, the face image of the person to be identified is converted into a feature vector and stored in the database. A method for converting a face image into a feature vector has been proposed, for example, in Non-Patent Document 5.
In the estimation step, the input face image is converted into a feature vector in the same way as in the learning step, and the person in the input face image is registered in advance by comparing with the feature vector of the pre-registered person. The estimation result of which of the people who made the match or which does not match is output.

In addition, various methods have been proposed for collecting facial images necessary for the learning step.
For example, Patent Document 1 proposes a device that collects images taken from various angles by instructing the user on the direction of the face when the user's face is registered in the system.
For example, Patent Document 2 proposes a device that smoothly performs a registration process by a robot instructing the user himself / herself when registering a user's face in the system.
For example, Patent Document 3 proposes a device that acquires a high-quality face image without imposing a load on the user by performing a registration process when the robot is held by the user.

Japanese Unexamined Patent Publication No. 2000-259834 Japanese Unexamined Patent Publication No. 2004-302645 International Publication No. 2018/084170

T. Mita et al., “Joint Haar-like features for face detection,” Tenth IEEE International Conference on Computer Vision (ICCV'05). S. Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” Twenty-ninth Conference on Neural Information Processing Systems (NIPS 2015). J. Redmon et al., “You Only Look Once: Unified Real-Time Object Detection,” 2016 IEEE conference on computer vision and pattern recognition (CVPR 2016). Y. Shuo et al., “WIDER FACE: A Face Detection Benchmark,” 2016 IEEE conference on computer vision and pattern recognition (CVPR 2016). Yoshihiko Kawai et al., "Study of Face Recognition Using Deep Neural Networks," 18th Information Science and Technology Forum (FIT 2019). G. B. Huang et al., “Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments,” University of Massachusetts, Technical Report 2007.

In a communication robot or an IoT device that performs a person identification process, it is common to register a user's face at the time of initial setting. At that time, as proposed in Patent Document 1 or Patent Document 2, a robot, an IoT device, or the like gives an instruction to the user by voice or text display, and the user turns his face in various directions according to the instruction. Face registration is done by moving. If the robot or IoT device is shared by the family, it is necessary to perform this face registration for the number of people. In addition, since the human face changes with aging, it may not be possible to correctly identify the person unless the registered information is updated regularly. Therefore, face registration with a robot or an IoT device causes a load on the user, and as the number of users used increases, the overall load increases accordingly.

As a means for solving this problem, for example, the method of Patent Document 3 can be used. This method can reduce the load on the user in face registration, but the robot that introduces this method must be designed so that it can be held, and a sensor that detects that the robot is also held inside. You need to implement the device. Therefore, there are restrictions on the design of the robot, and there is a problem that it cannot be realized by the IoT device, so it cannot be said that the method is highly versatile.

In addition, since communication robots and IoT devices are often used at home, it is assumed that there are people who have genetically similar faces in the set of users. Most of the methods for converting a face image into a feature vector as in Non-Patent Document 5 perform learning using a large-scale face image data set as in Non-Patent Document 6. Such a large facial image dataset contains people with various attributes (age, gender, race, etc.), and by learning with such a dataset, general-purpose facial feature extraction can be performed. It is expected that it will be possible. However, in the case of a system such as a communication robot or an IoT device that is supposed to be used at home, the attributes of the person that actually needs to be identified are limited. Then, in the conventional general-purpose face feature extraction method, people with similar faces are placed at close positions in the feature space, so that the distance between the feature vectors is short even though they are different people, which causes misidentification. There was a challenge.

An object of the present invention is to provide a person identification device, a person identification method, and a person identification program that can perform highly accurate identification even when there are users with similar faces without requiring prior face registration by the user. do.

The learning data generation device according to the present invention has a face detection unit that detects a region of a person's face image from a group of images taken by a camera, and two images taken within a time less than a threshold, from the face image. Correspondence relationship estimation unit that determines the same person based on the coordinates, assigns the same person ID to the face image of the same person, and assigns another person ID to another face image, and associates the face image with the person ID. A learning data recording unit that records a pair of the person IDs obtained from a group of images taken at the same time, two face images having the same person ID, and the person ID and the pair. It is provided with a learning data set generation unit for generating triplet format training data in which the face image of the person ID constituting the above is composed of one sheet, and outputs the training data for person identification using the face image as an input. ..

The learning data generation device includes a person direction estimation unit that derives a person direction based on the coordinates of the face image, and the correspondence estimation unit has a difference of less than a threshold value from two images taken at the same time. When the person direction is derived, the same person ID may be assigned to the two face images corresponding to the person direction.

In the correspondence estimation unit, the difference is less than the threshold when the person direction whose difference is less than the threshold is derived from the two images taken at the same time, or from the two images taken within the time less than the threshold. When the person direction is derived, the same person ID may be assigned to the two face images corresponding to the person direction, and another person ID may be assigned to the other face images.

Multiple cameras are placed at positions where the same person is not captured at the same time, and the correspondence estimation unit detects two face images whose distance between coordinates is less than the threshold from two images taken within the time less than the threshold. If this is the case, the same person ID may be assigned to the two face images, and another person ID may be assigned to the other face images.

The person identification device according to the present invention uses the training data output from the training data generation device to learn a feature extraction model, and converts the face image into a feature vector by the feature extraction model. For each user ID that integrates the person ID based on the distance between the feature vectors, the feature recording unit that records the center of gravity vector of the feature vector and the face image included in the newly captured image are described as the feature. The face feature extraction unit that converts to a feature vector by the extraction model and the feature vector obtained by the face feature extraction unit are compared with the center of gravity vector for each user ID, and the user ID at a distance less than the threshold is the person's. It is provided with a person estimation unit that outputs an estimation result.

The feature recording unit may receive an input of the number of users to be identified, cluster the center of gravity vector for each person ID to the number of users, and assign the user ID.

The feature recording unit may present a face image corresponding to the user ID, accept input of a user name, and record the face image together with the center of gravity vector.

When the feature recording unit is instructed to use a face image that is not the identification target among the presented face images, the feature recording unit may exclude the person ID including the instructed face image and perform clustering again.

The learning data generation method according to the present invention is a face detection step of detecting a region of a person's face image from a group of images taken by a camera, and two images taken within a time less than a threshold of the face image. Correspondence estimation step of determining the same person based on the coordinates, assigning the same person ID to the face image of the same person, and assigning another person ID to another face image, and associating the face image with the person ID. A learning data recording step of recording a pair of the person IDs obtained from a group of images taken at the same time, two face images having the same person ID, and the person ID and the pair. The computer executes a training data set generation step of generating triplet format training data consisting of one face image of a person ID constituting the above, and the training data for person identification using the face image as an input. Is output.

The learning data generation program according to the present invention is for making a computer function as the learning data generation device.

According to the present invention, it is possible to identify a person in the vicinity with high accuracy without performing prior face registration.

It is a figure which shows the functional structure of the person identification apparatus in 1st Embodiment. It is a figure which shows the processing example of the person direction estimation part in 1st Embodiment. It is a figure which shows the processing example of the correspondence relation estimation part in 1st Embodiment. It is a figure which shows the processing example of the learning data recording part in 1st Embodiment. It is a figure which shows the processing example of the learning data set generation part in 1st Embodiment. It is a figure which shows the processing example of the model learning part in 1st Embodiment. It is a figure which shows the processing example of the feature vector generation part in 1st Embodiment. It is a figure which shows the processing example of the person estimation part in 1st Embodiment. It is a flowchart which illustrates the flow of the person identification method in 1st Embodiment. It is a figure which shows the functional structure of the person identification apparatus in 2nd Embodiment. It is a figure which shows the processing example of the correspondence relation estimation part in 2nd Embodiment. It is a figure which shows the functional structure of the person identification apparatus in 3rd Embodiment. It is a figure which shows the processing example of the person registration part in 3rd Embodiment.

The person identification device according to the embodiment of the present invention collects facial images of users in the vicinity during operation of a communication robot, an IoT device, or the like, automatically constructs a data set for learning, and uses the constructed data set. Learn a deep learning model at a specific timing.
At this time, the person identification device detects the face area of the person captured by the camera and stores the labeled data in the database. When labeling, the property that the people who were close to each other in the previous and next frames are the same person, and the people who are shown at the same time are different people is used.

[First Embodiment]
Hereinafter, the first embodiment of the present invention will be illustrated and described. The person identification device 1 of the present embodiment is a device mounted on a communication robot or an IoT device and automatically identifies a person in the vicinity by using a camera array having a plurality of cameras as an input device.

FIG. 1 is a diagram showing a functional configuration of the person identification device 1.
The person identification device 1 is an information processing device provided with various interfaces in addition to the control unit 10 and the storage unit 20, and the control unit 10 executes software (person identification program) stored in the storage unit 20. , Various functions of this embodiment are realized.

The control unit 10 includes a face detection unit 11, a person direction estimation unit 12, a correspondence estimation unit 13, a learning data recording unit 14, a learning data set generation unit 15, a model learning unit 16, and a feature vector generation unit. A 17 (feature recording unit), a face feature extraction unit 18, and a person estimation unit 19 are provided.
In addition to the person identification program, the storage unit 20 includes a person memory 21, a face image database (DB) 22, a person pair database (DB) 23, a face feature extraction model 24, and a face feature database (DB) 25. And.

The face detection unit 11 acquires an image from a camera array connected to a communication robot, an IoT device, or the like, and detects the face of a person in the acquired image.
A camera array is, for example, a device in which a plurality of cameras are embedded in a cylindrical housing, and if eight cameras are installed at intervals of 45 degrees in the horizontal direction, images in all directions can be collected. It should be noted that the camera array does not necessarily have to collect images in all directions, and it is sufficient that the position where the user is present is captured when the communication robot, IoT device, or the like is used.
Further, it is not limited to simultaneous shooting by a plurality of cameras constituting the camera array, and even if one or several cameras continuously shoot the surroundings at intervals sufficiently faster than the movement of a person. good.

For the detection of the face image, for example, a method using a cascade detector as proposed in Non-Patent Document 1 or object detection by deep learning as proposed in Non-Patent Document 2 or Non-Patent Document 3 Techniques can be used. The face detection unit 11 performs face detection processing on each image acquired from each camera mounted on the camera array, and outputs coordinate information indicating the position of the face region in the image to the person direction estimation unit 12.
When the person identification device 1 completes a series of processes and outputs the person estimation result, the person identification device 1 acquires a new image from the camera array and executes the process of the face detection unit 11 again.

The person direction estimation unit 12 estimates the direction in which the person is located based on the position of the face detected by the face detection unit 11 in the image.
For the estimation of the person direction, for example, the method proposed in the following document A is used. In this case, the person direction estimation unit 12 prepares in advance an angle conversion table for each camera in the camera array, which associates the positions of the pixels of the camera image with the angles seen from a robot or the like equipped with the person identification device 1. Then, by using these, the direction of the person is estimated.
Reference A: Yuta Hagio et al., "Prototype and verification of a communication robot for watching TV with humans," IEICE Technical Report, vol. 119, no. 446, CNR2019-46, pp. 7-12, 2020.

FIG. 2 is a diagram showing a processing example of the person direction estimation unit 12.
In this example, for the sake of simplicity of explanation, the number of cameras in the camera array is two.
Here, the direction of the person is estimated using the face detection result output from the face detection unit 11. Specifically, the person direction estimation unit 12 converts the center coordinates of the face region obtained as the face detection result into an angle by an angle conversion table prepared in advance for each camera.

In this example, one person is shown in camera A and two people are shown in camera B. The person direction estimation unit 12 refers to the numerical value corresponding to the center coordinate of the face area in the angle conversion table for each camera, and as the estimation result of the person direction, 75 ° from the camera A and 75 ° and 115 from the camera B. ° is obtained.
The angle conversion table is actually the same size as the number of pixels of the camera, but here, it is simplified for the sake of explanation. The numerical value stored in the angle conversion table is set in units of 1 ° in this example, but an angle finer than 1 ° may be set.
Further, in this example, only the horizontal angle is estimated, but a two-dimensional numerical value (angle) may be stored in the angle conversion table and expanded so that the vertical angle can also be estimated.

The correspondence estimation unit 13 estimates the correspondence between people from the past person direction data stored in the person direction and the person memory 21 obtained by the person direction estimation unit 12, and the learning data recording unit 14 and the face feature. The data is output to the extraction unit 18 and the data in the person memory 21 is updated.

The person ID, the face image, the person direction, and the update time are recorded in the person memory 21, and the contents are updated every time the processing of the correspondence estimation unit 13 is performed.
Here, the person ID is an identifier for managing users who are presumed to be the same person by comparing the frames before and after (two images close in time), and when issuing a new person ID, the person ID is used. A person ID that does not overlap with the person ID used in the past is used.

First, the correspondence estimation unit 13 estimates the face image of the same person based on the person direction obtained from the person direction estimation unit 12. This is a process of estimating that a part of the angles of view of a plurality of cameras mounted on the camera array overlaps and the same person is simultaneously photographed from different angles by the plurality of cameras. The correspondence estimation unit 13 estimates by assuming that the persons existing in the same direction at substantially the same time are the same person when viewed from the communication robot or the IoT device. If there is an estimation result of an angle within the same person direction threshold set in advance with respect to the person direction estimated from images acquired from a plurality of cameras, the correspondence estimation unit 13 estimates these people as the same person. do. If a plurality of people exist within the same person direction threshold value and are not uniquely determined, the control unit 10 interrupts a series of processes, acquires a new image again, and executes the process from the face detection unit 11. ..

After that, the correspondence estimation unit 13 compares the estimated person direction with the past person direction stored in the person memory 21 and estimates whether the person is the same person. Here, it is presumed that when a specific person exists in a certain direction, the next acquired image also exists in a close direction.
Specifically, in the correspondence estimation unit 13, the person direction of the person obtained from the person direction estimation unit 12 is within the person direction of the person recorded in the person memory 21 and the person movement threshold value set in advance. In this case, these are regarded as the same person and the same person ID is assigned. If there is a person who cannot be regarded as the same person as the person recorded in the person memory 21 among the persons obtained from the person direction estimation unit 12, the correspondence estimation unit 13 has not used in the past. Person ID is assigned and recorded in the person memory 21.
The correspondence estimation unit 13 updates the data in the person memory 21 and outputs the updated information to the learning data recording unit 14 and the face feature extraction unit 18.

Further, when a timeout threshold is set in advance and the correspondence estimation unit 13 has data in the person memory 21 whose elapsed time from the update time exceeds the timeout threshold, the correspondence estimation unit 13 transfers these data from the person memory 21. After the deletion, the same person is estimated and the person ID is assigned.
For example, the time-out threshold value is set in a sufficiently short time with respect to the movement time of the distance corresponding to the person movement threshold value. As a result, even if a detection omission such as a failure in face detection occurs, a person ID can be assigned by collating with a person detected several frames before existing in the person memory 21.

FIG. 3 is a diagram showing a processing example of the correspondence relationship estimation unit 13.
In this example, the same person direction threshold is set to 5 °, the person movement threshold is set to 10 °, and the timeout threshold is set to 5 seconds. The person direction estimation unit 12 estimates the person directions of 75 °, 75 °, and 115 ° for three people, and the person memory 21 stores data that a person with a person ID of 3 exists in the direction of 72 °. Has been done.

First, among the persons obtained from the person direction estimation unit 12, the person with the data ID of 01 and the person with the data ID of 02 are within 5 ° of the same person direction threshold value in the correspondence estimation unit 13. , Presumed to be the same person. The data ID is an identifier given to the output data of the person direction estimation unit 12.

Next, the data with the person ID 3 recorded in the person memory 21 is in the 72 ° direction, and the person with the data ID 01 and the person with the data ID 02 are within 10 ° of the person movement threshold. Therefore, the correspondence estimation unit 13 assigns 3 as a person ID to the person whose data ID is 01 and the person whose data ID is 02. On the other hand, since the data whose data ID is within the estimated angle of 03 and the person movement threshold value is not included in the person memory 21, the correspondence estimation unit 13 newly assigns 4 as the person ID to the person whose data ID is 03. assign.
The correspondence estimation unit 13 updates the person memory 21 with these data. Since there is no data in the data this time in which the elapsed time from the update time exceeds the timeout threshold of 5 seconds, there is no data to be deleted from the person memory 21.

The learning data recording unit 14 records data in the face image DB 22 and the person pair DB 23 based on the face image and the person ID output from the correspondence estimation unit 13.
A set of an image ID, a face image, and a person ID is recorded in the face image DB 22, and a pair of person IDs existing at the same time is recorded in the person pair DB 23. The person identification device 1 identifies a person by utilizing the fact that persons having different person IDs that exist at the same time are facial images of different persons.

FIG. 4 is a diagram showing a processing example of the learning data recording unit 14.
In this example, two face images having a person ID of 3 and one face image having a person ID of 4 are input from the correspondence estimation unit 13.
The learning data recording unit 14 first adds these data to the face image DB 22, and then adds data indicating that the person IDs 3 and 4 exist at the same time to the person pair DB 23.

The learning data set generation unit 15 uses the data of the face image DB 22 and the person pair DB 23 to generate a data set for learning the weights of the face feature extraction model 24 (deep learning model). This process is executed, for example, periodically at a predetermined timing for training the face feature extraction model 24.
Here, a set of a face image and a person ID is recorded in the face image DB 22 by the processing of the learning data recording unit 14, and the face image having the same person ID is the face image of the same person. Further, the pair of person IDs recorded in the person pair DB 23 is a different person because they are people who were in different directions at the same time. The training data set generation unit 15 uses these properties to generate training data.

The training data is a triplet format in which three images are set, a second image (normal example) that is the same person as the first face image, and a third face image (negative) that is a different person. Example). By combining the data of the face image DB 22 and the person pair DB 23, various combinations of Triplet data can be generated.

FIG. 5 is a diagram showing a processing example of the learning data set generation unit 15.
In this example, a pair of person IDs "1-2" and "3-4" is recorded in the person pair DB 23. This indicates that a person with a person ID of 1 and a person with a person ID of 2 are different from each other, and a person with a person ID of 3 and a person with a person ID of 4 are different from each other.

Based on this information, the learning data set generation unit 15 extracts a face image from the face image DB 22 and generates learning data in Triplet format. In the training data, the face image 1 (reference source) and the face image 2 (normal example) are face images with the same person ID, and the face image 1 (reference source) and the face image 3 (negative example) are different person IDs. And, it becomes a pair recorded in the person pair DB23. For example, assuming that a face image having an image ID of 1 is used as a reference source, a face image having an image ID of 3 is a positive example, and a face image having an image ID of 2 is a negative example.

The model learning unit 16 learns the face feature extraction model 24, which is a deep learning model, using the triplet-format learning data generated by the learning data set generation unit 15.
As the deep learning model used here, for example, the Triplet-Based Variational Autoencoder (TVAE) proposed in the following document B can be used as an architecture.
Document B: H. Ishifaq et al. , "TVAE: Triplet-Based Variational Autoencoder using Metric Learning," ICLR 2018 Workshop.

TVAE is a kind of Variational Autoencoder (VAE) that inputs data in Triplet format, and like VAE, the difference between the reconstructed data obtained by inputting the latent variable that is the output of the encoder to the decoder and the input data. By using Triplet Ross, which evaluates the distance between the positive and negative examples, as the loss function, in addition to the Reaction Loss, which evaluates the data, the latent variables can be placed close to each other. , Negative examples of inputs are architectures that train latent variables to be placed far away.

The model learning unit 16 updates the network weight so as to minimize the loss by using the data set generated by the learning data set generation unit 15, and records it in the face feature extraction model 24.
As the initial value of the weight of the face feature extraction model 24, a trained model learned by using a large-scale face image data set as in Non-Patent Document 6 may be used.

FIG. 6 is a diagram showing a processing example of the model learning unit 16.
The model learning unit 16 acquires data from the data set generated by the learning data set generation unit 15, and performs learning processing, that is, updating the weights of the deep learning model. The target to be updated by the learning process is the weights of the Encoder Network and the Decoder Network.

The model learning unit 16 learns so that the image can be restored by using Reaction Loss, and the distance of the latent variable (for example, the average μ of the Gaussian distribution) is close in the positive example and negative by using Triplet Ross. Let's learn to be far away.
In this example, three Encoder Networks and three Encoder Networks are shown, but the same weights are used for these networks.

The feature vector generation unit 17 updates the face feature DB 25 that records the face feature data of a known person by using the face feature extraction model 24 updated by the model learning unit 16.
The user ID, the center of gravity vector, and the person ID list are recorded in the face feature DB 25.
Here, since the person IDs existing in the face image DB 22 and the person pair DB 23 are given one after another to the person framed in to the camera, the same person ID is used even if they are actually the same person. Not at all. Therefore, the feature vector generation unit 17 classifies the persons included in the face image DB 22 as follows by using the Euclidean distance of the feature vector extracted from the face image, and the classification result is used as the user ID for the face feature. Record in DB25.

First, the feature vector generation unit 17 acquires a face image from the face image DB 22, inputs these images into the Encoder Network of the face feature extraction model 24 updated by the model learning unit 16, and converts them into a feature vector. .. The feature vector is the output corresponding to the average of the outputs of the Encoder Network.

Next, the feature vector generation unit 17 calculates the vector of the center of gravity for each person ID with respect to the acquired feature vector.
Then, the feature vector generation unit 17 extracts arbitrary two vectors from the calculated center of gravity vector for each person ID, and compares the Euclidean distance of these vectors with the preset person feature threshold value. If the Euclidean distance between the two vectors is less than the person feature threshold, the two person IDs are considered to be the same person, and if they are greater than or equal to the person feature threshold, the two person IDs are considered to be different people. The feature vector generation unit 17 calculates the center of gravity vector again for the person IDs regarded as the same person, and registers them in the face feature DB 25.
The person feature threshold value may be determined based on the value of the margin used in the calculation of Triplet Ross of the model learning unit 16.

FIG. 7 is a diagram showing a processing example of the feature vector generation unit 17.
First, the feature vector generation unit 17 acquires a face image having a person ID of 1 to 4 from the face image DB 22, and converts it into a feature vector using the Encoder Network of the face feature extraction model 24 learned by the model learning unit 16. Then, the center of gravity vector for each person ID is calculated.

Next, the feature vector generation unit 17 calculates the Euclidean distance between the respective center of gravity vectors, and compares the calculated distance with the person feature threshold value. In this example, since the Euclidean distance between the centroid vectors having the person IDs 2 and 3 is less than the person feature threshold value, it is regarded as the same person and recorded in the face feature DB 25.
Although the feature vector is expressed in a two-dimensional space here, a higher-dimensional feature vector may be used. Further, the distance between the centroid vectors has been described as the Euclidean distance, but the distance is not limited to this, and other distances such as the Mahalanobis distance may be used.

The face feature extraction unit 18 converts the face image obtained from the correspondence estimation unit 13 into a feature vector by the Encoder Network of the face feature extraction model 24. This conversion process is the same as the process of the feature vector generation unit 17.
If there are multiple images with the same person ID among the face images input from the correspondence estimation unit 13, the face feature extraction unit 18 selects any one of them and converts it into a feature vector. Alternatively, the feature vectors of all or part of the plurality of images are statistically processed (for example, averaged) to obtain the feature vectors of the corresponding person ID.
Although it can be expected that the identification accuracy of a person is improved by adopting the averaged feature vector, the processing load and the processing time increase, so that the processing method may be selected depending on the situation.

The person estimation unit 19 compares each feature vector input from the face feature extraction unit 18 with the center of gravity vector of the user ID registered in the face feature DB 25, and calculates the Euclidean distance. When the person estimation unit 19 finds a user ID whose calculated distance is less than the person feature threshold, the person estimation unit 19 estimates that the person in the face image is the user. Further, in the person estimation unit 19, when the center of gravity vector whose Euclidean distance from the feature vector obtained from the face feature extraction unit 18 is less than the person feature threshold does not exist in the face feature DB 25, the input face image is a face feature. It is estimated to be an unknown user that does not exist in DB25.

FIG. 8 is a diagram showing a processing example of the person estimation unit 19.
In this example, the center of gravity vectors for three users A to C are recorded in the face feature DB 25, and the feature vectors for three people with person IDs 14 to 16 are input from the face feature extraction unit 18. ..

The person estimation unit 19 confirms whether the center of gravity vector whose Euclidean distance is less than the person feature threshold exists in the face feature DB 25 for each of the input feature vectors. In this example, since the feature vector having a person ID of 14 and the center of gravity vector of the user B and the feature vector having a person ID of 15 and the center of gravity vector of the user C satisfy this condition, the person with the person ID of 14 is the user B. A person with a person ID of 15 is presumed to be user C. Further, since the center of gravity vector whose Euclidean distance from the feature vector of the person whose person ID is 16 is less than the person feature threshold does not exist in the face feature DB 25, it is estimated that the person with the person ID 16 is an unknown user. ..

The model learning function in the person identification device 1 is composed of a learning data set generation unit 15, a model learning unit 16, and a feature vector generation unit 17, and various patterns can be considered for the timing of learning. For example, in addition to the method of executing periodically at a fixed cycle, the method of executing when a face image estimated to be an unknown user exists in the person estimation unit 19, and the face image of a different person ID in the person estimation unit 19 The method to execute when it is presumed to be the same user can be mentioned.

FIG. 9 is a flowchart illustrating the flow of the person identification method.
Here, as an example, a case where learning is performed when the face images of different person IDs are estimated to be the same user by the person estimation unit 19 is shown.

In step S1, the control unit 10 captures and acquires an image from each camera in the camera array.
In step S2, the face detection unit 11 detects a face region from each acquired image.

In step S3, the control unit 10 determines whether or not the face region has been detected. If this determination is YES, the process proceeds to step S4, and if the determination is NO, that is, if no face area is detected, the process returns to step S1.

In step S4, the person direction estimation unit 12 estimates the person direction from the center coordinates of each detected face area.
In step S5, the correspondence relationship estimation unit 13 estimates the correspondence relationship with the person in the past frame, that is, whether or not they are the same person, with reference to the person memory 21.

In step S6, the control unit 10 determines whether or not the correspondence between the persons can be uniquely estimated. If this determination is YES, the process proceeds to step S7, and if the determination is NO, the process returns to step S1.

In step S7, the learning data recording unit 14 records the face image and the person ID in the face image DB 22, and the pair of person IDs (pairs of different people) that existed at the same time in the person pair DB 23.

In step S8, the face feature extraction unit 18 converts the face image into a feature vector using the face feature extraction model 24.
In step S9, the person estimation unit 19 compares the feature vector obtained in step S8 with the center of gravity vector of each user recorded in the face feature DB 25, and performs person estimation.

In step S10, the control unit 10 determines whether or not a person with a different person ID is determined to be the same user. If this determination is YES, the process proceeds to step S11 to learn the face feature extraction model 24, and if the determination is NO, the process ends.

In step S11, the learning data set generation unit 15 acquires data from the face image DB 22 and the person pair DB 23, and generates a data set for learning.
In step S12, the model learning unit 16 learns the weights of the face feature extraction model 24 using the learning data set.
In step S13, the feature vector generation unit 17 converts the face image recorded in the face image DB 22 into a feature vector using the face feature extraction model 24, and updates the face feature DB 25.

According to the present embodiment, the person identification device 1 detects a region of a person's face image from a group of images taken by a camera, and converts two images taken within a time less than a threshold into the coordinates of the face image. Based on this, the same person is determined, the same person ID is assigned to the face image of the same person, and another person ID is assigned to another face image. Then, the person identification device 1 records the face image in association with the person ID, and records the pair of person IDs obtained from the image group taken at the same time as another person. The person identification device 1 generates triplet-format learning data including two face images having the same person ID and one face image of a person ID forming a pair with the person ID.
Thereby, the person identification device 1 can learn the deep learning model by using the triplet format training data composed of the face image of a certain person, the face image of the same person, and the face image of a different person.

That is, the person identification device 1 converts the face image into a feature vector by the face feature extraction model 24 learned using the generated learning data, and the user who integrates the person ID based on the distance between the feature vectors. The center of gravity vector of the feature vector is recorded for each ID. Then, when the face image included in the newly captured image is converted into a feature vector by the face feature extraction model 24, the person identification device 1 compares this feature vector with the center of gravity vector for each user ID and is less than the threshold value. The user ID at the distance of is output as the estimation result of the person.
As a result, the person identification device 1 can automatically record the feature vector (center of gravity vector) for each user ID without performing prior face registration by the user, and can identify the surrounding people with high accuracy.

The person identification device 1 derives the person direction based on the coordinates of the face image, and when the person direction whose difference is less than the threshold is derived from the two images taken at the same time, the person direction is directed to these person directions. The same person ID is assigned to the two corresponding face images.
Therefore, the person identification device 1 can appropriately determine the same person from the face images obtained from the plurality of camera images.
Further, the person identification device 1 is the same as the two face images corresponding to these person directions when the person direction whose difference is less than the threshold is derived from the two images taken within the time less than the threshold. Assign a person ID and assign another person ID to another face image.
Therefore, the person identification device 1 can appropriately associate facial images existing in the same direction as the same person in a short period of time.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described.
This embodiment is a person identification device in which a camera is installed in each room such as a house or an office and automatically identifies a person who acts in the space where the camera is installed. That is, unlike the camera array of the first embodiment, it is premised that the same person is not captured by a plurality of cameras at the same time.

FIG. 10 is a diagram showing a functional configuration of the person identification device 2 in the present embodiment.
In the present embodiment, as compared with the first embodiment, there is no person direction estimation unit 12, the functions of the correspondence estimation unit 13 are different, and the data recorded in the person memory 21 is different.

The correspondence estimation unit 13 compares the coordinate information of the face area output from the face detection unit 11 with the coordinate data of the past person stored in the person memory 21, estimates the correspondence of the person, and records the learning data. The data is output to the unit 14 and the face feature extraction unit 18, and the data in the person memory 21 is updated.
The person ID, face image, camera ID, coordinates, and update time are recorded in the person memory 21, and the contents are updated every time the correspondence estimation unit 13 performs processing. The camera ID is an identifier of the camera that captured the person, and the coordinates are the center coordinates of the face area of the person in the image. The person ID, the face image, and the update time are the same as those in the first embodiment.

The correspondence estimation unit 13 compares the center coordinates of the face area with the coordinates of the past person stored in the person memory 21, and the person obtained by the face detection unit 11 is recorded in the person memory 21. Estimate whether it is the same person as the person who is. Here, it is assumed that when the face of a specific person is present at a certain coordinate of a camera image, this person is located close to the original coordinate in the next image captured by the same camera.

Specifically, when the Euclidean distance between the coordinates of the face area obtained from the face detection unit 11 and the coordinates of the person recorded in the person memory 21 is within the preset person movement threshold, the same person is used. The same person ID is assumed to be assigned. Further, if there is a person who cannot be regarded as the same person as the person recorded in the person memory 21 among the persons obtained from the face detection unit 11, a new person ID that has not been used in the past is assigned. It is recorded in the person memory 21.
The deletion from the person memory 21 by the timeout threshold is the same as in the first embodiment. Then, the correspondence estimation unit 13 updates the data in the person memory 21 and outputs the updated information to the learning data recording unit 14 and the face feature extraction unit 18.

FIG. 11 is a diagram showing a processing example of the correspondence relationship estimation unit 13.
In this example, the person movement threshold value is set to 10 and the timeout threshold value is set to 5 seconds. The center coordinates of the face region obtained from the face detection unit 11 are the coordinates of the camera A (200,300), the coordinates of the camera B (250,280), and the coordinates of the camera B (560,250). Further, the person ID: 3 recorded in the person memory 21 is the coordinates of the camera A (198,304), and the person ID: 4 is the coordinates of the camera B (563,242).

When calculating the Euclidean distance between face images taken by the same camera, the coordinates between the coordinates with data ID 01 and the coordinates with person ID 3 are 4.47, the coordinates with data ID 02 and the coordinates with person ID 4 are 4. The distance is 315.30, and the distance between the coordinates with the data ID of 03 and the coordinates with the person ID of 4 is 3.61.
Since the person movement threshold value is 10, the correspondence estimation unit 13 assigns a person ID: 3 to a person with a data ID of 01 and a person ID: 4 to a person with a data ID of 03. On the other hand, since the person memory 21 does not include the coordinates whose Euclidean distance is within the person movement threshold with respect to the coordinates whose data ID is 02, the correspondence estimation unit 13 newly applies the coordinates whose data ID is 02 to the person whose data ID is 02. Person ID: 5 is assigned.

The correspondence estimation unit 13 updates the person memory 21 with these data. Since there is no data in the data this time in which the elapsed time from the update time exceeds the timeout threshold of 5 seconds, there is no data to be deleted from the person memory 21.
Further, the subsequent processing is the same as that of the first embodiment.

In the present embodiment, the description has been made on the premise that a plurality of cameras are input, but the present invention is not limited to this, and the number of cameras may be one. Further, the camera to be used has been described as taking a rectangular area, but the present invention is not limited to this, and for example, an omnidirectional camera capable of taking a picture of the entire surroundings may be used.

According to the present embodiment, a plurality of cameras are arranged at positions where the same person is not captured at the same time, and the person identification device 2 has a distance between coordinates of less than the threshold from two images taken within a time less than the threshold. When two face images are detected, the same person ID is assigned to these two face images, and another person ID is assigned to the other face images.
Therefore, the person identification device 2 can identify a person in each room in a space where an unspecified number of people enter and exit, and various IoT devices (lighting or home appliances) based on the arrangement of people in each room. Can be controlled.

[Third Embodiment]
Hereinafter, a third embodiment of the present invention will be described.
In the case of a communication robot or an IoT device equipped with a conventional face recognition device, there are many forms in which a user's name is registered at the same time as performing face registration processing. The person identification device of the present embodiment clusters the feature vectors acquired by automatic learning into a specified number, and registers the feature vectors including the input user's name in the database.

FIG. 12 is a diagram showing a functional configuration of the person identification device 3 in the present embodiment.
Compared with the first embodiment, the person identification device 3 of the present embodiment includes a person registration unit 17a (feature recording unit) instead of the feature vector generation unit 17, and the data recorded in the face feature DB 25 is different.
Although the functional configuration based on the configuration of the first embodiment is shown here, the functional configuration may be a modification of the feature vector generation unit 17 and the face feature DB 25 based on the second embodiment.

In the present embodiment, a communication robot or an IoT device equipped with a person identification device 3 is operated for about 1 to several days, and after sufficient data for learning is recorded in the face image DB 22 and the person pair DB 23, a learning data set is performed. Processing by the generation unit 15, the model learning unit 16, and the person registration unit 17a is executed. Further, the processing of the face feature extraction unit 18 and the person estimation unit 19 is not performed until the data is saved in the face feature DB 25 by the person registration unit 17a.

The person registration unit 17a converts the face image registered in the face image DB 22 into a feature vector using the face feature extraction model 24 learned by the model learning unit 16, and shows the result of clustering to the user. The data including the user name is registered in the face feature DB 25.
A user ID, a user name, a center of gravity vector, and a person ID list are recorded in the face feature DB 25.

First, the person registration unit 17a acquires a face image from the face image DB 22, and converts these images into a feature vector using the Encoder Network of the face feature extraction model 24 learned by the model learning unit 16. The feature vector is the output corresponding to the average of the outputs of the Encoder Network.
Next, the person registration unit 17a calculates the vector of the center of gravity for each person ID with respect to the acquired feature vector.

After that, the person registration unit 17a requests the user to input the number of users to be identified. For example, when a communication robot or an IoT device equipped with a person identification device 3 is used in a home, the number of family members is input. Subsequently, the person registration unit 17a clusters the center of gravity of the feature vector calculated for each person ID according to the number of people input by the user by the k-means method or the like.

Next, the person registration unit 17a randomly extracts several images from the facial images of the person IDs belonging to each cluster and presents them to the user. The user inputs the name of a person in several presented facial images.
At this time, if the presented face image contains a face image of a person who is not the identification target, the user inputs to that effect to the person identification device 3. In this case, the person registration unit 17a performs clustering again on the image from which the center of gravity vector of the person ID included in the presented image has been removed, and again performs several face images of the person ID belonging to each cluster. Present to the user.

Specifically, re-clustering when a non-target facial image is presented is performed by, for example, the following procedure.
(1) For each cluster, one face image is presented from the person ID to which the person belongs. However, an upper limit of the number of face images to be presented may be set in advance, and the number of images exceeding this upper limit may be limited so as not to be presented. The user confirms the presented face image list for each cluster, and selects a cluster that includes a person not to be identified.
(2) One face image of each person ID belonging to the cluster selected by the user is presented in a list. The user selects a face image that is not to be identified from the list.
(3) The person ID corresponding to the face image selected by the user is removed, and the result of clustering again is presented.

When a name is input for all the presented images of each cluster, the person registration unit 17a calculates the center of gravity of the center of gravity vector of the person ID belonging to the cluster, and uses the user ID, the user name, the center of gravity vector, and the cluster as the center of gravity. The included person ID list is registered in the face feature DB 25.

FIG. 13 is a diagram showing a processing example of the person registration unit 17a.
Face images with person IDs 1 to 5 are acquired from the face image DB 22, and the person registration unit 17a features each of these face images using the Encoder Network of the face feature extraction model 24 learned by the model learning unit 16. It is converted into a vector, and the center of gravity vector for each person ID is calculated.

After that, the user inputs the number of users to be identified. In this example, since "3 people" is input, the person registration unit 17a clusters the center of gravity vector for each person ID into three by the k-means method.

Next, the person registration unit 17a presents to the user several face images randomly selected from the face images of the person ID included in each cluster, and asks the user to input the name of the person. In this example, "Taro", "Hanako", and "Jiro" are input respectively. The person registration unit 17a registers the vector that is the center of gravity of the center of gravity vector of the person ID included in each cluster in the face feature DB 25 together with these input names.
By using the face feature DB 25 registered in this way, the person estimation unit 19 can also acquire the name of the estimated person.

According to the present embodiment, the person identification device 3 accepts the input of the number of users to be identified, clusters the center of gravity vector for each person ID to this number of users, and assigns the user ID.
Therefore, the person identification device 3 can cluster the feature vector (center of gravity vector) based on the actual number of users, whereby the user can be appropriately identified.

The person identification device 3 presents a face image corresponding to the user ID, accepts the input of the user name, and records it together with the center of gravity vector.
Therefore, the person identification device 3 can identify the face image and output the user name as the estimation result.

When the person identification device 3 is instructed to use a face image that is not the identification target among the presented face images, the person identification device 3 excludes the person ID including the instructed face image and performs clustering again.
Therefore, the person identification device 3 can record the feature vector (center of gravity vector) excluding the face image group that is not the identification target, so that the identification accuracy of the person can be improved.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Moreover, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects according to the present invention are not limited to those described in the embodiments.

In the above-described embodiment, unused IDs are assigned to the person IDs recorded in the face image DB 22 and the person pair DB 23, but an upper limit may be set for the number of person IDs. In this case, the data associated with the old person ID may be deleted and this ID may be reused.

The camera for acquiring the face image has been described as not moving, but for example, when the camera is provided in a movable robot or the like, the direction of the person changes with the movement, so that the contents of the person memory 21 are reset. Will be processed again.

In the present embodiment, the configuration and operation of the person identification device (1, 2, 3) have been mainly described, but the present invention is not limited to this, and the present invention includes each component and is a method or program for identifying a person. It may be configured as.

Further, a program for realizing the functions of the person identification device (1, 2, 3) is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read and executed by the computer system. It may be realized by.

The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built in a computer system.

Furthermore, a "computer-readable recording medium" is a communication line that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. ..

1, 2, 3 Person identification device (learning data generation device)
10 Control unit 11 Face detection unit 12 Person direction estimation unit 13 Correspondence relationship estimation unit 14 Learning data recording unit 15 Learning data set generation unit 16 Model learning unit 17 Feature vector generation unit (feature recording unit)
17a Person registration department (feature recording department)
18 Face feature extraction unit 19 Person estimation unit 20 Storage unit 21 Person memory 22 Face image DB
23 Person pair DB
24 Face feature extraction model 25 Face feature DB

Claims (10)

A face detector that detects the area of a person's face image from a group of images taken by a camera,
From two images taken within the time less than the threshold, the same person is determined based on the coordinates of the face image, the same person ID is assigned to the face image of the same person, and another person is assigned to another face image. Correspondence estimation unit that assigns ID and
A learning data recording unit that records a face image in association with the person ID and records a pair of the person IDs obtained from a group of images taken at the same time.
A learning data set generation unit that generates triplet-format learning data including two face images having the same person ID and one face image of the person ID and the person ID constituting the pair. Prepare,
A learning data generation device that outputs the learning data for identifying a person by inputting a face image. A person direction estimation unit for deriving a person direction based on the coordinates of the face image is provided.
The claim that the correspondence estimation unit assigns the same person ID to two face images corresponding to the person direction when the person direction whose difference is less than the threshold is derived from the two images taken at the same time. The learning data generation device according to 1. In the correspondence estimation unit, the difference is less than the threshold when the person direction whose difference is less than the threshold is derived from the two images taken at the same time, or from the two images taken within the time less than the threshold. The learning data generation device according to claim 2, wherein when the person direction is derived, the same person ID is assigned to two face images corresponding to the person direction, and another person ID is assigned to another face image. Multiple cameras are placed in positions where the same person cannot be seen at the same time.
When two face images in which the distance between coordinates is less than the threshold are detected from the two images taken within the time less than the threshold, the correspondence estimation unit has the same person ID in the two face images. The learning data generation device according to claim 1, wherein a person ID is assigned to another face image. A model learning unit that learns a feature extraction model using the learning data output from the learning data generator according to any one of claims 1 to 4.
A feature recording unit that converts the face image into a feature vector by the feature extraction model and records the center of gravity vector of the feature vector for each user ID that integrates the person ID based on the distance between the feature vectors.
A face feature extraction unit that converts a face image included in a newly captured image into a feature vector by the feature extraction model, and a face feature extraction unit.
A person identification unit including a person estimation unit that compares the feature vector obtained by the face feature extraction unit with the center of gravity vector for each user ID and outputs a user ID at a distance less than the threshold value as a person estimation result. Device. The person identification device according to claim 5, wherein the feature recording unit receives an input of the number of users to be identified, clusters the center of gravity vector for each person ID to the number of users, and assigns the user ID. The person identification device according to claim 6, wherein the feature recording unit presents a face image corresponding to the user ID, accepts input of a user name, and records the face image together with the center of gravity vector. The seventh aspect of claim 7, wherein when the feature recording unit is instructed to use a face image that is not to be identified among the presented face images, the person ID including the instructed face image is excluded and clustering is performed again. Person identification device. A face detection step that detects the area of a person's face image from a group of images taken by a camera,
From two images taken within the time less than the threshold, the same person is determined based on the coordinates of the face image, the same person ID is assigned to the face image of the same person, and another person is assigned to another face image. Correspondence estimation step to assign ID and
A learning data recording step of recording a face image in association with the person ID and recording a pair of the person IDs obtained from a group of images taken at the same time.
A training data set generation step of generating triplet-format learning data including two face images having the same person ID and one face image of the person ID and the person ID constituting the pair. Computer run,
A learning data generation method for outputting the learning data for identifying a person using a face image as an input. The learning data generation program for operating a computer as the learning data generation device according to any one of claims 1 to 4.

Images