JP2002157592A - Method and device for registering personal information and recording medium recording its program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To register personal information which is robust concerning the direction of a face and the change of display without preparing a dictionary in advance. SOLUTION: A face picture extracting part 11 extracts a face picture from an inputted picture. A similarity degree calculating part 12 collates the extracted face picture with a face picture registered in a dictionary 2 and calculates a similarity degree for each person. A person picture output part 13 displays the pictures in a display part 5 in order from the one with the highest similarity degree. A dictionary register part 14 registers the extracted picture in the dictionary 2 when an operator judges that the extracted face picture is the new one after viewing the display of the display part 5 and operates a key.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method and an apparatus for registering personal information.

[0002]

2. Description of the Related Art When a video is used secondarily, consideration for portrait rights is indispensable. However, even if you want to clear the portrait right,
It has been difficult to identify individuals who appear in the video for past videos. This is because the information on the portrait right, personal name, affiliation, etc. were not managed together with the video. In order to add portrait right information, it is first necessary to cut out a person from the video, but if it is performed manually, it takes time, so automation is desired.

[0003] An authentication system using a face image has been developed, and is applied to entry / exit management of buildings and the like. In a typical system, a dictionary in which one or more frontal face images are registered for one person is created in advance, and authentication is performed by comparing the captured frontal face with the dictionary. Furthermore, according to the face recognition technology (Japanese Patent Application Laid-Open No. 7-302327), authentication can be performed regardless of the direction of the face by registering not only the front face but also a plurality of face images taken by changing the shooting direction in the dictionary. I have to.

[0004]

It is conceivable to apply the above conventional technology (face recognition technology) to personal information registration.
There are the following problems.

[0005] 1) It is difficult to prepare a dictionary in advance. For example, it is difficult to create a face image dictionary of a deceased actor.

2) Misrecognition often occurs because the direction and expression of the face greatly change. In the case of the authentication system, the user can consciously turn his / her face to the camera or keep a constant expression, but in a drama or a conversation program, the orientation and expression of the face cannot be kept constant.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and does not require the preparation of a dictionary in advance, and is a method of registering personal information that is robust to changes in face orientation and facial expressions.
It is an object to provide a device and a recording medium on which a personal information registration program is recorded.

[0008]

According to the present invention, a face area is detected from an input image, and the detected face area image is registered in a dictionary storing the face image in association with a person number. Calculate the degree of similarity for each person by comparing with
The face images are presented to the operator in order of similarity.

Although the authentication system needs to be completely automatic, the person information registration method of the present invention allows human intervention, so that an operator can easily correct an error caused by erroneous recognition. Further, by dynamically configuring the dictionary according to the operator's operation, there is no need to prepare the dictionary in advance.

In addition, in the authentication system, it is necessary to use a feature that does not change with the passage of time (to enable recognition even when the time has elapsed at the time of dictionary registration and at the time of authentication). , Eyeglasses, makeup, etc., which do not change significantly over time, cannot be used. However, the person registration system of the present invention takes advantage of the fact that it is possible to assume that hairstyles and clothes do not frequently change in one program. By registering such information in a dictionary, it is robust against changes in face direction and facial expressions.

[0011]

Next, embodiments of the present invention will be described with reference to the drawings.

Referring to FIG. 1, a personal information registration device according to an embodiment of the present invention includes an input unit 1, a dictionary 2, a processing unit 3, a storage unit 4, and a display unit 5.

The input unit 1 is for inputting an image and various instructions of an operator.

In the dictionary 2, a face image of a person is registered. Various construction methods of the dictionary 2 are conceivable. The simplest method is to store all the normalized face images in association with the person number n, as shown in the schematic diagram of FIG. In the dictionary of FIG. 4, the face image of the person number n is stored in the n-th row. The face image is normalized with respect to image size and brightness. Of course, this increases the data size of the dictionary and takes time to collate, so various methods of summarizing the dictionary have been developed. For example, the average of the face images registered for the person number n can be stored as a representative, or the registered face images can be reduced by vector quantization of the face image by the k-average method or the like. . Further, instead of storing the face image as it is, it is also possible to compress it into a smaller number of feature vectors and then register it in the dictionary. When referring to the face region, it often points to a region including the eyes, the mouth, and the nose (for example, 51 in FIG. 6). In this embodiment,
Information such as a hairstyle and clothes can be used, and it is desirable to register an area including a part of hair and clothes (for example, 52 in FIG. 6) as one of the features in the dictionary 2 in order to increase the recognition rate. .

The processing unit 3 compares the extracted face image with a face image registered in the dictionary 2 to extract a face image from the input image, and determines the degree of similarity for each person. A similarity calculation unit 12 for calculating, a person image output unit 13 for outputting face images to the display unit 5 in descending order of similarity, and registering input person information in the dictionary 2 or adding a face area image to the dictionary 2 It includes a dictionary registration unit 14 for performing the operations. The storage unit 4 stores an input image and a calculation result in the middle of the similarity calculation.

FIG. 2 is a flowchart showing the operation of the personal information registration device.

In step 21, a video deck or a digitized video file is read from the input unit 1 and images are input one by one. It is also possible to input images sampled at fixed time intervals.

In step 22, the face area detecting section 11 detects a face area from the image. For example, two face regions 31 and 32 surrounded by a rectangle are detected from the image shown in FIG.
For this detection, for example, a technique disclosed in Japanese Patent Application Laid-Open No. 9-50528 “Person detection device” can be used. In this technique, a face area is roughly obtained from color information, and positioning is performed using a mosaic feature.
Also, when the input is a video, there is a high correlation between temporally adjacent images. Therefore, face area tracking techniques (for example, Shin'ichi Satoh, Yuichi Nakamura, Takeo Kana
de: Name-It; Naming and Detecting Faces in News Vi
deos, IEEE Multimedia, pp.22-35, January-March, 19
By using (99), the amount of calculation can be reduced. Further, a configuration is possible in which an operator designates a face area with a mouse or the like, and the face area is detected only in the vicinity of the face area. This can be used to add a face area that could not be automatically detected.

Here, the face image detecting method disclosed in Japanese Patent Application Laid-Open No. 9-50528 will be described. For an input image, first, the presence or absence of a person skin color area is determined, the mosaic size is automatically determined for the person skin color area, the candidate area is tessellated, and the distance from the person face dictionary is calculated. The presence or absence of a face is determined, and a human face is cut out.

In step 23, it is checked whether all the face areas have been processed. When all face images have been processed, the process proceeds to step 21.

In step 24, the similarity calculator 12 calculates the similarity by comparing the detected face area image with the face image in the dictionary 2. The similarity is calculated for each person registered in the dictionary 2, and the similarity with the person with the person number n is represented as r (n). The larger the similarity r (n), the higher the possibility that the face area image is the person with the person number n.

The method of comparing the dictionary 2 with the face area image and calculating the similarity may have various variations. Explaining with the dictionary shown in FIG. 4, if there are N face images registered for the person number n, the correlation between the input face area images is calculated for each of the N face images, and the N face images are calculated. The largest correlation value is returned as the similarity r (n).

In step 25, the person image output unit 13 arranges the face images registered in descending order of the similarity r (n) and arranges them in the display unit 5.
And present it to the operator.

FIG. 5 shows an example of the user interface. An area 40 is an area for displaying a video, and the face area detected in step 22 is displayed by being surrounded by a frame 41. The button 42 is a button for playing / stopping a video. The area 44 is a list of characters registered in the dictionary 2, and displays the face images of the characters, the names of persons, and the contact information in parallel. When step 25 is performed,
The areas 44 are rearranged, and the person most similar to the face area 41 of interest is displayed so as to be at the top. As in the case where the conversion candidates are displayed in a list in the kana-kanji conversion, the operation of the operator is made more efficient by presenting the person information to the operator in descending order of similarity.

Step 26 waits for an operator input. When the operator presses the button 45 of FIG. 5, it is determined that a new person not registered in the dictionary 2 has appeared,
Move to step 27. When the operator selects one of the character lists displayed in the area 44 with a mouse or the like, it is determined that a person already registered in the dictionary 2 has been selected, and the process proceeds to step 28. When the operator presses the button 46, the detected face area 41 is determined to be erroneously detected, and
The processing related to the area is cancelled, and the process proceeds to step S23.

In step 27, since a new person who is not registered in the dictionary 2 appears, information such as a person's name, affiliation, and contact information is input from the keyboard of the input unit 1 and the like.
4 registers in the dictionary 2.

In step 28, the face area image 4 is stored in the dictionary 2.
Add 1. By adding the face area image to dictionary 2,
Recognition that does not depend on facial expressions or face orientation becomes possible. As described above, when a face image having a high degree of similarity is already registered in the dictionary 2, a configuration in which a face area image is not added to the dictionary 2 is of course possible.

The processing of steps 24 to 28 is repeated for all face areas. When all face areas have been processed, the next image is input in step 21.

As a result, temporal person information indicating who appeared at what time can be obtained. This information can be displayed on a timeline as shown in an area 49 in FIG. In the area 49, the horizontal axis represents time, the vertical axis represents characters, and the rectangle represents a time section.

By using temporal person information, it is possible to find out who was performing at a certain time, to find out how long a performer is performing for each performer, or to select a certain scene and perform in it. Can be output as a list. It is possible to play back only a scene in which a certain person appears, or to skip a scene in which a person with a portrait right problem appears. A system for paying a performer to a performer based on the appearance time and the number of appearance cuts can be constructed.

As shown by the buttons 47 and 48 in FIG.
It can also provide the ability to save dictionaries and load saved dictionaries. For example, it is possible to prepare a dictionary in which characters are registered for each program, and perform a person registration operation while switching the dictionary.

The processing of the processing section 3 of FIG. 1, ie, the processing of FIG. 2, is recorded as a personal information registration program on a computer-readable portable recording medium such as a floppy disk, CD-ROM, or magneto-optical disk. It can be executed on a computer such as a personal computer.

[0033]

As described above, according to the present invention,
There is no need to prepare a dictionary in advance, and it is possible to provide a personal information registration method, apparatus, and program storage medium that are robust to changes in face direction and facial expressions.

By using temporal person information, it is possible to find out who was performing at a certain time, to check the length of the performing time for each performer, or to select a certain scene and perform in that scene. Can be output as a list. It is also possible to play back only a scene in which a certain person appears, or to skip a scene in which a person with a portrait right problem appears. A system for paying a performer to a performer based on the appearance time and the number of appearance cuts can be constructed.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a person information registration device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of the person information registration device of FIG. 1;

FIG. 3 is an explanatory diagram of face area detection.

FIG. 4 is a schematic diagram for explaining a dictionary construction method.

FIG. 5 is a diagram illustrating an example of a user interface.

FIG. 6 is a diagram for explaining a method of setting a face configuration.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Input part 2 Dictionary 3 Processing part 4 Storage part 5 Display part 11 Face area extraction part 12 Similarity calculation part 13 Person image output part 14 Dictionary registration part 21-28 Step 31, 32 Face area 40 Video display area 41 Frame 42, 45, 46, 47, 48 buttons 44, 49 areas 51, 52 areas

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Haruhiko Kojima 2-3-1 Otemachi, Chiyoda-ku, Tokyo F-term in Nippon Telegraph and Telephone Corporation (reference) 5B057 AA19 BA02 CA12 CA16 DA07 DB02 DC34 5L096 BA02 EA35 HA09 JA01 JA03 KA03

Claims (7)

[Claims] An image inputting step, a face area detecting step from an input image, and a face registered in a dictionary storing the detected face area image in association with a face number corresponding to a person number. A person information registration method, comprising: calculating a similarity for each person by comparing the image with an image; and presenting face images to an operator in order of the similarity. 2. The method according to claim 1, further comprising the step of registering, in the dictionary, features that change significantly over time as characteristics of a person. 3. The method according to claim 1, further comprising the step of dynamically updating the dictionary based on an instruction from an operator.
Or the method of 2. 4. An input means for inputting an image, a display means for displaying a face image, a face image is detected from the input image, and the detected face area image is corresponded to a person number. A personal information registration device having a processing unit that calculates a similarity for each person by comparing the image with a face image registered in a dictionary stored with the dictionary and displays the face images on the display unit in the order of the similarity. 5. The apparatus according to claim 3, wherein the processing unit registers, in the dictionary, a feature that greatly changes over time as a feature of a person. 6. The apparatus according to claim 4, further comprising an input unit for inputting an instruction by an operator, wherein the processing unit dynamically updates the dictionary according to the instruction input by the operator. 7. A recording medium recorded for causing a computer to execute the method according to claim 1 as a personal information registration program.