KR20110067013A - Method and apparatus for reducing the number of photo in photo album - Google Patents

Abstract

PURPOSE: A method and an apparatus for reducing the number of registration photos of an individual database are provided to efficiently manage massive photographs by giving the name corresponding to the individual included in a photo image saved through a face recognition method, and detecting and displaying rapidly the photo image with the given name. CONSTITUTION: A face feature clustering unit(400) proceeds face feature clustering based on the distance comparison between face features. A representative feature selector(420) selects the representative feature among the clustered face features. A face deleting unit(440) deletes a face photo corresponding to the feature which is not selected as representative from a face database. The face feature clustering unit comprises a node set generator which generates a node set expressing an instance as a node and a node graph generator which generates a node graph by calculating the similarity distance between the nodes.

Description

Method and apparatus for reducing the number of registered photographs {Method and apparatus for reducing the number of photo in photo album}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a photo album, and more particularly, to a method and apparatus for reducing the number of people DB registered photos, a semi-automatic registration method and apparatus for a photo album system, and a photo album system using the same.

With the spread of digital cameras and scanners in homes, the number of photographic images owned by individuals is rapidly increasing. Such photographic images are usually stored in the form of digital data in a hard disk or other storage medium of a computer. Accordingly, there is a need to quickly and easily search for photographic images including a specific person among numerous stored photographic images.

In order to search for a photo image containing a specific person, it is possible to have information about the name of the person included in the photo image for each photo image. However, in order to do this, the name of the person included in each photo image is registered in advance. The process is necessary. However, the method of registering a name corresponding to each photo image is not only time-consuming but also cumbersome.

In order to solve these problems, US 2004/0264780, entitled "Face Annotation for Photo Management", requires that only a few photographic images be manually registered for a person, and among the photographic images stored in a large database, A method of supporting a search for a picture including a person is disclosed.

The method disclosed herein recognizes a face in a photographic image using features such as attire, face, eyes, nose, mouth, and generates a candidate name list for a given photographic image. Allows you to choose a name for the person. Also, after extracting a feature from the face of the queried picture, the name of the most similar face may be displayed.

However, this method does not have a function of facilitating naming and face registration for a given photo database DB, and also does not have a function of updating a face registered with the latest picture.

In addition, since a face of a person varies so much according to time, place, expression, and lighting, there is a limit in performance of automatically assigning a name to a photo through a conventional face recognition method.

An object of the present invention is to provide a method and apparatus for reducing the number of person DB registered photos in a photo album system.

Another object of the present invention is to provide a semi-automatic registration method and apparatus for a photo album system.

Another technical problem to be solved by the present invention is to provide a registration, query, search and browsing functions, and to give a name corresponding to a person included in a stored picture image through a face recognition method, and to give a picture image with a given name. It is to provide a photo album system for quickly detecting and displaying.

According to an aspect of the present invention, there is provided a method of reducing the number of registered person DB registered pictures, the method comprising: performing facial feature clustering by comparing distances between facial features; Selecting a representative feature from the clustered facial features; And deleting a face picture corresponding to a feature not selected as a representative from the face DB.

According to an aspect of the present invention, there is provided a method of reducing the number of registered person DB registered pictures, when (a) a facial feature registered in a face database is an instance, generating a node set representing a node as a node. step; (b) calculating a similarity distance between nodes and linking nodes smaller than a predetermined reference value to generate a node graph; (c) calculating a link sum of each node in the node graph; (d) calculating a current representative level by considering the link island and the previous representative level for each node; (e) selecting a node having the largest level of representativeness as a representative node, and removing the representative node and nodes directly linked to the representative node from the node graph; (f) repeating steps (c) to (e) for remaining nodes until there are no remaining nodes; And (g) deleting a picture corresponding to a node other than the representative node from the person DB.

In the step (d), it is preferable to calculate the current representative level as shown in Equation 1 in consideration of the link island and the previous representative level for each node.

[Equation 1]

는 현재의 대표성 레벨을,

는 이전의 대표성 레벨을, a(0<a<1)는 aging rate을,

는 각 노드의 링크섬을 나타낸다.)(here,

The current representative level,

Is the previous representative level, a (0 <a <1) is the aging rate,

Represents the linksum of each node.)

In accordance with an aspect of the present invention, there is provided a device for reducing a number of registered person DB images, including: a facial feature clustering unit configured to perform facial feature clustering as a distance comparison between facial features; A representative feature selection unit for selecting a representative feature among the clustered face features; And a face deleting unit for deleting a face picture corresponding to a feature not selected as a representative from the face DB.

The face feature clustering unit may include: a node set generation unit generating a node set representing a node as a node when the face feature registered in the face DB is an instance; And a node graph generation unit for calculating a similarity distance between nodes and linking nodes smaller than a predetermined reference value to generate a node graph, wherein the representative feature selection unit includes a node graph generation unit for each node in the node graph. A link sum calculator configured to calculate a link sum; A representative level calculator for calculating a current representative level in consideration of the link island and a previous representative level for each node; And a representative node selector configured to select a node having the largest representative level as a representative node, wherein the face deleting unit removes the representative node and nodes directly linked to the representative node from the node graph. And a person deletion unit for deleting a picture corresponding to a node other than the representative node from the person DB.

It is preferable that the representative level calculator calculates a current representative level as shown in Equation 1 in consideration of the link island and the previous representative level for each node.

According to another aspect of the present invention, there is provided a semi-automatic registration device of a photo album system according to the present invention, comprising: a multi-instance search unit for performing a multi-instance search for each person using registered faces; A similarity calculator that queries all unregistered images as targets, and displays registered faces as targets having similarity distances lower than a predetermined threshold; And a face DB registration unit which assigns a name to a face selected by the user from among the faces shown and registers it in the face database.

According to another aspect of the present invention, there is provided a method for semi-automatic registration of a photo album system, the method comprising: performing a multi-instance search for each person by using registered faces when a photo is input; Querying all non-registered images as targets, wherein the registered images are targeted to show faces with a similarity distance lower than a predetermined threshold; And assigning a name to a face selected by the user from among the faces shown, and registering the face in the face database.

According to another aspect of the present invention, there is provided a photo album system, including: a registration database storing facial features and names of the photo image and the specific person for each specific person in the photo image; A registration unit which extracts a facial feature from a picture and registers the pictures with the extracted face in the registration database; A query unit to estimate a name of a person in the picture when a new unregistered picture is input; And when a person's name to be searched comes in, searching for a face containing a name to be searched from among faces given a name, searching for a picture with the corresponding face, and a search unit showing the pictures.

The photo album system according to the present invention preferably further comprises a browsing unit for showing the pictures of successful face detection. Preferably, the facial feature is determined based on at least one of a face position, an eye position, and a facial descriptor. The registration unit facial feature extraction unit for extracting a facial feature when a picture comes in; A manual registration unit which shows pictures with a face extracted from the feature and registers the name when a name is given by the user; And displaying a face within a predetermined similarity distance by performing a multi-instance search for each person, and if the face is selectively given a name by the user, a semi-auto registration unit for registering the name.

The semi-automatic registration unit is a multi-instance search unit for performing a multi-instance search for each person using the registered face, if a picture is input; A similarity calculator that queries all unregistered images as targets, and displays registered faces as targets having similarity distances lower than a predetermined threshold; And a face DB registration unit for assigning a name to a face selected by the user from among the faces shown and registering it in the face database. The registration unit preferably further includes a registration DB reduction unit for reducing the number of instances in the multi-instance search.

The registration DB reduction unit includes a face feature clustering unit that performs face feature clustering by comparing distances between face features; A representative feature selection unit for selecting a representative feature among the clustered face features; And a face deleting unit for deleting a face picture corresponding to a feature not selected as a representative from the face DB.

A computer readable recording medium having recorded thereon a program for executing the invention described above is provided.

According to the present invention, if a large number of Photo Album photos are registered manually only for a few photos, the remaining photos can be quickly registered. In addition, the registered pictures can be quickly searched by the name of the person, so that a large number of pictures can be managed efficiently.

1 is a block diagram of a photo album system according to an embodiment of the present invention.
2 shows a more detailed configuration of the register 120 in a block diagram.
3 is a block diagram illustrating a more detailed configuration of the semi-automatic registration unit 240.
4 is a block diagram showing a more detailed configuration of the registration DB reduction unit 260.
FIG. 5 illustrates the block diagram of FIG. 4 in more detail.
Figure 6 shows the overall system of the photo album system according to the present invention.
7 is a flowchart illustrating a facial feature extraction process.
8 is a flowchart illustrating a manual registration process.
9 is a flowchart illustrating a semi-automatic registration process.
10 is a flowchart illustrating a method for reducing the number of person DB registered pictures according to the present invention.
FIG. 11 illustrates the flow chart shown in FIG. 10 in more detail.
12 is a diagram illustrating a method for reducing the number of person DB registered pictures according to the present invention using a node graph.
13 is a flowchart illustrating the query process.
14 is a flowchart illustrating a process of estimating a person's name for a plurality of pictures.
15 is a flowchart illustrating a process of finding a picture by name.
Figure 16 illustrates one embodiment of manual registration in registration.
17 illustrates one embodiment for semi-automatic registration in registration.
18 illustrates one embodiment of name estimation in a picture in a query.
19 illustrates an embodiment of estimating person names for a plurality of pictures in a query.
20 illustrates one embodiment of finding photos by name in a search.
Fig. 21 shows an example before reduction of the number of photos in the reduction of the registration DB.
Fig. 22 shows an example after reduction of the number of pictures in registration DB reduction.
FIG. 23 illustrates an embodiment of displaying all the portrait pictures in browsing. FIG.

Hereinafter, with reference to the accompanying drawings will be described in more detail a photo album system and a face recognition method in the system according to the present invention.

In the photo album system according to the present invention, since a face of a person varies too much according to time, place, expression, lighting, etc. to recognize a face by the conventional face recognition method, a plurality of faces may be registered for a person. Use a multi-instance search. In addition, by extending the registered face for a person, the user manually registers only a few faces, and similar faces similar to the registered face are added after user confirmation, and the latest faces are also registered. To help. If the number of faces registered for one person becomes too large, a method of reducing the number of registered faces is used by the representative face selection method.

1 is a block diagram of a photo album system according to an embodiment of the present invention. Referring to FIG. 1, the photo album system according to the present invention includes a registration database 100, a registration unit 120, a query unit 140, and a search unit 160. The photo album system preferably further includes a browsing unit 180. FIG. 6 illustrates the overall structure of the photo album system according to the present invention, and includes enrollment, query, searching, and browsing.

The registration database 100 stores the feature and name of the face of the photo image and the specific person for each specific person in the photo image.

The registration unit 120 extracts a facial feature on the picture and registers the pictures with the extracted face in the registration database 100. If the user manually registers only a few pictures, the registration unit 120 not only supports semi-automatically registering a large number of pictures, but also supports a function of updating a registration DB with the latest pictures.

2 is a block diagram illustrating a more detailed configuration of the registration unit 120, and includes a facial feature extraction unit 200, a manual registration unit 220, and a semi-automatic registration unit 240. The registration unit 120 preferably further includes a registration DB reduction unit 260.

The facial feature extractor 200 extracts facial features when a picture comes in. Preferably, the facial feature is determined based on at least one of a face position, an eye position, and a facial descriptor. 7 is a flowchart illustrating a process of extracting a facial feature, when input pictures are input, faces are detected for all pictures in a folder designated by the user (step 700). Then, eyes are detected for all detected faces. In operation 720, image normalization is performed using the detected eyes. In operation 740, the normalization of a face is performed by extracting feature points such as eyes from the detected face region and converting the image into a size and position suitable for recognition. It means the process. For this normalized image, a face descriptor is extracted. (Step 760) Face descriptor extraction is automatic face recognition such as PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), etc. adopted in MPEG 7. Algorithms can be used. Facial features are determined by the extracted facial descriptors, facial position, eye position, and the like.

The manual registration unit 220 shows pictures with faces extracted from features, and registers the name when a name is given by the user. FIG. 8 is a flowchart illustrating the manual registration process, in which thumbnails of pictures with extracted faces are displayed as thumbnails (step 800), and a user manually selects and assigns a number of pictures per person (820). Step) If there are a plurality of faces in a picture, it can be selected by using a mouse (step 840). A face to which a name is given is registered under the name (step 860). FIG. 16 illustrates an embodiment of manual registration.

The semi-auto registration unit 240 performs a multi-instance search for each person to show a face within a predetermined similarity distance, and if a name is selectively given to the face, the name is registered. .

3 is a block diagram illustrating a more detailed configuration of the semi-automatic registration unit 240, and includes an instance retrieval unit 300, a similarity calculation unit 320, and a face DB registration unit 340. When the picture is input, the instance search unit 300 performs a multi-instance search for each person by using the registered faces. 17 shows one embodiment for semi-automatic registration.

The similarity calculator 320 queries all images that are not registered and queries registered images as targets, and shows faces having a similarity distance lower than a predetermined threshold.

The face DB registration unit 340 assigns a name to a face selected by the user among the faces shown and registers it in the face database.

9 is a flowchart illustrating a semi-automatic registration process, when a picture is input, faces that are less than a threshold between face descriptors are selected by using a person feature registered for each person (step 900). Can be entered. If the user needs to make a correction (step 920), the face feature is assigned a name (step 940), and then a person is registered (step 960). To perform. All registered images are queried, and registered images are targeted to show faces with a similarity distance lower than a threshold. The user can delete images that have been incorrectly retrieval and add them to the database only for those that have been properly retrieval. This greatly expands the number of faces registered under one name and increases face recognition performance. By adjusting the threshold, you can adjust the ratio of false rejection and false alarms.

The registration DB reduction unit 260 reduces the number of instances in the multi-instance search. In multi-instance retrieval, when instances grow too much, you need to scale them down. There is no need to duplicate similar models, and more instances increase the search time. You also need to update to the latest model. The registration DB reduction unit 260 allows the old representative to drop out according to a given distance threshold, and puts the concept of a representative level so that the representative of the previous set is more likely to be selected in the next set. Frequent use can degrade the performance of multi-instance searches. The registration DB reduction unit 260 may be one invention as a person DB registration picture number reduction device itself.

4 is a block diagram illustrating a more detailed configuration of the registration DB reduction unit 260, and includes a facial feature clustering unit 400, a representative feature selection unit 420, and a face deletion unit 440. The face feature clustering unit 400 performs face feature clustering by comparing distances between face features. The representative feature selection unit 420 selects a representative feature from the clustered facial features. The face deleting unit 440 deletes a face picture corresponding to a feature not selected as a representative from the face DB.

5 illustrates the block diagram of FIG. 4 in more detail, wherein the facial feature clustering unit 400 includes a node set generator 500 and a node graph generator 510. When the face feature registered in the face database is an instance, the node set generation unit 500 generates a node set representing the instance as a node. The node graph generator 510 calculates a similarity distance between nodes and links nodes smaller than a predetermined reference value to generate a node graph.

The representative feature selector 420 includes a link island calculator 520, a representative level calculator 530, and a representative node selector 540. The link island calculator 520 calculates a link sum of each node in the node graph. The representative level calculator 530 calculates a current representative level by considering the link island and the previous representative level for each node. The representative node selector 540 selects a node having the largest representative level as the representative node.

The face deleting unit 440 includes a node removing unit 550 and a person deleting unit 560. The node removing unit 550 removes the representative node and nodes directly linked to the representative node from the node graph. The person deleting unit 560 deletes a picture corresponding to a node other than the representative node from the person DB.

10 is a flowchart illustrating a method for reducing the number of person DB registered pictures according to the present invention. First, face feature clustering is performed by distance comparison between face features (step 1000). Then, a representative feature is selected from the clustered face features (step 1010), and a face photograph corresponding to a feature not selected as a representative is selected. Delete from face DB (step 1020).

a = aging rate(0<a<1), 예를 들어 a = 0.9))FIG. 11 illustrates the flow chart shown in FIG. 10 in more detail. First, when a face feature registered in a face database is an instance, a node set representing a node as a node is generated (step 1100). The node set at t consists of a representative node at t-1 and a newly added node. (T = threshold for distance, representative level of newly added node,

a = aging rate (0 <a <1), for example a = 0.9)

A node graph is generated by linking nodes smaller than a predetermined reference value by calculating a similarity distance between nodes (step 1110). For example, if a distance between nodes is smaller than T, a link is connected. do. The link sum of each node is calculated in the node graph (step 1120). That is, a link sum is calculated for each node. (Recursive link is also included.) For each node, the current representative level is calculated by considering the link island and the previous representative level. (Step 1130) The node having the largest representative level is selected as the representative node. Remove the representative node and the nodes directly linked to the representative node from the node graph (step 1140). Steps (c) to (e) are performed on remaining nodes when there are no remaining nodes. Repeat step until step 1150. Delete the picture corresponding to the node other than the representative node from the person DB (step 1160).

In operation 1130, the current representative level may be calculated by using Equation 1 in consideration of the link island and the previous representative level for each node.

[Equation 1]

는 현재의 대표성 레벨을,

는 이전의 대표성 레벨을, a(0<a<1)는 aging rate을,

는 각 노드의 링크섬을 나타낸다. 도 12는 상술한 본 발명에 의한 인물 DB 등록사진수 축소방법을 노드그래프를 이용하여 도시한 것이다. 도 21은 등록 DB 축소에 있어서, 사진 수 축소 전의 예를 도시한 것이며, 도 22는 등록 DB축소에 있어서, 사진수 축소 후의 예를 도시한 것이다.here,

The current representative level,

Is the previous representative level, a (0 <a <1) is the aging rate,

Represents the link sum of each node. 12 illustrates a method for reducing the number of registered person DB registered pictures according to the present invention using a node graph. FIG. 21 shows an example before the number of photos is reduced in registration DB reduction, and FIG. 22 shows an example after the number of photos is reduced in registration DB reduction.

Meanwhile, the querying unit 140 shown in FIG. 1 estimates the name of a person in the picture when a new unregistered picture is input. The query finds the name of the face when a new unregistered picture is entered. The distance between the face of the query and all registered faces is obtained and the name of the face having the smallest distance is displayed. If you have multiple faces in a photo, repeat the same operation for each face. If the guessed name is wrong, you can correct it. FIG. 13 is a flowchart illustrating the query process and extracts facial features when a photo is input (step 1300). The name of the face having the smallest distance between facial descriptors is displayed (step 1310). In step 1320, the face feature is assigned a name (step 1330). Then, the person is registered (step 1340). FIG. 18 illustrates an embodiment of name estimation in a photo in a query.

In addition, the querying unit 140 may estimate a person's name for a plurality of pictures. For example, if you take a picture with a digital camera and then move the pictures to one folder at a time, the thumbnails are estimated by estimating the person's name on several sheets. The names of the characters are shown just below the face. If it's wrong, you can double click to fix it. FIG. 14 is a flowchart illustrating a process of estimating a person's name for a plurality of pictures. When a picture is input, a face feature is extracted. (Step 1400) Referring to all registered person features, the face having the smallest distance between face descriptors is extracted. Identify the name (step 1410) Arrange the thumbnails and display the estimated name (step 1420). If the guessed name is wrong, the user can modify it as needed (step 1430). (Step 1440), and register a person. (Step 1450) FIG. 19 shows an embodiment of estimating a person's name for a plurality of pictures in a query.

In order to register the latest picture, after performing a query (estimating the name of a person in the picture or a person name for a plurality of pictures) with respect to the unregistered pictures, the latest faces may be registered.

When the name of the person to search for comes in, the search unit 160 searches for a face containing a name to be searched from among faces given a name, searches for a picture with the corresponding face, and shows the pictures. Show all faces of a given name. There is no error because only the photos that have already been "named" are shown. Like a regular search engine, you can search not only "Kim Jeong-bae" but also "Kim Jung-bae & Hwang Won-jun" or "Kim Jung-bae".

15 is a flowchart illustrating a process of finding a picture by name. When a person name to be searched is input, the face having a name to be searched is identified by referring to the faces assigned to the name (step 1500). (Step 1520), and arrange the pictures in thumbnails (step 1540). FIG. 20 shows an embodiment of finding pictures by name in a search.

The browsing unit 180 shows thumbnails of pictures that succeed in face detection. FIG. 23 illustrates an embodiment of displaying all the portrait pictures in browsing. FIG.

The present invention can be embodied as code that can be read by a computer (including all devices having an information processing function) in a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording devices include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the specific embodiments described above, but the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

100 ... Registration Database 120 ... Register
140 ... Query 160 ... Search
180 ... Browsing

Claims (6)

Performing facial feature clustering by distance comparison between facial features;
Selecting a representative feature from the clustered facial features; And
And deleting a face picture corresponding to a feature not selected as a representative from the face DB. (a) when a facial feature registered in a facial database is an instance, generating a node set representing the instance as a node;
(b) calculating a similarity distance between nodes and linking nodes smaller than a predetermined reference value to generate a node graph;
(c) calculating a link sum of each node in the node graph;
(d) calculating a current representative level by considering the link island and the previous representative level for each node;
(e) selecting a node having the largest level of representativeness as a representative node, and removing the representative node and nodes directly linked to the representative node from the node graph;
(f) repeating steps (c) to (e) for remaining nodes until there are no remaining nodes; And
and (g) deleting a picture corresponding to a node other than the representative node from the person DB. The method of claim 2, wherein step (d)
[Equation 1]

(here,

The current representative level,

Is the previous representative level, a (0 <a <1) is the aging rate,

Represents the linksum of each node.)
And calculating the current representative level as shown in Equation 1 in consideration of the link island and the previous representative level for each node. A facial feature clustering unit that performs facial feature clustering as a distance comparison between facial features;
A representative feature selection unit for selecting a representative feature among the clustered face features; And
And a face deletion unit for deleting a face picture corresponding to a feature not selected as a representative from the face DB. The method of claim 4, wherein the facial feature clustering unit
A node set generation unit generating a node set representing a node as a node when the facial feature registered in the face DB is an instance; And
Comprising a node graph generation unit for generating a node graph by linking nodes smaller than a predetermined reference value by calculating the similarity distance (distance) between the nodes,
The representative feature selection unit
A link sum calculator configured to calculate a link sum of each node in the node graph;
A representative level calculator for calculating a current representative level in consideration of the link island and a previous representative level for each node;
A representative node selector configured to select a node having the largest level of representativeness as a representative node,
The face deleting unit
A node removing unit removing the representative node and nodes directly linked with the representative node from the node graph; And
And a person deletion unit for deleting a picture corresponding to a node other than the representative node from the person DB. The method of claim 5, wherein the representativeness level calculation unit
[Equation 1]

(here,

The current representative level,

Is the previous representative level, a (0 <a <1) is the aging rate,

Represents the linksum of each node.)
And calculating the current representative level as shown in Equation 1 in consideration of the link island and the previous representative level for each node.

Images