WO2014094284A1

WO2014094284A1 - Learning an adaptive threshold and correcting tracking error for face registration

Info

Publication number: WO2014094284A1
Application number: PCT/CN2012/087074
Authority: WO
Inventors: Qianxi ZHANG; Jie Zhou; Wei Zhou
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2012-12-20
Filing date: 2012-12-20
Publication date: 2014-06-26
Anticipated expiration: 2015-06-20

Abstract

There are provided a system and a method for performing user registration in a face recognition system. Image flows are extracted from a sequence of images by following inter-frame constraints, i. e. within a scene, the position of an individual is approximately constant and his/her face should be detected in subsequent frames roughly in the same position, and intra-frame constraints, i. e. images appearing in the same frame must belong to different individuals. Thresholds that are used to classify image flows are updated according to the extracted image flows. Existing classification is updated using the extracted image flows and the updated threshold, including class merging and class splitting, thus correcting the tracking errors in previous image extractions.

Description

LEARNING AN ADAPTIVE THRESHOLD AND CORRECTING TRACKING

ERROR FOR FACE REGISTRATION

TECHNICAL FIELD

The present principles relate generally to face registration in face recognition and, more particularly, to learning an adaptive threshold and correcting tracking error for face registration.

BACKGROUND

Face recognition is a promising biometric technology which can be widely used in practical applications, such as security systems. Although the traditional face recognition system is able to recognize users automatically, the process of registration for a new user is usually complicated. For example, the registration process for face recognition typically involves the identification (ID) of the new user being entered as an input to the registration process as well as several face images of the new user being taken. The conditions of those images such as, for example, lighting and the viewing angle of the face, are strictly enforced in order to provide a suitable result for the registration process. Thus, the human-computer interaction process of face registration limits the suitable scope and applicable crowd to which face registration can be readily applied.

SUMMARY

These and other drawbacks and disadvantages of the prior art are addressed by the present principles, which are directed to learning an adaptive threshold and correcting tracking error for face registration in a face recognition system.

According to one aspect of the present principles, there is provided a method for performing face registration in a face recognition system. The method includes extracting image flows from a sequence of frames responsive to at least one inter-frame constraint and at least one intra-frame constraint. The method further includes identifying classes from the image flows responsive to an adaptive threshold. Each of the classes corresponds to a respective one of a plurality of individuals.

According to another aspect of the present principles, there is provided a system for performing face registration in a face recognition system. The system includes a flow extractor for extracting image flows from a sequence of frames responsive to at least one inter-frame constraint and at least one intra-frame constraint. The system further includes a class updating unit for identifying classes from the image flows responsive to an adaptive threshold. Each of the classes corresponds to a respective one of a plurality of individuals.

These and other aspects, features and advantages of the present principles will become apparent from the following detailed description of exemplary embodiments, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present principles may be better understood in accordance with the following exemplary figures, in which:

FIG. 1 shows an exemplary processing system 100 to which the present principles can be applied, in accordance with an embodiment of the present principles;

FIG. 2 shows an exemplary system 200 for registering a new user with a face recognition system, in accordance with an embodiment of the present principles;

FIG. 3 further shows the image access device 210 of FIG. 2, in accordance with an embodiment of the present principles;

FIG. 4 further shows the class updating unit 230 of FIG. 2, in accordance with an embodiment of the present principles;

FIG. 5 shows an exemplary method 500 for providing updated classes for a user registration process for a face recognition system, in accordance with an embodiment of the present principles; and

FIG. 6 shows an exemplary method 600 for image flow extraction for a user registration process for a face recognition system, in accordance with an embodiment of the present principles.

DETAILED DESCRIPTION

The present principles are directed to learning an adaptive threshold and correcting tracking error for face registration in a face recognition system. In an embodiment, the adaptive threshold is learned and the tracking error is corrected based on inter-frame and/or intra-frame constraint. Advantageously, the present principles simplify the face registration process while making the face registration process more reliable and adaptable. In an embodiment, the present principles provide a system and a method that enable the automatic registration of new users in a face recognition system. For example, in an embodiment, face images from a video source are captured, and it is then automatically judged whether to register a new user based on the captured face images.

The face registration process involves the distribution of images in a high dimensional space. However, the distribution of the images in a high dimensional space is complicated. Thus, it is difficult to identify an image directly according to the Euclidean distance between different samples. For example, the distance between two images of the same person and different light conditions is most likely larger than the distance between two different persons and the same light conditions.

Accordingly, we hereinafter describe the use of inter-frame information and intra-frame information in the face registration process of face recognition. Either type of information, namely inter-frame or intra-frame, can be used, or both can be used, depending upon the specific implementation.

Thus, in an embodiment, inter-frame information of a video source can be used to overcome the above identified problems of the prior art. For example, it has been assumed that within a scene, the position of an individual is approximately constant and his/her face should be detected in subsequent frames roughly in the same position. Accordingly, in an embodiment, this type of inter-frame information can be applied to the face registration process to improve the resultant accuracy of the face registration process. It is to be appreciated that the preceding inter-frame information is merely illustrative and, given the teachings of the present principles provided herein, one of ordinary skill in the art will contemplate this and various other inter-frame information to which the present principles can be applied, while maintaining the spirit of the present principles.

Moreover, in an embodiment, intra-frame information can be applied to the face registration process. For example, in an embodiment, the intra-frame information can be used to determine that two face images captured from one frame must belong to different persons. Accordingly, this type of intra-frame information can be applied to the face registration process to improve the resultant accuracy of the face registration process. It is to be appreciated that the preceding intra-frame information is merely illustrative and, given the teachings of the present principles provided herein, one of ordinary skill in the art will contemplate this and various other intra-frame information to which the present principles can be applied, while maintaining the spirit of the present principles.

In an embodiment, the intra-frame information can be used to correct a cluster error that typically occurs in the face registration process. Moreover, in an embodiment, the intra-frame information can be used to learn an adaptive threshold used to judge whether a sub-flow of face images belong to a new user. In an embodiment, the adaptive threshold can serve a significant role in face registration, since it is difficult, if not impossible, to find a fixed threshold that is applicable to any crowd of people so as to provide meaningful results in the face registration process. For example, the threshold to distinguish a family of users should be lower than the threshold used for non-family users.

The cluster error can be caused by a user tracking error during image capture or some other reason(s), as is known to those of ordinary skill in the art. Tracking error involves a sub-flow of face images that includes images of two or more persons. In prior art systems, the face registration process would register the sub-flow with respect to one person, thus disadvantageously resulting in the two persons being recognized as the same user. However, the present principles can correct for cluster error. Accordingly, if images of two or more persons appear in the same frame, the present principles can advantageously split the "registered user" into two or more registered users to correct the cluster error.

FIG. 1 shows an exemplary processing system 100 to which the present principles may be applied, in accordance with an embodiment of the present principles. The processing system 100 includes at least one processor (CPU) 102 operatively coupled to other components via a system bus 104. A read only memory (ROM) 106, a random access memory (RAM) 108, a display adapter 1 10, an input/output (I/O) adapter 1 12, a user interface adapter 1 14, and a network adapter 198, are operatively coupled to the system bus 104.

A display device 1 16 is operatively coupled to system bus 104 by display adapter 1 10. A disk storage device (e. g. , a magnetic or optical disk storage device) 1 18 is operatively coupled to system bus 104 by I/O adapter 1 12.

A mouse 120 and keyboard 122 are operatively coupled to system bus 104 by user interface adapter 214. The mouse 120 and keyboard 122 are used to input and output information to and from system 100. A transceiver 196 is operatively coupled to system bus 104 by network adapter 198.

Of course, the processing system 100 may also include other elements (not shown), as readily contemplated by one of ordinary skill in the art, as well as omit certain elements. For example, various other input devices and/or output devices can be included in processing system 100, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and/or wired input and/or output devices can be used. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized as readily appreciated by one of ordinary skill in the art. These and other variations of the processing system 100 are readily contemplated by one of ordinary skill in the art given the teachings of the present principles provided herein.

Moreover, it is to be appreciated that system 200 described below with respect to FIG. 2 is a system for implementing respective embodiments of the present principles. Part or all of processing system 100 may be implemented in one or more of the elements of system 200.

Further, it is to be appreciated that processing system 100 may perform at least part of the method described herein including, for example, at least part of method 300 of FIG. 3 and/or at least part of method 400 of FIG. 4. Similarly, part or all of system 200 may be used to perform at least part of method 300 of FIG. 3 and/or at least part of method 400 of FIG. 4.

FIG. 2 shows an exemplary system 200 for registering a new user with a face recognition system, in accordance with an embodiment of the present principles. The system 200 includes an image access device 210, a flow extractor 220, and a class updating unit 230. The image access device 210 receives and/or otherwise includes one or more videos from which face images can be extracted. Accordingly, the output of the image access device 210 includes extracted images. The flow extractor 220 inputs extracted images and outputs extracted image flows. The class updating unit 230 inputs extracted image flows and outputs updated classes. Each of the classes corresponds to a respective one of multiple individuals/persons. These elements are described in further detail herein below.

FIG. 3 further shows the image access device 210 of FIG. 2, in accordance with an embodiment of the present principles. The image access device 210 includes a video access device 21 1 and an image extractor 212. The image extractor 212 receives input pictures and/or movies from the video access device 21 1 . The image extractor 212 extracts images from the pictures and/or movies to output extracted images.

FIG. 4 further shows the class updating unit 230 of FIG. 2, in accordance with an embodiment of the present principles. The class updating unit 230 includes a class creator 231 , a class merger 232, and a class splitter 233. The class creator 231 inputs extracted image flows from the flow extractor 220, as well as existing classes if there is any, and outputs updated classes. The classer merger 232 inputs the updated classes from the class creator 231 and outputs further updated classes with some classes being merged. The class splitter 233 inputs the further updated classes from the class merger 232 and outputs even further updated classes with some classes being split when necessary. The output of the class merger 232 serves as the output of system 200. Note that these three modules may have different connection relationships according to different requirement, and one or more of them can be omitted from the class updater. For example, if class merging is considered as unnecessary, class merger can be omitted and the class creator can be directly connected to the class splitter.

FIG. 5 shows an exemplary method 500 for providing updated classes for a user registration process, in accordance with an embodiment of the present principles. At step 510, a video sequence is accessed, for example, via image access device 210. At step 520, image flows Fi, F₂, F_n are extracted under one or more intra-frame constraints (hereinafter simply "intra-frame constraint" for the sake of brevity) using the flow extractor 220. At step 530 a threshold Thn is updated, where Thn denotes the low threshold. If the difference between two image flows is lower than the low threshold Thn, then the image flows/classes belong to the same class. One methodology for updating the threshold Thn is described below. At step 540, it is determined whether image flow F,- is/represents a new class based on Thr_h, where 777/>, denotes the high threshold. If so, then the method proceeds to step 550. Otherwise, the method proceeds to step 555. At step 550, image flow F,- is added as a new class. In other words, if the difference between the image flow F,- and each of the existing classes is higher than the high threshold Thr_h, the image flow F, belongs to a new class. On the other hand, if the difference is lower than the high threshold Thr_h, that means F, may belong to one or more existing classes. At step 555, those existing classes Cn, ... , C,-k that image flow F, belongs to are found. At step 560 these classes Cn, Cik are merged into one class. The process repeats for every image flow extracted in step 520. In one implementation, the above steps 530-560 are realized in the class creator 231 and class merger 232 of the class updating unit 230.

The next step is to examine the classes to see if there are any redundant classes that can be merged. At step 570, one or more or all pairs of image flows F,- and F_j are examined to determine whether F,- and F_s- belong to the same class Ck- If so, then the method proceeds to step 580. Otherwise, the method proceeds to the next pair of image flow until all the required pairs are examined, and the process moves to step 590. At step 580, class Ck is split into two classes, for example, using class splitter 233. Since the extraction of image flows in step 520 has taken into consideration of the intra-frame constraint, images in different image flows would belong to different classes, and the class Ck, which contains images from both image flows, needs to be split into two classes. Finally, at step 590, classes are updated to a new set of classesCi, C2, C_m.

FIG. 6 shows an exemplary method 600 for image flow extraction for a user registration process, in accordance with an embodiment of the present principles. An input to the method 600 includes extracted images, such as face images.

At step 610, a loop is commenced for each pair of images in the extracted images. Thus, the loop relates to a pair-of-images-basis, as the steps that are encompassed by the loop are applied to each respective pair of images from among the extracted images that are input to method 600. At step 620, it is determined whether an inter-frame constraint is satisfied or not. An example of such a determination method can be found in A Franco, D Maio, D Maltoni, "Incremental template updating for face recognition in home environments" Pattern Recognition, 2010. If the inter-frame constraint is satisfied, then the method proceeds to step 630 and the current pair of images is put into the same image flow. Otherwise, the method proceeds to step 680, whereby the images in the current pair of images are put into different image flows. The same process repeats for all the image pairs. Starting at step 640, another loop is commenced for each pair of images in each image flow. Thus, this loop relates to a pair-of-images-per-image-flow-basis, as the steps that are encompassed by the loop are applied to each respective pair of images in each respective flow from among the flows determined by steps 630 and 680. At step 650, it is determined whether an intra-frame constraint is satisfied. If so, then the method proceeds to step 660 and the current image flow is split into two flows. Otherwise, the method proceeds to step 670 and the current flow is kept unchanged. The final outputs of this process are the extracted image flows.

A description willnowbegivenregarding a possible application to which the presentprinciplescanbeapplied, in accordance with an embodiment of the presentprinciples.

In a home scenario where face detection is used with respect to members of the same family, a particular family member will likely have a high similarity with other family members, which will increase the difficulty of automatic face registration. At the same time, it is common that a corresponding camera used for face registration and recognition will capture images in which multiple family members simultaneously appear. Thus, one concern about automatic face registration in a home scenario is how to minimize the error of categorizing different members into the same class. To solve the problem, a threshold adjustment mechanism is introduced to improve the robustness and reliability of automatic face registration by making full use of the above intra-frame and inter-frame information.

The present principles determine the distance between two image flows which are constrained by inter-frame information and introduce intra-frame information to the face registration process to learn an adaptive threshold 777/7 for the system. In each image flow, it is initially assumed that only one member appears. First, face detection is conducted. If it is determined and/or otherwise believed that there are multiple users simultaneously appearing in the image flow, the image flow will be split into n sub-flows in order that only one user is present in each sub-flow.

It is assumed that two image sub-flows (Fi, F₂)belong to the same person if the distance between the two image sub-flows is less than the threshold 777/ . If the threshold higher than the distance between the two image sub-flows(F?, F₂) with the intra-frame constraint, then the assumption fails due to the counterexample^ , Fi). Thus, to fulfill the assumption, we choose the minimum value from among the distances of intra-frame sub-flows as the threshold 777/ . Learning an adaptive threshold is meaningful, as the same ensures the face registration process is efficient with respect to different crowds of users.

Furthermore, since a main concern in automatic face registration in the home scenario is how to minimize the error of categorizing different members into the same class, the target of the threshold adjustment mechanism is gradually adjusted to strengthen this threshold.

The intra-frame information can also help to correct the tracking error of the inter-frame. A class is split into two classes, if two sub-flows of images which have the intra-frame constraint both belong to that class. To split the two classes, for each element of the class, i. e. the face image, the system computes its respective distance to the two sub-flows and adds the element to the new class which represents the nearer sub-flow. That makes the system more stable and reliable than a system that only uses the inter-frame information.

A detailed discussion of various aspects of the present principles will now be given, in accordance with one or more embodiments of the presentprinciples.

A sub-flow is defined as the images which are constrained by inter-frame information. An example method of identifying sub-flow under the inter-frame constraint can also be found in A Franco, D Maio, D Maltoni, "Incrementaltemplateupdating for face recognition in home environments"Pattern Recognition, 2010. It is assumed that the images of one sub-flow belong to the same class. Each class represents one person. In fact, when there are multiple users simultaneously appearing in an image flow for a period of time, we can determine the corresponding additional classes corresponding to each of the users. By using face segmentation technology, the above image flow can be split into n sub-flows. In each image sub-flow, there is only one user appearing.

To determine which class a sub-flow belongs to, the distance between two sub-flows is first defined to measure the similarity of the two sub-slows.

In an embodiment, every image is a high dimension column vector^ and the distance between two images is defined as follows:

d( i, ₂) = ( i- 2)^T( i- 2), 0 ) where _xand ₂ represent two images, and the distance is the Euclidean distance, which is only used as an example here. Other forms of distances can also be applied.

Let F =

be two image sub-flows. F includes n elements/images and F₂ includes m elements/images. Then, there aren x m distances of images between the two sub-flows, but only the minimum subset of the distances is meaningful for clustering. If the distances of the subset are small enough, then the two sub-flows can be clustered to the same class. That is, find k images from each sub-flow, and the minimum average value of the k x /cdistances is set to be the distance of the sub-flows, which can be expressed as follows:

D(F_lt F₂) = mm_x^_Pi:X^_P2 {^∑_Xl ._Mk _{Xz j≡x} d( _lit x_2j)}, (2) where ^means a subset of F_rwhich has k elements. If k is set to be 1 , the distance is just the smallest distance between the images from different sub-flows. A k value that is greater than 1 , allows the method to be robust to the interference of noise, and thus is preferred.

In scenarios whereD( ₁, ₂)is difficult to calculate, the following expression is used to get the approximation:

D(F_lt F₂) = min _aver^ ._eP2 jmin _aver^._ePi{d(x_li, x_2j)} , (3) where min _aver _eF{/(x_£)} means the average of the /c*^h smallest value of /(x_£) , x_£ £ F. It is to be noted that£⁾(F₁, F₂)≠ D(F₂, F₁)and D(F₁, F₂), D(F₂, F₁)≤ D(F₁, F₂) . The distances between a class and an image sub-flow and the distances between two classes can be defined in the same way.

The input sub-flows are constrained by the intra-frame information that they must belong to different classes. This leads to that the low threshold Thn must be lower than the distance of each pair of the sub-flows. Otherwise, there exists at least one pair of sub-flows whose distance there between is lower than Tftr_/, and this pair of sub-flows would be considered as belong to the same class which is in contradiction to the intra-frame information/constraint. Consequently, Thn should be updated and one example is as follows:

Thr_t = min mm_i≠j{D (F_£, F_y)}}. (5)

Sub-flow F_/ is determined as belonging to Cy, if D(C_j, F_£) < Thr_h

To decide whether F,- is a new class, a high threshold Thr_h is needed. If the distance between F,- and each class is higher than Thr_h, it is recognized as a new class. It is to be appreciated that the present principles are not constrained to any particular methodology for learning and/or otherwise determining Thr_h and, thus, any methodology can be used, while maintaining the spirit of the present principles. In an embodiment, Thr_h can be a fixed parameter.

By following the above procedure, if F,- and F_j, which are two image sub-flows according to the intra-frame constraint, are classified to the same class Ck, conflict occurs. To solve the conflict, the newest information is relied upon. If the newest information turns out to be incorrect, it can be corrected by the later information. As a result, it is believed that Ck includes images of at least two classes and should be split according to the distance between elements/face images of the class and the two sub-flows.

In an embodiment, face detection can be conducted at an initiation step (not shown) of method 500, and can be repeated after the classes Ci, C2, C_m are updated by step 590 of method 500. If it is thought that there are multiple users simultaneously in a particular image flow, the particular image flow will be split into n sub-flows (Fi, F_n)so that only one user is in each sub-flow. If F,- is not seen as a new class and belongs to several classes _f, then merge these classes _f. If F,- and F_j, are seen as belonging to the same class Ck, it means that there is an error in the current class sets, and Ck should be split and the database of the class sets should be updated.

We note that the terms "flow" and "sub-flow" may be used interchangeably herein, as it is nonetheless clear to one of ordinary skill in the art that in the case where an image flow is split into two image sub-flows, each of the sub-flows can be considered an image flow in and of itself.

Also, as used herein, the words "picture" and "image" are used interchangeably and refer to a still image or a picture from a video sequence. As is known, a picture may be a frame or a field.

These and other features and advantages of the present principles may be readily ascertained by one of ordinary skill in the pertinent art based on the teachings herein. It is to be understood that the teachings of the present principles may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.

Most preferably, the teachings of the present principles are implemented as a combination of hardware and software. Moreover, the software may be implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPU"), a random access memory ("RAM"), and input/output ("I/O") interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present principles is not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.

Claims

1 . A method for performing user registration, comprising:

extracting image flows responsive to at least one inter-frame constraint and at least one intra-frame constraint, wherein said image flows are extracted from a sequence of frames containing a plurality of individuals; and

registering said plurality of individuals by identifying classes from the image flows responsive to an adaptive threshold, each of the classes corresponding to a respective one of said plurality of individuals.

2. The method of claim 1 , wherein the adaptive threshold is determined to be a value that is equal to or less than a minimum distance from among the distances between each pair of the image flows.

3. The method of claim 1 , further comprising

updating the adaptive threshold responsive to distance values determined between the image flows.

4. The method of claim 1 , wherein the identifying classes comprises at least one of: creating a new class, merging a set of existing classes and splitting a single existing class.

5. The method of claim 4, wherein the new class is created when one of the image flows has a distance to any of the classes greater than a second threshold, wherein said second threshold is higher than said adaptive threshold.

6. The method of claim 4, wherein the first set of existing classes are merged when a same image flow belongs to each of the first set of existing classes.

7. The method of claim 6, wherein the same image flow is determined to belong to a class when a distance between the same image flow and the class is below the adaptive threshold.

8. The method of claim 4, wherein the single existing class is split if the single existing class contains more than one of the image flows.

9. The method of claim 1 , wherein the inter-frame constraint comprises applying a presumption that a position of an individual within a same scene is approximately constant in multiple frames of the same scene.

10. The method of claim 1 , wherein the intra-frame constraint comprises applying a presumption that images appearing in a same frame belong to different classes.

1 1 . A system for performing face registration, comprising:

a flow extractor for extracting image flows responsive to at least one inter-frame constraint and at least one intra-frame constraint, wherein said image flows are extracted from a sequence of frames containing a plurality of individuals; and

a class updating unit for identifying classes from the image flows responsive to an adaptive threshold, each of the classes corresponding to a respective one of said plurality of individuals.

12. The system of claim 1 1 , wherein the adaptive threshold is determined to be a value that is equal to or less than a minimum distance from among distances between each pair of the image flows.

13. The system of claim 1 1 , wherein the adaptive threshold is updated responsive to distance values determined between the image flows.

14. The system of claim 1 1 , wherein the class updating unit comprises at least one of: a class creator, a class merger and a class splitter.

15. The system of claim 14, wherein the class creator creates new class when one of the image flows has a distance to any of the classes greater than a second threshold, wherein said second threshold is higher than said adaptive threshold.

16. The system of claim 14, wherein the class merger merges the first set of existing classes when a same image flow belongs to each of the first set of existing classes.

17. The system of claim 16, wherein the same image flow is determined to belong to a class when a distance between the same image flow and the class is below the adaptive threshold.

18. The system of claim 14, wherein the class splitter splits the single existing class if the single existing class contains more than one of the image flows.

19. The system of claim 1 1 , wherein the intra-frame constraint comprises applying a presumption that images appearing in a same frame belong to different classes.

20. A computer readable storage medium comprising a computer readable program for performing face registration in a face recognition system, wherein the computer readable program when executed on a computer causes the computer to perform the following steps:

extracting image flows from a sequence of frames responsive to at least one inter-frame constraint and at least one intra-frame constraint; and

identifying classes from the image flows responsive to an adaptive threshold, each of the classes corresponding to a respective one of a plurality of individuals.