WO2016015170A1

WO2016015170A1 - A method for face recognition and a system thereof

Info

Publication number: WO2016015170A1
Application number: PCT/CN2014/000716
Authority: WO
Inventors: Xiaoou Tang; Xiaogang Wang; Zhenyao ZHU; Ping Luo
Original assignee: Individual
Current assignee: Individual
Priority date: 2014-07-28
Filing date: 2014-07-28
Publication date: 2016-02-04
Anticipated expiration: 2017-01-28
Also published as: CN106663186B; CN106663186A

Abstract

Disclosed are a multi-view perceptron system and a multi-view perceptron method. The disclosed system comprises an identity feature determination unit, a view representation capture unit, a feature combination unit and a recovery unit. The identity feature determination unit is configured to determine a plurality of identity features for an input face image in a given view point of the image. The view representation capture unit is configured to capture a view representation of the input face image. The feature combination unit is configured to yield one or more features for face recovery from the determined identity features and the view representation. The recovery unit is configured to generate a face image from the generated face for recovery, and then unite the generated face image and the view representation to a view label of the generated face image.

Description

A METHOD FOR FACE RECOGNITION AND A SYSTEM THEREOF

Technical Field

[0001] The present application relates to a method for face recognition and a system thereof.

Background

[0002] The performance of face recognition systems depends heavily on facial representation, which is naturally coupled with many types of face variations, such as face views, illuminations, and view expressions. As face images are often observed in different viewpoints (angle of view), a major challenge is to untangle the face identity and view representations.

[0003] Substantial efforts have been dedicated to extract identity features by hand, such as LBP, Gabor, and SIFT. The best practice of face recognition extracts the above features on the landmarks of face images with multiple scales and concatenates them into high dimensional feature vectors. Deep neural nets have been applied to learn features from raw pixels.

[0004] Deep neural net is inspired by the understanding of hierarchical cortex in human brain and mimicking some aspects of its activities. Human not only can recognize identity, but can also imagine face images of a person under different viewpoints, making face recognition in human brain robust to view changes. In some sense, human brain can infer a 3D model from a 2D face image, even without actually perceiving 3D data.

Summary

[0005] In one aspect of the present application, disclosed is a method for multi-view perceptron, comprising:

determining a plurality of identity features for an input face image in a given view point of the image; capturing a view representation of the input face image;

determining one or more features for face recovery from the determined identity features with the view representation; and

generating a face image from the generated features for recovery, and then unite the generated face image and the captured view representation to a view label of the generated face image.

[0006] In one aspect of the present application, disclosed is also a multi-view perceptron system, comprising:

an identity feature determination unit configured to determine a plurality of identity features for an input face image in a given view point of the image;

a view representation capture unit configured to capture a view representation of the input face image;

a feature combination unit configured to yield one or more features for face recovery from the determined identity features and the view representation; and

a recovery unit configured to generate a face image from the generated face for recovery, and then unite the generated face image and the view representation to a view label of the generated face image.

[0007] According to embodiments of the present application, identity feature determination unit, the view representation capture unit, the feature combination unit and the recovery unit may be coupled together to form a biological neural network. The parameters of the biological neural network, i.e. weight and biases, may be determined through maximizing a lower-bound of a probability distribution formed from the generated the face image, the view representation in view of the view labels of the input face image.

[0008] In one aspect of the present application, disclosed is also

computer-readable storage medium for recording instructions executable by one or more processors to,

determine a plurality of identity features for an input face image in a given view point of the image; capture a view representation of the input face image;

determine one or more features for face recovery from the determined identity features with the view representation; and

generate a face image from the generated features for recovery, and then unite the generated face image and the captured view representation to a view label of the generated face image.

Brief Description of the Drawing

[0009] Exemplary non-limiting embodiments of the present invention are described below with reference to the attached drawings. The drawings are illustrative and generally not to an exact scale. The same or similar elements on different figures are referenced with the same reference numbers.

[0010] Fig. 1 is a schematic diagram illustrating a system for face recognition consistent with one disclosed embodiments.

[0011] Fig.2is a schematic diagram illustrating neural network simulated for the system for face recognition according to one embodiment of the present application.

[0012] Fig. 3 is a schematic flowchart illustrating face recognition consistent with some disclosed embodiments of the present application.

[0013] Fig. 4 is a schematic flowchart illustrating a training process for the neural networks consistent with some disclosed embodiments of the present application.

[0014] Fig. 5 is a schematic diagram illustrating a system for face recognition consistent with another disclosed embodiment of the present application.

[0015] Fig. 6 is a schematic flowchart illustrating face test procedure consistent with some disclosed embodiments of the present application.

Detailed Description

[0016] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When appropriate, the same reference numbers are used throughout the drawings to refer to the same or like parts.

[0017] Fig. 1 is a schematic diagram illustrating an exemplary multi-view perceptron system 100 according to one embodiment of the present application.

[0018] The multi-view perceptron system 100 receives face

different viewpoints, and output images y of the same identity

in different viewpoints and their view labels v, i.e. \ \^y_i ^ _» ¹'i_fejj_.__{1 .}__{l fe}__1?where x_y- is the input image of the z^'-th identity under the j-th viewpoint, _¾ denotes the output image of the same identity in the k-th viewpoint, and v_ikis the view label of the output and may be a M dimensional binary vector with the c-th element as 1 and the remaining zeros.

[0019] It shall be appreciated that the system 100 may be implemented using certain hardware, software, or a combination thereof. In addition, the embodiments of the present invention may be adapted to a computer program product embodied on one or more computer readable storage media (comprising but not limited to disk storage, CD-ROM, optical memory and the like) containing computer program codes.

[0020] In the case that the system 100 is implemented with software, the system lOOmay include a general purpose computer, a computer cluster, a mainstream computer, a computing device dedicated for providing online contents, or a computer network comprising a group of computers operating in a centralized or distributed fashion.

[0021] Referring to Fig. 1 again, where the apparatuslOOO is implemented by the hardware, it may comprise a deterministic unit (neurons) 10 configured to learn the identity features ½ for an input face image x in a given arbitrary view, and a random unit (neurons) 20 configured to capture a view representations h^v of the input face image x. The view representation h^v is naturally coupled with many types of face variations, such as viewpoints (angle of view), illuminations, and face expressions. [0022] In one embodiment of the present application, the identity feature determination unit 10 operates to determine a plurality of identity features for an input face image in a given viewpoint (angle of view) of the image. In one embodiment of the present application, the identity feature determination unit lOmay generate a first plurality of identity features \ί_\ from the input face image in accordance with an activation function, i.e. sigmoid function σ(χ) and then to generate a second plurality of identity features i₂ ^d based on the generated first identity features \ί_\ . For example, the identity feature determination unit 10 may generate the first plurality of identity features Zi'/and then the identity features based on the generated first identity features h! by rule of h^li = a U_Qx) Formula 1) h^l2 = σ(ί/₁ ι^ί ₁ ^ίί) Formula 2)

Where U₀ and U^. are predetermined values of weight, which may be numbers ranged from 0 to 1 as will be discussed later.

[0023] As shown in Fig. 1, the multi-view perceptron system 100 further comprises a feature combination unit 30 configured to yield one or more features for face recovery from the determined identity features and the view representation. In one embodiment, the feature combination unit 30 may combine the generated second identity features t with the generated view representation h_v to yield one or more third features /i^for face recovery and then generate one or more fourth features for face recovery /i₄ from the yielded third features/13. For example, the third and the fourth features /13 and /i₄for face recovery may be determined by rule of

13 = ff([t/₂ i₂ ^d, K₂ i₂]) Formula 3) h^r ₄ = a([U₃h^r ₃, V₃h ]) Formula 4)

Where {U₂, V₂} and {U₃,V₃} are predetermined values of weight, which may be number ranged from 0 to 1. [0024] Referring to Fig. 1 again, the multi-view perceptron system 100 may further comprise a recovery unit 40 configured to generate a face image y from the generated recovery features /i₄, and then unite the generated face image y and the view representation h_v to a view label of the generated face image. In one embodiment of the present application, the view point of face image y and the view label v may be determined by rule of

y = a U₄h ) Formula 5)

v = o([U_sy, W V ) Formula 6)

Where t/₄ and t/₅ are predetermined values of weight, which may be number ranged from 0 to 1.

[0025] In one embodiment of the present application, the system 100 may be implemented as network which mimics a biological neural network and are formed by a plurality of artificial nodes, known as "neurons" or "units", which are connected together. In theory, an artificial neuron is a mathematical function conceived as a model of biological neurons. The artificial neuron receives one or more inputs (representing dendrites) and sums them to produce an output (representing a neuron's axon). The above motioned U₀, Όγ, U₂, Uz, U , V₂, V^, W₂and represent the weights and biases of the formed neural network. In other words, the above motionedt/o, U₁, U₂, U-_i, t/₄, V₂, V₃, W₂and W₃ may be determined based on a neural network independently. Figure 2 illustrates a schematic configuration of neural network according to one embodiment of the present application.

[0026] The parameters (also referred to "weights and biases" herein) of MVP system, i.e. U₀, U₁, U₂, U2, U4, V₂, V2, W₂and are learned by maximizing a data log-likelihood: log p y, v|/i^id; ) E = Formula 7)

[0027] The lower-bound log∑_hv q(h^{v P{y,V,h}^ ^;E)is reached by letting q(h^v) = p(h^v \y, v; 0^old). Importance sampling is used to estimate the true posteriorp(/i^v |y, v; 0^old). The joint probability p y, v \h^v; 0^old) is the importance weight.

[0028] Herein after, the specific training process 400 will be discussed in reference to Fig. 4.

[0029] In step S401, parameters ©, i.e. U_Q, U_x, U₂, U₃, U₄, V₂, V₃, W₂and W₃ are randomly initialized with a value that is ranged from 0 to 1.

[0030] In step S402,it samples a number of view representation i_j, based on the current parameters Θ. Specifically, the view representation h₂ is sampled from a prior distributionq /iO, i.e. uniform distribution. In other words, the set of h^v are assigned with values such that h₂ has a uniform distribution, i.e.{/i^v} ~ U (0, 1). And then, h₃ is generated from h₂ through W₂ of the current parameters Θ.

[0031] In step S403, a face image x is inputted to the identity feature determination unit 10, i.e. the lowest layer in the simulated net as shown in Fig. 2 so as to generate the first and the second identity features in accordance with the formulas 1) and 2) based on the randomly initialized Uo and Uj. The combination unit

30 then operates to combine the generated second identity features h^l ₂ with the assigned h₂ to yield one or more third features h₃ for face recovery and then generate one or more fourth features for face recovery /i₄ from the yielded third features/13 in accordance with formulas 3) and 4). The recovery unit 40 then generates the face image y from the generated recovery features /i₄, and then unite the generated face image y and the view representation h_v, which is assigned with value, to a view label of the generated face image by rule of formulas 5) and 6).

[0032] In step S404, it uses the generated face image y and the view labels v to form/compute a prior distribution (i.e. importance weights) corresponding to the different view representations h_v, which may be represented as p(h^v \y, v; 0^old). In particular, it samples the view representation h^v from the uniform distribution, i.e. h ~ U (0, 1), then the expectation is the weighted summation by the importance weight p y, v \h^v; 0^old), which may be represented in accordance with Formula 7).

[0033] In step S405, Gradient accent is used to maximize the lower-bound of the importance weight p y, v \h^v; 0^old). The lower-bound may be particularized as log∑_ft q(h^v)— ^' q(h^v) ' ^{as snown m} F°^rmul^a 7). And then it calculates the gradient of the lower-bound, i.e. VE, where the gradient is computed by averaging over all thegradients with respect to the importance samples. Importance sampling is a basic sampling algorithm, which estimates a complex distribution p(x) with a proposal distribution q(x). As p(x) is too complex to sample direct, in the embodiments of the present application, it can sample from a simple distribution, i.e. uniform distribution, and the ratio p(x)/q(x) are known as importance weights, which correct the bias introduced by sampling from a different distribution, as below:

E(f) = J f(z)p(z)dz = J f(z) g q(z)dz * Eg f(_zi) Formula 8)

Where -÷ is the importance weight.

q(_Zl)

[0034] In step S406, it updates the parameters by gradient accent by rule of:

0^t+1 = 0^l + aVe¹ Formula 9)

[0035] In step S407, it is determined if the lower bound is reached or convergence of a data-likelihood of the joint probability is observed, if not, steps S402-S407 are iterated, otherwise, the parameters E (U_Q, U_t, U₂, U₃, U₄, V₂, V₃, W₂ and Wz) are learnt/determined.

[0036] Hereinafter, a process 200 for face recognition consistent with some disclosed embodiments will be discussed. As shown in Fig. 3, process 200 comprises a series of steps that may be performed by one or more of processors, which may be embedded or arranged on the computer, may be performed by each module/unit of the system 100 to implement a data processing operation. For purpose of description, the following discussion is made in reference to the situation where each module/unit of the system lOOis made in hardware or the combination of hardware and software. The skilled in the art shall appreciate that other suitable devices or systems shall be applicable to carry out the following process and the system 100 are just used to be an illustration to carry out the process.

[0037] At step S201 , a plurality of identity features for an input face image in a given viewpoint of the image will be determined. In one embodiment of the present application, the first plurality of identity features Yi\ is generated from the input face images in accordance with an activation function, and then a second plurality of identity features is to generated based on the generated first identity features h '^'.

For example, the first plurality of identity features i^and the identity features h may be to generated based on the generated first identity features \ί_\ by rule of Formula 1) and Formula 2).

[0038] At step S202, the process 100 captures a view representations h^vof the input face image x.

[0039] At step S203, the process yields one or more features for face recovery from the determined identity features and the view representation. In one embodiment of the present application, the generated second identity features h-^is combined with the generated view representation h_v to yield one or more third features /i^for face recovery and then generate one or more fourth features for face recovery /i₄ from the yielded third features/13. For example, the third and the fourth features /13 and /i₄for face recovery may be determined by rule of Formula 3) and Formula 4) as discussed in the above.

[0040] At step S204, the face image y will be generated from the generated recovery features /i₄, and then the generated y is united with the view representation h_v to a view label of the generated face image. In one embodiment of the present application, the face image y and the v may be determined by rule of Formula 5)and Formula 6). [0041] Fig. 5 illustrates a multi-view perceptron system 500 according to another embodiment of the present application. The system 500 may reconstruct a full spectrum of multi-view images for all the possible view labels v of a given image.

[0042] As shown in Fig. 5, the system 500may comprise a identity feature determination unit 10, a view representation capture unit 20, a feature combination unit 30, a recovery unit 40 and an image selection unit 50. Fig. 6 illustrates a process 600 for the system 500 to reconstruct the full spectrum of multi-view images for all the possible view labels v of the given image. The cooperation of the units 10-50 will be discussed in reference to Fig. 6 as below.

[0043] In step S601, the identity feature determination unit 10 operates to learn a plurality of identity features for input face images x with a given view label v. In step S601, view representation capture unit 20 operates to capture a view

representations h^v of the input face images x. In step S603, the feature combination unit 30 operates to combine the generated second identity features h₂ ^d with the view representation h_v to yield one or more third features b.3 for face recovery and then generate one or more fourth features for face recovery h₄ from the yielded third featuresb.3. In step S604, the recovery unit 40 operates to generate face image y from the generated recovery features h₄, and then the generated y (which may be represented as a set of outputs {y_s}_s_₁) and the view representation h_v to a view label of the generated face image. Since the configuration of the units 10-40 are the same of those of Fig. 1 and the processes for steps S601-S604 are the same as steps S201-S204, the detailed description thereof is omitted.

[0044] And then in step S605, the image selection unit 50 operates to compute the probabilitiesp(f |y_s , h^v of {y_s}_s_₁ and then chooses one of the set of y_s that produces the largest probability, i.e. to determine the most similar image to the input image x. For the input image x with all the possible view labels v, the system 500 repeats the above procedure to obtain the most similar image to the input x with different view labels v, such that a full spectrum of multi-view images are reconstructed for all the possible view labels v of the input image x.

[0045] According to one embodiment of the present application, given a face image x, a set of corresponding output images {y_z} may be generated through the above step S601-S605, where z indicates the index of the values of view we generated (or interpolated). If one of y_zthat is the most similarimage to x is selected from the output images {y_z}, the view label of the z-th output y_zmay be assigned to the face image x.

[0046] Although the preferred examples of the present invention have been described, those skilled in the art can make variations or modifications to these examples upon knowing the basic inventive concept. The appended claims is intended to be considered as comprising the preferred examples and all the variations or modifications fell into the scope of the present invention.

[0047] Obviously, those skilled in the art can make variations or modifications to the present invention without departing the spirit and scope of the present invention. As such, if these variations or modifications belong to the scope of the claims and equivalent technique, they may also fall into the scope of the present invention.

Claims

What is claimed is:

1. A method for face recognition, comprising:

determining a plurality of identity features for an input face image in a given viewpoint of the image;

capturing a view representation of the input face image;

generating a face image from the generated features for recovery; and uniting the generated face image and the captured view representation to a view label of the generated face image.

2. A method according to claim 1, wherein the view representation is naturally coupled with at least one of face viewpoints, face illuminations, and face expressions.

3. A method according to claim 1, wherein the determining, the capturing, the generating and uniting are carried out by a biological neural network.

4. A method according to claim 3, further comprising:

training parameters of the biological neural network through maximizing a lower-bound of prior distribution represented by the generated face image, the view representation in view of the view labels of the input face image.

5. A multi-view perceptron method according to claim 4, wherein the training further comprises:

initializing the parameters randomly;

assigning a plurality of the view representation with values such that the view representation has a uniform distribution;

generating the face image and the view label based on an inputted image and the view representation;

constructing a joint probability p(y, v |h^v; 0^old) from the generated face image, the view label, the view representation, and a current value of said parameters;

calculating a gradient Va0^tof a lower-bound of the joint probability; updating the parameters by rule of 0^t+1 = Θ¹ + VQ¹; and

iterating the above steps until convergence of a data-likelihood of the joint probability is reached.

6. A method according to claim 1 , further comprising:

computing probabilitiesp(v |y_s , h^v) of a plurality of the generated face images, and

determining a most similar image to the input image from the generated face images based on the computed probabilities.

7. A method according to claim 6, wherein, for all the possible view labels for the input image, the method further comprises:

forming a full spectrum of multi-view images from the determined most similar images to the input image with different view labels.

8. A method according to claim 1 , wherein the step of determining a plurality of identity features further comprises:

determining a first plurality of identity features from the input face image in accordance with an activation function; and

determining a second plurality of identity features based on the generated first identity features in accordance with the activation function.

9. A method according to claim 8, wherein the step of determining one or more features for face recovery further comprises:

combining the generated second identity features with the generated view representation to yield one or more third features for face recovery; and

generating one or more fourth features for face recovery from the yielded third features in accordance with the activation function.

10. A system for face recognition, comprising:

an identity feature determination configured to determine a plurality of identity features for an input face image in a given viewpoint of the image;

a recovery unit configured to generate a face image from the generated features for face recovery, and then to unite the generated face image and the view representation to a view label of the generated face image.

11. A system according to claim 10, wherein the view representation is naturally coupled with at least one of face viewpoints, face illuminations, and face expressions.

12. A system according to claim 10, wherein the identity feature determination unit, the view representation capture unit, the feature combination unit and the recovery unit are coupled together to form a biological neural network.

13. A system according to claim 12, wherein parameters of the biological neural network are determined through maximizing a lower-bound of the importance weight, the weight being represented as a probability distribution based on the generated face image, the view representation in view of the view labels of the input face image.

14. A system according to claim 13, wherein the parameters of the biological neural network are determined by: initializing the parameters E randomly;

calculating a gradient Va0^tof a lower-bound of the joint probability; updating the parameters by rule of 0^t+1 = Θ¹ + ανΌ¹; and

15. A system according to claim 10, further comprising:

an image selection unit configured to compute probabilitiesp(f |y_s , h^v) of a plurality of the generated face images, and to determine a most similar image to the input image from the generated face images based on the computed probabilities.

16. A system according to claim 15, wherein, for all the possible view labels for the input face, the determined most similar images to the input image with different view labels form a full spectrum of multi-view images.

17. A system according to claim 10, wherein the identity feature determination unit is further configured to:

determine a first plurality of identity features from the input face image in accordance with an activation function; and

determine a second plurality of identity features based on the generated first identity features in accordance with the activation function.

18. A system according to claim 17 wherein the feature combination unit is further configured to:

combine the generated second identity features with the generated view representation to yield one or more third features for face recovery; and

generate one or more fourth features for face recovery from the yielded third features in accordance with the activation function.