TWI419059B - Method and system for example-based face hallucination - Google Patents

Description

Sample-based face super-resolution method and system

The present disclosure relates to an example-based face hallucination method and system.

The face super-resolution technology can have many applications, such as surveillance, face recognition, face expression estimation, and face age estimation. . Face image is different from the general low-resolution to high-resolution image problem, because the face has its structure, and some errors in the details may cause the whole high-resolution face to become unreasonable, for example, when the eyes The shape becomes non-elliptical, or the shape of the mouth is distorted. For the entire image, the error rate is very small, but the effect is great.

Based on the characteristics of the face structure, unique face super-resolution technology has been proposed. For example, in U.S. Patent No. 7,379,611, an image super-resolution method is disclosed which infers the low resolution by extracting the original sketch priors data from the input low resolution image. Enter the high resolution detail of the image. In the document of U.S. Patent Publication No. 2008/0267525, a method of performing super-resolution amplification by capturing an edge feature in an image is disclosed.

In the paper "Image Hallucination Using Neighbor Embedding over Visual Primitive Manifolds" disclosed by Wei Fan et al., a learning-based image hallucination method is proposed, which is obtained by capturing the original image. Primitive features, and use the original features of multiple training sets to combine the high-resolution original features of the target image. In the paper "LPH super-resolution and neighbor reconstruction for residue compensation" disclosed by Yueting Zhuang et al., a two-stage face super-resolution technique is proposed, which utilizes the characteristics of manifold learning, that is, each input. Images can be similarly distributed in the manifold domain. By calculating the low-resolution image, the linear combination coefficients in the manifold domain are patched, and then the same linear combination coefficient and radius function are used. Function) to combine high-resolution images, this technique uses manifold learning to combine high-resolution images.

In the paper "An example-based face hallucination method for single-frame, low-resolution facial images" disclosed by Jeong-Seon Park et al., a sample-based super-resolution method for face is proposed, which utilizes the main analysis ( Principle Components Analysis (PCA), disassembles and trains as a base image from low-resolution images, and aligns faces through warping. As shown in the example flow of the first figure, a high-resolution face is calculated by computing a linear combination of low-resolution base images and using the same combination in a high-resolution base image.

In the paper "Face hallucination using OLPP and Kernel Ridge Regression" disclosed by BGVijay Kumar et al., a face super-resolution technique is proposed, which uses the orthogonal locality preserving projection (OLPP) in the manifold learning method. The method is to reduce the dimension of the small segment of the face image, and use the probability model in the low-dimensional space to estimate the largest possible high-resolution small segment. The reconstructed face image is corrected by using the prediction model of Kernel Ridge Regression (KRR).

The embodiment of the present disclosure can provide a sample-based face super-resolution method and system.

In one embodiment, the disclosed person is directed to a sample-based face super-resolution method. The method comprises: preparing a training database with a plurality of training images, and obtaining a low-resolution facial image to be enlarged; using the manifold learning, the plurality of training images I _{train in the} training database and the The low-resolution face image is projected to the same manifold domain, wherein the projected low-resolution face image is represented by the symbol y _L , and the projected training image is represented by the symbol y _train ; the N-projected training image y _{In the train} , a best matching training set is selected for y _L ; and a base decomposition is performed on the training set and y _L to learn a set of base images, including high and low resolution prototype faces of the training set. a low resolution prototype face of y _L ; and a high resolution face image reconstructed from the low resolution face image using the set of base images. The difference between the high-resolution prototype face of this training set and the prototype face of y _L meets a threshold requirement.

In another embodiment, the disclosed person is directed to a sample-based face super-resolution system. The system comprises a training database for collecting and storing a plurality of training images; a projection module, receiving the plurality of training images, and obtaining a projection matrix by a first-class shape learning method, and then training the plurality of exercises The image and the input low-resolution face image are projected into the same manifold domain, and the N-projected training image y _train and a projected low-resolution face image y _{L are obtained} ; a matching module (matching module) From the N-projected training image y _train , a training set that k and y _L can match is selected, k≦N; a basic decomposition module, which is decomposed using the base To extract a low resolution prototype face and a set of high-resolution prototype faces of the training set, and the difference between the low-resolution prototype face and the set of high-resolution prototype faces is in line with the threshold. a value requirement; and a face hallucination module that reconstructs the low low-resolution prototype face that meets the threshold value and the set of high-resolution prototype faces as a set of base images A high-resolution face image that resolves facial images.

The above and other objects and advantages of the present invention will be described in detail with reference to the accompanying drawings.

In a general spatial domain, the difference in the face of different faces cannot be clearly expressed. Therefore, in the present disclosure, a face database is converted to another space that can represent a face difference, that is, a manifold. area. The embodiment of the present disclosure projects a face image in an input image and a training database into a manifold domain through a manifold learning method, and filters a suitable face image to establish a face image. Prototype faces, and then combine high-resolution face images with appropriate combination parameters and prototype faces.

In general, the reconstructed face image R can be expressed as

Where I is the face image originally input, P is the prototype face, and α is the coefficient of reconstruction. Therefore, through the calculated combination parameters of the low-resolution face prototype, such as weightings, the high-resolution face prototype can be multiplied by the same combination parameters to obtain the reconstructed high-resolution face image. As shown in the example in the second figure, the originally input low-resolution face image I is equal to the linear combination of M low-resolution face prototypes, wherein α ₁ to α _M are M combination parameters, The reconstructed high-resolution face images are obtained by multiplying M high-resolution face prototypes by the same combination parameters, α ₁ to α _M . The calculation of these combined parameters can be found using one of the following examples:

α *=(( P _L ) ^T ‧ P _L ) ^-1 ‧( P _L ) ^T ‧ I _L

Among them, P _L is a low-resolution prototype face, and I _L is a low-resolution input face image.

The third figure is a schematic diagram illustrating a sample-based method of super-resolution of a face, consistent with certain disclosed embodiments. In the example of the third figure, first, a training database 310 is prepared, and a low-resolution face image 312 to be enlarged is obtained. The training database 310 can collect training face images, and generally includes a variety of training images. Then, using the manifold learning 314, for example, a transformation matrix A is obtained by using a dimensionality reduction method, and the N training images and the low-resolution facial image 312 in the training database 310 are projected into the same manifold domain, wherein the projected The low-resolution face image 312 is represented by the symbol y _L , and the projected training image is represented by the symbol y _train . From a set of N projected training images y _train , a training set 316 that matches y _L is selected. Decomposed using a base (basis decomposition) 318, this training set to extract the high resolution set of face prototypes y _L and P _H prototype face P _L, a set of learning until the substrate until the image, wherein the prototype of this group of high-resolution The difference between the face and the prototype face of y _L meets a threshold requirement. Then, the high-resolution face image 322 of the low-resolution face image 312 is reconstructed by using the set of base images, that is, the sample-based face super-resolution 320 is performed.

A technique of manifold learning is introduced in the training database 310, and a projection matrix can be learned from the training database 310, and the projection matrix can project the face image in the training database 310 to the first-class shape. The difference in different facial images can be clearly expressed. The input low-resolution face image 312 can also be projected to the same manifold domain through the same projection matrix. It is assumed that a projection matrix A is obtained by a first-order shape learning method, such as a dimensionality reduction algorithm, and a formula for projecting a low-resolution input image into a manifold domain can be expressed as y _L = A ^T I _L . A projection matrix A is obtained by any manifold learning algorithm. In general, the dimension of the projection matrix A is much smaller than the dimension of the original data in the training database 310. For example, the dimension of the high-resolution facial image is 64×. 64=4096, and the projected dimension may only select 100.

In a set of training images y _train with N projections, a method, such as a k-NN algorithm, can be used to filter out a set of {y _train } that can be matched with y _L , for example, after N projections. In the training image, find the face image y _{train that} k is most similar to y _L. A set of basic images is obtained through the k most similar facial image y _train .

The fourth diagram is an example flow diagram that further illustrates how to filter the training set in the manifold domain, consistent with certain disclosed embodiments. In the example of the fourth figure, from the N pieces of the projected training images, a training set of the face images y _train that best matches K and y _L is selected, as shown in step 410; K≦N. Then, through the training set, a base decomposition, such as principal component analysis, can be used as a base decomposition function to extract a low-resolution prototype face P _L and a set of high-resolution prototype faces of the training set, as in step 420. Shown. When the difference between the low-resolution prototype face P _L and the set of high-resolution prototype faces is less than a threshold, the low-resolution prototype face P _L and the set of high-resolution prototype faces are regarded as a set of base images, such as Step 440 is shown. Otherwise, the value of k is incremented, step 430, and then steps 410 through 420 are repeated until a set of high resolution prototype faces that meet the threshold requirements are selected.

This set of base images will vary with different input low resolution images and is the basis for reconstructing low resolution face images 312. Since the input low-resolution face image is subjected to the above-mentioned re-training action on the base image before the synthesis, it is possible to avoid reconstructing when the user inputs the face image and the face image in the training database is too large. Dissimilar facial images; it can also avoid the appearance of high-resolution images after reconstruction due to the problem of excessive facial image differences.

That is to say, once a suitable training face image set is selected from the face image database in the manifold domain, a set of prototype faces meeting the threshold value can be extracted through the base decomposition, and the set of prototype faces is also used. It is used as a base image to reconstruct high-resolution images. In step 410, a cost function can be used to determine the k value. The k value determined by this cost function minimizes the difference between the linear combination of a set of prototype faces of this training set and the low resolution face image 312.

As shown in the curve example of the fifth figure, when the k value is equal to 330, for a low-resolution face image of a test, from the N (for example, N=400) projected training images, 330 can be selected. A training set of the face image y _train that best matches the y _L , the difference between the linear combination of a set of prototype faces of the training set and the low-resolution face image of the test is minimal. In the fifth figure, the horizontal axis represents the number of screen images y _train , that is, the k value, and the vertical axis represents the above difference value.

After obtaining a low-resolution image, pre-processing such as alignment and brightness averaging may be performed first, and then the projected image of the training image is used to select a face image matching the low-resolution image. A high-resolution version of the face image that is close to the originally input low-resolution face image is reconstructed through the aforementioned combined parameter calculation and combined with the calculated parameters and the extracted base image. This is the design principle of sample-based super-resolution for this disclosure. This design principle also ensures that low-resolution images projected onto the manifold domain are similar in distribution to high-resolution images. The examples of the sixth A map and the sixth B graph are respectively the distribution of the low-resolution image projected onto the manifold domain and the distribution of the high-resolution image, wherein only the first two dimensions after projection are selected for observation. It can be seen that the sixth A map and the sixth B graph have similar distributions. That is to say, the distribution of the low-resolution image after the plurality of training images in the training database 310 are projected to the manifold domain is similar to the distribution of the high-resolution image.

In view of the above, the seventh figure is an exemplary diagram illustrating a sample-based face super-resolution system consistent with certain disclosed embodiments. In the example of the seventh figure, the super-resolution system 700 can include a training database 310, a projection module 720, a matching module 730, a base decomposition module 740, and a face super-resolution module 750. .

The training database 310 is used to collect and store a plurality of training images 712. The projection module 720 receives the plurality of training images 712 from the training database 310, and passes the first-order shape learning, such as the dimensionality reduction algorithm, to obtain the projection matrix A , and then the plurality of training images 712 and the input low-resolution people. The face image 312 is projected into the same manifold domain, that is, the projected training image y _train and the projected low-resolution face image y _{L are obtained} . The matching module 730 filters out a training set 732, k≦N with k training images that can be matched with y _L from the N projected training images y _train , for example, using a k-NN algorithm. The base decomposition module 740 uses base decomposition to extract a low resolution prototype face P _L of the low resolution face image 312 and a set of high resolution prototype faces P _{H of the} training set.

When the difference between the prototype face P _L and the set of high-resolution prototype faces does not meet the threshold requirement, the base decomposition module 740 increments the k value, and the matching module 730 extracts the training images from the N projections y _train According to the incremental k value, another training set is re-screened, and then a set of high-resolution prototype faces of the other training set are extracted through the base decomposition module 740 until the prototype face P _L and the final one The difference between the high-resolution prototype faces of the group meets this threshold requirement. The face super-resolution module 750 reconstructs the high-resolution person of the low-resolution face image 312 by using the prototype face P _L and the final set of high-resolution prototype faces meeting the threshold value as a set of base images. The face image 322 is obtained by combining the appropriate weights with the set of base images to obtain a high-resolution face image.

As shown in the example of the eighth diagram, the face super-resolution system 700 can be implemented in a computer system 800. The computer system 800 includes at least one memory device 810 and a processor 820. The memory device 810 can be used to implement the training database 310. The processor 820 is provided with a projection module 720, a matching module 730, a base decomposition module 740, and a face super-resolution module 750. The processor 820 can receive the input low-resolution facial image 312 and read the training image of the memory device 810 to execute the projection module 720, the matching module 730, the base decomposition module 740, and the face super-resolution. The function of the module 750 is used to generate a high-resolution face image 322 of the low-resolution face image 312 by combining appropriate parameter combinations.

Using the embodiment of the present disclosure, an experimental example of face simulation enhancement is provided. In this experimental example, the training set includes 483 face images, high resolution of the face image and test images (test images 1~4). The low resolution is 64 x 64 and 16 x 16, respectively. Using principal component analysis as a base decomposition function, a total of 100 prototype faces were extracted. The k-value of the k-NN algorithm can be incremented between 100 and 483 and automatically determined. The example table in Figure 8 provides a comparison of objective data for reconstruction quality. It can be clearly seen from the objective data of the ninth figure that the implementation examples of the present disclosure can effectively improve the reconstruction quality of the face image compared to the three conventional techniques.

The tenth figure is an example table providing detection rate comparisons for five different techniques, consistent with some of the disclosed embodiments. Among them, face recognition is performed according to Multilinear PCA (MPCA) technology; in the training phase, there are 62 training images with 14 subject objects; in the identification stage, there are 12 images, total 6 Item of the item. In the example of the tenth figure, the additive Gaussian noise only and the averaging filter are used to remove the additivity Gaussian noise only (additive Gaussian noise). Noise with averaging filtering) to compare the detection rates of these different technologies. It can be clearly seen from the data of the tenth figure that the embodiment of the present disclosure has a high detection rate as compared with the four conventional techniques. The implementation example can filter useful training sets from the training database. When the input image and the image in the database are different, the high-resolution version that is not like the original face image will not be reconstructed, and the manifold will not be seriously changed. The distribution of local face components in the domain.

In summary, the embodiment of the present disclosure provides a sample-based face super-resolution method and system. The embodiment of the present disclosure utilizes manifold learning and iteratively improves the base image at the time of reconstruction by filtering out a training set that is more closely matched to the input low-resolution face image. Therefore, it is possible to effectively enhance the reconstruction quality of a low-resolution face image both objectively and subjectively. Compared with the prior art, the embodiment of the present disclosure can reconstruct a reconstructed high-resolution version that is closer to the original face image, and can avoid the high resolution after the reconstruction due to the problem of excessive facial image difference. image.

The above is only an embodiment of the present invention, and the scope of the present invention cannot be limited thereto. That is, the equivalent changes and modifications made by the scope of the present invention should remain within the scope of the present invention.

310. . . Training database

312. . . Low resolution face image

y _L . . . Low resolution face image after projection

y _train . . . Projected image after projection

314. . . Manifold learning

316. . . Matchable training set

318. . . Substrate decomposition

320. . . Face super resolution

322. . . High resolution face image

410. . . From the N projected training images, a new training set of k and y _L matching face images y _{train is selected} .

420. . . Through this new training set, a base decomposition is used to extract a low-resolution prototype face P _L and a set of high-resolution prototype faces.

430. . . Incremental k value

440. . . Combine this low-resolution prototype face P _L with this set of high-resolution prototype faces as a set of base images

700. . . Face super-resolution system

712. . . Training image

720. . . Projection module

732. . . a training set with k training images

730. . . Matching module

740. . . Substrate decomposition module

750. . . Face super-resolution module

P _L . . . Low resolution prototype face

y _L . . . Low resolution face image after projection

P _H . . . High resolution prototype face

y _train . . . Projected image after projection

800. . . computer system

810. . . Memory device

820. . . processor

The first figure is an example flow diagram illustrating a conventional sample-based face super-resolution method.

The second figure is an example schematic diagram showing how to reconstruct a high-resolution face image through the calculated combination parameters of the low-resolution face prototype.

The third figure is a schematic diagram illustrating a sample-based method of super-resolution of a face, consistent with certain disclosed embodiments.

The fourth diagram is an example flow diagram illustrating how to screen a training set in a manifold domain consistent with certain disclosed embodiments.

The fifth figure is a graph of a curve, which illustrates that the number of screen images y _train , that is, the value of k, is related to the difference value, and is consistent with some of the disclosed embodiments.

6A and 6B are schematic diagrams showing examples of the distribution of low-resolution images projected onto the manifold field and the distribution of high-resolution images, respectively, consistent with certain disclosed embodiments.

The seventh diagram is a schematic diagram illustrating a sample-based facial hyper-resolution system consistent with certain disclosed embodiments. .

The eighth figure is an exemplary diagram illustrating that the face super-resolution system of the sixth figure can be implemented in a computer system consistent with certain disclosed embodiments.

The ninth diagram is an example table that provides a comparison of objective data for the quality of facial image reconstruction, consistent with certain disclosed embodiments.

The tenth figure is an example table that provides a comparison of detection rates for five different techniques, consistent with some of the disclosed embodiments.

700. . . Face super-resolution system

712. . . Training image

720. . . Projection module

732. . . a training set with k training images

730. . . Matching module

740. . . Substrate decomposition module

750. . . Face super-resolution module

P _L . . . Low resolution prototype face

y _L . . . Low resolution face image after projection

P _H . . . High resolution prototype face

310. . . Training database

312. . . Low resolution face image

y _train . . . Projected image after projection

314. . . Manifold learning

322. . . High resolution image

Claims (13)

A sample-based super-resolution method for a face, comprising: preparing a training database with a plurality of training images, and obtaining a low-resolution facial image to be enlarged; using a first-class shape learning technique, determining a projection matrix, projecting the plurality of training images of the training database to the first-class shape to clearly distinguish the projected images of the plurality of training images; using the projection matrix, the plurality of training databases The training image and the low-resolution face image are projected to the manifold domain, wherein y _L represents a projected image of the low-resolution face image, and y _train represents the plurality of training images of the training database. The projected training image; from the N projections of the training image y _train , a training set that matches y _L is selected, and the number of the plurality of training images is N; by performing the training set and y _L Base decomposition, learning a set of base images, including a set of high and low resolution prototype faces of the training set and a low resolution prototype face of y _L ; and reconstructing the low resolution face using the set of base images A high-resolution face image like the one. The face super-resolution method according to claim 1, wherein the training set finds k persons most similar to the projected image y _L in the N-projection training image y _train Face image, k≦N. The face super-resolution method according to claim 1, wherein the manifold learning technique uses a dimension reduction algorithm to obtain the projection matrix. The face super-resolution method according to claim 1, wherein the difference between the set of high-resolution prototype faces of the training set and the low-resolution prototype face of y _L meets a threshold requirement. The method of super-resolution of a face according to claim 1, wherein the substrate decomposition is performed by using principal component analysis as a substrate decomposition function to extract the set of base images. The method for super-resolution of a face according to claim 1, wherein the training set matching the y _L is filtered from the N pieces of the projected training image y _train , and the training set is matched with the y _L is an exploded substrate, the set of the learning image substrate, further comprising: a training image y from the N _train slides were screened out and the image y after the projection of the low-resolution face image of the k best matching Zhang _L The face image y _train ,k is a positive integer, k≦N; through the k face images y _train , using the base decomposition to extract the low resolution face image of y _L and the set of high resolution prototypes a face; and when the difference between the low-resolution prototype face and the set of high-resolution prototype faces meets a threshold requirement, the low-resolution prototype face of y _L and the set of high-resolution prototype faces are regarded as the group Base image; otherwise, after incrementing the k value, repeat the above two steps until the set of base images is obtained. The method according to claim 6, wherein the method uses a cost function to determine a k value, and the k value determined by the cost function causes a linear combination of a set of prototype faces of the training set and the The difference between low-resolution face images is minimal. A sample-based face super-resolution system, the system comprising: a training database for collecting and storing a plurality of training images; a projection module, receiving the plurality of training images, and obtaining the first-order shape learning technology After a projection matrix, the plurality of training images and an input low-resolution face image are projected into the same manifold domain to obtain N projections of the training image y _train and a projected low-resolution face image. y _L , the projection matrix is determined by the manifold learning technique to clearly distinguish the N projections of the training image y _train ; a matching module, from the N projections of the training image y _train , A training set of k face images that best match y _L is selected, k≦N; a base decomposition module, which uses base decomposition to extract a low-resolution prototype face of y _L for the training set and y _L And a set of high-resolution prototype faces of the training set, such that the difference between the low-resolution prototype face of y _L and the set of high-resolution prototype faces meets a threshold requirement; and a face super-resolution module, Will meet the threshold requirement Prototype resolution face with the prototype set of high-resolution face image as a set of base, to reconstruct a high-resolution low-resolution face image of the facial image. The face super-resolution system of claim 8, wherein when the difference between the low-resolution prototype face of y _L and the set of high-resolution prototype faces does not meet the threshold requirement, The base decomposition module increments the k value, and the matching module rescreens another training set according to the incremental k value from the N projected training images y _train , and then extracts another through the base decomposition module. A set of high-resolution prototype faces until the difference between the low-resolution prototype face of y _L and the final set of high-resolution prototype faces meets the threshold value requirement. For example, the face super-resolution system described in claim 8 of the patent scope, the face super-resolution system is implemented in a computer system, the computer system is at least The method includes: a memory device, the memory device is used to implement the training database; and a processor, wherein the processor is provided with the projection module, the matching module, the base decomposition module, and the face An ultra-resolution module, and receiving the input low-resolution facial image, and performing the functions of the projection module, the matching module, the base decomposition module, and the face super-resolution module, and Combining a plurality of parameters to generate the high-resolution face image of the low-resolution face image. The face super-resolution system of claim 8, wherein the value of k is automatically determined. The face super-resolution system of claim 8, wherein the low-resolution image after the plurality of training images are projected into the manifold domain is similar to the distribution of the high-resolution image. The face super-resolution system of claim 8, wherein the face super-resolution module obtains the high-resolution face image by combining a plurality of appropriate parameter combinations with the set of base images.