KR20040085503A - Apparatus and method for reconstructing facial image - Google Patents

Abstract

PURPOSE: A face image recovery apparatus and a method therefor are provided to recover a damaged face image or a face image with low resolution as the entire face image with an original state or a face image with high resolution through a simplified calculation process. CONSTITUTION: A damage region detecting unit(120) detects damage region information from a certain face image. A shape/texture information extracting unit(130) recognizes a non-damage region in an input face image on the basis of damage region information received from the damage region detecting unit. The shape/texture information extracting unit extracts shape information of an input face image displayed as a deformation field with respect to a certain reference face image, and extracts texture information of an input face image mapped to the reference face image from an input face image with respect to the non-damage region. An entire face image recovering unit(140) extracts shape information and text information corresponding to shape information and texture information of the non-damage region on the basis of a plurality of shape eigen-vectors and texture eigen-vectors, and calculates a shape eigen-vector factor and a texture eigen-vector factor through linear projection operation with shape information and texture information of the non-damage region. The entire face image recovering unit linearly overlaps the shape eigen-vector factor and the texture eigen-vector factor with the shape eigen-vector and the texture eigen-vector, and recovers shape information and texture information with respect to the entire face image. A face image generating unit(160) synthesizes shape information and texture information with respect to the entire face image received through the entire face image recovering unit, and generates a face image.

Description

Apparatus and method for reconstructing facial image}

The present invention relates to an apparatus and method for restoring a face image, and more particularly, to a face image (ie, a damaged face image) photographed in a state where a part of the face is covered by a predetermined means (eg, sunglasses, mask, glasses, etc.). The present invention relates to a face image restoring apparatus and method for restoring a face image in an original state (that is, a state in which a face is not covered by a predetermined means) and restoring a face image photographed at a low resolution to a high resolution face image.

In recent years, the number of criminals who obtain or copy another person's cash card or credit card in an illegal manner and withdraw cash through ATMs of banks is increasing. In addition, masked robbers often enter the banks and rob their cash. In order to reduce such criminal activity, each bank installs a security device (for example, CCTV) and manages the face images taken by a separate database by photographing the faces of people who are looking for a bank or an ATM. In addition, the criminal arrest rate is being increased by utilizing facial images taken when a cash extortion incident occurs. However, face images photographed from a long distance or part of the face is blocked by means such as sunglasses, masks, hats, etc. (hereinafter referred to as damaged face images) have a disadvantage in that they cannot be effectively used for criminal arrest.

In order to solve the above disadvantages, a technique of restoring the entire face image from a damaged face image in which part of the face is covered by sunglasses, a mask, a hat, or the like, or restoring a low resolution face image captured at a distance to a high resolution face image is used. It is becoming.

Conventionally, a technique for restoring an entire face image from a damaged face image is mainly used for restoring the entire face image by hand using simple image processing software, or by using a library of face components such as eyes, nose, and mouth. Techniques for restoring damaged face areas have been used. In this case, since the quality of the restored image is determined by the experience or skill of the operator, the accuracy of the restored image is inferior, and further, the division of the image restoration work is impossible.

There is also a technique for reconstructing a damaged face image by repeating a statistical gradient procedure. In this case, the speed is slow by the iterative process, and since a certain number of points having an error value with the original face image among the randomly selected points for restoring must be excluded in every iterative restoration, accurate restoration is difficult.

In addition, a method for restoring a damaged facial image is disclosed in 'Face Image Compression and Restoration of Damaged Facial Image Using a Small Number of Feature Points', filed by the applicant (Domestic Application No. 2000-0029845). . However, the above-described damaged face image restoration method is a more suitable technique for face image compression, and the accuracy of restoration is limited because the number of restored feature points must be smaller than the number of eigenvectors used for principal component analysis, and the SVD (Singular Value Decomposition) There is a disadvantage in that a large amount of computation is required to find the pseudo inverse matrix.

Meanwhile, as a technique for restoring a high resolution face image, a technique for restoring a high resolution face image with a plurality of low resolution face images has been studied. In this case, two or more face images are required, and only one face image has a disadvantage in that it is impossible to restore a high resolution face image. In addition, when restoring a high resolution face image of a predetermined level or more, there is a disadvantage in that face images photographed at a large number of different viewpoints are required.

In addition, there is a method of obtaining a high resolution face image using only one face image by using a classic simple interpolation technique used for general image processing, but the face image obtained through the simple interpolation technique has a disadvantage in that the quality is very poor.

The present invention has been made to solve the conventional problems as described above, the face image taken in a state in which a part of the face is covered by a predetermined means and the face image taken in a low resolution through a simplified calculation process It is an object of the present invention to provide a face image restoration apparatus and method for restoring a full face image and a high resolution face image in a state.

1 is a schematic configuration diagram of a face image restoration apparatus according to an embodiment of the present invention;

2 is a flowchart illustrating a method of restoring a face image according to an embodiment of the present invention;

3A is a flowchart illustrating a shape information and texture information restoration process for an entire face image according to an embodiment of the present invention;

3B is a flowchart illustrating a process of restoring shape information and texture information for a high resolution face image according to an embodiment of the present invention;

4 is a flowchart illustrating a method of restoring a face image according to another embodiment of the present invention;

5 is a flowchart illustrating a process of restoring shape information and texture information for a high resolution full face image according to another embodiment of the present invention;

6 is a first exemplary view showing a schematic configuration of a face recognition system using a face image restoration apparatus according to an embodiment of the present invention;

7 is a second exemplary view showing a schematic configuration of a face recognition system using a face image restoration apparatus according to an embodiment of the present invention.

* Description of the symbols for the main parts of the drawings *

100: face image restoration device 110: face image input unit

120: damaged area detection unit 130: shape / texture information extraction unit

140: full face image restoration unit 150: high resolution face image restoration unit

160: face image generation unit 170: face model information DB

According to an aspect of the present invention, there is provided an apparatus for recovering a face image, including: a damaged area detector for deriving damaged area information from a predetermined face image; A type of input face image that is recognized as an intact area in the input face image based on the damaged area information received from the damaged area detection unit, and is displayed as a deformation field for a predetermined reference face image in the input face image for the intact area. A shape / texture information extracting unit for extracting information and texture information of an input face image mapped to the reference face image; Based on a plurality of shape eigenvectors and texture eigenvectors provided, only shape information and texture information corresponding to the shape information and texture information of the intact area are extracted, and linear and projection of the shape information and texture information of the intact area, respectively. Calculation of shape eigenvector coefficients and texture eigenvector coefficients through calculation, and by linearly overlapping the calculated eigenvector coefficients and texture eigenvector coefficients with shape eigenvectors and texture eigenvectors, respectively, shape information and texture information of the entire face image A full face image restorer to restore the image; And a face image generator for generating a face image by synthesizing the shape information and the texture information of the entire face image received through the full face image restoration unit.

On the other hand, face image restoration method according to an embodiment of the present invention for achieving the above object comprises the steps of (a) deriving the damage area information from a predetermined face image; (b) receiving the damaged area information derived in step (a), recognizing the intact area in the input face image based on the input damaged area information, and then referring to a predetermined reference in the input face image for the intact area. Extracting shape information of an input face image displayed as a deformation field for a face image and texture information of the input face image mapped to the reference face image; (c) After extracting only the shape information and texture information corresponding to the shape information and texture information of the undamaged region based on a plurality of shape unique vectors and texture eigenvectors, the extracted shape information and texture information are intact. Compute shape eigenvector coefficients and texture eigenvector coefficients to match the shape information and texture information of the area, and linearly overlap the calculated shape eigenvector and texture eigenvector coefficients with shape eigenvectors and texture eigenvectors respectively. Restoring shape information and texture information of the image; (d) the shape information and texture information of the undamaged region extracted in step (b) or the shape information and texture information of the entire face image restored in step (c) are received, Determining whether the resolution is equal to or greater than the preset reference resolution; (e) if it is determined that the resolution of the inputted shape information and texture information is less than or equal to a preset reference resolution, a low resolution corresponding to the inputted shape information and texture information based on a plurality of preformed shape intrinsic vectors and texture inherent vectors; Extracting only shape information and texture information, and calculating shape eigenvector coefficients and texture eigenvector coefficients for matching the extracted low resolution shape information and texture information with the input shape information and texture information; (f) a high resolution shape eigenvector and texture eigenvector consisting of only high resolution shape information and texture information derived based on the calculated shape eigenvector coefficients and texture eigenvector coefficients; Restoring shape information and texture information of the high-resolution face image by linearly overlapping each other; And (g) receiving shape information and texture information of the reconstructed high resolution face image in step (f), and synthesizing the input shape information and texture information to generate a face image corresponding to the high resolution full face image. It is composed of the features.

In addition, the facial image restoration method according to another embodiment of the present invention for achieving the above object comprises the steps of: (a) detecting a damaged area in a low-resolution face image damaged a predetermined area; (b) Recognizing an undamaged region in the input face image based on the damaged region detected in step (a), and displaying a deformation length for a predetermined low resolution reference face image in the low resolution input face image for the intact region Extracting shape information of an input face image and texture information of an input face image mapped to the low resolution reference face image; (c) Receive the shape information and texture information of the low resolution input face image extracted in the step (b), the unique low resolution shape consisting of only the low resolution shape information and texture information corresponding to the input shape information and texture information Deriving a vector and a texture eigenvector based on a plurality of shape eigenvectors and a texture eigenvector; (d) calculating the shape eigenvector coefficients and texture eigenvector coefficients for matching the derived low resolution shape eigenvectors and texture eigenvectors with the inputted shape information and texture information, and then calculating the plurality of shape eigenvectors and texture eigenvectors. Restoring the shape information and the texture information of the high resolution full face image by linearly overlapping the high resolution shape eigenvector and the high resolution texture eigenvector composed of only the high resolution shape information and the texture information derived based on the above; And (e) generating a face image by synthesizing the shape information and texture information of the high resolution full face image restored in step (d).

Hereinafter, exemplary embodiments of a face image restoration apparatus and method and a face recognition system using a face image restoration apparatus will be described with reference to the accompanying drawings.

1 is a schematic configuration diagram of a face image restoration apparatus according to an embodiment of the present invention. Referring to FIG. 1, the face image restoration apparatus 100 according to the present invention includes a face image input unit 110 and a damaged area detection unit. 120, the form / texture information extractor 130, the full face image restorer 140, the high resolution face image restorer 150, the face image generator 160, and the face model information DB 170. It is composed.

Here, the face model information DB 170 stores face model information, which is information representing statistical characteristics of shape information and texture information of m face images obtained through a Principal Component Analysis process. The face model information includes the shape eigenvectors needed to restore the entire face image. And textures Eigenvectors required for reconstruction of high-resolution facial images And textures Included.

The face image input unit 110 may be a device that receives a face image and may be implemented by, for example, a device such as a scanner or a camera.

When the damaged image is transmitted through the face image input unit 110, the damaged region detector 120 determines whether the damaged region exists in the delivered face image, and when the damaged region exists, the damaged region is detected as a local damaged region. Derivation of damaged area information indicating that On the other hand, if the damaged area does not exist in the transferred face image, the damaged area information indicating that the damaged area is not detected is derived.

Here, the damaged area indicates a hidden area when a part of the face image is covered by an object such as sunglasses, a mask, glasses, or the like, or when a part of the face image is covered by overlapping with another object, shadow, camera noise, or the like. Such damaged areas may be manually controlled using a graphical interface (e.g., Marquee Tool used in Adobe's Photoshop or Lasso Tool, etc.) or a semi-automatic or automatic segmentation method (e.g., Magic Wand Tool used in Adobe's Photoshop). Used area setting method, etc.), etc. can be detected. On the other hand, as a method of detecting a damaged area, it is obvious that various methods may be used in addition to the above-described method.

The shape / texture information extraction unit 130 receives the damaged area information through the damaged area detection unit 120, and recognizes the remaining area except the damaged area in the input face image as an undamaged area based on the transmitted damaged area information. By automatically or manually acquiring a corresponding relationship between the input face image and the predetermined reference face image for the corresponding intact region, the input face image for the intact region represented by the deformation length or position difference information for the reference face image is obtained. Shape information and texture information represented by color or contrast information of an input face image of an undamaged region mapped to the reference face image are extracted.

Here, if the transmitted damaged area information is information on the local damaged area, the shape / texture information extractor 130 recognizes the remaining area of the input face image except the local damaged area as a local undamaged area, and has already described the above process. After extracting the shape information and texture information of the input face image for the local undamaged region through the shape information and texture information of the input face image for the extracted local undamaged region is transmitted to the full face image restoration unit 140. .

On the other hand, the shape / texture information extractor 130 recognizes the entire area of the input face image as an entire undamaged area (or a whole face area) if the transmitted damaged area information is information about undetected damage area. After extracting the shape information and texture information of the input face image of the entire undamaged region through the above-described process, the shape information and texture information of the input face image of the extracted entire undamaged region are extracted. Forward to 150.

The full face image restoring unit 140 receives the shape information and texture information of the input face image of the local undamaged area from the shape / texture information extracting unit 130 and corresponds to the shape information of the local undamaged area. A sub shape eigenvector consisting only of information is derived based on a plurality of shape eigenvectors for restoring the entire face image stored in the face model information DB 170. And, the optimal shape eigenvector coefficient for matching the derived sub shape eigenvectors with the shape information of the local undamaged region After calculating, linearly overlap the calculated shape eigenvector coefficients and the shape eigenvectors stored in the face model information DB 170 to restore the shape information of the entire face image.

In addition, the full face image restoration unit 140 stores a plurality of textures for restoring the entire face image stored in the face model information DB 170 using a sub texture eigenvector consisting of only texture information corresponding to the texture information of the local undamaged region. Derived based on eigenvectors. And, the optimal texture eigenvector coefficient for matching the derived sub texture eigenvector with the texture information of the local undamaged region And linearly superimpose the calculated texture eigenvector coefficients and the texture eigenvectors stored in the face model information DB 170 to restore texture information for the entire face image. Here, the entire face image refers to the original face image which is not damaged by restoring the damaged area with respect to the input face image.

The high resolution face image restoration unit 150 receives shape information and texture information of the entire face image restored from the full face image restoration unit 140 or the entire undamaged region extracted from the shape / texture information extraction unit 130. Receives shape information and texture information of input face image for.

When the resolution of the received shape information and texture information is analyzed and the resolution is less than or equal to a preset reference resolution, the resolution of the received shape information and texture information is low resolution, and a low resolution shape corresponding to the received shape information is recognized. A plurality of shape eigenvectors for restoring a high resolution face image stored in the face model information DB 170 with a low resolution eigenvector consisting only of information Derived based on

Subsequently, an optimal shape eigenvector coefficient for matching the derived low resolution shape eigenvector with the received shape information To calculate.

Then, a high resolution shape eigenvector consisting of only high resolution shape information is derived based on a plurality of shape eigenvectors stored in the face model information DB 170, and then the resulting high resolution shape eigenvector and the calculated shape eigenvector coefficients are linear. By overlapping, shape information for a high resolution face image is restored.

In addition, a plurality of texture eigenvectors for restoring a high resolution face image stored in the face model information DB 170 with a low resolution texture eigenvector consisting of only low resolution texture information corresponding to the received texture information. After deriving based on, the optimal texture eigenvector coefficient for matching the derived low resolution texture eigenvector with the received texture information To calculate.

After deriving a high resolution texture eigenvector consisting of only high resolution texture information based on a plurality of texture eigenvectors stored in the face model information DB 170, linearly deriving the derived high resolution texture eigenvector and the calculated texture eigenvector coefficient. By overlapping, the texture information for the high resolution face image is restored.

Meanwhile, when the resolution of the received shape information and texture information is greater than or equal to a preset reference resolution, the high resolution face image restoration unit 150 reconstructs the entire face restored from the shape information and texture information (that is, the entire face shape restoration unit 140). Recognizing that the resolution of the shape information and texture information of the image or the shape information and texture information of the input face image for the entire undamaged region extracted from the shape / texture information extraction unit 130 is high resolution, the shape information and texture Do not restore the resolution of the information to a new high resolution.

Here, the reference resolution is the resolution of the shape information and the texture information of the face image of 64 x 64 pixels, which is generally the minimum value that can be recognized by a human or face recognition system. On the other hand, since the minimum resolution that can be recognized may differ for each person or face recognition system, it is natural that the reference resolution value may vary.

The face image generator 160 receives shape information and texture information of the high resolution face image restored through the high resolution face image restorer 150 from the high resolution face image restorer 150, or the high resolution face image restorer 150. Receives shape information and texture information of the high resolution face image that is higher than or equal to the reference resolution restored by the full face image restoration unit 140 from 150, and synthesizes the received shape information and texture information to the high resolution full face image. A corresponding face image is generated. Here, the process of generating the face image by synthesizing the shape information and the texture information is called warping, and the warping process is well known in the art, and thus a detailed description thereof will be omitted.

2 is a flowchart illustrating a method of restoring a face image according to an exemplary embodiment of the present invention. As shown in FIG. 2, the method for restoring a face image according to an embodiment of the present invention receives a predetermined face image (S100), determines whether a damaged area exists in the input face image (S110), As a result, if a damaged area exists at a predetermined position of the input face image, the damaged area information is derived based on the damaged area (S120).

Then, based on the damaged area information derived in step s120, the damaged area is identified in the input face image, and the remaining area except the damaged area is recognized as a local intact area, and then the input face image for the corresponding local intact area is detected. Shape information and texture information of the extracted (S130). Here, the process of extracting the shape information and the texture information of the input face image for the local undamaged area (s130) has already been mentioned when the operation of the shape / texture information extraction unit 130 shown in FIG. 1 is duplicated. Omit.

Subsequently, the shape information and texture information of the input face image of the local undamaged region extracted in step S130 are received, and the entire face image is based on the shape information and the texture information of the input face image of the local undamaged region. The shape information and texture information for the data is restored (s140). Here, the description of step S140 will be described in detail with reference to FIG. 3A.

On the other hand, it is determined whether the resolution of the shape information and texture information of the entire face image reconstructed in step S140 is equal to or greater than the preset reference resolution (S150).

If the resolution of the shape information and texture information of the restored full face image is greater than or equal to a preset reference resolution, the determination result (s150) recognizes that the resolution of the shape information and texture information is high resolution and synthesizes the shape information and texture information. Thus, a face image corresponding to the high resolution full face image is generated (S160).

On the other hand, if the resolution of the shape information and texture information of the entire face image restored in step s140, the determination result (s150) or less than the preset reference resolution, the resolution of the shape information and texture information for the restored full face image Recognizing that it is a low resolution, and restores the shape information and texture information for a high-resolution face image with the shape information and texture information (that is, the shape information and texture information for the low-resolution full face image (s170), the restored high resolution face The face image is generated by synthesizing the shape information and the texture information of the image (S180). Here, the description of step S170 will be described in detail with reference to FIG. 3B.

Also, as a result of determining whether the damaged area exists based on the input face image (s110), when it is determined that the damaged area does not exist in the input face image, the damaged area information indicating the detection of the damaged area based on the corresponding input face image After recognizing the entire region of the input face image as the entire intact region based on the damaged region information, the shape information and the texture information of the input face image for the entire intact region are extracted (s190). . Here, the extraction process (s190) of the shape information and texture information of the input face image for the entire undamaged area is already mentioned when the operation of the shape / texture information extraction unit 130 shown in FIG. 1 is omitted. .

In operation S200, it is determined whether the resolution of the shape information and the texture information of the input face image of the entire undamaged region extracted in step S190 is greater than or equal to the preset reference resolution.

If the resolution of the shape information and texture information of the input face image of the entire undamaged region is less than or equal to a preset reference resolution, the resolution of the shape information and the texture information of the input face image of the entire undamaged region is determined. Recognizes that the image is low resolution, and restores the shape information and the texture information for the high resolution face image with the shape information and the texture information (that is, the shape information and the texture information of the input face image of the entire undamaged region of low resolution) (s210). ). At this time, since step s210 operates in the same manner as step s170, a redundant description will be omitted.

Then, the shape information and the texture information of the high resolution face image restored in step S210 are synthesized to generate a face image corresponding to the high resolution full face image (S220).

On the other hand, if the resolution of the shape information and texture information of the input face image for the entire undamaged area is greater than a predetermined reference resolution, the damaged region does not exist in the input face image and the image of the input face image is not determined. Since the resolution is higher than the preset reference resolution, the operation is terminated without performing any further work.

3A is a flowchart illustrating a process of restoring shape information and texture information for a whole face image (S140) according to an embodiment of the present invention. Referring to FIG. 3A, first, shape information and texture information of an input face image of a local undamaged region extracted in step S130 are received.

Then, the shape inherent vector stored in the face model information DB 170 with the shape information of the input face image (hereinafter, referred to as shape information of the local undamaged area) for the input local undamaged area. ( ; Is the number of shape eigenvectors, Search for a predetermined number of points in a reference face image), extracts only the shape information corresponding to the shape information of the local undamaged region based on the corresponding shape eigenvector, and consists of only the extracted shape information. ( ; Deduces the number of points in the local undamaged region of the input face image (S141).

Subsequently, an optimal shape eigenvector coefficient for matching the sub shape eigenvector with the shape information of the local undamaged area by comparing the sub shape eigenvector derived in step S141 with the shape information of the local undamaged area. After calculating (s142), the calculated shape eigenvector coefficients And eigenvectors ( ) Are linearly superimposed, respectively, to restore the shape information of the entire face image (s143).

In addition, the texture information of the input face image of the input local undamaged region may be performed by performing steps s144, s145, and s146 operating in the same manner as the above-described steps s141, s142, and s143. Restore texture information for.

Here, the process of restoring the shape information and the texture information of the entire face image with the shape information and the texture information of the input face image for the local undamaged region will be described in more detail as follows.

This process is based on the premise that the face is modeled as a linear superposition of shape and texture prototypes, and the existing SVD (Singular) with the shape information and texture information of the input face image for the intact area (local intact area). Find an optimized approximation (i.e., shape eigenvector coefficients and texture eigenvector coefficients) of the deformation close to the circular shape of the input face image (input face image) using a much simpler linear projection method than Value Decomposition. The process can be summarized.

First, the process of modeling the face as a linear superposition of shape and texture prototypes is as follows.

Under the assumption that all the face images and the reference face images are already given, the shape information of the face image is encoded into a deformation field of the reference face image, and the texture information of the face image is mapped to the reference face image. Color or contrast information of the facial image is encoded. That is, the shape information of the face image Points on the reference face image The texture information of the face image by the positional difference between the corresponding points of the face and the input face image. Points on the reference face image It is defined as the color or contrast value for the corresponding point of the input face image.

And, for each point of shape information obtained independently ( ; here Is the shape average consisting of the average number of predetermined points in the reference face image) , For each point of texture information Texture Averages of Means Finding the difference in form Covariance of And texture difference Covariance of Obtain

The values obtained in this way are processed by principal component analysis. Eigenvectors of the shape of covariance for face images Vector unique texture with Obtain Shape eigenvectors Vector unique texture with The face model based on may be expressed as Equation 1 below.

here ego, Is the number of face images.

Through this process, the shape eigenvectors , Texture unique vector Is stored in the face model information DB 170 and used when restoring the shape information and texture information of the entire face image.

Next, a process of finding an optimized approximation of the deformation close to the circular shape of the input face image using the simple linear projection method using only the shape information and the texture information of the input face image for the local undamaged region will be described. . Since only the shape information and texture information of the input face image for the local undamaged region are given, only an optimal approximation value can be estimated. Therefore, the following describes the process of finding the optimal solution (value) under these conditions.

First, each point in the local undamaged region of the input face image Shape information in And a plurality of shape eigenvectors stored in the face model information DB 170 The shape information extracted based on (i.e., each point in the local undamaged region) Shape eigenvectors consisting only of shape information corresponding to Error function is the sum of the squares of the difference of linear overlapping values It is defined as Error function Is defined as in Equation 2 below.

Where the error function is for a local intact region. Coefficients of the shape eigenvectors satisfying that condition by setting the condition that must have a minimum value Obtain This optimal solution In order to find), Equation 3 is defined as follows.

At this time, Are predetermined points within the local undamaged region of the input face image, Are points for the local undamaged region of the input face image ( ), Is a shape unique vector stored in the face model information DB (170) Is the number of. Generally, silver 10 times larger than

Here, Equation 4 is converted into a matrix form in order to simplify the equation 3, the optimal solution by converting into a matrix form as shown in Equation 4 Can be obtained more easily by the linear projection method. In addition, summarizing Equation 4 may be expressed as Equation 5 below.

here,

At this time, It is assumed that the column vectors of are linearly independent. As a result, as defined in Equation 5, alpha is each point in the local intact region of the input face image. Shape information in And the shape eigenvector stored in the face model information DB 170. Shape inherent vector consisting only of shape information corresponding to shape information of each point in the local undamaged region extracted based on [ ] (Ie, subproject eigenvectors). This process is performed in the shape eigenvector coefficient calculating step s142 of FIG. 3a.

Therefore, the equation 5 is summarized, the optimal solution Can be obtained from equation (6). Finally, the shape information for the entire face image is restored by the equation (7) defined based on the equations (1) and (6).

At this time, Are predetermined points in the reference face image, Is the number of points on the reference face image, and the shape information of the entire face image. Is the optimal shape eigenvector coefficient as described above in step s143 of FIG. And the shape eigenvectors stored in the face model information DB 170 Is restored by linear overlapping.

Similarly, by applying the above equations (Equations 1 to 7) to the texture information, it is possible to restore the texture information of the entire face image with the texture information of the local undamaged region of the input face image.

3B is a flowchart illustrating a process of restoring shape information and texture information for a high resolution face image (S170) according to an embodiment of the present invention. Referring to FIG. 3B, first, when it is determined that the resolution of the shape information and texture information of the entire face image reconstructed in step S150 is less than or equal to a preset reference resolution, the shape information of the restored full face image and Recognizing that the resolution of the texture information is low resolution, and receives the shape information and texture information for the entire face image.

Then, a plurality of shape inherent vectors stored in the face model information DB 170 with the shape information on the input full face image (that is, the shape information on the low resolution full face image). ( ; Is the shape eigenvector coefficient, Is the number of predetermined points in the low resolution reference face image, Search for a predetermined number of points in a high resolution reference face image), extracts only the low resolution shape information corresponding to the shape information of the low resolution full face image based on the shape eigenvector, and includes a low resolution composed only of the extracted shape information. Shape eigenvectors To derive (s171).

Subsequently, the low resolution shape eigenvector derived in step s171 is compared with the shape information of the low resolution full face image to match the low resolution shape eigenvector to the shape information of the corresponding low resolution full face image. Shape eigenvector coefficients To calculate (s172).

Meanwhile, the shape inherent vector stored in the face model information DB 170. ( High resolution shape eigenvector consisting of extracted shape information only To derive (s173).

Then, the optimal shape eigenvector coefficient calculated in step s172 And the high resolution shape eigenvectors derived in step s173. By linearly overlapping, the shape information for the high resolution face image is restored (s174).

In addition, the steps of operating the same as the above-described steps (s171, s172, s173, s174) with respect to the texture information (ie, the texture information of the low-resolution full-face image) for the entire face image (s175, s176, s177, s178) to restore the texture information for the high resolution face image.

Here, the process of restoring the shape information and texture information of the high resolution face image will be described in more detail.

This process is based on the assumption that the face is modeled as a linear superposition of shape and texture prototypes, and has only the shape and texture information of the low resolution full face image, and the deformation required by the low resolution full face image using the simple linear projection method. It can be summarized as a process of finding an optimized approximation of (i.e., shape eigenvector coefficients and texture eigenvector coefficients).

First, the process of modeling the face as a linear superposition of shape and texture prototypes will be described. Here, basically, the process of modeling the face by linear superposition of shape and texture prototypes is the same as described above. However, here, the shape information of the face image is encoded by the concatenation of the transform field for each of the low resolution reference face image and the high resolution reference face image, and the texture information of the face image is mapped to the low resolution reference face image and the high resolution reference face image. It is encoded by linking color or contrast information of the input face image.

That is, the shape information of the face image Is the point on the low-resolution reference face image. And positional differences between the corresponding points on the input face image and the point on the high-resolution reference face image Texture information of the face image by linking the positional difference between the corresponding points of the image and the input face image Is the point on the low-resolution reference face image. Color or contrast values for the corresponding points in the input face image for, and points on the high-resolution reference face image. It is defined as the concatenation of color or contrast values for the corresponding points of the input face image for.

Therefore, the shape information of the face image Is Texture information of face image Is Can be defined as here, Is Wow Means the position difference with respect to the corresponding point in the direction, Means the color or contrast value for the corresponding point.

Also, as already explained, the shape average Texture average , Form difference Covariance of , Texture difference Covariance of Obtain Principal component analysis process Eigenvectors of the shape of covariance for face images Vector unique texture with Obtain Shape eigenvectors Vector unique texture with The face model based on may be expressed as Equation 8 below.

,

here, to be.

Next, a process of finding an optimal approximation of the deformation required by the low resolution full face image input using a simple linear projection method using only the shape information and the texture information of the low face full face image will be described. Since only shape information and texture information of the low resolution full face image are given, only an optimal approximation value can be estimated under this condition. Therefore, the following describes the process of finding the optimal solution (value) under these conditions.

First, each point in the full face image of the input low resolution Shape information in And a plurality of shape eigenvectors stored in the face model information DB 170 Shape inherent vector consisting only of low resolution shape information corresponding to shape information of each point in the low resolution full face image extracted based on (Ie, sum of squares of differences between linearly overlapping low-resolution eigenvectors) It is defined as Error function Is defined as in Equation 9 below.

here, Are the dots on the low resolution full face image, Is the number of points on the low-resolution full-face image, Silver shape eigenvectors Is the number of. And, in general, silver 10 times larger than

In this case, the error function for the low resolution full face image Coefficients of shape eigenvectors (i.e., low resolution shape eigenvectors) that satisfy the condition by setting the condition that must have a minimum value Error function Optimal solution to minimize Obtain Such an optimal solution In order to calculate the equation (10) is defined as follows.

At this time, if the equation 10 is converted into a matrix form for simplicity, it is expressed as Equation 11 below.

As defined in Equation 11, the optimal solution by converting to matrix form Can be obtained more easily by the linear projection method. Equation 11 is summarized as in Equation 12 below.

here,

At this time, It is assumed that the column vectors of are linearly independent. As a result, as defined in Equation 12, Is the shape information at each point in the low resolution full face image. And the shape eigenvector stored in the face model information DB 170. Shape eigenvectors consisting only of shape information corresponding to shape information of each point in the low resolution full face image extracted based on [ (Ie, a low resolution shape eigenvector) is calculated through a linear projection calculation process, and this process is performed in the shape eigenvector coefficient calculating step s172 of FIG. 3B.

Therefore, the equation (12) is the best solution Can be obtained from equation (13). Finally, the shape information for the high resolution face image is restored by Equation 14 defined based on Equations 8 and 13.

That is, shape information for high resolution face image Is the optimal shape eigenvector coefficient calculated from Eq. And the shape eigenvectors stored in the face model information DB 170 Shape eigenvector consisting only of high resolution shape information extracted based on (I.e., high resolution shape eigenvectors) is reconstructed by linearly superimposing. This process is performed in steps s173 and s174 of FIG. 3b.

Similarly, by applying the above equations (Equation 8 to Equation 14) to the texture information, it is possible to restore the texture information for the high-resolution face image with the texture information for the low-resolution full face image.

4 is a flowchart illustrating a method of restoring a face image according to another exemplary embodiment of the present invention. 4 is an embodiment of a method for restoring a face image under the assumption that a damaged low resolution face image is always input. On the other hand, since the damaged low-resolution face image is always input, the full face image restoration unit 140 and the high resolution face image restoration unit 150 shown in FIG. 1 are preferably integrated into one.

Referring to FIG. 4, first, a method for restoring a face image according to another embodiment of the present invention receives a damaged low resolution face image (s300), detects a damaged area after checking a damaged area in the input damaged low resolution face image. In operation S310, damage area information is derived based on the detected damage area. At this time, the damaged area information always includes information on the local damaged area.

Then, based on the damaged region information derived in step s310, the remaining region except for the damaged region is recognized as a local undamaged region in the low-resolution input face image, and then the low-resolution input face image and the predetermined resolution for the local undamaged region are determined. Shape information of the low resolution input face image of the local uninjured region represented by the deformation length or position difference information of the low resolution reference face image by automatically or manually obtaining a correspondence relationship with the low resolution reference face image, and the low resolution reference face image The texture information represented by the color or contrast information of the input face image mapped to the extracted is extracted (S320).

The shape information and texture information of the low resolution input face image of the local undamaged region extracted in step S320 are restored, and the shape information and texture information of the high resolution full face image are restored (S330). In this case, a detailed description of the operation s330 will be described with reference to FIG. 5.

Finally, the face image is generated by synthesizing the shape information and the texture information of the high resolution full face image restored in step S330 (S340). Here, since the detailed description of the operation s340 has already been mentioned when the operation of the face image generator 160 shown in FIG. 1 is already described, it will be omitted.

5 is a flowchart illustrating a shape information and texture information restoration process s330 of a high resolution full face image according to another exemplary embodiment of the present invention. Referring to FIG. 5, first, shape information and texture information of a low resolution input face image of a local undamaged region extracted in step S320 are received.

Then, the shape intrinsic vector stored in the face model information DB 170 is searched with the shape information of the low resolution input face image of the input local intact region, and the local intact region is based on the shape inherent vector. After extracting only the low resolution shape information corresponding to the shape information of the low resolution input face image with respect to the low resolution shape eigenvector consisting of only the extracted shape information (s331).

Subsequently, an optimal shape eigenvector coefficient is calculated for matching the low resolution shape eigenvector derived in step S331 with the shape information of the low resolution input face image for the local undamaged region (S332).

Meanwhile, after extracting only high resolution shape information based on the shape unique vector stored in the face model information DB 170, a high resolution shape unique vector including only the shape information is derived (s333), and the derived high resolution shape unique vector and the calculated By linearly superimposing the shape eigenvector coefficients, shape information of the high resolution full face image is restored (s334).

Also, textures of the high resolution full face image are performed by performing steps s335, s336, s337, and s338, which are performed in the same manner as described above, with respect to the texture information of the low resolution input face image of the input local undamaged region. Restore the information.

Here, the process of restoring the shape information and the texture information of the high resolution full face image with the shape information and the texture information of the low resolution input face image of the local undamaged region shown in FIG. A process of restoring the shape information and the texture information of the entire face image with the shape information and the texture information of the input face image, and the high resolution face image with the shape information and the texture information of the low resolution full face image with reference to FIG. Since it combines the process of restoring the shape information and texture information for, the detailed description of each step (s331, s332, s333, s334, s335, s336, s337, s338) shown in FIG. .

6 is a first exemplary view showing a schematic configuration of a face recognition system using a face image restoration apparatus according to an embodiment of the present invention. Referring to FIG. 6, the face recognition system 200 includes a recognition feature information extractor 210, a similarity comparison unit 220, a result output unit 230, and a face feature information DB 240.

The recognition feature information extracting unit 210 receives a face image from the face image generating unit 160 of the face image restoring apparatus 100, and then uses feature information necessary for face recognition from the input face image (hereinafter, referred to as recognition feature information) ). Here, the recognition characteristic information includes a normalized image of a face image based on face components (for example, eyes, nose, mouth, etc.), a coefficient obtained by principal component analysis of a face image, and a ratio representing a correlation between face components. Information, wavelet transform coefficients of the face image, and the like may be used.

The similarity comparison unit 220 receives the recognition feature information from the recognition feature information extractor 210, and compares the similarity between the input recognition feature information and each of the plurality of recognition feature information stored in the facial feature information DB 240 provided. After the calculation, the calculated similarity is extracted from the facial feature information DB 240 by comparing each calculated similarity. In this case, the similarity between the two recognition feature information may be calculated using the correlation between the recognition feature information, the Euclidean distance, and the like. Here, more various equations may be used for the similarity calculation method, and the similarity calculation method is well known to those skilled in the art, and thus a detailed description thereof will be omitted.

The result output unit 230 transmits the face image information (eg, photo) and human information (eg, name, age, etc.) corresponding to the recognition feature information extracted by the similarity comparison unit 220 from the face feature information DB 240. Extract it and print it out.

Meanwhile, the facial feature information DB 240 stores recognition feature information for each of a plurality of face images for comparing the similarity with the recognition feature information extracted by the recognition feature information extractor 210, and also recognizes each feature feature. It stores face image information and personal information about.

7 is a second exemplary view showing a schematic configuration of a face recognition system using a face image restoration apparatus according to an embodiment of the present invention. Referring to FIG. 7, the face recognition system 300 includes a similarity comparison unit 310, a result output unit 320, and a facial feature information DB 330.

At this time, the face recognition system 300 receives the shape information and texture information of the high resolution face image restored through the high resolution face image restoration unit 150 of the face image restoration apparatus 100, and inputs the shape information and texture information. After the feature is set as the recognition feature information, the shape feature and texture information is retrieved, and the face feature information DB 330 is searched to extract the shape information and texture information having the greatest similarity with the input shape information and texture information. The facial image information and recognition information corresponding to the extracted shape information and texture information may be output.

Here, unlike the face recognition system 200 illustrated in FIG. 6, the face recognition system 300 illustrated in FIG. 7 is a high resolution face restored through the high resolution face image restoration unit 150 of the face image restoration apparatus 100. By using the shape information and the texture information of the image as recognition feature information, a separate device (ie, recognition feature information extraction unit) for extracting recognition feature information may not be provided.

Meanwhile, the similarity comparison unit 310 illustrated in FIG. 7 operates in the same manner as the similarity comparison unit 220 illustrated in FIG. 6. However, only the data used for comparing the similarity is different. In addition, unlike the facial feature information DB 240 shown in FIG. 6, the facial feature information DB 330 shown in FIG. 7 stores shape information and texture information of each of a plurality of face images, and each shape information and It stores face image information and human information corresponding to texture information.

Finally, the face recognition systems 200 and 300 utilizing the result information of the face image restoration apparatus 100 according to the embodiment of the present invention as input information are conventional systems, and may be implemented by various methods other than the above-described operation method. Naturally, a face recognition system can also be used.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown not in the above description but in the claims, and all differences within the equivalent scope should be interpreted as being included in the present invention.

According to the present invention, a damaged face image or a low resolution face image can be restored to a full face image or a high resolution face image in its original state through a more simplified calculation process.

On the other hand, the present invention in conjunction with the existing face recognition system, by restoring a damaged face image or a low-resolution face image provided as input information to the face recognition system to the original state face image or by reconstructing the face image of a high resolution There is an advantage that can improve the result of the face recognition system.

Claims (10)

A damaged area detector for deriving damaged area information from a predetermined face image; A type of input face image that is recognized as an intact area in the input face image based on the damaged area information received from the damaged area detection unit, and is displayed as a deformation field for a predetermined reference face image in the input face image for the intact area. A shape / texture information extracting unit for extracting information and texture information of an input face image mapped to the reference face image; Based on a plurality of shape eigenvectors and texture eigenvectors provided, only shape information and texture information corresponding to the shape information and texture information of the intact area are extracted, and linear and projection of the shape information and texture information of the intact area, respectively. Calculation of shape eigenvector coefficients and texture eigenvector coefficients through calculation, and by linearly overlapping the calculated eigenvector coefficients and texture eigenvector coefficients with shape eigenvectors and texture eigenvectors, respectively, shape information and texture information of the entire face image A full face image restorer to restore the image; And And a face image generation unit for generating a face image by synthesizing the shape information and the texture information of the whole face image received through the full face image restoring unit. The method of claim 1, It is determined whether the resolution of the shape information and the texture information of the predetermined face image is greater than or equal to the preset reference resolution. When the resolution of the shape information and texture information is less than or equal to a preset reference resolution, low resolution shape information and texture information corresponding to the shape information and texture information are extracted based on a plurality of shape unique vectors and texture unique vectors. The shape eigenvector coefficients and texture eigenvector coefficients are calculated through the linear projection operation with the shape information and texture information, respectively, and the calculated shape eigenvector coefficients and texture eigenvector coefficients are based on a plurality of shape eigenvectors and texture eigenvectors. And a high resolution face image restoring unit for restoring the shape information and the texture information for the high resolution face image by linearly overlapping the high resolution shape eigenvectors and the texture eigenvectors, respectively. The damage zone detection unit of claim 1 or 2, wherein the damage area detection unit If it is determined that a part of the face image is damaged, damage area information indicating that the damaged area is detected as a local damage area is derived, and if it is determined that all of the face image is not damaged, the damaged area indicating that the damaged area is not detected Facial image restoration device, characterized in that to derive the information. The method of claim 3, wherein the form / texture information extraction unit If the received damaged area information is information on the local damaged area, the remaining area except the local damaged area in the input face image is designated as a local undamaged area, and the shape information and texture information of the input face image for the local undamaged area. After extracting and passing to the full face image restoration unit, If the received damaged region information is information on the undetected damage region, the entire region of the input face image is designated as all intact region, and after extracting the shape information and texture information of the input face image for the entire intact region, Face image restoration apparatus characterized in that the transfer to the high-resolution face image restoration unit. The method of claim 4, wherein the full face image restoration unit The shape / texture information extracting unit receives shape information and texture information of the input face image of the local undamaged area, and based on the plurality of shape eigenvectors and texture eigenvectors provided, the shape of the received local undamaged area. A shape / texture eigenvector derivation unit for deriving a sub-form eigenvector and a texture eigenvector comprising only shape information and texture information corresponding to the information and texture information; The subform eigenvectors and subtexture eigenvectors and the shape information and texture information of the local undamaged region are linearly projected, respectively, and the derived subtype eigenvectors and texture eigenvectors are used to form information of the corresponding local undamaged regions. A shape / texture eigenvector coefficient calculator for calculating shape eigenvector coefficients and texture eigenvector coefficients for matching to texture and texture information, respectively; And linearly overlapping the calculated shape eigenvector coefficients and the shape eigenvectors and linearly overlapping the calculated texture eigenvector coefficients and texture eigenvectors. Facial image restoration device. The method of claim 5, wherein the high-resolution face image restoration unit A resolution determination unit that determines whether the resolution of the shape information and texture information of the entire face image restored by the full face image restoration unit is equal to or greater than the reference resolution; If it is determined that the resolution of the shape information and the texture information of the entire face image is less than or equal to the reference resolution, the low resolution corresponding to the shape information and the texture information of the whole face image is based on a plurality of shape unique vectors and texture unique vectors. A low resolution shape / texture eigenvector derivation unit for deriving a low resolution eigenvector and a texture eigenvector comprising only the shape information and texture information; The derived low resolution shape eigenvectors and texture eigenvectors are linearly projected with the shape information and texture information of the entire face image, respectively, and the low resolution shape eigenvectors and texture eigenvectors are applied to the shape information and texture information of the corresponding full face image. A shape / texture eigenvector coefficient calculating unit for calculating a shape eigenvector coefficient and a texture eigenvector coefficient for matching respectively; A high resolution shape / texture eigenvector derivation unit for deriving a high resolution shape eigenvector and texture eigenvector consisting of only high resolution shape information and texture information based on a plurality of preformed shape eigenvectors and texture eigenvectors; Linearly superimpose the calculated shape eigenvector coefficients and the derived high resolution shape eigenvectors to restore shape information for a high resolution face image, and linearly overlap the calculated texture eigenvector coefficients and derived high resolution texture eigenvectors to obtain a high resolution face image. Facial image restoration apparatus comprising a linear overlapping portion for restoring texture information for. The apparatus of claim 6, wherein the face image generation unit And a face image by synthesizing the shape information and the texture information of the high resolution face image restored by the high resolution face image restoration unit. The method of claim 4, wherein the high-resolution face image restoration unit A resolution determination unit for determining whether the resolution of the shape information and texture information of the input face image of the entire undamaged region extracted by the shape / texture information extraction unit is equal to or greater than a preset reference resolution; If it is determined that the resolution of the shape information and texture information of the entire undamaged region is less than or equal to a preset reference resolution, the shape information and texture information of the entire undamaged region are based on a plurality of shape inherent vectors and texture eigenvectors. A low resolution shape / texture eigenvector derivation unit for deriving a low resolution eigenvector and a texture eigenvector comprising only corresponding low resolution shape information and texture information; The derived low resolution shape eigenvectors and texture eigenvectors are linearly projected with the shape information and texture information of the entire undamaged area, respectively, and the low resolution shape eigenvectors and texture eigenvectors are used for the shape information and texture of the entire undamaged area. A shape / texture eigenvector coefficient calculator for calculating a shape eigenvector coefficient and a texture eigenvector coefficient for matching each information; A high resolution shape / texture eigenvector derivation unit for deriving a high resolution shape eigenvector and texture eigenvector consisting of only high resolution shape information and texture information based on a plurality of preformed shape eigenvectors and texture eigenvectors; Linearly superimpose the calculated shape eigenvector coefficients and the derived high resolution shape eigenvectors to restore shape information for the high resolution full face image, and linearly overlap the calculated texture eigenvector coefficients and high resolution texture eigenvectors Facial image restoration apparatus comprising a linear overlapping portion for restoring texture information for the image. (a) deriving damaged area information from a predetermined face image; (b) receiving the damaged area information derived in step (a), recognizing the intact area in the input face image based on the input damaged area information, and then referring to a predetermined reference in the input face image for the intact area. Extracting shape information of an input face image displayed as a deformation field for a face image and texture information of the input face image mapped to the reference face image; (c) After extracting only the shape information and texture information corresponding to the shape information and texture information of the undamaged region based on a plurality of shape unique vectors and texture eigenvectors, the extracted shape information and texture information are intact. Compute shape eigenvector coefficients and texture eigenvector coefficients to match the shape information and texture information of the area, and linearly overlap the calculated shape eigenvector and texture eigenvector coefficients with shape eigenvectors and texture eigenvectors respectively. Restoring shape information and texture information of the image; (d) the shape information and texture information of the undamaged region extracted in step (b) or the shape information and texture information of the entire face image restored in step (c) are received, Determining whether the resolution is equal to or greater than the preset reference resolution; (e) if it is determined that the resolution of the inputted shape information and texture information is less than or equal to a preset reference resolution, a low resolution corresponding to the inputted shape information and texture information based on a plurality of preformed shape intrinsic vectors and texture inherent vectors; Extracting only shape information and texture information, and calculating shape eigenvector coefficients and texture eigenvector coefficients for matching the extracted low resolution shape information and texture information with the input shape information and texture information; (f) a high resolution shape eigenvector and texture eigenvector consisting of only high resolution shape information and texture information derived based on the calculated shape eigenvector coefficients and texture eigenvector coefficients; Restoring shape information and texture information of the high-resolution face image by linearly overlapping each other; And (g) receiving shape information and texture information of the reconstructed high resolution face image in step (f), and synthesizing the input shape information and texture information to generate a face image corresponding to the high resolution full face image; Facial image restoration method characterized in that made. (a) detecting a damaged area in a low resolution face image in which a predetermined area is damaged; (b) Recognizing an undamaged region in the input face image based on the damaged region detected in step (a), and displaying a deformation length for a predetermined low resolution reference face image in the low resolution input face image for the intact region Extracting shape information of an input face image and texture information of an input face image mapped to the low resolution reference face image; (c) Receive the shape information and texture information of the low resolution input face image extracted in the step (b), the unique low resolution shape consisting of only the low resolution shape information and texture information corresponding to the input shape information and texture information Deriving a vector and a texture eigenvector based on a plurality of shape eigenvectors and a texture eigenvector; (d) calculating the shape eigenvector coefficients and texture eigenvector coefficients for matching the derived low resolution shape eigenvectors and texture eigenvectors with the inputted shape information and texture information, and then calculating the plurality of shape eigenvectors and texture eigenvectors. Restoring the shape information and the texture information of the high resolution full face image by linearly overlapping the high resolution shape eigenvector and the high resolution texture eigenvector composed of only the high resolution shape information and the texture information derived based on the above; And (e) generating a face image by synthesizing the shape information and texture information of the high resolution full face image restored in step (d).