KR20120113912A - The detection and recovery method of occlusion of a face image using a correlation based method - Google Patents

Abstract

PURPOSE: A method for detecting and restoring a hidden face image by using a correlation is provided to minimize a noise generated around a hidden face area and surrounding areas by detecting a hidden part of a face image based on the correlation between pixels included in the face images. CONSTITUTION: A correlation coefficient of pixel values included in a face image is calculated with a learning face image(S130). A difference is calculated by receiving a test face image partially hidden from the face image and comparing pixel values of the test face image with pixel values predicted through pixel values having a high correlation(S230). A hidden area included in the test face image is detected(S240). The face image is restored by predicting pixel values of the hidden area through pixel values of an unhidden part having the high correlation according to the correlation coefficient and replacing the predicted pixel values to the pixel values of the hidden area(S300). [Reference numerals] (AA) Unhidden part; (BB) Hidden part; (S100) Learning; (S110) Inputting learning images; (S120) Calculating average and standard deviation; (S130) Calculating a correlation coefficient; (S200) Detection; (S210) Inputting test image; (S220) Estimating pixel value; (S230) Difference image; (S300) Restoration; (S310) Re-estimating pixel value; (S330) A reference value is satisfied?; (S340) Image finally restored

Description

{The Detection and Recovery Method of Occlusion of a Face Image using a Correlation Based Method}

The present invention relates to a method for detecting and restoring a masked portion of a face image. In particular, a masked portion of a face image is detected using a correlation coefficient and a detected face region is restored using a correlation coefficient before a face recognition step. The present invention relates to a method for detecting and restoring a masked face image using correlation.

All over the world, surveillance cameras are installed not only in the industrial sector but also in many places such as urban areas, convenience stores and banks. The reason why there are so many surveillance cameras around us is that it is very convenient to monitor a large place or several places at a glance with a small number of surveillance cameras, and to see the situation when a problem occurs through the recorded video. Because.

But surveillance cameras are not always useful in every situation. If criminals are captured by surveillance cameras with their faces covered in sunglasses or masks, it is likely that they will not be able to capture the important features of the criminal's identity even when a problem occurs. This means that it is difficult to determine the identity of criminals by using face recognition, which is recently expanding its scope of application. Therefore, detecting the hidden part of the image and restoring the detected area before the face recognition step is more important in the image processing field in order to more accurately identify the identity.

As a representative method for detecting and restoring an image, especially a masked part of a face image, there is a method of applying a principal component analysis (PCA). A method of removing glasses from a face image or a method of removing noise from a face by using the same has been proposed. In addition, research on hidden parts of images has been conducted in various ways such as Fast Recursive PCA, which reconstructs the hidden parts of images faster by developing methods of applying existing principal component analysis methods, and Tipping and Bishop's research using Probabilistic PCA. Is going on.

1 is a conceptual diagram illustrating a method of detecting and restoring a hidden part of a face image using the conventional principal component analysis method (PCA). The face restoration method using the principal component analysis method includes a learning process (S10), a detection process (S20), and a restoration. It is made through the process (S30).

[Learning Course]

The learning process (S10) consists of first receiving the training image (S11), performing normalization (S12), calculating a covariance matrix (S13), and then obtaining a transformation matrix (W) (S14). In other words, the final calculation in the learning process S10 is a transformation matrix W composed of as many eigenvectors as the number of dimensions to be reduced.

라 표현하고, 학습을 위해 하나의 열벡터로 표현한다. 그 다음 전체 얼굴 영상들의 평균 영상을 구하여 정규화를 수행한 후(S12), 각 얼굴 영상들로부터 평균을 뺀다. 위의 내용을 식으로 표현하면 다음과 같다.When the learning image is input (S11), N learning face images having the number of pixels d are

And a single column vector for learning. Then, the average image of the entire face image is obtained and normalized (S12), and then the average is subtracted from each face image. If the above content is expressed as an expression, it is as follows.

A covariance matrix of images obtained by subtracting the mean from each face image is obtained (S13).

을 선택한다. 선택된 고유 값과 관련된 고유벡터를 구하고 변환 행렬

을 만든다(S14). 고유벡터는 원래의 차원 수인 d차원만큼 얻을 수 있는데, 축소하고자 하는 차원 수인 d' 차원만큼만 취하고 나머지는 버리면 된다.Perform eigenvalue analysis on the covariance matrix to determine the d 'largest eigenvalues

. Find the eigenvectors associated with the selected eigenvalues and transform matrix

(S14). The eigenvectors can be obtained by the number of dimensions, d, which is the original dimension, but take only the d 'dimension, which is the number of dimensions to be reduced, and discard the rest.

The transformation matrix W is obtained through the above learning process.

[Detection and Restoration Process of Hidden Face Image]

가 입력되면(S21) 이를 정규화하여(S22) 재구성한 후(S23), 테스트 얼굴 영상과 재구성된 얼굴 영상의 차를 계산하여(S24), 얼굴 영상의 가려진 부분을 검출하게 된다(S25)(S26). 또한, 복원과정(S30)에서는 상기 검출과정(S20)을 통하여 검출된 가려진 부분을 재구성한 영상을 이용하여 복원하게 되는데(S31), 이러한 복원은 기준치를 만족할 때까지 반복되어(S32), 최종 복원된 영상이 생성되게 된다(S33). 상기 입력되는 테스트 얼굴 영상들은 임의의 너비와 높이를 가진 사각형으로 얼굴의 일부분을 가린 영상을 의미한다.In the detection process (S20), a test face image

Is input (S21), and normalized it (S22), and then reconstructed (S23), and calculating a difference between the test face image and the reconstructed face image (S24), and detecting a hidden portion of the face image (S25) (S26). ). In addition, the restoration process (S30) is to restore using the reconstructed image detected by the masked portion detected through the detection process (S20) (S31), such restoration is repeated until the reference value is satisfied (S32), the final restoration The generated image is generated (S33). The input test face images mean an image in which a part of the face is covered by a rectangle having an arbitrary width and height.

Each test face image is normalized to the average image obtained in the previous learning process (S10) (S22), and then a weight value is obtained for the eigenvector finally derived in the learning step, and each test face image is used. Are reconstructed in the form of a weighted sum of eigenvectors (S23).

Where x is a test face image and x 'is a new face image reconstructed by the transformation matrix. The difference between each pixel of the reconstructed face image x 'and the test face image x is determined to be a masked area depending on whether or not a predetermined boundary value is exceeded (S24). After determining the hidden region in the face image (S26), a reconstruction process is performed by reconstructing the reconstructed face image and the test face image into a new image called "x" (S30). The pixel value of the reconstructed face image is filled, and the pixel value determined to be the unobscured portion is filled with the pixel value of the test face image input as an original input (S31).

는 가려지지 않은 영역을 뜻하고,

는 가려진 영역을 뜻한다. 이렇게 새롭게 구성된 영상

로 여기고 위의 과정을 기존에 정해 놓은 기준치가 수렴할 때까지 계속 반복한다(S32). 모든 반복 과정을 거지고 기준치가 수렴했을 때 생성된 영상을 최종적으로 복원된 영상으로 결정한다(S33).
here

Means an unobstructed area,

Is the hidden area. This newly composed video

It is regarded as and the above process is repeated until the reference value established previously converges (S32). After all repetitive processes and the reference value converges, the generated image is determined as the finally reconstructed image (S33).

Finally, the hidden part of the face image is restored through the above process. The face image restoration method using the conventional PCA uses the all the pixel information of the face image in the face image restoration process, which makes the calculation process complicated. There was a problem of slow computation.

In addition, the face image reconstruction using the principal component analysis (PCA) has a problem in that the masked portion does not detect all the pixels of the masked portion in the detected image or a lot of noise occurs in pixels other than the masked portion. In addition, the face image restoration method using the principal component analysis (PCA) has a problem that the boundary line between the hidden portion and the unobscured portion of the reconstructed image is not smoothly connected and broken, and the reconstructed image is also crushed. There was a problem.

The present invention has been proposed to solve the above-described problems of the prior art, and an object of the present invention is to generate a noise of a face image in which a masked region is detected by using correlation between pixels in a process of detecting a portion of the face image. The present invention provides a method for detecting a masked face image using a correlation that can reduce the error.

In addition, another object of the present invention is to restore the face image by using only a small number of pixel information that has a high correlation with the masked portion in the process of restoring the masked portion of the face image so that the face image restoration speed can be made faster, The present invention provides a method for restoring a masked face image using a correlation that prevents a crushed face image from being formed and smoothly forms a boundary between a masked portion and a portion that is not covered.

In order to achieve the above object, the method for detecting and restoring a masked face image using correlation according to the present invention is a face image detection and restoration method for detecting and restoring a hidden portion of an input face image. A learning step of calculating a correlation coefficient of each pixel value included in the learning face image; Receives a test face image in which part of the face image is masked, and compares each pixel value of the test face image with a pixel value predicted by a plurality of highly correlated pixel values according to the correlation coefficient calculated through the learning step. Detecting a hidden region included in the test face image by calculating a; The pixel value of the masked region detected in the detecting step is predicted as pixel values of the plurality of unobscured parts having a high correlation according to the correlation coefficient calculated in the learning step, and the pixel value of the masked region is estimated as the predicted pixel value. And a restoration step of restoring a face image instead.

The learning step calculates an average and a standard deviation of each pixel value included in the input learning face image, and calculates a correlation coefficient between each pixel using the average and standard deviation. The correlation coefficient is selected so that some large values exceeding the reference value are used in the detection and recovery steps.

The detecting may include reconstructing each pixel value into a predicted image by predicting a plurality of pixel values highly correlated with each pixel of the test face image, and correlating with each pixel value of the calculated test face image. Comparing the pixel values reconstructed with the high pixel values, and selecting pixels in which the difference of each pixel value exceeds the reference value and determining the area as a hidden area.

The plurality of pixel values having a high correlation with each pixel value of the test face image predicted in the reconstructing each pixel value of the test face image has a value that exceeds a reference value among the correlation coefficients calculated in the learning step. It is desirable to predict a pixel value with some correlation coefficient value.

A plurality of pixel values highly correlated with each pixel value of the test face image predicted in the step of reconstructing each pixel value of the test face image is calculated by the following equation.

[Mathematical Expression]

은 n번째 얼굴 영상에서의 i번째 픽셀 값이고,

는 i번 째, j번 째 샘플 평균을 나타낸다.Where N is the number of face images,

Is the i th pixel value in the n th face image,

Denotes the i th and j th sample averages.

On the other hand, it is preferable that the plurality of pixel values having a high correlation with the pixel values of the masked area predicted in the reconstructing step are the pixel values of the test face area not determined as the area masked in the detecting step.

In addition, the plurality of pixel values having a high correlation with the pixel values of the masked region predicted in the reconstructing step may include pixels having some correlation coefficient values having a value exceeding a reference value among correlation coefficients calculated in the learning step. Preferred by value.

The method for detecting and restoring a masked face image using correlation according to the present invention minimizes noise generated in the masked face area and the surrounding area by detecting a hidden part of the face image by using the correlation between each pixel included in the face image. It can be effective. In addition, the method for detecting and restoring a masked face image using correlation according to the present invention has a high restoration speed by restoring a face image by using pixels having a high correlation with the masked face area when a hidden area is detected in the face area. It is possible to minimize crushing in the image and to smoothly form a boundary between the masked and unhidden portions of the restored face image.

1 is a conceptual diagram illustrating a method of detecting and restoring a hidden part of a face image using a conventional principal component analysis method (PCA);
2 is a conceptual diagram illustrating a process of detecting and restoring a hidden part of a face image using correlation according to the present invention;
3 is a diagram illustrating a correlation coefficient map for a specific portion of a face in an actual face image using the correlation coefficient prediction value according to the present invention;
FIG. 4 is a diagram illustrating a set of various pixels having a high correlation with respect to an eye, a nose, a mouth, and a cheek part of the face image of FIG. 2;
5 is a view showing an influence relationship between a face image portion and pixels having a high correlation with the face image portion;
6 illustrates an example in which a process of detecting a hidden portion and restoring the detected portion by receiving a test face image by using a principal component analysis method and a correlation coefficient is shown.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a conceptual diagram illustrating a process of detecting and restoring a hidden part of a face image by using a correlation according to the present invention.

As shown in FIG. 2, the method for detecting and restoring a face using correlation according to the present invention is performed through a learning process S100, a detection process S200, and a restoration process S300.

The learning process S100 receives a face image that is not covered by any part of the face image as a learning face image, and calculates a correlation coefficient of each pixel value included in the learning face image. In the learning process (S100), after calculating the average and standard deviation of each pixel included in the input learning face image (S110) (S120), the correlation coefficient of each pixel is calculated using this (S130).

The detection process (S200) receives a test face image covering a part of the face image, compares each pixel value of the test face image with a plurality of pixel values having a high correlation according to the correlation coefficient calculated through the learning step, and makes a difference. By calculating the step of detecting the hidden area included in the test face image. To this end, the detection process (S200) calculates each pixel value of the input test face image (S210), predicts and estimates a plurality of pixel values having a high correlation with each pixel of the test face image (S220). After reconstructing each pixel value with the extracted image, the difference is calculated by comparing the pixel value of the test face image reconstructed with the pixel value having a high correlation with each pixel value of the test face image (S230), and the difference between each pixel value. Selects a pixel that exceeds the reference value and determines the area to be a masked area (S240) (S250).

In addition, the reconstructing process (S300) predicts and estimates the pixel value of the hidden region detected in the detecting process (S200) as a plurality of pixel values having a high correlation according to the correlation coefficient calculated in the learning process (S100) ( S310), the face image is restored by replacing the pixel value of the region covered by the estimated pixel value (S320). The face image restoration is repeatedly performed until the restored image satisfies the preset reference value as the final restored image. In operation S330, a final reconstructed image is generated in operation S340.

Hereinafter, each process of the face image detection and reconstruction method using the correlation performed through the above process will be described in detail.

First, prior to the description of the face detection and restoration method using the correlation according to the present invention, the correlation coefficient used in the present invention will be described.

The correlation coefficient is a value representing the correlation between pixels of the image. Assuming that each pixel of the image is a single random variable, the correlation coefficient is an index indicating how correlation exists between the two random variables. The correlation coefficient is normalized by dividing the covariance by the standard deviation and has a value from -1 to 1. The correlation coefficient means that the closer to 1 or -1 value, the greater the relationship between the two random variables, and the closer to 0, the less relevant.

The correlation coefficient between two random variables is calculated through Equation 9 below.

는 i번째 확률변수이고,

는 j번 째 확률변수를 의미한다. 또한,

는 두 확률변수의 공분산을 나타내고,

는 각각의 확률변수의 표준편차를 나타내며,

를 의미한다.From here,

Is the i th random variable,

Is the j th random variable. Also,

Represents the covariance of two random variables,

Represents the standard deviation of each random variable,

Means.

In the input image, each pixel can be thought of as one random variable, so one image can be thought of as a realization of one d-dimensional probability vector, and the correlation between each pixel is determined through the correlation coefficient defined in Equation 9 above. Can be inferred. In the case of a masked image, it is a basic concept of the present invention that a pixel value of a hidden portion can be predicted from pixel values of an unhidden portion having a high association with the corresponding pixel.

Assuming that N learning face images are given, the expected value E (.) Of the two random variables in Equation 9 cannot be known in the real situation, so the predicted value of the correlation coefficient using N samples is expressed as in Equation 10 below. You can get it.

은 n번째 학습 영상에서의 i번째 픽셀 값이고,

는 i번 째, j번 째 샘플 평균을 나타낸다.From here

Is the i th pixel value in the n th training image,

Denotes the i th and j th sample averages.

FIG. 3 shows the correlation coefficient maps for four specific parts of the face, ie eyes, nose, mouth, and cheek, in the real face image using the correlation coefficient prediction value.

In the correlation map of FIG. 3, the lighter (white) portion is the portion where the absolute value of the correlation coefficient is higher, and the darker (black) portion is the portion where the absolute value of the correlation coefficient is low. Here, looking at the correlation coefficient map for the eye, it can be seen that pixels that are highly related to one pixel of the left eye, that is, the area around the left eye and the right eye are brightly displayed. Using the correlation coefficient like this, it can be expected that better information can be obtained by using not only the information of the near pixels but also the information of the pixels far from the pixel.

의 픽셀 값을 모른다고 가정하고,

의 픽셀 값은 알고 있다고 가정하면,

의 픽셀 값은 해당 픽셀과 연관도가 높은 여러 픽셀들의 집합

을 이용해서 구할 수 있는데, 이때 이용하는 것이 jointly Gaussian 방법이다. If, of face image

Suppose you don't know the pixel value of,

Suppose you know the pixel value of

A pixel value of is a set of pixels that are highly related to that pixel.

This can be obtained using the jointly Gaussian method.

로 표시된 픽셀은

픽셀을 나타낸 것이다.FIG. 4 illustrates a set of several highly correlated pixels of the eye, nose, mouth, and cheek parts of the face image of FIG. 3.

Pixels marked with

It represents a pixel.

일 때,

번째 픽셀 값이 v라고 가정하고

의 확률을 식으로 표현하면 다음의 수학식 11과 같이 나타낼 수 있다.if,

when,

Suppose the first pixel value is v

When the probability of is expressed by the equation, it can be expressed as Equation 11 below.

는 상수이기 때문에

는

에 비례한다고 볼 수 있다. 그러므로

는

를 생략하고

에 대해 나타낼 수 있는데,

를 정리하면 다음의 수학식 12와 같이 나타낼 수 있다. From here,

Is a constant

The

It can be regarded as proportional to. therefore

The

Omit

Can represent

In summary, it can be expressed as Equation 12 below.

은 다음의 수학식 13과 같다.From here

Equation 13 is as follows.

Through Equation 12, the expected value and the variance value may be arranged as shown in Equations 14 and 15 below.

이 경우에

번째 픽셀 값이 v,

번째 픽셀 값이 w 라고 가정할 때,

값을 구하기 어려우므로 가중합계(weighted sum)에 의해 구하게 된다. 이때, 가중치(weight)는 신뢰도(confidence)로 1/variance, 즉

이다. 만일 신뢰도(confidence)가 높으면 가중치(weight) 값을 크게 주고, 신뢰도(confidence)가 낮으면 가중치(weight) 값을 작게 준다.
if,

In this case

The first pixel value is v,

Assuming that the th pixel value is w,

Since the value is difficult to obtain, it is obtained by weighted sum. In this case, the weight is 1 / variance, that is, confidence

to be. If the confidence is high, the weight value is increased. If the confidence is low, the weight value is made small.

Hereinafter, a process of detecting and restoring the image hidden in the face image using the above-described correlation will be described.

As described in FIG. 2, the face detection and restoration method using correlation according to the present invention is performed through a learning process, a detection process, and a restoration process.

The learning process receives a learning image and calculates the correlation of each pixel, and the detection process receives a test image and calculates a difference of each pixel value by using the correlation of each pixel calculated through the learning process. The hidden part of the test image is detected, and the reconstruction process reconstructs the hidden part detected in the detection process by using the correlation of each pixel.

The learning process receives an unoccluded face image and obtains a correlation coefficient for every pixel of all face images. That is, as shown in Equation 10, first, the average image and the standard deviation image of all the face images are obtained, the difference images of the average image are obtained from the face images, and then the correlations are obtained for each pixel of the obtained difference images. Coefficients are obtained, and only a part of large values exceeding a reference value among the correlation coefficient values are stored and used for later detection and restoration. In the embodiment of the present invention, if there are 10,000 pixels in the face image, only a correlation coefficient of about 1,000 having a high double correlation coefficient is selected and used, so that a detection and reconstruction process using this correlation coefficient can be quickly performed. In addition, the average image and the standard deviation image, which were previously calculated to obtain the correlation coefficient value, are also stored for later use.

When the learning process is performed through the above process, the test face image is input to perform a detection process of detecting a hidden portion of the test image. The test face image is an image in which a part of the face is covered with a rectangle having an arbitrary width and height at an arbitrary position, as in PCA described in the background art.

The input test face images are normalized to the average image obtained through the learning process, and then the pixel values of each normalized test face image are estimated using correlation. If it is assumed that the first pixel value of each of the normalized test face images is not known, the pixel value of the first pixel may be predicted by using pixel values of various pixels that are highly related to the pixel. In this case, the corresponding pixel value is obtained by the weighted sum of several pixel values whose correlation coefficient value exceeds a certain value. In order to more easily understand the content of the weighted value and the predicted pixel value, Expressed in the following equations (16) to (18).

는 i번 째 픽셀(모른다고 가정한 첫 번째 픽셀)과 j번 째 픽셀 사이의 상관관계에 대한 가중치(weight) 값, 즉 앞에서 언급한 신뢰도(confidence)를 의미한다. 또한,

는 i번 째 픽셀과 연관도가 높은 픽셀들로, 그 픽셀들의 값은 우리가 알고 있다고 여긴다. 이를 간단히 그림으로 나타내면 도 5과 같은데, 도 5의 왼쪽은 얼굴 영상 부분을 간단하게 나타낸 도면이고, 오른쪽 그림은 상관관계가 높은 픽셀들이

에 영향을 주는 것을 나타낸 도면이다.From here

Denotes a weight value for the correlation between the i th pixel (the first pixel assuming unknown) and the j th pixel, that is, the aforementioned confidence. Also,

Is a pixel that is highly related to the i th pixel, and the values of those pixels are assumed to be known to us. This is shown in Fig. 5 in brief. The left side of Fig. 5 is a simplified view of the face image portion, and the right side shows pixels having high correlation.

It is a diagram showing that affects.

로 나타낸 것은 픽셀 값을 알지 못해 값을 추정해야 하는 i번 째 픽셀을 나타낸 것이고,

는 i번 째 픽셀과 연관도가 높은 픽셀들을 그림으로 도시화한 것이다. 결론적으로 의 픽셀 값은 연관도가 높은 각각의

의 픽셀 값들의 가중합계(weighted sum)에 의해 결정된다.
Looking at Figure 5,

Is the i-th pixel for which to estimate the value without knowing the pixel value.

Is a pictorial representation of pixels highly related to the i th pixel. In conclusion The pixel value of is highly relevant for each

Is determined by the weighted sum of the pixel values.

In the above, it is assumed that the first pixel value is not known. Now assume that each pixel value of each of the normalized test face images is not known, each pixel value of the test face images may be regarded as obtained by a weighted sum of highly correlated pixel values.

와 상관계수를 이용하여 예측된 픽셀 값

와의 차이값이 일정 경계값보다 크면 그 부분은 임의의 너비와 높이를 가지는 사각형에 의해 가려진 부분이라 판단되게 되는데, 이를 식으로 표현하면 다음의 수학식 19와 같다. At this time, the actual pixel value of each of the test face image data

Pixel values predicted using and correlation coefficient

If the difference between and is greater than a certain boundary value, the part is determined to be a part covered by a rectangle having an arbitrary width and height, which is expressed by Equation 19 below.

는 가려지지 않은 영역을 의미하고,

는 가려진 영역을 의미한다.From here

Means an unobstructed area,

Means the hidden area.

By using Equation 19, it is determined whether a specific part of each test face image is a hidden part, thereby detecting a hidden part of the face image.

When the hidden area of the test face image is detected through the above process, the detected hidden area is reconstructed using the correlation between pixels.

Unlike in the detection process, it is possible to know in advance which part of the face image is hidden so that the pixel value is unknown. Therefore, in order to fill the pixel values of this portion one by one, only the remaining portions except for each pixel, that is, the portion assuming that the pixel value is known, are used to find the pixels highly related to the hidden pixels. An unknown pixel value is obtained by using Equation 16, Equation 17, and Equation 18, and the predicted pixel values are replaced with the existing pixel values only when the condition of Equation 20 is satisfied in Equation 19 below. do.

로 대체될 때까지 상기의 과정을 반복하여 마지막으로 얻어진 영상을 의미한다.
If the part assuming that the pixel value is known is the area around the pixel for which the value is to be predicted and the part is also a hidden area, the hidden pixel does not satisfy the condition of Equation 19 and is determined as the hidden area. On the contrary, if the part assuming that the pixel value is known belongs to the unobscured region, the predicted pixel value is filled with the predicted value because the predicted pixel value satisfies the condition of Equation 19. Therefore, if the pixels assumed to be known belong to the hidden region, they are first filled with predicted pixel values from the pixels belonging to the latter case, and then inferred by the predicted values. Eventually, the finally reconstructed face image is obtained by each pixel value of the hidden region of the image.

It means the last image obtained by repeating the above process until replaced by.

Through the above process, the hidden part of the test face image is detected, and the pixel of the detected hidden part is filled through other highly related pixels according to the correlation to restore the face image.

6 illustrates an example in which a process of detecting a hidden portion and restoring the detected portion by receiving a test face image by using a principal component analysis method and a correlation coefficient is shown.

The face images used in this experiment were 1521 gray images with BioID data, and each image consisted of 23 frontal faces. Each image has a size of 100 × 100 pixels, and the human face of each image is fixed at the center of both eyes. This data was taken in real-world conditions and was obtained in a lighting environment with very large changes in each image.

In the experiment using the conventional principal component analysis method, randomly selected 1000 pieces of 1521 pieces were used as learning images, and the remaining 521 pieces were used as test images. In the experiment using the correlation of the present invention, the correlation coefficients were obtained for all 1521 chapters, and then the same test images used in the principal component analysis were used. Since the bioID data does not cover a part of the face of the image, for experiments, as shown in the test image of FIG. 6, a rectangle having an arbitrary width and height is arranged at an arbitrary position of the face.

인 변환 행렬을 구했다. 테스트 영상으로 들어온 얼굴 부분이 가려진 데이터를 위의 배경기술에서 설명한 수학식 6을 이용하여 재구성한다. 그 다음, 재구성된 얼굴 영상과 테스트 영상인 가려진 얼굴 영상과의 차이를 구한다. 차 영상에서 각각의 픽셀 값이 일정 경계값을 넘으면 영상에서 가려진 부분이라고 간주하고, 일정 경계값을 넘지 않으면 가려지지 않은 부분이라 여긴다. 이를 그림으로 나타내기 위해서 경계값을 넘는 부분의 값은 1로 정하고, 넘지 않는 부분은 0으로 정하여 이를 도시화하였다. 도 6의 주성분 분석법 검출 영상에서와 같이, 가려진 부분을 어느 정도 찾지만 가려지지 않은 영역을 가려진 영역이라 판단하거나 가려진 영역에서도 0의 값을 가지는 픽셀들이 존재하는 등 여러 잡음이 발생하는 것을 볼 수 있다. 이제 앞의 수학식 8에서 언급했듯이, 새로운

에 영상의 가려진 부분이라 간주된 각 픽셀에는 위에서 재구성된 영상의 각 픽셀 값으로 채우고, 나머지 가려지지 않은 부분에는 테스트 얼굴 영상의 가려지지 않은 부분으로 채운다. 새롭게 생성된 영상

를 테스트 영상과 같이 취급하여 위의 과정을 실험에서 정한 기준치가 수렴할 때까지 반복한다. 이렇게 해서 최종적으로 복원된 영상은 도 6의 주성분 분석법 복원 영상에서 확인할 수 있다. In the experiment using the conventional principal analysis method, 1000 learning face image data, which is 10000 dimensions in the learning stage, is reduced to 50 dimensions using the principal component analysis method.

The phosphorus transformation matrix was obtained. The data masked from the face part of the test image is reconstructed using Equation 6 described in the background art above. Then, the difference between the reconstructed face image and the masked face image, which is a test image, is obtained. If each pixel value in the difference image exceeds a certain boundary value, it is regarded as a part that is hidden in the image. In order to show this as a figure, the value of the part exceeding the boundary value is set to 1, and the part not exceeding is set to 0 and illustrated. As shown in the principal component analysis detection image of FIG. 6, it can be seen that various noises are generated, such as searching for a part of the hidden part but determining the unobscured area as the hidden area, or having pixels having a value of 0 even in the hidden area. . Now, as mentioned in Equation 8,

Each pixel considered to be a hidden part of the image is filled with the value of each pixel of the reconstructed image above, and the remaining unhidden part is filled with an unhidden portion of the test face image. Newly created video

Repeat the above procedure until the reference values established in the experiment converge, treating them as test images. The finally reconstructed image can be confirmed in the reconstructed image of the principal component analysis method of FIG. 6.

On the other hand, in the experiment using the correlation according to the present invention, 1521 bioID data are obtained using the equation (10) to obtain a correlation coefficient for each pixel. Among them, only 1000 large correlation coefficients are stored separately. First, the hidden part of the image is found by using the correlation coefficient values stored separately. In this experiment, 20 correlation coefficients were extracted from the larger of 1000 and jointly Gaussian was used only when the correlation coefficient was above 0.9. If 20 correlation coefficient values are not more than 0.9 without knowing a specific pixel value, an unknown pixel value is estimated by weighting an image pixel value corresponding to each of the 20 correlation coefficient values. In this case, when the difference between the estimated pixel value and the pixel value of the test image is larger than a predetermined boundary value, it is regarded as a hidden area, and when smaller than the boundary value, it is regarded as an unobstructed area. Here, as in the experiment of the principal component analysis method, the hidden region is plotted as a value of 1 and the uncovered region is set to a value of 0, which can be confirmed in the correlation coefficient detection image of FIG. 6. Referring to the correlation coefficient detection image of FIG. 6, although the noise that pops up more than the actual hidden region occurs, the noise generated in the region farther from the hidden region than the method using the principal component analysis method does not exist and is within the hidden region. You can see that noise does not occur. In addition, in the reconstructed image, the visual confirmation of the correlation coefficient reconstructed image of FIG. 6 also leads to a cleaner result than the reconstructed image of the principal component analysis method.

As shown in FIG. 6, even when the test face image data are directly compared with the eye, blurring of the image reconstructed using the principal component analysis method is more severe than the images reconstructed using the correlation. You can see that the boundary with the unobscured area is not smoothly connected (discontinuity). In addition, while the method using the principal component analysis method uses all the pixels, the method using the correlation coefficient proposed in the present invention has the advantage of using only a few pixels.

The test face images of the BioID data were reconstructed with the principal component analysis method, and the reconstructed portions of the image were reconstructed using the correlation. The images reconstructed by the two methods can be compared with the reconstruction performance by calculating an error with the image before the image, that is, the ground truth image.

라고 하고, 상관관계를 이용하여 복원된 영상을

라고 하자. 또한, 테스트 얼굴 영상 데이터가 임의의 너비와 높이를 가진 사각형으로 가려지기 전의 실제 기준(ground truth) 영상 데이터를

라고 하자. 복원된 영상

또는

가 얼마나 실제 영상

와 비슷하게 복원되었는지를 알기 위해 L1-norm 오차와 L2-norm 오차를 구해 비교해 본다. 오차를 구하기 위한 식들은 아래의 수학식 21과 수학식 22와 같다. Finally reconstructed image using principal component analysis

And reconstructed image using correlation

Let's say In addition, the ground truth image data before the test face image data is obscured by a rectangle of arbitrary width and height can be obtained.

Let's say Reconstructed image

or

How much real footage

To find out whether the reconstruction is similar to, calculate L1-norm error and L2-norm error. Equations for calculating the error are as shown in Equations 21 and 22 below.

Table 1 below is a chart comparing L1-norm error and L2-norm error of the reconstructed image using the principal component analysis method and the correlation calculated through Equation 21 and Equation 22.

As shown in Table 1, L1-norm average error is 17052.45 principal component analysis method, 9997.02 method using the correlation, the method using the correlation is 7070.43 error smaller than the principal component analysis method. The L2-norm mean error is 788.06 in principal component analysis method and 499.21 in correlation method, and the correlation method is smaller by 288.85 error than principal component method. As a result, even when the reconstructed picture was compared with the naked eye, the method using the correlation was found to have a small error. Even when comparing the calculated L1-norm error with the L2-norm error, the method using the correlation was small. You can check once again.

As described above, the method of detecting and restoring the hidden part of the image using the correlation according to the present invention has less blurring and less information than the conventional method using the principal component analysis method. ground truth) It can be seen that a good result can be obtained with a small error from the image.

The present invention is not limited to the above-described embodiments, and various modifications within the equivalent scope of the technical idea of the present invention and the claims to be described below by those skilled in the art to which the present invention pertains. Of course, modifications can be made.

Claims (8)

In the face image detection and restoration method for detecting and restoring the hidden portion of the input face image,
(a) a learning step of receiving a learning face image and calculating a correlation coefficient of each pixel value included in the learning face image;
(b) receiving a test face image in which a part of the face image is hidden, and predicting pixel values predicted by a plurality of pixel values having a high correlation according to each pixel value of the test face image and a correlation coefficient calculated through the learning step. A detecting step of detecting a hidden area included in the test face image by comparing and calculating a difference;
(c) predicting pixel values of the masked region detected in the detecting step with pixel values of a plurality of unobscured parts having a high correlation according to the correlation coefficient calculated in the learning step, and And restoring a face image by substituting pixel values. The method of claim 1,
In the learning step (a), the average and standard deviation of each pixel value included in the input learning face image are calculated, and the correlation coefficient between each pixel is calculated using the average and standard deviation. A method of detecting and restoring a masked face image. The method of claim 2,
The correlation coefficient between each pixel calculated in the learning step (a) is selected by some large values exceeding the reference value, and used in the detection step and the reconstruction step. . The method of claim 1,
The detecting step (b)
(b-1) reconstructing each pixel value into a predicted image by predicting a plurality of pixel values highly correlated with each pixel of the test face image;
(b-2) comparing the pixel values reconstructed with the pixel values having a high correlation with each pixel value of the calculated test face image, and selecting a pixel whose difference between each pixel value exceeds a reference value and determining the area as a masked area; A method for detecting and restoring a masked face image using correlations comprising the steps of: a. The method of claim 4, wherein
A plurality of pixel values having a high correlation with each pixel value of the test face image predicted in the step (b-1) of reconstructing each pixel value of the test face image is based on a reference value among the correlation coefficients calculated in the learning step. A method for detecting and restoring a masked face image using correlations, wherein the exceeding value is predicted as a pixel value having a part of a large correlation coefficient value. The method of claim 4, wherein
A plurality of pixel values having a high correlation with each pixel value of the test face image predicted in step (b-1) of reconstructing each pixel value of the test face image may be calculated by the following equation. A method for detecting and restoring hidden face images using relations.
[Mathematical Expression]

Where N is the number of face images,

Is the i th pixel value in the n th face image,

Denotes the i th and j th sample averages. The method of claim 1,
The plurality of pixel values having a high correlation with the pixel values of the masked area predicted in the restoring step (c) are the pixel values of the test face area that are not determined to be the masked area in the detecting step. Facial image detection and restoration method. The method of claim 1,
The plurality of pixel values having a high correlation with the pixel values of the masked region predicted in the reconstruction step (c) may include pixels having some correlation coefficient values larger than a reference value among the correlation coefficients calculated in the learning step. A method for detecting and restoring a masked face image using a correlation characterized in that it is predicted by a value.

Images