KR102463353B1 - Apparatus and method for detecting fake faces - Google Patents

Abstract

The present invention provides an apparatus and method for detecting and determining whether a manipulated face included in an image or video is forged or tampered with. The detection device may include: an acquisition unit configured to acquire an image; a separation unit separating a first area representing a face and a second area not representing a face in the image; an extractor configured to extract a first feature point of the first region and extract a second feature point of the second region; and a determining unit configured to determine whether the face is forged by comparing the first feature point and the second feature point.

Description

Fake face detection device and method

The present invention relates to an apparatus and method for detecting and determining whether a face included in an image or video is forged/falsified/fake.

Invasion of personal privacy using manipulated/forged faces in data such as digital images or videos is continuously increasing.

It is necessary to establish a system to block personal privacy violations through policies such as the Personal Information Protection Act.

In order to build the system, a means for determining whether a manipulated or forged face is used in multimedia data such as digital images or moving pictures should be provided.

Korean Patent Application Laid-Open No. 10-2017-0006355 discloses a technique for classifying and judging fake faces based on motion vectors and feature vectors.

Korean Patent Publication No. 10-2017-0006355

An object of the present invention is to provide a detection device and a detection method for detecting a manipulated or forged face in multimedia data such as various images and moving pictures including real-time.

The detection apparatus of the present invention includes: an acquisition unit for acquiring an image; a separation unit separating a first area representing a face and a second area not representing a face in the image; an extractor configured to extract a first feature point of the first region and extract a second feature point of the second region; and a determining unit configured to determine whether the face is forged by comparing the first feature point and the second feature point.

The detection method of the present invention comprises the steps of acquiring an image; separating a first area representing a face and a second area not representing the face in the image; extracting a first feature point of the first region and extracting a second feature point of the second region; and determining whether the face is forged by comparing the first feature point and the second feature point.

Damage due to fake data using forged or forged faces is increasing. Just as portal companies and the government are preparing countermeasures, countermeasures against fake content are necessary.

The detection apparatus and detection method of the present invention may separate a face region and a non-face region in an image, and determine whether a face is forged by comparing feature points extracted from each region.

1 is a schematic diagram showing a detection device of the present invention.
2 is a schematic diagram showing the operation of the detection device of the present invention.
3 is a flowchart illustrating a detection method of the present invention.
4 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In this specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when a component is referred to as 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components in the middle It should be understood that there may be On the other hand, in the present specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Also, in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It should be understood that the existence or addition of other features, numbers, steps, acts, components, parts, or combinations thereof, is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of a plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a schematic diagram showing a detection device of the present invention. 2 is a schematic diagram showing the operation of the detection device of the present invention.

The detection apparatus shown in FIG. 1 may include an acquiring unit 110 , a separating unit 130 , an extracting unit 150 , and a determining unit 170 .

The acquisition unit 110 may acquire image i. The image i is digital data and may be displayed as a visual still image through a display means such as a display or displayed as a moving picture.

The separator 130 may separate a first region ① representing a face in the image i and a second region ② that is not a face. The detection apparatus of the present invention may read, determine, and detect whether a face is authentic or not in an image including a face. In this case, the separator 130 may separate the face region included in the image by using various image processing techniques or machine learning techniques. The face region separated by the separator 130 may correspond to the first region. The remaining area after separating the first area from the first image may correspond to the second area.

When a video is input, the acquisition unit 110 may extract a frame from the video. For example, if the frames per second of a specific video are 30 frames, the acquisition unit 110 may extract 30 still images per second from the video. The separating unit 130 may separate the first region ① and the second region ② from the extracted frame (still image).

The extraction unit 150 may extract a feature point of the first area from the first area ①. In this case, the feature point of the first region extracted by the extraction unit 150 may correspond to the first feature point d1.

The extraction unit 150 may extract a feature point of the second region from the second region ②. In this case, the feature point of the second region extracted by the extraction unit 150 may correspond to the second feature point d2.

The extractor 150 may extract the first feature point d1 and the second feature point d2 using machine learning or deep learning.

The determining unit 170 may determine whether the face is forged by comparing the first feature point d1 and the second feature point d2.

The determination unit 170 may be provided with a first analysis unit 171 that analyzes whether the first region ① is forged or not, and a second analysis unit 172 that analyzes whether the second region ② is forged.

The determination unit 170 determines that the integrity of any one of the comparison result of the first characteristic point d1 and the second characteristic point d2, the analysis result of the first analysis unit 171, and the analysis result of the second analysis unit 172 is defective. The image can then be identified as a forgery.

The first analyzer 171 may acquire a color or black-and-white face image included in the first area ①.

The first analyzer 171 may Fourier transform each pixel of the face image with respect to a brightness value to convert it into a frequency domain.

The first analyzer 171 may represent the absolute value of the Fourier-transformed result value on a log-scale, and then convert the position so that the low-frequency component is centered.

The first analyzer 171 may divide the position-transformed result value into a plurality of concentric circles, calculate an average energy value of each concentric circle, and generate a first one-dimensional feature vector.

The first analyzer 171 may calculate a local binary pattern code from the user's face image.

The first analyzer 171 may histogram the calculated local binary pattern code.

The first analyzer 171 may generate the second one-dimensional feature vector by normalizing the histogram to be equal to one.

The first analysis unit 171 compares the first one-dimensional feature vector with the first reference forged facial feature vector generated from the forged face image using the support vector machine, thereby first determining whether the face image is a forged face. can be judged.

The first analysis unit 171 compares the second one-dimensional feature vector with a second reference forged facial feature vector generated from the forged face image using the support vector machine to determine whether the face image is a forged face or not. can be judged.

The first analysis unit 171 may finally determine whether the face image is a forged face by combining the result of the first determination and the result of the second determination.

The first analysis unit 171 prepares one or more fake face images, generates each first one-dimensional feature vector from the fake face images, and uses the first one-dimensional feature vectors as a first reference in the support vector machine. The first reference fake facial feature vector may be generated through the step of learning as the fake facial feature vector.

The detection apparatus of the present invention may determine whether a forged face is made by using one or more face images acquired from a general color camera or a black and white camera.

The content is mainly composed of three methods.

First, a fake face is a method of detecting a fake face as a difference in the entire frequency band component of the acquired face image using the characteristic that the detail and three-dimensional effect are lowered compared to the real face.

Second, since the fake face shows different characteristics from the real face in terms of material, the difference in the local binary pattern, which is easy to analyze the image texture How to detect fake faces using

Third, a method for detecting fake faces by combining the aforementioned feature extraction method using frequency components and local binary patterns in a feature level, score level, or decision level method.

It will be described in more detail.

The first is a method using the power spectrum of the entire frequency band. To this end, first, the input face image is Fourier-transformed and changed to the frequency domain.

As an example of the Fourier transform, the following Equation 1 may be used.

Here, f(x,y) is the brightness value at (x,y) coordinates of an image of size H×W, and u,v are frequency variables.

F is the result of the Fourier transform of f, and the component with frequency 0 is shifted to come to the center.

After that, the low-frequency value is transformed to be located at the center of the Fourier-transformed result, and a predetermined number of concentric circles are drawn based on the center of the Fourier-transformed result, and the average energy value between each concentric circle is calculated. Finally, the calculated average energy value is expressed as a one-dimensional vector and subjected to a min-max normalization process. Through these processes, a feature vector is extracted.

Such a power spectrum-based feature vector may be extracted from the entire face image, but may also be arbitrarily segmented into a face image, and feature vectors may be extracted from each segmented face image fragment and merged into one. By extracting these feature vectors from real face images and fake face images and learning them with a classifier such as a support vector machine, it is possible to generate a fake face detector.

The second is a method using the local binary pattern. To this end, a feature vector can be obtained by first calculating the codes of the local binary pattern from the input face image, then making a histogram of these codes and normalizing the histograms so that the sum of the histograms becomes 1. Such a local binary pattern-based feature vector may be extracted from the entire face image, or may be extracted from arbitrarily divided face image fragments and combined into one. By extracting these feature vectors from real face images and fake face images and learning them with a classifier such as a support vector machine, it is possible to generate a fake face detector.

The third is a method of combining the power spectrum-based method and the local binary pattern-based method described above. The above two methods can be combined into three approaches as follows. First, the feature vector extracted based on the power spectrum and the feature vector extracted based on the local binary repeater are combined at the feature vector level and learned using a classifier such as a support vector machine, so that it can be used for fake face detection. Second, the score obtained by passing the feature vector extracted based on the power spectrum through a classifier such as a support vector machine, and the score obtained by passing the feature vector extracted based on the local binary pattern through a classifier such as a support vector machine, etc. It can be used for fake face detection by combining it in , and learning it through a classifier such as a support vector machine. Third, the decision result obtained by passing the feature vector extracted based on the power spectrum through a classifier such as a support vector machine, and the decision result obtained by passing the feature vector extracted based on the local binary pattern through a classifier such as a support vector machine. It can be combined at the judgment result level and used for fake face detection.

In summary, a face image taken by a color or black-and-white camera is converted into the frequency domain and the position is transformed so that the low-frequency component is at the center. A fake face detector is created by creating it and learning it using a support vector machine or the like. Such a power spectrum-based feature vector may be extracted from the entire face image, or may be extracted from arbitrarily divided pieces of the face image and combined into one.

In addition, a one-dimensional feature vector is generated by calculating a local binary pattern code from a face image captured by a color or black-and-white camera and then histogramd, and a fake face detector is created by learning it using a support vector machine or the like. Such a local binary pattern-based feature vector may be extracted from the entire face image, or may be extracted from arbitrarily divided pieces of the face image and combined into one. Finally, the power spectrum-based fake mask detection method and the local binary pattern-based fake mask detection method can be combined at three levels to detect fake faces. First, a fake face detector can be made by combining the feature vector extracted based on the power spectrum and the feature vector extracted based on the local binary repeater at the feature vector level and learning using a classifier such as a support vector machine. Second, the score obtained by passing the feature vector extracted based on the power spectrum through a classifier such as a support vector machine and the score obtained by passing the feature vector extracted based on the local binary pattern through a classifier such as a support vector machine are combined at the score level. Can be used. Third, the judgment result obtained by passing the feature vector extracted based on the power spectrum through a classifier such as a support vector machine, and the judgment result obtained by passing the feature vector extracted based on the local binary pattern through a classifier such as a support vector machine, etc. It can be used to determine whether or not a fake face is combined in

The determining unit 170 may generate a polygonal or closed-curve window disposed on the boundary line between the first area ① and the second area ②.

The window may include both a portion of an edge of the first area ① and a portion of an edge of the second area ②.

The determining unit 170 may compare the first feature point existing on one side of the window with the second feature point existing on the other side of the window based on the boundary line.

The determining unit 170 moves the window along the boundary line, and whenever the window moves, the first feature point and the second feature point existing in the window may be compared.

The determination unit 170 may determine the integrity between the first area ① and the second area ② using the comparison result.

In order to further improve the accuracy of forgery or not, the second area may be divided into a plurality of areas.

For example, the separation unit 130 may divide the second region ② into a body region t1 excluding the face from the human body and a remaining peripheral region t2.

The extractor 150 may extract body feature points from the body region t1. The extractor 150 may extract peripheral feature points from the peripheral area t2.

The determining unit 170 may compare the first feature point with the body feature point. The determining unit 170 may compare the first feature point with the surrounding feature point. The determining unit 170 may compare the body feature point with the surrounding feature point.

If a defect occurs in any one of the comparison result between the first feature point and the body feature point, the comparison result between the first feature point and the surrounding feature point, and the comparison result between the body feature point and the peripheral feature point, the face included in the image i is can be identified as counterfeit.

According to the present embodiment, the synthesis of a forged body against the background, the synthesis of a forged face against the body, and the synthesis of a forged face against the background can be comprehensively determined. In addition, various types of forgery of the image i can be accurately determined.

3 is a flowchart illustrating a detection method of the present invention.

The detection method of FIG. 3 may be performed by the detection apparatus illustrated in FIG. 1 .

The detection method of the present invention may first acquire an image i (S510). As an operation performed by the acquiring unit 110 , the acquiring unit 110 may include a camera that generates image i, or may include a communication unit that communicates with an external camera in wired or wireless communication.

Next, in the image i, the first region ① representing the face and the second region ② not representing the face may be separated ( S520 ). As an operation performed by the separating unit 130 , the separating unit 130 may perform chamfering along the so-called face outline through deep learning, machine learning, image analysis, and the like.

A first feature point d1 of the first region ① may be extracted, and a second feature point d2 of the second region ② may be extracted ( S530 ). This is an operation performed by the extraction unit 150 . When the second region ② is subdivided into a body region t1 and a peripheral region t2, the extractor 150 may extract a body feature point d21 for the body region t1 and a peripheral feature point d22 for the peripheral region t2.

By comparing the first feature point d1 and the second feature point d2, it may be determined whether the face is forged (S540). This is an operation performed by the determining unit 170 .

4 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 4 may be a device (eg, a detection device, etc.) described herein.

In the embodiment of FIG. 4 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . Also, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, and methods described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100 .

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110 . Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the embodiment of the present invention is not implemented only through the apparatus and/or method described so far, and a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded may be implemented. And, such an implementation can be easily implemented by those skilled in the art from the description of the above-described embodiments.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also present. It belongs to the scope of the invention.

110...acquisition unit 130...separation unit
150...extraction unit 170...discrimination unit
171...First analysis unit 172...Second analysis unit

Claims (9)

an acquisition unit for acquiring an image;
a separating unit separating a first area representing a face and a second area not representing a face in the image;
an extractor configured to extract a first feature point of the first region and extract a second feature point of the second region;
a determining unit that determines whether the face is forged by comparing the first feature point and the second feature point;
The determining unit generates a polygonal or closed curve-shaped window disposed on a boundary line between the first area and the second area,
The window includes both a portion of an edge of the first area and a portion of an edge of the second area,
The determining unit compares a first feature point existing on one side of the window with a second feature point existing on the other side of the window based on the boundary line,
The determining unit moves the window along the boundary line, and each time the window moves, the first feature point and the second feature point existing in the window are compared,
The determination unit determines the integrity between the first area and the second area using the comparison result,
If the determination unit determines that the integrity is defective, the detection device determines that the image is a forgery.
According to claim 1,
When the video is input, the acquisition unit extracts a frame from the video,
The separator separates the first region and the second region from the frame.
According to claim 1,
The extraction unit is a detection device for extracting the first feature point and the second feature point using machine learning or deep learning.
delete According to claim 1,
A first analysis unit for analyzing whether the first area is forged or not, and a second analysis unit for analyzing whether the second area is forgery are provided;
The determination unit determines that the image is a forgery if it is determined that the integrity of any one of the comparison result of the first characteristic point and the second characteristic point, the analysis result of the first analysis unit, and the analysis result of the second analysis unit is defective,
The first analysis unit,
Obtaining a color or black-and-white face image included in the first area,
Fourier transforms each pixel of the face image with respect to the brightness value and transforms it into the frequency domain,
After representing the absolute value of the Fourier-transformed result value in log-scale, the position is transformed so that the low-frequency component is at the center,
The position-transformed result value is divided into a plurality of concentric circles, and an average energy value of each concentric circle is calculated to generate a first one-dimensional feature vector,
Calculate a local binary pattern code from the face image,
Histogram the calculated local binary pattern code,
A second one-dimensional feature vector is generated by normalizing the sum of the histogram to be 1,
First determining whether the face image is a forged face by comparing the calculated first one-dimensional feature vector with a first reference fake facial feature vector generated from a forged face image using a support vector machine,
By comparing the calculated second one-dimensional feature vector with a second reference fake facial feature vector generated from a forged face image using a support vector machine, secondary determination is made as to whether the face image is a forged face,
A detection device for finally determining whether the face image is a forged face by combining a result of the first determination and a result of the second determination.
6. The method of claim 5,
The first analysis unit prepares one or more fake face images, generates each of the first one-dimensional feature vectors from the fake face images, and uses the generated first one-dimensional feature vectors in the support vector machine. A detection apparatus for generating the first reference forged facial feature vector through the step of learning as the first reference forged facial feature vector.
delete delete A detection method performed by a detection device, comprising:
acquiring an image;
separating a first area representing a face and a second area not representing the face in the image;
extracting a first feature point of the first region and extracting a second feature point of the second region;
determining whether the face is forged by comparing the first feature point and the second feature point;
The step of determining whether the face is forged,
generating a polygonal or closed curve-shaped window disposed on a boundary line between the first region and the second region;
The window includes both a portion of an edge of the first area and a portion of an edge of the second area,
Comparing the first feature point existing on one side of the window with the second feature point existing on the other side of the window based on the boundary line,
moving the window along the boundary line, and comparing the first feature point and the second feature point existing in the window whenever the window moves,
Determining the integrity between the first region and the second region using the comparison result,
A detection method for determining that the image is a forgery when it is determined that the integrity is defective.

Images