KR102530240B1 - Face fake detection method based on remote photoplethysmography signal and analysis apparatus - Google Patents

Abstract

Provided is a non-contact photoplethysmography (PPG)-based facial image forgery determination method, which includes the steps of: an analysis device receiving a facial image; the analysis device detecting a region of interest in the facial image; the analysis device detecting a PPG signal based on a change in skin color in the region of interest; the analysis device detecting the heart rate based on the PPG signal; and the analysis device inputting at least one of the PPG and the heart rate into a pre-trained learning model to calculate a value indicating whether the facial image has been forged or altered.

Description

Non-contact PPG-based face image forgery detection method and analysis device

The technology to be described below relates to a technique for determining forgery or alteration of a face image.

Biometric authentication is used in various fields. Face authentication is an authentication method using video and is used in various devices such as smartphones, laptops, and access control systems. Meanwhile, a spoofing attack on face authentication has also become an issue. Representative spying attacks include attacks using printed photos, played videos, and 3D masks.

Korean Patent Registration No. 10-2215557

The technology to be described below is intended to provide a technique for determining forgery of a face image by extracting a photoplethysmography (PPG) signal from a face image.

The non-contact PPG-based face image forgery detection method includes the steps of receiving a face image by an analysis device, detecting a region of interest from the face image by the analysis device, and the analysis device using a PPG signal based on a change in skin color of the region of interest. detecting a heart rate based on the PPG signal by the analysis device, and whether the face image is forged or falsified by inputting at least one of the PPG signal and the heart rate to a previously learned learning model. Calculating a probability value representing

An analysis device for classifying face image forgery based on non-contact PPG includes an input device for receiving a face image, a storage device for storing a learning model for classifying image forgery based on a PPG signal and heart rate, and a skin color for a region of interest in the face image. and an arithmetic device for calculating a probability value indicating whether the face image is forged or falsified by inputting the PPG signal and heart rate detected based on the change into the learning model.

The technology to be described below can easily determine forgery or alteration of a face image using non-contact PPG-related information extracted from a face image. The technology described below can detect forgery and falsification of facial images even in a device with limited performance, such as a smartphone, and can be used in biometric authentication applications and fintech fields.

1 is an example of a non-contact PPG based face image forgery detection system.
2 is an example of a flowchart for a forgery detection process of a face image.
3 is an example of a process of extracting a PPG signal and heart rate from a face image.
4 is an example of a process of forgery discrimination of a face image using a neural network model.
5 is an example of an analysis device for determining forgery or alteration of a face image.

Since the technology to be described below can have various changes and various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood to include all modifications, equivalents, or substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. may be used to describe various elements, but the elements are not limited by the above terms, and are merely used to distinguish one element from another. used only as For example, without departing from the scope of the technology described below, a first element may be referred to as a second element, and similarly, the second element may be referred to as a first element. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

In terms used in this specification, singular expressions should be understood to include plural expressions unless clearly interpreted differently in context, and terms such as “comprising” refer to the described features, numbers, steps, operations, and components. , parts or combinations thereof, but it should be understood that it does not exclude the possibility of the presence or addition of one or more other features or numbers, step-action components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is to be clarified that the classification of components in the present specification is merely a classification for each main function in charge of each component. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each component to be described below may additionally perform some or all of the functions of other components in addition to its main function, and some of the main functions of each component may be performed by other components. Of course, it may be dedicated and performed by .

In addition, in performing a method or method of operation, each process constituting the method may occur in a different order from the specified order unless a specific order is clearly described in context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The technology to be described below is a technology for discriminating forgery and falsification of a face image based on non-contact PPG.

Non-contact PPG means a PPG signal calculated using an image. Non-contact PPG is also called remote PPG (rPPG).

The analysis device analyzes the face image of the subject and determines whether the image is forged or altered. The analysis device may be implemented with various devices capable of image and data processing. For example, the analysis device may be implemented as a PC, a server on a network, a smart device, a chipset in which a dedicated program is embedded, and the like. The analysis device may be an authentication device that performs face authentication. However, since the technology to be described below is a technique for determining forgery or alteration of a face image before authentication of a user using a face, description will be focused on the content for forgery or alteration detection.

The analysis device may determine forgery or falsification of the face image using the learning model.

The learning model means a machine learning model. Machine learning models include decision trees, random forests, K-nearest neighbors (KNNs), Naive Bayes, support vector machines (SVMs), and artificial neural networks (ANNs).

ANN is a statistical learning algorithm that mimics biological neural networks. Various neural network models are being studied. A deep learning network (DNN) can model complex non-linear relationships like a general artificial neural network. Various types of DNN models have been studied. For example, there are a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Generative Adversarial Network (GAN), and Relation Networks (RL).

1 is an example of a non-contact PPG based face image forgery detection system.

1(A) is an example of a system 100 for determining forgery or alteration of a face image. 1(A) is an example of determining forgery or forgery of a face image using an algorithm or model built into a single device. In FIG. 1(A), the analysis device may be a terminal device 110 such as a smart phone, a laptop computer, a kiosk, an access control device (entrance gate), an automated teller machine (ATM), and the like.

The terminal device 110 obtains user A's face using a camera. The terminal device 110 extracts a face region from a face image and extracts a non-contact PPG signal and related data. The terminal device 110 determines whether the current face image is forged or altered based on the extracted information. A detailed forgery detection process will be described later.

Furthermore, the terminal device 110 may transmit a forgery determination result of a face image to a server that performs face image-based authentication or a server that requires face image forgery information.

1(B) is another example of a system 200 for determining forgery or alteration of a face image. The system 200 includes a terminal device 210 and a server 220 . In FIG. 1(B), the analysis device is the server 220.

The terminal device 210 may be a device such as a smart phone, a laptop computer, a kiosk, an access control device, or an ATM. The terminal device 210 obtains user A's face using a camera. The terminal device 210 transmits the acquired image to the server 220 .

The server 220 extracts a face region from the face image, and extracts a non-contact PPG signal and related data. The server 220 determines whether the current face image is forged based on the extracted information. The server 220 may deliver the forgery determination result (forgery information) to the terminal device 210 .

Meanwhile, in some cases, the terminal device 210 may perform a part of the facial image forgery detection process and deliver the performed result to the server 220 . In this case, the server 220 may perform the rest of the face image forgery detection process. For example, some of the processes performed by the terminal device 210 may be at least one of processes such as image pre-processing, face region detection, PPG signal detection, and heart rate detection.

2 is an example of a flowchart of a forgery and forgery discrimination process 300 of a face image.

The analysis device receives a face image (310). The face image may be acquired by the analysis device directly with a camera or may be received from another object. At this time, the face image must be an image (a plurality of frames) for a certain period of time in order to detect the PPG signal. In this case, the frames may be two or more consecutive frames or two or more frames having a predetermined time interval.

The analysis device detects a face region from the input image (320). The analysis device may use the entire face area. Furthermore, the analysis device may classify a region of interest (ROI) among facial regions. The region of interest may be a region in which changes in blood flow can be easily observed among skin regions of the face. For example, the region of interest may be a region such as a cheek, nose, ear, or forehead. Meanwhile, the ROI itself may be the entire face region instead of the background or other body parts.

The analysis device may detect a face region or a region of interest using a conventional image processing technique. For example, the analysis device may detect a face region or a region of interest based on landmarks based on image features.

Furthermore, the analysis device may use a segmentation model for detecting a face region or a region of interest in an input image. The segmentation model may be a deep learning-based model. For example, the segmentation model may be a model such as U-net or Fully Convolutional Network (FCN). The segmentation model must be trained using face images in advance.

The analysis device may pre-detect whether the current image is a normal image or a forgery attempt image before extracting the PPG signal based on the face region or the region of interest (330). At this time, the detected attack scenarios include a bending attack, a slight shaking, and a face covering. That is, this is a case in which the face image itself does not have a certain quality or standard suitable for authentication. The analysis device may detect whether there is an attempt such as bending, shaking, or covering a face in the current image by detecting characteristics of the image using image processing technology. If the analysis device determines that there is a corresponding forgery attempt in the current video (YES in 330), authentication may be terminated (340). Alternatively, the analysis device may issue an alarm that the current input image is abnormal and perform a process of obtaining a face image again.

If the pre-detection step is passed because there is no physical forgery attempt (NO in 330), the analysis device may determine the non-contact PPG and heart rate based on the change in skin color of the face region or region of interest (350). Heart rate can be detected using the PPG signal. Therefore, PPG and heart rate can be referred to as PPG-related information.

The analysis device may determine forgery or alteration of the face image by inputting the PPG-related information extracted from the image to the previously learned learning model (360). The analysis device may determine forgery or alteration of the current face image based on the probability value output by the learning model.

3 is an example of a process 400 of extracting a PPG signal and heart rate from a face image.

The analysis device detects a face region from the input face image (410).

The analysis device may detect a skin area for observing a skin color change in the face area (420). Here, the skin region may be the aforementioned region of interest.

The analysis device detects a non-contact PPG (rPPG) signal using the skin region (430). The PPG signal can be estimated through changes in skin color. That is, the rPPG signal is calculated by analyzing a series of consecutive images. The analysis device can detect non-contact PPG as follows using an image processing technique.

Since blood absorbs more light than surrounding tissue, the amount of light absorbed and the amount of reflected light decreases when blood passes through blood vessels. Therefore, the analysis device can detect non-contact PPG based on periodic changes in skin color.

The analysis device may obtain a non-contact PPG signal using a color difference-based method after performing skin pixel filtering using skin colors in the YCbCr range for the face area. RGB image-based skin color change analysis can cause distortion depending on environmental factors. Therefore, the analyzer can convert the RGB frame into YCbCr and perform skin pixel clustering in the Cb-Cr plane.

Thereafter, the analysis device may expand the distribution of fine skin pixels by randomly expanding the Cb and Cr components n times from the cluster central value P (Cb-central value, Cr-central value). The analyzer may detect the heart rate using the expanded Cb and Cr signals (440). The analyzer may determine the number of beats per minute (bpm) for a certain period of time by applying a window of a certain size to the PPG signal. Since the bpm determined at this time is also a non-contact method, it can be named non-contact bpm or rbpm (remote bpm).

On the other hand, the analyzer may remove the individual's breathing tendency and remove noise using a Butterworth filter in order to improve signal quality. That is, the analysis device may perform additional signal processing required in the face image forgery detection process.

3 is an example in which an analysis device detects non-contact PPG (rPPG) and heart rate using a traditional image processing technique. Furthermore, the analysis device may use a deep learning model that extracts PPG or heart rate based on feature values extracted from an input image. In this case, the analysis device may determine PPG or heart rate based on a value calculated by inputting a face image to a deep learning model.

Thereafter, the analysis device may determine whether the current face image is forged or falsified using the PPG-related information. At this time, the analysis device may use a learning model learned in advance. 4 is an example of a process 500 of determining forgery or alteration of a face image using a neural network model.

The analysis device extracts input data to be input to the learning model from the face image frame (550).

Data that the analysis device can extract from the face image include rPPG and heart rate (bpm). Meanwhile, the analysis device may take as input data raw data that is a pixel value of a face image itself or a region of interest.

Input data may be a combination of various data. For example, the input data may be any one of (i) rPPG + heart rate, (ii) rPPG + raw data, (iii) heart rate + raw data, and (iv) rPPG + heart rate + raw data.

Meanwhile, the analysis device may convert a constant value of input data into a constant vector value by one-hot encoding. The analysis device may process various types of input data into a single matrix and input the data to the learning model.

The analysis device may classify facial image forgery by inputting the input data to a previously learned learning model (520). The analysis device may classify facial image forgery based on the probability value output by the learning model. For example, the analysis device may determine that the value output by the learning model is forgery when it is close to 0, or determine the original image when it is close to 1.

The learning model may be a deep learning model that calculates a certain classification value by receiving images and vectors.

In FIG. 4, (A) at the bottom illustrates a CNN-based model. A CNN-based model can extract features from input data and output a forgery probability value based on feature values. In this case, the input data may be at least one of PPG, heart rate, and raw data measured at a specific point in time.

In FIG. 4, (B) at the bottom is an example of a deep learning model that processes time series data. For example, the deep learning model may be an RNN-based model. In this case, the deep learning model may receive and process continuous input data according to time and finally output a forgery probability value.

The learning model or deep learning model of FIG. 4 must be trained in advance. Learning data consists of input data extracted from an image and the label value of the corresponding image (whether forged or altered). The learning process corresponds to a process in which the parameters of the model layer are optimized.

The researcher built a deep learning model that uses rPPG, heart rate, and raw data extracted from face images as input data. The researcher built the model using AlexNet. The researcher trained the model, and then verified whether the input image captured by the mobile camera in real time was an image of a real person or a forged image. Table 1 below shows the classification accuracy of the model built by the researcher. The researcher evaluated the accuracy according to the type of mobile device (Galaxy, iPhone) and type of attack (printed photo, video playback). Looking at Table 1 below, it can be seen that the built model discriminates forgery of face images with very high accuracy.

original footage printed photo play video average Galaxy (Samsung) 99.12 99.19 95.72 98.01 iPhone (Apple) 93.98 98.04 100 97.34 Sum 96.55 95.61 97.86 97.67

5 is an example of an analysis device 600 that determines forgery or alteration of a face image. The analyzer 600 is a device corresponding to the analyzers 110 and 220 of FIG. 1 . The analysis device 600 may be physically implemented in various forms. For example, the analysis device 600 may have a form of a computer device such as a PC, a network server, and a data processing dedicated chipset.

The analysis device 600 may include a storage device 610, a memory 620, an arithmetic device 630, an interface device 640, and a communication device 650. Although FIG. 5 does not show a camera that captures a face image, the analysis device 600 may include a camera. However, FIG. 5 will mainly describe operations after capturing a face image.

The storage device 610 may store an input face image.

The storage device 610 may store image processing programs or codes used for image processing.

The storage device 610 may store a learning model for face image forgery.

The storage device 610 may store a result of forgery and falsification determination for a face image.

The memory 620 may store data and information generated in the process of analyzing whether the analysis device 600 has forged or falsified the face image.

The interface device 640 is a device that receives certain commands and data from the outside. The interface device 640 may receive a face image from a physically connected input device or external storage device. The interface device 640 may receive a learning model for falsification classification of a facial image from a physically connected input device or an external storage device. The interface device 640 may transmit the facial image forgery analysis result to an external object.

The communication device 650 refers to a component that receives and transmits certain information through a wired or wireless network. The communication device 650 may receive a face image from an external object. The communication device 650 may receive a learning model for classifying forgery or alteration of a face image. The communication device 650 may transmit the face image forgery analysis result to an external object.

Since the interface device 640 and the communication device 650 are configured to send and receive certain data from a user or other physical object, they can also be collectively referred to as input/output devices. If the function of receiving necessary data is limited, the interface device 640 and the communication device 650 may be referred to as input devices.

The arithmetic device 630 may use the learning model in the storage device 610 to analyze whether the face image has been forged or altered.

The arithmetic unit 630 may detect a face region or a region of interest from an input image using an image processing technique. Alternatively, the arithmetic unit 630 may detect a face region or a region of interest from an input image using a separate segmentation model.

The arithmetic unit 630 may detect in advance whether there is a forgery attempt (bending, shaking, face covering, etc.) in the input image.

The arithmetic unit 630 may detect the PPG signal based on the change in skin color in the face region or the region of interest. The detection process is as described above. Alternatively, the calculator 630 may calculate a PPG signal for a face region or a region of interest using a separate deep learning model.

The calculator 630 may calculate the heart rate (bpm) based on the time-sequential PPG signal.

The arithmetic device 630 may input PPG-related information and/or raw data (pixel values of a face region or a region of interest) to a learning model to determine whether the face image is forged or altered.

The computing device 630 may input any one of various types of input data to the learning model. The type of input data is as described above. During this process, the arithmetic unit 630 may prepare input data through one-hot encoding. The learning model calculates a probability value indicating forgery or alteration of the face image.

The arithmetic unit 630 may calculate a probability value by inputting an image (raw data) of the region of interest at a certain point in time, a PPG signal detected at the same point in time, and a heart rate detected at the same point in time to the learning model.

The arithmetic unit 630 calculates a probability value by inputting the image (raw data) of the region of interest acquired time-sequentially in a certain time interval, the PPG signal detected in the same time interval, and the heart rate detected in the same time interval to the learning model. may be

The arithmetic device 630 may determine forgery of face image forgery based on a value output by the learning model.

The arithmetic device 630 may be a device such as a processor, an AP, or a chip in which a program is embedded that processes data and performs certain arithmetic operations.

In addition, the forgery detection method for face image as described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, the various applications or programs described above are CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM) Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporary readable media include static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (Enhanced SDRAM). SDRAM, ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM) and Direct Rambus RAM (DRRAM).

This embodiment and the drawings accompanying this specification clearly represent only a part of the technical idea included in the foregoing technology, and those skilled in the art can easily understand it within the scope of the technical idea included in the specification and drawings of the above technology. It will be obvious that all variations and specific examples that can be inferred are included in the scope of the above-described technology.

Claims (10)

receiving a facial image by an analysis device;
Predetermining, by the analysis device, whether there is forgery or alteration of one of a bending attack, a fine shaking attack, and a face masking attack in the face image;
detecting, by the analysis device, a region of interest from the face image when it is determined in the preliminary determination that there is no forgery;
detecting, by the analysis device, a photoplethysmography (PPG) signal based on a change in skin color of the region of interest;
detecting, by the analyzer, a heart rate based on the PPG signal; and
Non-contact PPG-based facial image forgery discrimination including the step of calculating a probability value indicating whether the facial image is forged or altered by inputting the PPG signal, the heart rate, and the image of the region of interest to a learning model learned in advance by the analysis device. method. delete According to claim 1,
The analysis device inputs the image of the region of interest at a certain point in time, the PPG signal detected at the certain point in time, and the heart rate detected at the certain point in time to the learning model to calculate the probability value. . According to claim 1,
The analysis device is based on non-contact PPG for calculating the probability value by inputting the image of the region of interest time-sequentially acquired in a certain time interval, the PPG signal detected in the time interval, and the heart rate detected in the time interval to the learning model. A method for detecting forgery of face image. delete According to claim 1,
The analyzer clusters skin pixels in the Cb-Cr plane of the YCbCr color space, expands the Cb and Cr components by n times based on the center value of the clustered clusters, and then detects the heart rate. method. an input device that receives a face image;
a storage device for storing a learning model for classifying image forgery based on a photoplethysmography (PPG) signal and heart rate; and
It is determined in advance whether there is forgery of one of a bending attack, a fine shaking attack, or a face covering attack in the face image, and if it is determined that there is no forgery in the preliminary determination, PPG detected based on skin color change for the region of interest of the face image signal, heart rate detected based on the detected PPG signal, and the image of the region of interest are input to the learning model to calculate a probability value indicating whether the face image is forged or not. Forgery of face image based on non-contact PPG An analysis device that classifies. According to claim 7,
The calculator classifies forgery and falsification of facial images based on non-contact PPG that calculates the probability value by inputting the image of the region of interest at a certain point in time, the PPG signal detected at the certain point in time, and the heart rate detected at the certain point in time to the learning model. analysis device. According to claim 7,
The calculator is based on non-contact PPG for calculating the probability value by inputting the image of the region of interest time-sequentially acquired in a certain time interval, the PPG signal detected in the time interval, and the heart rate detected in the time interval to the learning model. An analysis device that classifies forgery and alteration of facial images. delete

Images