KR20210013793A - Deception detection method using biometric information - Google Patents

Abstract

Disclosed is a method for detecting deception using biometric information such as a voice, face image, and thermal image. The method comprises the steps of: generating first training data including feature values extracted from voice data of a speaker, who is a target for deception detection; further learning an artificial neural network learned through second training using the first training data; and detecting whether a target voice is false by inputting a target feature value extracted from the target voice of the speaker into the artificial neural network.

Description

Lie detection method using biometric information {DECEPTION DETECTION METHOD USING BIOMETRIC INFORMATION}

The present invention relates to a lie detection method using biometric information, and more particularly, to a lie detection method using voice, face image, and thermal image.

A common lie detection method is a method of detecting lies by attaching a sensor to the skin and measuring GSR (Galvanic Skin Response), ECG (electrocardiogram, electrocardiogram), and PPG (photoplethysmogram, pulse wave).

The answer of the subject of lie detection is unified as “no”, and when the answer is answered, whether or not a lie is determined through a change in signal.

In the case of a method of detecting a lie by attaching a sensor to the skin, since various sensors attached to the skin are very sensitive, there are restrictions on the movement of the lie detection target and the location of the lie detection. Therefore, it is difficult to perform lie detection as needed regardless of time and place.

In order to overcome this limitation, in recent years, various methods for detecting a lie in a non-contact manner for a lie detection target have been developed using video or audio.

Related prior documents include Korean Patent Application Publication Nos. 2012-0066275, 2018-0112303, and Korean Patent Registration No. 10-1784058.

The present invention is to provide a method capable of detecting a lie without contacting a lie detection object.

In addition, the present invention is to provide a method for detecting lies by using changes in voice, facial expression, and blood flow due to cognitive dissonance that occur when lying.

In addition, the present invention is to provide a method for detecting lies using an artificial neural network.

According to an embodiment of the present invention for achieving the above object, there is provided a method comprising: generating first training data including feature values extracted from voice data of a talker who is a lie detection target; Additionally learning an artificial neural network learned through second training data by using the first training data; And detecting whether the target voice is false by inputting a target feature value extracted from the target voice of the talker into the artificial neural network.

In addition, according to another embodiment of the present invention for achieving the above object, the step of learning a first artificial neural network by using first training data including a face image marked with a landmark; Predicting a landmark in a target face image of a talker who is a lie detection target using the first artificial neural network; And determining whether or not the talker has a false statement according to the predicted movement of the landmark by using a pre-learned second artificial neural network.

In addition, according to another embodiment of the present invention for achieving the above object, by using the first training data including a thermal image face image marked with a landmark, learning a first artificial neural network; Predicting a landmark in a target thermal image of a talker who is a lie detection target using the first artificial neural network; Dividing the target thermal image face image into a plurality of regions of interest by using the predicted landmark; And inputting a target feature value extracted from the region of interest for the talker into a pre-learned second artificial neural network to detect whether or not the talker's statement is false or intention to make a false statement. A lie detection method is provided.

According to the present invention, lies can be detected in a non-contact method with respect to a lie detection target.

In addition, according to the present invention, an artificial neural network is additionally learned using voice data of not only the learner, but also the talker who is a lie detection target, and reflects the personal voice characteristics of the talker, which is a lie detection target, to the artificial neural network, thereby improving lie detection performance. have.

In addition, according to the present invention, not only can the talker's lies be detected using the talker's facial expression change, but also the talker's lies can be detected using the talker's fine expressions that intentionally hide the expression change.

In addition, according to the present invention, not only can detect whether or not the speaker's short-answer statement is false by using the change of the talker's facial blood flow, but also can detect the lie, and also detect the intention of false statement about the speaker's long statement. can do.

1 is a diagram illustrating a lie detection apparatus using biometric information according to an embodiment of the present invention.
2 is a view for explaining a lie detection method using voice according to an embodiment of the present invention.
3 is a diagram illustrating a method of generating first training data according to an embodiment of the present invention.
4 is a diagram illustrating a lie detection method using a face image according to an embodiment of the present invention.
5 is a diagram illustrating a sample face image in which a landmark is displayed.
6 is a view for explaining a lie detection method using a thermal image according to an embodiment of the present invention.
7 is a diagram illustrating a face image of a target thermal image on which a predicted landmark is displayed.
8 is a diagram illustrating a target thermal image face image in which an ROI is displayed.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a lie detection apparatus using biometric information according to an embodiment of the present invention.

Referring to FIG. 1, a lie detection apparatus according to an embodiment of the present invention includes a biometric information generation unit 110 and a lie determination unit 120.

The biometric information processing unit 110 extracts biometric information necessary for lie detection by processing biometric information in the form of raw data. The raw data may include voice data and a face image of a lie detection target answering the administrator's question, and the face image may be an RGB image or a thermal image.

When a person lies, a change in voice, a change in facial expression, and a change in blood flow occur due to cognitive dissonance, and an embodiment of the present invention uses the change in voice, facial expression, and blood flow as biometric information for lie detection. The biometric information processing unit 110 extracts such biometric information from voice data and face images.

According to an embodiment, the biometric information processing unit 110 extracts a feature value representing a voice change from voice data of a talker who is a lie detection target, and the feature values include a pitch value, an intensity value, and a jitter value ( zitter), a shimmer, and a Mel-Frequency Cepstral Coefficient (MFCC) coefficient. These feature values are feature values that are frequently used for speech analysis, and the biometric information processing unit 110 may extract these feature values using various published algorithms.

According to another embodiment, the biometric information processing unit 110 may extract a landmark representing an expression change from a face image of a talker who is a lie detection target, and the landmark is a coordinate for a specific part of the face such as eyes, nose, and mouth. Can be a value. The biometric information processing unit 110 may predict a landmark from a face image using a pre-learned artificial neural network.

According to another embodiment, the biometric information processing unit 110 tracks a plurality of regions of interest (ROIs) from a thermal image of a talker who is a lie detection target, and extracts a feature value representing a change in blood flow of the face from the region of interest. I can. The region of interest may include eyebrows, eyes, nose, and mouth, and the feature values are the mean, maximum, median, and standard deviation values for the pixel values included in the region of interest. deviation), a kurtosis value, and a skewness value. The biometric information processing unit 110 may predict the region of interest in the thermal image of the face using a pre-learned artificial neural network.

The lie detection unit 120 detects a lie of a talker using an artificial neural network learned in advance. When the talker makes a statement about the manager's question, a lie about the statement is detected. Raw data obtained while the talker makes a statement is processed by the biometric information processing unit 110, and the processed biometric information is input to an artificial neural network, and the artificial neural network classifies the talker's statement as true or false, and then determines the classification result. Print.

The artificial neural network used in the biometric information processing unit 110 and the lie detection unit 120 may be an artificial neural network based on Long Short Term Memory (LSTM) as an embodiment, and the machine learning algorithm used according to the embodiment may vary. I can.

As described above, an embodiment of the present invention uses voice, facial images, and thermal images that can be obtained in a non-contact state without attaching various sensors to the skin of the talker, thereby detecting lies without being restricted to time and place. You can do it.

Hereinafter, an embodiment of a lie detection method using voice, face image, and thermal image will be described in more detail. This lie detection method may be performed in a computing device including a processor, and as an example, may be performed in the lie detection apparatus described above.

FIG. 2 is a diagram for explaining a lie detection method using voice according to an embodiment of the present invention, and FIG. 3 is a diagram for explaining a method for generating first training data according to an embodiment of the present invention.

Referring to FIG. 2, the lie detection apparatus according to an embodiment of the present invention generates first training data including feature values extracted from voice data of a talker who is a lie detection target (S210). The feature value included in the first training data includes at least one of a pitch value, an intensity value, a jitter value, a seamer value, and an MFCC, and may be a feature value sampled for each preset section.

Then, the lie detection apparatus additionally learns (S220) the artificial neural network learned through the second training data, using the first training data. That is, the artificial neural network is in a state in which it is learned by the second training data, and the lie detection apparatus additionally learns the artificial neural network by using the first training data.

The second training data includes the same feature values as the first training data, and includes feature values for each of the case where the statement of the learning talker used for learning of the artificial neural network is true and false. Learning is performed by inputting the feature values obtained from the speech data when the learning talker asserts true to the artificial neural network, and at this time, the true is given as a label for the artificial neural network. In addition, the feature value obtained from the speech data in the case of the learning talker stating false is input to the artificial neural network to perform learning, and the false is given as a label for the artificial neural network.

Like the learning process using the second training data, the lie detection apparatus additionally learns an artificial neural network using the first training data. The first training data includes feature values extracted from voice data for a case where the talker, which is a lie detection target, states true and each case that states false.

In step S220, the lie detection apparatus may selectively use a feature value having high learning efficiency among several feature values in order to further improve the lie detection performance. In order to select a feature value with high learning efficiency, the lie detection apparatus learns the artificial new network a plurality of times using a combination of feature values included in the first training data in step S220, and learns each result a plurality of times. One of the combinations of feature values can be selected by using.

For example, the lie detection device learns an artificial neural network multiple times by using various feature value combinations consisting of four feature values excluding one of the five feature values, and a combination of feature values with the best detection performance as a result of learning. Can be used as the first training data.

Then, the lie detection apparatus inputs a target feature value extracted from the target voice of the talker, which is a lie detection target, into the artificial neural network, and detects whether the target voice is false (S230). In step S230, the lie detection apparatus may detect whether the target voice is false using the target feature value corresponding to the combination of the feature values selected as described above.

The artificial neural network can classify the target voice as true if the feature value and the target feature value when the speaker states true in the first and second training data are similar, and the speaker is false in the first and second training data. If the feature value and the target feature value in the case of the statement are similar, the target voice can be classified as false.

According to an embodiment of the present invention, by additionally learning an artificial neural network using voice data of not only the learner but also the talker who is a lie detection target, and reflects the personal voice characteristics of the talker, which is a lie detection target, to the artificial neural network, lie detection performance Can increase.

On the other hand, as described above, the first and second training data include feature values extracted from voice data for the case where the talker, who is the target of lie detection, states true and each case that states false. For example, one of a precutaneous base dead value (GSR), a pulse wave value (PPG), and an electrocardiogram value (ECG) may be used to classify a case where the talker states true and if the speaker states false.

Referring to FIG. 3, in step S310, the lie detection apparatus receives one of a GSR value, a PPG value, and an ECG value acquired through an electrode attached to the skin of the talker in step S310, and whether the talker's statement is true or false. Whether it is determined (S320). The lie detection apparatus generates a feature value for false from voice data of the statement determined to be false (S330), and generates a feature value for true from voice data of the statement determined to be true (S340).

As a result, the first and second training data may include a feature value of voice data of a talker who is a lie detection target, classified as false or true using one of a skin warfare base value, a pulse wave value, and an electrocardiogram value.

FIG. 4 is a diagram for explaining a lie detection method using a face image according to an embodiment of the present invention, and FIG. 5 is a diagram illustrating a sample face image in which a landmark is displayed.

Referring to FIG. 4, the lie detection apparatus according to an embodiment of the present invention learns a first artificial neural network (S410) by using first training data including a face image on which a landmark is displayed. Here, the face image may be an RGB image, and may be a 2D image or a 3D image according to embodiments. Also, it may be a video recorded with audio.

As shown in FIG. 5, a sample face image on which a landmark is displayed may be input to the first artificial neural network to perform learning. In the subsequent prediction step, the first artificial neural network determines a position similar to the position of the landmark included in the first training data. , In the input face image, it can be predicted as a landmark. In FIG. 5, a green dot represents a landmark.

Returning to FIG. 4 again, the lie detection apparatus predicts a landmark in the target face image of the talker, which is a lie detection target, using the first artificial neural network (S420). The target face image is input to the first artificial neural network, and the first artificial neural network may predict a landmark from the target face image based on the learning result.

In this case, the lie detection apparatus may predict the landmark in units of a preset frame, and may predict the landmark from the target face image in the section before the talker's statement, the statement section, and the section after the statement. The pre-statement section is a section from a preset time point until the current question is started by the administrator, and the statement section corresponds to a section in which the speaker makes an answer to the current question. And the section after the statement corresponds to the section after the current question and before the next question starts.

The lie detection device can detect the talker's statement section in the target face image by using the talker's voice included in the image, and detect the section before and after the statement in the target face image based on the statement section. have.

When the landmark is predicted, the lie detection apparatus determines whether or not the talker falsely states according to the movement of the landmark predicted in step S420 by using the pre-learned second artificial neural network (S430). The movement of the landmark corresponds to the change of the talker's facial expression, and eventually, the lie detection device may determine whether the talker has a false statement through the change of the talker's expression.

The second training data used for learning of the second artificial neural network includes landmarks of the face image of the learning talker for the case where the learning talker states true and each case where the learning talker states false, and the section before the statement and the statement section And a time series landmark coordinate value extracted according to a preset sampling period for each section after the statement. A landmark in the case of the learning talker stating true is input to the second artificial neural network to perform learning, and at this time, the true is given as a label for the second artificial neural network. In addition, the landmark is inputted to the second artificial neural network when the learning talker makes a false statement, and learning is performed. At this time, the false is given as a label for the second artificial neural network.

The predicted landmark is input to the second artificial neural network in step S430, and is extracted according to a preset sampling period for each section before the statement, the section after the statement, and the section after the statement so that it can correspond to the second training data used in the learning process. It may be a time series landmark coordinate value. The lie detection apparatus may determine whether or not the talker makes a false statement by using the movement of the landmark, which is the change in coordinate values.

The second artificial neural network classifies the speaker's statement as a true statement if the movement pattern of the landmark and the predicted movement pattern of the landmark are similar when the learning talker states that it is true, and when the learning talker makes a false statement. If the movement pattern of the landmark and the predicted movement pattern of the landmark are similar, the speaker's statement can be classified as a false statement.

Alternatively, according to the embodiment, the second artificial neural network is, in the case where the speaker for learning has made a false statement, the movement pattern of each section before the statement, the statement section, and after the statement, and the section before the statement of the predicted landmark, the statement section and after the statement If the motion patterns for each section are similar, the speaker's statement can be classified as false.

On the other hand, when a person lies, a micro expression, which is a minute change occurring within a very short time, may occur on the face due to cognitive dissonance. Unlike general facial expressions, fine facial expressions occur for a very short time, which is difficult for humans to recognize, and occur unconsciously, so it is difficult for a person to intentionally control fine facial expressions, unlike normal facial expressions.

The lie detection apparatus according to another embodiment of the present invention may determine whether the talker has falsely stated by using a movement corresponding to a fine facial expression among the movements of the landmarks predicted in step S420.

To this end, in step S430, the lie detection apparatus generates a coordinate value of the predicted landmark according to time, using the prediction result of the landmark, and, among the generated coordinate values, the length of time at which a fine expression occurs. The coordinate values where the change occurs in the section is extracted.

As an example, as described above, when a time-series landmark coordinate value extracted according to a preset sampling period is generated, the lie detection device uses a band pass filter that filters a signal in a 4 to 25 Hz band, and a fine expression is generated. It is possible to extract a coordinate value in which a change occurs in a time interval of a length of time.

In addition, the lie detection apparatus may input the extracted coordinate values into the second artificial neural network to determine whether the talker has falsely stated. Of course, for this, in the learning process of the second artificial neural network, a landmark coordinate value corresponding to a fine expression may be used.

As described above, according to an embodiment of the present invention, not only can the talker's lies be detected by using the talker's facial expression change, but also the talker's lies can be detected using the talker's fine expression that intentionally hides the expression change. have.

6 is a diagram illustrating a lie detection method using a thermal image according to an embodiment of the present invention, and FIG. 7 is a diagram illustrating a target thermal image face image in which a predicted landmark is displayed. 8 is a diagram illustrating a target thermal image face image in which an ROI is displayed.

Referring to FIG. 6, the lie detection apparatus according to an embodiment of the present invention learns a first artificial neural network (S610) by using first training data including a thermal image of a face on which a landmark is displayed.

Similar to FIG. 4, a sample thermal image of a landmark marked face image may be input to the first artificial neural network to perform learning. In the subsequent prediction step, the first artificial neural network determines the position of the landmark included in the first training data. Similar positions can be predicted as landmarks in the input thermal face image.

The lie detection apparatus uses a first artificial neural network to predict a landmark in a target thermal image of a talker, which is a lie detection target (S620). As shown in FIG. 7, a landmark may be predicted in the target thermal image face image, and a red dot in FIG. 7 represents the predicted landmark. The target thermal image face image is input to the first artificial neural network, and the first artificial neural network may predict a landmark from the target thermal image face image based on the learning result.

Then, the lie detection apparatus divides the target thermal image face image into a plurality of regions of interest by using the landmark predicted in step S620 (S630). As an embodiment, the lie detection apparatus may classify the predicted landmarks into landmarks located for each specific part of a face, and generate an ROI by connecting the classified landmarks with a solid line. Thus, this region of interest corresponds to a specific part of the face. As shown in FIG. 8, the region of interest may correspond to the eyebrows, brows, nose, throat, chin, and cheeks.

The lie detection apparatus inputs the target feature value extracted from the region of interest divided in step S630 into a pre-learned second artificial neural network, and detects whether the talker's statement is false or the intention of false statement (S640). The lie detection apparatus may extract at least one of an average value, a maximum value, a median value, a standard deviation value, a kurtosis value, and a skewness value for a pixel value included in the ROI as a target feature value.

The second training data used for learning of the second artificial neural network includes feature values for a case where the learning talker states true and each case that states false, and includes a preset feature value for a region of interest. When the learning talker states true, the feature value is inputted to the second artificial neural network to perform learning, and at this time, the true is given as a label for the second artificial neural network. In addition, when the learning talker states false, the feature value is inputted to the second artificial neural network to perform learning, and in this case, false is given as a label for the artificial neural network.

At this time, the statement of the speaker for learning is a short-answer statement. That is, it is a statement that answers "no" to the manager's question, and the feature value for the learning talker who performs such a short-answer statement is used as the second training data.

Accordingly, the lie detection apparatus may determine whether the talker's short-answer statement is false, and determine whether the talker is false by extracting the target feature value from the target thermal image obtained when the talker performs the short-answer statement.

The second artificial neural network can classify the talker's statement as true if the feature value when the talker states true in the second training data and the target feature value are similar, and if the talker makes a false statement in the second training data If the feature value of is similar to the target feature value, the speaker's statement can be classified as false.

On the other hand, unlike voices and facial expressions, the rate at which the change in blood flow occurs is not fast. In other words, when the talker tells a lie, changes in voice and facial expression occur during the statement, whereas changes in blood flow occur during the statement and may change even after the statement.

Using this point, the lie detection apparatus according to an embodiment of the present invention not only detects whether the speaker's short-answer statements that take a short time are false or not, but also detects the intention of false statements about the speaker's long-form statements that take a long time. can do. When the speaker makes a statement in sentence form, false and true may be mixed in such a statement, and if false, it may be judged that the statement is made with false intentions.

To this end, the second training data used for the learning of the second artificial neural network may include feature values extracted from the thermal image face image of the talker for learning to state in a sentence form, and the lie detection device may state in a sentence form. By extracting a feature value from the thermal image of the talker, which is the target of the lying detection target, the intention of false statement regarding the talker's statement can be detected.

As described above, according to an embodiment of the present invention, not only can the talker's short-answer statement be false or not, but also the talker's long statement, by using the change of the talker's facial blood flow. Intent to falsely state against can be detected.

The technical details described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In addition, as described above, the present invention has been described by specific matters such as specific components and limited embodiments and drawings, but this is only provided to help a more general understanding of the present invention, and the present invention is limited to the above embodiments. It is not, and a person of ordinary skill in the field to which the present invention belongs can make various modifications and variations from this description. Therefore, the spirit of the present invention is limited to the described embodiments and should not be defined, and all things that are equivalent or equivalent to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. .

Claims (13)

Generating first training data including feature values extracted from voice data of a talker who is a lie detection target;
Additionally learning an artificial neural network learned through second training data by using the first training data; And
Inputting a target feature value extracted from the target voice of the talker into the artificial neural network to detect whether the target voice is false
Lie detection method using voice comprising a.
The method of claim 1,
The feature value included in the first training data is
It includes at least one of a pitch value, an intensity value, a jitter value, a seamer value, and MFCC,
The step of additionally learning the artificial neural network
Learning the artificial new public network multiple times by using a combination of feature values included in the first training data; And
Selecting one of a combination of feature values using the results of each of the plurality of times learned
Lie detection method using voice comprising a.
The method of claim 2,
The step of detecting whether the target voice is false
Using a target feature value corresponding to the combination of the selected feature values, detecting whether the target voice is false
Lie detection method using voice.
The method of claim 1,
The first training data is
Including the feature values of voice data classified as false or true using one of the precutaneous base value (GSR), pulse wave value (PPG), and electrocardiogram value (ECG)
Lie detection method using voice.
Learning a first artificial neural network by using first training data including a face image in which the landmark is displayed;
Predicting a landmark in a target face image of a talker who is a lie detection target using the first artificial neural network; And
Determining whether or not the talker makes a false statement according to the predicted movement of the landmark using a pre-learned second artificial neural network
Lie detection method using a face image comprising a.
The method of claim 5,
The step of predicting the landmark
Predicting a landmark in the target face image in the section before the talker's statement, the statement section, and the section after the statement,
A lie detection method using facial images.
The method of claim 5,
Detecting a statement section of the talker in the target face image by using the talker's voice; And
Based on the statement section, detecting a section before the statement and a section after the statement in the target face image
Lie detection method using a face image further comprising.
The method of claim 5,
The step of determining whether the talker makes a false statement
Among the motions of the predicted landmarks, using a motion corresponding to a micro expression, determining whether the talker has a false statement
A lie detection method using facial images.
The method of claim 8,
The step of determining whether the talker makes a false statement
Generating a coordinate value of the landmark over time using the prediction result of the landmark;
Extracting a coordinate value in which a change occurs in a time section of a length in which the fine expression occurs, from among the coordinate values; And
Inputting the extracted coordinate values into the second artificial neural network to determine whether the talker is false
Lie detection method using a face image comprising a.
Learning a first artificial neural network by using first training data including a thermal face image in which the landmark is displayed;
Predicting a landmark in a target thermal image of a talker who is a lie detection target using the first artificial neural network;
Dividing the target thermal image face image into a plurality of regions of interest by using the predicted landmark; And
Inputting a target feature value extracted from the region of interest for the talker into a pre-learned second artificial neural network to detect whether the talker's statement is false or false statement intention
Lie detection method using a thermal image comprising a.
The method of claim 10,
The target feature value is
Including at least one of an average value, a maximum value, a median, a standard deviation value, a kurtosis value, and a skewness value for the pixel values included in the region of interest.
Lie detection method using thermal imaging.
The method of claim 10,
The step of detecting the talker's lies
To detect whether the talker's short-answer statement is false
Lie detection method using thermal imaging.
The method of claim 10,
The step of detecting the talker's lies
To detect false intentions of the talker's long statement
Lie detection method using thermal imaging.

Images