KR20220136005A - Method and apparatus that detects spoofing of biometric information - Google Patents

Abstract

According to one embodiment, provided are a method and a device for detecting spoofing of biometric information, which may extract an embedding vector from an intermediate layer of a neural network which detects whether biometric information is stolen from an image including biometric information of a user, detect whether the biometric information is firstly stolen based on the embedding vector, and detect whether the biometric information is secondly stolen by using an output vector output from an output layer of the neural network depending on whether the biometric information is firstly stolen.

Description

Method and apparatus for detecting whether biometric information is stolen

The following embodiments relate to a method and apparatus for detecting whether biometric information is stolen.

With the development of smartphones and various mobile/wearable devices, the importance of security authentication technology for biometric information is increasing. Biometric authentication is being introduced in various fields due to the convenience and accessibility of the authentication method through biometric information. For example, fingerprint recognition technology, which is one of security authentication technologies, is widely used due to advantages such as convenience, security, and economic feasibility. Generally, in fingerprint recognition, a user's fingerprint image is acquired through a sensor, and the user is authenticated by comparing the acquired fingerprint image with a pre-registered fingerprint image. In this case, when a precisely manipulated fake fingerprint pattern is input to the sensor, the fingerprint recognition device may not distinguish the fake fingerprint pattern and may recognize the fake fingerprint pattern as a biometric fingerprint. For example, when a material engraved with a fingerprint, such as rubber, silicone, gelatin, epoxy, or latex, comes into contact with the sensor, the fingerprint engraved on the material may be recognized as a human fingerprint. In terms of security of biometric information, it is important to distinguish fake biometric information from real biometric information.

According to an embodiment, the method of detecting whether biometric information is stolen includes extracting an embedding vector from an intermediate layer of a neural network that detects whether the biometric information is stolen from an image including the user's biometric information ; detecting whether the biometric information is first stolen based on the embedding vector; and detecting whether the biometric information is second stolen using an output vector output from an output layer of the neural network according to whether the first theft is detected.

The step of detecting whether the first stealing may include calculating a first score based on the embedding vector; and detecting whether the first stealing is performed based on the first score.

The step of detecting whether the first stealing by the first score includes: determining whether the first score is a score that falls within a first threshold range for determining whether the first stealing; and when the first score is determined to be a score that falls within the first threshold range, detecting whether the first steal is performed based on the first score.

The first threshold range is a first threshold value corresponding to a maximum probability that the first score can be determined as a score corresponding to stolen information and a minimum probability that the first score can be determined as a score corresponding to actual information. may be determined based on the corresponding second threshold.

The step of detecting whether the first stealing based on the first score is determined by using a pre-trained classifier to determine whether the first score is a score belonging to a range determined as the theft information or a score belonging to a range determined as actual information. It may include a step of classifying.

The method of detecting whether the biometric information has been stolen may further include outputting a classification result of the classifier when the first stealing is detected based on the first score.

The method of detecting whether the biometric information has been stolen may include: when the first stealing is not detected by the first score, extracting a 1-2 embedding vector from a next intermediate layer after the intermediate layer; calculating a 1-2 score based on the 1-2 embedding vector; and detecting whether the first stealing is performed based on the 1-2 score.

The step of detecting whether the first stealing based on the 1-2 score is whether the 1-2 score is a score belonging to a range determined as the theft information using a pre-trained 1-2 classifier or as actual information. It may include classifying whether the score is within the determined range.

The method of detecting whether the biometric information has been stolen may include: when the first stealing is not detected by the first score, extracting a 1-2 embedding vector from a next intermediate layer after the intermediate layer; calculating a 1-2 score based on the 1-2 embedding vector; fusing the first score and the 1-2 score; and classifying whether the fused score is a score belonging to a range determined as the theft information or a score belonging to a range determined as the actual information, and may further include the step of detecting whether the first stolen or not.

The step of detecting whether the second theft is detected may include detecting whether the second theft is based on a second score based on the output vector when the first theft is not detected.

The image includes a first image and a second image including the user's biometric information, and the step of extracting the embedding vector is one of a first neural network for detecting whether the biometric information is stolen from the first image. extracting one or more 1-1 embedding vectors from one or more first intermediate layers; and extracting one or more 1-2 embedding vectors from one or more second intermediate layers of a second neural network for detecting whether the biometric information is stolen from the second image.

The second image may be an image obtained by capturing the user's biometric information at a different time from the first image by a sensor, or may include an image generated through image processing on the first image.

The step of detecting whether the first stealing may include: calculating a 1-3 score based on the 1-1 embedding vector and the 1-2 embedding vector; and detecting whether the biometric information is first stolen based on the 1-3 score.

The step of detecting whether the biometric information is first stolen based on the 1-3 score includes: determining whether the 1-3 score is a score that falls within a first threshold range for determining whether the first theft or not ; and detecting whether the first steal is based on the 1-3 score when the 1-3 score is determined to be a score that falls within the first threshold range.

The step of detecting whether the second is stolen may include: if the first steal is not detected by the 1-3 score, obtaining a first output vector from a first output layer of the first neural network; obtaining a second output vector from a second output layer of the second neural network; and detecting whether the second stealing is performed based on the first output vector and the second output vector.

The step of detecting whether the second theft is based on the first output vector and the second output vector may include whether the biometric information is second stolen based on a third score based on the first output vector and the second output vector. It may include the step of detecting

The method for detecting whether the biometric information is stolen may include: generating a 1-3 embedding vector based on the 1-1 embedding vector and the 1-2 embedding vector; and calculating a 1-3 score based on the 1-3 embedding vector, wherein the step of detecting whether or not the first stealing is performed may include calculating a 1-3 score based on the 1-1 embedding vector. and detecting whether the biometric information is first stolen based on the 1st score, the 1-2th score calculated based on the 1-2th embedding vector, and the 1-3th score.

The generating of the 1-3 embedding vector may include generating the 1-3 embedding vector by combining the 1-1 embedding vector and the 1-2 embedding vector; and generating the 1-3 embedding vector by element-wise summing the 1-1 embedding vector and the 1-2 embedding vector.

The step of detecting whether the first stealing is performed is to determine whether the 1-1 score is a score that belongs to a range determined as the stolen information or a score that belongs to a range determined as actual information using a pre-trained 1-1 classifier. 1-1 classifying; performing a 1-2 classifying step using a pre-trained 1-2 classifier to determine whether the 1-2 score is a score within a range determined as the stolen information or a score that belongs to a range determined as the real information; classifying the 1-3 score as a score belonging to a range determined as the stolen information or a score belonging to a range determined as the actual information using a pre-trained 1-3 classifier; and detecting whether the first robbery is performed using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result.

The step of detecting whether or not the first theft is performed using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result includes the 1-1 classification result and the 1-2 classification result. The method may include detecting whether the first stealing is performed based on a classification result and a weighted sum of the 1-3 classification results.

The step of detecting whether or not the first theft is performed using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result includes the 1-1 classification result and the 1-2 classification result. The method may include detecting whether or not the first robbery is used according to a classification result and a majority vote based on the 1-3 classification result.

The biometric information may include any one of a fingerprint, an iris, and a face of the user.

According to an embodiment, an apparatus for detecting whether biometric information is stolen includes: a sensor for capturing an image including a user's biometric information; and extracting an embedding vector from an intermediate layer of a neural network that detects whether the biometric information is stolen from the image, and detects whether the biometric information is first stolen based on the embedding vector, and whether the first theft is detected and a processor configured to detect whether the biometric information is second stolen by using an output vector output from an output layer of the neural network according to whether the biometric information is stolen.

The processor may calculate a first score based on the embedding vector, and detect whether the first steal is performed based on the first score.

The processor determines whether the first score is a score that falls within a first threshold range for determining whether the first stealing occurs, and when the first score is determined as a score that falls within the first threshold range, the first score is determined to be within the first threshold range. Based on the score, whether the first stealing may be detected.

The first threshold range is a first threshold value corresponding to a maximum probability that the first score can be determined as a score corresponding to stolen information and a minimum probability that the first score can be determined as a score corresponding to actual information. may be determined based on the corresponding second threshold.

The processor may classify whether the first score is a score within a range determined as the stolen information or a score that belongs to a range determined as the real information by using a pre-trained classifier.

The processor may output a classification result of the classifier when the first stealing is detected by the first score.

If the first stealing is not detected by the first score, the processor extracts a 1-2 embedding vector from a next intermediate layer after the intermediate layer, and a first based on the 1-2 embedding vector. A -2 score is calculated, and based on the 1-2 score, it is possible to detect whether the first stealing occurs.

The processor may classify whether the 1-2 score is a score that belongs to a range determined as stolen information or a score that belongs to a range determined as real information by using a pre-trained 1-2 classifier.

If the first stealing is not detected by the first score, the processor extracts a 1-2 embedding vector from a next intermediate layer after the intermediate layer, and a first based on the 1-2 embedding vector. -2 score is calculated, the first score and the 1-2 score are fused, and the fused score is classified as a score belonging to a range determined as stolen information or a score belonging to a range determined as real information By doing so, it is possible to detect whether the first theft is made.

The processor may detect whether the second theft is based on a second score based on the output vector when the first theft is not detected.

The image includes a first image and a second image including the user's biometric information, and the processor detects whether the biometric information is stolen from the first image. One or more first intermediate layers of a first neural network. Extracts one or more 1-1 embedding vectors from the second image, and extracts one or more 1-2 embedding vectors from one or more second intermediate layers of a second neural network that detects whether the biometric information is stolen from the second image. can

The second image may be an image obtained by capturing the user's biometric information at a different time from the first image by a sensor, or may include an image generated through image processing on the first image.

The processor may calculate a 1-3 score based on the 1-1 embedding vector and the 1-2 embedding vector, and detect whether the biometric information is first stolen based on the 1-3 score. have.

the processor determines whether the 1-3 score is a score that falls within a first threshold range for determining whether the first stealing occurs, and when the 1-3 score is determined to be a score that falls within the first threshold range , whether the first stealing may be detected based on the 1-3 score.

When whether the first stealing is not detected by the 1-3 score, the processor obtains a first output vector from a first output layer of the first neural network, and a second output of the second neural network A second output vector may be obtained from the layer, and whether the second stealing may be detected based on the first output vector and the second output vector.

The processor may detect whether the biometric information is second stolen based on a third score based on the first output vector and the second output vector.

the processor generates a 1-3 embedding vector based on the 1-1 embedding vector and the 1-2 embedding vector, and calculates a 1-3 score based on the 1-3 embedding vector; The biometric information is obtained based on the 1-1 score calculated based on the 1-1 embedding vector, the 1-2 score calculated based on the 1-2 embedding vector, and the 1-3 score. First, it is possible to detect whether theft is stolen.

The processor combines the 1-1 embedding vector and the 1-2 embedding vector to generate the 1-3 embedding vector, or uses the 1-1 embedding vector and the 1-2 embedding vector as elements. By adding stars, the 1-3 embedding vector may be generated.

The processor 1-1 classifies whether the 1-1 score is a score that belongs to a range determined as stolen information or a score that belongs to a range determined as real information by using a pre-trained 1-1 classifier, Using a pre-trained 1-2 classifier, the 1-2 classifies whether the 1-2 score is a score that belongs to the range determined by the stolen information or a score that belongs to the range determined by the real information; Using the trained 1-3 classifier, classifying 1-3 whether the score 1-3 is a score belonging to a range determined by the stolen information or a score belonging to a range determined by the real information, and Whether the first stealing may be detected using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result.

The processor may detect whether the first robbery is performed based on a weighted sum of the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result.

The processor may detect whether the first robbery is performed according to a majority vote based on the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result.

According to an embodiment, the electronic device includes: a fingerprint sensor for capturing an image including user's fingerprint information; extracting one or more embedding vectors from an intermediate layer of a neural network that detects whether the fingerprint information has been stolen from the image, and detecting whether the fingerprint information is first stolen based on the one or more embedding vectors; a processor configured to detect whether the fingerprint information is second stolen using an output vector output from an output layer of the neural network according to whether the fingerprint information is detected; and an output device for outputting at least one of whether the first stealing and the second stealing.

The fingerprint sensor may include at least one of an ultrasonic fingerprint sensor, an optical fingerprint sensor, and an electrostatic fingerprint sensor.

1 is a diagram illustrating an environment for detecting whether biometric information is stolen, according to an exemplary embodiment;
2 is a flowchart illustrating a method of detecting whether biometric information is stolen, according to an exemplary embodiment;
3 is a diagram illustrating a structure of a neural network according to an embodiment.
4 is a diagram for explaining a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to an embodiment;
5 is a flowchart illustrating a method of first detecting whether biometric information is stolen, according to an exemplary embodiment;
6 illustrates a first threshold range according to an embodiment;
7 to 10 are diagrams for explaining a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to embodiments;
12 is a block diagram of an apparatus for detecting whether biometric information is stolen, according to an embodiment;
13 is a block diagram of an electronic device according to an embodiment;

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit described in the embodiments.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted.

1 is a diagram illustrating an environment for detecting whether or not biometric information is stolen, according to an exemplary embodiment. Referring to FIG. 1 , according to an embodiment, a detection apparatus 100 including a sensor 110 for sensing user's biometric information (eg, fingerprint) and registered fingerprint images 121 , 122 , 123 are detected. An enrolled fingerprint database 120 is shown comprising. Hereinafter, a case in which the user's biometric information is a fingerprint will be described as an example for convenience of explanation, but the present invention is not limited thereto. The biometric information may include various information such as an iris, a palm, and a face in addition to a fingerprint.

The detection apparatus 100 may acquire the input fingerprint image 115 in which the user's fingerprint appears through the sensor 110 . The sensor 110 may be, for example, an ultrasonic fingerprint sensor that captures a user's fingerprint, an optical fingerprint sensor, an electrostatic fingerprint sensor, or an image sensor, but is not limited thereto.

Fingerprint registration may be performed for fingerprint recognition. The registered fingerprint images 121 , 122 , and 123 may be pre-stored in the registered fingerprint database 120 through a fingerprint registration process. For personal information protection, the registered fingerprint database 120 may store features extracted from the registered fingerprint images 121 , 122 , and 123 instead of storing the registered fingerprint images 121 , 122 , and 123 as they are. The registered fingerprint database 120 is stored in a memory (not shown) included in the detection device 100 or an external device (not shown) such as a server or a local cache capable of communicating with the detection device 100 . can be stored in

When the input fingerprint image 115 for authentication is received, the detection device 100 detects the fingerprint (hereinafter, referred to as 'input fingerprint') displayed on the input fingerprint image 115 and the registered fingerprint images 121 to 123 . The user's fingerprint may be recognized by comparing the registered fingerprints. The detection device 100 may compare the characteristics of the input fingerprint and the characteristics of the registered fingerprints.

Biometric authentication may consist of a matching process that checks whether the person attempting authentication has access rights and an anti-spoofing (ASP) process that determines whether biometric information is forged/falsified. In the spoofing process, since authentication is attempted by imitating, falsifying, or duplicating the user's biometric information, it is not easy to improve the speed and accuracy of spoofing detection.

For example, the input fingerprint image 115 senses a fake fingerprint, and the fingerprint pattern of the input fingerprint image 115 is matched with any one of the registered fingerprint images 121 , 122 , and 123 . In a similar case, there is a possibility that the authentication for the fake fingerprint is successful. The detection apparatus 100 may prevent spoofing by determining whether the input fingerprint shown in the input fingerprint image 115 is a forged fingerprint or a real fingerprint of a person. Here, 'theft' refers to fake biometric information rather than live biometric information, and may be understood to include, for example, duplication, forgery, and alteration of biometric information.

According to an embodiment, the detection device 100 itself may operate as an anti-spoofing device for detecting whether an input fingerprint is a forged fingerprint, or a separate anti-theft device may include Hereinafter, the anti-theft device may be referred to as a 'detection device'.

As will be described in detail below, the detection device 100 according to an embodiment uses a plurality of pre-equipped real fingerprint features, an unspecified number of pre-equipped counterfeit fingerprint features, and/or registered fingerprint features of a device user. It may be determined whether the input fingerprint is a forged fingerprint.

2 is a flowchart illustrating a method of detecting whether biometric information is stolen, according to an exemplary embodiment. In the following embodiment, each operation may be sequentially performed, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 2 , an apparatus for detecting whether biometric information is stolen (hereinafter, 'detecting device') according to an embodiment detects whether biometric information has been stolen through the processes of steps 210 to 230. The process is shown. The processes described below are, for example, after the matching process of checking whether the user who is attempting user authentication by biometric information in the biometric authentication process has access to the system, that is, whether it is a registered user. It is assumed that the description is performed, but is not necessarily limited thereto.

In step 210 , the detection apparatus extracts an embedding vector from an intermediate layer of a neural network that detects whether the biometric information is stolen from an image including the user's biometric information.

The detection apparatus may receive an image including biometric information sensed from the user by various sensor(s) and input it into a neural network that detects whether the biometric information is stolen. The neural network may be trained to extract features or feature vectors suitable for detecting whether biometric information is forged. The neural network may be learned using an unspecified number of real biometric information and an unspecified number of fake biometric information. The embedding vector extracted by the neural network embeds feature information suitable for detecting whether or not biometric information is forged, and the embedding vector may be referred to as a 'feature vector'. The neural network may be, for example, a deep neural network (DNN) shown in FIG. 3 below, or a convolutional neural network (CNN), but is not necessarily limited thereto.

The sensor(s) may include, for example, but not limited to, an ultrasonic fingerprint sensor, an optical fingerprint sensor, an electrostatic fingerprint sensor, a depth sensor, an iris sensor, an image sensor, and the like. As the sensor, any one of these may be used, or two or more may be used. The biometric information sensed by the sensor may be, for example, the input fingerprint image 115 shown in FIG. 1 , an iris image or a face image.

There may be a single image or a plurality of images including the user's biometric information. When there is a single image including the user's biometric information, one neural network is used, and when there are multiple images including the user's biometric information, a number of neural networks corresponding to the plurality of images may be used.

In order to infer a score for each image, a neural network may exist for each image, and there may be one or more embedding vectors extracted from each neural network.

The structure and operation of the detection device for detecting whether the biometric information is stolen from a single image including the user's biometric information will be described in more detail with reference to FIGS. 4 to 7 below. In addition, the structure and operation of the detection device for detecting whether the biometric information is stolen from a plurality of (eg, two) images including the user's biometric information will be described in more detail with reference to FIGS. 8 to 11 below. explained as

In step 220 , the detection device detects whether the biometric information is first spoofed based on the embedding vector extracted in step 210 . For example, the detection device infers whether or not the first thievery has been obtained by combining or ensembles the embedding vectors extracted in the same step, or according to a plurality of results of the individual embedding vectors or the inference results of the individual embedding vectors. It can be inferred whether The detection device may calculate the first score based on the embedding vector extracted in step 210 , for example. The detection device may detect whether the first stealing is performed based on the first score. Hereinafter, in the present specification, 'first stealing' may be understood as whether or not theft is detected by a score ('first score') calculated based on an embedding vector extracted from an intermediate layer of a neural network.

The first score may correspond to, for example, a similarity score calculated by a comparison result between authentication information (eg, a registration feature vector of a fingerprint image) included in a registration database and an embedding vector, but is not necessarily limited thereto. does not The first score is calculated by, for example, a pre-trained classifier or regression neural network, or by a Shallow Deep Neural Network (DNN) as shown in FIG. 4 below. may be calculated, but is not necessarily limited thereto.

In step 220, the detection device may determine whether the first score is a score that falls within a first threshold range for determining whether the first stealing. The first threshold range is, for example, a range in which biometric information is a score determined by stolen information, as in section 630 of FIG. 6 below, and a range and section 650 in which biometric information is live. It can be understood as a range that can be clearly distinguished as a score determined by information. The first threshold range includes a first threshold (eg, rejection threshold 620 ) corresponding to a maximum probability that the first score may be determined to be a score corresponding to stolen information, and a score in which the first score corresponds to actual information. It may be determined based on a second threshold (eg, the acceptance threshold 640 ) corresponding to the minimum probability that can be determined as . When the first score is determined to be a score that falls within the first threshold range, the detection device may detect whether the first steal is performed based on the first score.

In step 220, the detection apparatus may classify whether the first score is a score belonging to a range determined as the stolen information or a score belonging to a range determined as the actual information using a pre-trained classifier. When whether the first stealing is detected by the first score, the detection apparatus may output a classification result of a classifier ('first classifier').

According to an embodiment, the first classifier may determine whether to first steal based on a limited result of an embedding vector extracted from an intermediate layer. When it is difficult to determine with the given information due to high uncertainty in the determination of whether the first fraud has occurred, the first classifier suspends the determination of whether or not the fraud has been made and is based on the output vector output from the output layer following the intermediate layer in the neural network. Determination of whether or not theft is stolen may be delegated to a second classifier that determines whether or not to steal ('second steal or not'). The first classifier may be referred to as an 'Early Decision Classifier (EDC)' in that it classifies whether the first classifier is stolen early by the embedding vector extracted from the intermediate layer prior to the output layer.

The number of first classifiers may vary according to how many levels the detection apparatus performs early determination in the intermediate layer. For example, when the detection apparatus performs early determination over one level of the intermediate layer, the number of first classifiers may be singular. Also, when the detection apparatus performs early determination over a plurality of levels of the intermediate layer, the number of first classifiers may be plural.

Unlike a binary classifier that generally classifies Accept or Reject with one threshold, for example, as shown in FIG. 6 below, the first classifier has two thresholds (rejection threshold). Through 620 and acceptance threshold 640), it can be divided into three classes: Accept, Reject, and Not Decide.

For example, if the first score is greater than the acceptance threshold 640 , the detection device may determine the image as real information, and if the first score is less than the rejection threshold 620 , the detection device may determine the image as stolen information. If the first score is greater than or equal to the rejection threshold 620 and less than or equal to the acceptance threshold 640 , the detection device determines whether to steal based on the second score calculated from the output vector rather than the first score. can

In step 230 , the detection device detects whether the biometric information is second stolen by using an output vector output from the output layer of the neural network according to whether the first theft is detected in step 220 . For example, when the first theft is not detected, the detection apparatus may detect the second theft based on the second score based on the output vector. The second score may correspond to, for example, a similarity score calculated by a comparison result between authentication information (eg, a registration feature vector of a fingerprint image) included in the registration database and an output vector, but is not necessarily limited thereto. does not Hereinafter, in the present specification, 'second theft' may be understood as whether or not theft is detected by a score ('second score') calculated based on an output vector output from an output layer of a neural network.

In step 230 , the detection apparatus may classify whether the second score is a score that belongs to a range determined as the stolen information or a score that belongs to a range determined as the real information by using a pre-trained second classifier. The second classifier may determine whether to steal by performing a binary decision of the actual information or the stolen information based on the second score.

3 is a diagram illustrating a structure of a neural network according to an embodiment. Referring to FIG. 3 , a schematic structure of a deep neural network (DNN) 300 corresponding to an example of a neural network is shown. The deep neural network 300 may be trained through deep learning.

The deep neural network 300 may include a plurality of layers 310 , 320 , and 330 including a plurality of nodes. The deep neural network 300 may include connection weights for connecting a plurality of nodes included in each of the plurality of layers 310 , 320 , and 330 to nodes included in another layer. The detection device obtains the deep neural network 300 from an internal database stored in a memory (for example, see the memory 1290 of FIG. 12 below), or a communication interface (for example, see the communication interface 1250 of FIG. 12) It can be obtained by receiving it from an external server through

For example, the deep neural network 300 may include a large number of artificial neurons connected by line-shaped edges. The artificial neuron is represented by a circular figure in FIG. 3 and may be referred to as a node. Artificial neurons may be interconnected through edges having a connection weight. The connection weight is a specific value of the edges, and may be referred to as a synapse weight or a connection strength.

The deep neural network 300 may include, for example, an input layer 310 , a hidden layer 320 , and an output layer 330 . The input layer 310 , the hidden layer 320 , and the output layer 330 may include a plurality of nodes. A node included in the input layer 110 is called an input node, a node included in the hidden layer 320 is called a hidden node, and a node included in the output layer 330 is It may be referred to as an output node.

The input layer 310 may receive an input for performing training or recognition and transmit it to the hidden layer 320 . The output layer 330 may generate an output of the deep neural network 300 based on a signal received from the hidden layer 320 . The hidden layer 320 is located between the input layer 310 and the output layer 330 and may change the training input of the training data transmitted through the input layer 310 to a value that is easy to predict. The input nodes included in the input layer 310 and the hidden nodes included in the hidden layer 320 may be connected to each other through connection lines having a connection weight. Hidden nodes included in the hidden layer 320 and output nodes included in the output layer 330 may be connected to each other through connection lines having a connection weight.

The hidden layer 320 may include a plurality of layers 320-1, ..., 320-N. For example, assuming that the hidden layer 320 includes the first hidden layer, the second hidden layer, and the third hidden layer, the output of the hidden node belonging to the first hidden layer is the second hidden layer. It can be connected to hidden nodes. Outputs of hidden nodes belonging to the second hidden layer may be connected to hidden nodes belonging to the third hidden layer.

The detection apparatus may input outputs of previous hidden nodes included in the previous hidden layer to the corresponding hidden layer through connection lines having a connection weight. The detection apparatus may generate outputs of hidden nodes included in the hidden layer based on values to which a connection weight is applied to outputs of previous hidden nodes and an activation function. According to an example, when the result of the activation function exceeds the threshold value of the current hidden node, the output may be ignited to the next hidden node. In this case, the current hidden node may maintain an inactive state without firing a signal to the next hidden node until a specific threshold activation intensity is reached through the input vectors.

The hidden layer 320 may be referred to as an 'intermediate layer' in that it is located in the middle between the input layer 310 and the output layer 330 . Hereinafter, 'hidden layer' and 'intermediate layer' may be understood to have the same meaning.

The detection apparatus according to an embodiment may train the deep neural network 300 through supervised learning. The detection device may be implemented as a software module, a hardware module, or a combination thereof. Supervised learning is a technique of inputting a training input of training data and a training output corresponding thereto to the deep neural network 300, and updating the connection weights of the connection lines so that output data corresponding to the training output of the training data is output. Training data is data comprising a pair of training input and training output.

3 is a diagram of a deep neural network 300 Although the structure is expressed as a node structure, embodiments are not limited to such a node structure. Various data structures may be used to store the neural network in memory.

According to an embodiment, the detection apparatus may determine the parameters of the nodes through a gradient descent technique based on the loss back propagated to the deep neural network 300 and output values of the nodes included in the deep neural network 300 . have.

For example, the detection apparatus may update a connection weight between nodes through lossy back-propagation learning. Loss back propagation learning, after estimating the loss through forward computation for the given training data, starts from the output layer 330 and is estimated in the reverse direction toward the hidden layer 320 and the input layer 110 . While propagating a loss, it is a method of updating the connection weight in the direction of reducing the loss.

The processing of the deep neural network 300 proceeds in the direction of the input layer 310, the hidden layer 320, and the output layer 330, but the update direction of the connection weights in the lossy backpropagation training is the output layer 330, the hidden layer 330 It may proceed in the direction of the layer 320 and the input layer 310 . One or more processors may use a buffer memory (buffer) to store a layer or a series of computational data in order to process the neural network in a desired direction.

The detection apparatus may define an objective function for measuring how close to optimal the currently set connection weights are, continuously change the connection weights based on the result of the objective function, and repeatedly perform training. For example, the objective function may be a loss function for calculating a loss between an output value actually output by the deep neural network 300 based on a training input of training data and an expected value desired to be output. The detection apparatus may update the connection weights in a direction to decrease the value of the loss function.

The deep neural network 300 according to an embodiment is trained to determine real information or stolen information through the network, and the output of the final output layer is trained to derive an optimal result, but in the training process, in addition to the output layer, the network is configured Each intermediate layer may also have the ability to discriminate between real information and stolen information. Since the output of the intermediate layer may also have a distinguishing power, if it is used, it is possible to determine whether the biometric information has been stolen before going to the final output layer 330 , thereby reducing the execution time.

The detection apparatus according to an embodiment may improve the detection speed of whether the theft is detected while minimizing a decrease in the accuracy of the detection of theft by utilizing the intermediate layer having the discrimination power in the stage before the deep neural network 300 derives the final result. In addition, the detection apparatus may minimize the decrease in accuracy by using the results of networks that receive different images as inputs to compensate for the decrease in accuracy due to the use of the output of the intermediate layer.

4 is a diagram for explaining a network structure and operation of an apparatus for detecting whether or not biometric information is stolen according to an embodiment, and FIG. 5 is a method for detecting whether or not biometric information is first stolen according to an embodiment It is a flow chart.

4 and 5 , for example, a 1-1 classifier 440 configured of shallow deep neural networks (Shallow DNN), and a 1-2 classifier 450 and a second classifier 460 are included. The structure and operation of the detection device 400 is shown.

Unlike conventional classifiers of deep neural networks having an end-to-end structure, the 1-1 classifier 440 and the 1-2 classifier 450 according to an embodiment are It is possible to classify whether or not biometric information is stolen from embedding vectors extracted from intermediate layers of the deep neural network 430 while receiving an image 410 including information and performing network inference.

The embedding vectors extracted from the intermediate layers of the deep neural network 430 may be applied to the 1-1 classifier 440 and the 1-2 classifier 450 and used to detect whether the biometric information is first stolen. have. The 1-1 classifier 440 and the 1-2 classifier 450 may be classifiers trained to classify an input image based on embedding vectors. The 1-1 classifier 440 and the 1-2 classifier 450 may be composed of, for example, shallow deep neural networks (Shallow DNN), which require less computation than a deep neural network, and may include early judgment ( Since the overhead due to Early Decision (ED) is small, it is possible to quickly detect whether the first robbery is performed without slowing down the speed.

An example of how the first-first classifier 440 and the first-second classifier 450 detect whether the first thievery is stolen is as follows.

The detection apparatus 400 may extract an embedding vector (eg, a 1-1 embedding vector) from an intermediate layer of the deep neural network 430 using the first-first classifier 440 . In operation 510 , the detection apparatus 400 may calculate a 1-1 score based on the extracted 1-1 embedding vector. The detection apparatus 400 may detect whether the first steal is performed based on the 1-1 score. In step 520 , the detection device 400 determines whether the 1-1 score is within a first threshold range for determining whether the first robbery using the pre-trained 1-1 classifier 440 . can decide The detection apparatus 400 may classify whether the 1-1 score is a score that belongs to a range determined as the first stolen information or a score that belongs to a range determined as the real information.

For example, when the 1-1 score is determined to be a score that falls within the first threshold range, in step 530 , the detection device may detect whether the first robbery is performed based on the 1-1 score.

If the first robbery is not detected by the 1-1 score, in step 540 , the detection device 400 uses the 1-2 classifier 450 to detect the middle layer of the deep neural network 430 . Then 1-2 embedding vectors can be extracted from the intermediate layer.

In operation 550 , the detection apparatus 400 may calculate a 1-2 score based on the 1-2 embedding vector.

In operation 560 , the detection apparatus may detect whether the first robbery is based on the 1-2th score calculated in operation 550 . In step 560 , the detection device 400 determines whether the 1-2 score is a score that falls within a first threshold range for determining whether the first stealing is performed using the pre-trained 1-2 classifier 450 . can decide The detection apparatus 400 may classify whether the 1-2 score is a score that belongs to a range determined as the first stolen information or a score that belongs to a range determined as the real information. For example, when the 1-2 score is determined to be a score that falls within the first threshold range, the detection device may detect whether the first robbery is based on the 1-2 score.

The detection apparatus 400 may output a classification result of the 1-2 classifier 450 when whether or not the first theft is detected based on the 1-2 score. Otherwise, when the first stealing is not detected by the 1-2 score, the detection apparatus 400 applies an output vector output from the output layer of the deep neural network 430 to the second classifier 460 to It is possible to detect whether the information is second stolen.

The detection apparatus 400 detects whether the biometric information is first stolen through the first-first classifier 440 and the first-second classifier 450 that perform early decision classification before deriving the output vector from the output layer. In this case, since it is possible to immediately detect whether the biometric information has been stolen without using an output vector, the time for determining whether the biometric information has been stolen can be shortened.

When determining whether biometric information has been stolen, the accuracy of the determination and the speed of detecting whether the biometric information has been stolen may correspond to a trade-off relationship. The detection apparatus 400 uses the 1-1 classifier 440 and the 1-2 classifier 450 for quick determination, but When the detection reliability is high, whether the first stealing is directly used, but when the detection reliability is low, the second stealing can be determined from the output vector by the second classifier 460 .

6 is a diagram illustrating a first threshold range according to an embodiment. Referring to FIG. 6 , a graph 601 showing the probability distribution of the first score corresponding to the stolen information and a graph 603 showing the probability distribution of the first score corresponding to the actual information are shown.

For example, if the first score calculated based on the embedding vector can clearly determine whether the biometric information falls within a range determined as real information or whether the biometric information falls within a range determined as stolen information , the detection device may directly detect whether or not theft is stolen based on the first score. On the other hand, when the first score cannot clearly determine whether the biometric information belongs to the range determined by the actual information, or whether the biometric information belongs to the range determined by the stolen information, the detection device is spoofed by the first score It is not possible to immediately decide whether

In an embodiment, the first threshold range may correspond to a probability range in which it is possible to clearly determine whether the first score is stolen. The first threshold range is for discriminating a first score for which a fraudulent or not unambiguous determination is made. When the first score does not immediately determine whether to steal, the detection device may determine whether to steal based on the second score.

In the graph 601 , the Reject Threshold 620 corresponds to the maximum value (Max (Spoof Score)) of the probability that the first score can be determined as belonging to a range in which biometric information is clearly determined to be stolen information. can do. In addition, in the graph 603, the Accept Threshold 640 is the minimum value (Min (Live Score)) of the probability that the first score can be determined as belonging to the range in which the biometric information is clearly determined to be real information. may be applicable.

The first threshold range includes a rejection threshold 620 corresponding to the maximum probability (Max (Spoof Score)) that the first score is determined to be stolen information, and the minimum probability (Min (Live Score)) that the first score is determined to be real information. may be determined based on the acceptance threshold 640 corresponding to . The first threshold range may correspond to a section 610 larger than the rejection threshold 620 in the graph 601 and smaller than the acceptance threshold 640 in the graph 603 . When the first score in the graph 601 belongs to the section 630 that is less than or equal to the rejection threshold 620 , the first score may be clearly determined as 'theft information'. Also, when the first score in the graph 603 belongs to the section 650 that is greater than or equal to the acceptance threshold 640 , the first score may be clearly determined as 'real information'.

When the first score belongs to the interval 630 and the interval 650 , the detection device determines that the first score falls within the first threshold range, and whether the first score is stolen directly by the first score ('first stolen or not') can be detected. On the other hand, if the first score belongs to the section 610 , the detection device determines that the first score does not belong to the first threshold range, and whether the first score is stolen by the second score calculated from the output vector ('second theft or not) ') can be detected.

7 is a diagram for describing a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to another embodiment. Referring to FIG. 7 , whether or not biometric information is stolen is not detected by the first score (eg, 1-1 score) of the embedding vector (eg, 1-1 embedding vector) extracted from the intermediate layer. In this case, detection of detecting whether biometric information is stolen by fusing the 1-1 score and the 1-2 score of the embedding vector (eg, 1-2 embedding vector) extracted from the next intermediate layer after the corresponding intermediate layer The structure of device 700 is shown.

When the image 701 including the user's biometric information is input to the deep neural network 710 , the detection device 700 performs 1-1 based on the 1-1 embedding vector extracted from the intermediate layer of the deep neural network 710 . score can be calculated. The detection apparatus 700 may detect whether the first thievery is performed based on the 1-1 score. The detection apparatus 700 may classify whether the 1-1 score belongs to a range determined as the stolen information or a score that belongs to a range determined as the real information using the 1-1 classifier 720 .

For example, when whether the first stealing is not detected by the 1-1 score, the detection apparatus 700 may extract the 1-2 th embedding vector from the next intermediate layer after the intermediate layer. After calculating the 1-2 score based on the 1-2 embedding vector, the detection device 700 may fuse the 1-1 score and the 1-2 score.

The detection device 700 may detect whether the first robbery is based on the fused score. The detection device 700 detects whether the first fraud is performed by classifying whether the score fused by the 1-2 classifier 730 is a score that belongs to a range determined as the stolen information or a score that belongs to a range determined as the actual information. can do.

When whether the first thievery is not detected by the 1-2 classifier 730 , the detection apparatus 700 may detect the second robbery from the output vector of the deep neural network 710 . The detection apparatus 700 may detect whether the second thievery is performed from the output vector of the deep neural network 710 by using the second classifier 740 .

8 is a diagram for explaining a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to another embodiment. Referring to FIG. 8 , a schematic structure of a detection apparatus 800 for detecting whether biometric information is stolen by extracting embedding vectors from two different images 801 and 803 is shown. The detection apparatus 800 may detect whether the biometric information is stolen by utilizing dynamics between the different images 801 and 803 .

The detection device 800 determines whether the biometric information is real information or stolen information by using intermediate layers of two or more networks (eg, the first neural network 810 and the second neural network 830) can do. The first neural network 810 and the second neural network 830 may be, for example, a convolutional neural network (Conv).

If reliability is not secured in the classifier 840 for classifying a score based on the embedding vector extracted from the intermediate layer of the first neural network 810 and the second neural network 830, the detection device determines whether the The determination of whether to steal ') may be withheld, and whether the first neural network 810 is output from the second neural network 830 may determine whether or not to steal ('second theft or not"). The classifier 840 may include, for example, a fully connected layer (FC (Fully-Connected) layer), but is not limited thereto.

The detection apparatus 800 may extract one or more 1-1 embedding vectors from one or more first intermediate layers of the first neural network 810 that detects whether biometric information is stolen from the first image 801 .

In addition, the detection apparatus 800 may extract one or more 1-2 embedding vectors from one or more second intermediate layers of a second neural network 830 that detects whether biometric information is stolen from the second image 803 . have.

In this case, the second image 803 captures the user's biometric information at a different time than the first image 801 , such as an image obtained in a matching process for checking whether the user attempting authentication has access rights. image, or an image generated through nonlinear image processing on the first image 801 , but is not limited thereto.

The detection apparatus 800 may calculate a 1-3 score based on the 1-1 embedding vector and the 1-2 embedding vector, and detect whether the biometric information is first stolen based on the 1-3 score. . The detection apparatus 800 may calculate a 1-3 score by fusing or combining the 1-1 embedding vector and the 1-2 embedding vector. The detection device 800 classifies whether the 1-3 score is a score that belongs to a range determined as the stolen information or a score that belongs to a range determined as the actual information using a pre-trained 1-3 classifier to classify the first thievery. whether or not it can be detected. When whether the first theft is detected based on the 1-3 score, the detection apparatus 800 may end the operation.

Contrary to this, when the first thievery is not detected by the 1-3 score, the detection apparatus 800 obtains from the output layers of the first neural network 810 and the second neural network 830). Based on the output vectors, it is possible to detect whether the second theft is done. For example, the detection apparatus 800 may obtain a first output vector from a first output layer of the first neural network 810 . The detection apparatus 800 may obtain a second output vector from a second output layer of the second neural network 830 . The detection apparatus 800 may detect whether the second is stolen based on the first output vector and the second output vector. The detection apparatus 800 may detect whether the biometric information is second stolen based on the third score based on the first output vector and the second output vector. The detection apparatus 800 may calculate the third score by giving different weights to each of the score based on the first output vector and the score based on the second output vector.

9 is a diagram for describing a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to another embodiment. Referring to FIG. 9 , a detection apparatus 900 for detecting whether biometric information is stolen by extracting embedding vectors from two different images 901 and 903 is illustrated.

The detection apparatus 900 determines whether the biometric information is real information or stolen information by using intermediate layers of two or more networks (eg, the first neural network 910 and the second neural network 930) can do.

The detection apparatus 900 may extract one or more 1-1 embedding vectors from one or more first intermediate layers of the first neural network 910 that detects whether biometric information is stolen from the first image 901 .

In addition, the detection apparatus 900 may extract one or more first-2 embedding vectors from one or more second intermediate layers of a second neural network 930 that detects whether biometric information is stolen from the second image 903 . have. In this case, the second image 903 is, for example, an image obtained by capturing the user's biometric information at a different time from that of the first image 901 by a sensor, or is generated through image processing on the first image 901 . may be an image, but is not necessarily limited thereto.

The detection device 900 calculates a 1-3 score by element-wise summing the 1-1 embedding vector and the 1-2 embedding vector, for example, and calculates a 1-3 score based on the 1-3 score. Whether the biometric information is first stolen may be detected. The detection apparatus 900 may determine, by the first classifier 950 , whether the 1-3 score is a score that falls within a first threshold range Th for determining whether the first stealing is performed. When the 1-3 score is determined to be a score that falls within the first threshold range, the detection apparatus 900 may detect whether the first robbery is performed based on the 1-3 score, and end the subsequent operation.

When the 1-3 score is determined to be a score that does not fall within the first threshold range, the detection apparatus 900 extracts a new 1-1 embedding vector from the next intermediate layer of the first neural network 910, and the second neural network 910 A new 1-2 embedding vector may be extracted from the next intermediate layer of the network 930 .

The detection apparatus 900 calculates a new 1-3 score by adding the 1-1 embedding vector and the 1-2 embedding vector extracted from the next intermediate layers for each element, and biometric information based on the new 1-3 score It is possible to detect whether the first theft of The detection apparatus 900 may determine whether the new 1-3 score is a score that falls within the first threshold range Th for determining whether the first stealing is performed by the second classifier 970 . As a result of the classification of the second classifier 970 , that is, when the first stealing is detected based on the new 1-3 score, the detection apparatus 900 may end the subsequent operation.

On the other hand, when the first thievery is not detected by the new 1-3 score, the detection apparatus 900 obtains from output layers of the first neural network 910 and the second neural network 930 ). Based on the one output vectors, it is possible to detect whether the second theft is done.

The detection apparatus 900 may detect whether the second thievery is performed by using the first output vector of the output layer of the first neural network 910 and the second output vector of the output layer of the second neural network 930 .

10 is a diagram for explaining a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to another embodiment. Referring to FIG. 10 , by combining the results obtained by classifying embedding vectors extracted from two different images 1001 and 1003 by different classifiers 1050 , 1060 , and 1070 , detection of whether biometric information is stolen or not. An apparatus 1000 is shown.

The detection apparatus 1000 may extract one or more 1-1 embedding vectors from one or more first intermediate layers of the first neural network 1010 for detecting whether biometric information is stolen from the first image 1001 .

The detection apparatus 1000 may extract one or more first-second embedding vectors from one or more second intermediate layers of the second neural network 1030 that detects whether biometric information is stolen from the second image 1003 .

The detection apparatus 1000 may generate the 1-3 th embedding vector based on the 1-1 th embedding vector and the 1-2 th embedding vector. The detection apparatus 1000 may, for example, simply combine the 1-1 embedding vector and the 1-2 embedding vector, or add the 1-1 embedding vector and the 1-2 embedding vector element-by-element. wise sum) to generate the 1-3 embedding vector.

The detection apparatus 1000 may calculate a 1-1 score based on the 1-1 embedding vector and may calculate a 1-2 th score based on the 1-2 embedding vector. Also, the detection apparatus 1000 may calculate a 1-3 score based on the 1-3 embedding vector.

The detection apparatus 1000 may detect whether the biometric information is first stolen based on the 1-1 score, the 1-2 score, and the 1-3 score.

The detection apparatus 1000 determines whether the 1-1 score is within a range determined as the stolen information or is real information by using the pre-trained first classifier (eg, the 1-1 classifier) 1050 . Whether the score falls within the range of 1-1 can be classified.

The detection apparatus 1000 determines whether the 1-3 score is within a range determined as the stolen information or as actual information by using the second classifier (eg, the 1-3 classifier) 1060 trained in advance. Whether it is a score that falls within the range can be classified into Sections 1-3.

The detection apparatus 1000 determines whether the 1-2th score is a score belonging to a range determined as stolen information or actual information using a third classifier (eg, 1-2 classifier) 1070 trained in advance. It is possible to classify the score in the first-2 range.

The detection apparatus 1000 may detect whether the first robbery is performed using the classification results (eg, the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result) of each classifier. .

The detection apparatus 1000 detects whether the first stealing is performed by a weighted sum or a majority vote 1080 of the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result. can do. For example, the detection apparatus 1000 may detect whether the first robbery is performed based on a weighted sum of the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result. In this case, the weight for each classification result is, for example, the highest first weight for the 1-1 classification result corresponding to the most recently acquired first image 1001 according to whether the time at which the image is acquired is preceded or not. is assigned, and the lowest third weight may be given to the results of the 1-2 classification corresponding to the previously acquired second image 1003 . Also, a second weight smaller than the first weight and greater than the third weight may be given to the results of classification 1-3 corresponding to the first image 1001 and the second image 1003 .

Alternatively, the detection apparatus 1000 may detect whether the first robbery is performed according to a majority vote based on the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result. For example, it is assumed that the 1-1 classification result is stolen information, the 1-2 classification result is real information, and the 1-3 classification result is real information. In this case, the detection apparatus 1000 may detect 'real information' corresponding to a majority vote among the theft information: 1 and the actual information: 2 as the first theft. Accordingly, the detection apparatus 1000 may output whether the first image 1001 and the second image 1003 have been stolen as 'real information'.

When the detection apparatus 1000 cannot detect whether the first robbery is detected using the classification result of each classifier, the first output vector of the output layer of the first neural network 1010 and the output of the second neural network 1030 Whether the second thievery is detected may be performed using the second output vector of the layer.

11 is a diagram for describing a network structure and operation of an apparatus for detecting whether biometric information is stolen, according to another embodiment. Referring to FIG. 11 , a detection apparatus 1100 using three embedding vectors for each of two images 1101 and 1103 and a first neural network 1110 and a second neural network 1130 is shown.

The first image 1101 is, for example, an image corresponding to a raw input of the sensor, and the second image 1103 is, for example, removing noise from the first image 1101 and/or The image may be an image obtained by performing non-linear image processing on the first image 1101 such as improving a signal to noise ratio (SNR) of the first image 1101 . The first image 1101 may be input to the first neural network 1110 , and the second image 1103 may be input to the second neural network 1130 .

The detection apparatus 1100 may operate by receiving embedding vectors extracted from intermediate layers of the neural networks 1110 and 1130 for each image.

The detection apparatus 1100 may include a plurality of classifiers corresponding to each level according to which level (position) of the intermediate layer of each neural network to extract the embedding vector from. Here, 'level' may be understood to mean a layer or an arrangement order of the layers.

The detection apparatus 1100 may include a 1-1 classifier 1140-1, a 1-2 classifier 1140-2, and a 1-3 classifier 1140-3 corresponding to the first level. . The detection apparatus 1100 may include a 2-1 classifier 1150-1, a 2-2 classifier 1150-2, and a 2-3 classifier 1150-3 corresponding to the second level. . The detection apparatus 1100 may include a 3-1 classifier 1160-1, a 3-2 classifier 1160-2, and a 2-3 classifier 1160-3 corresponding to the third level. .

For example, the 1-1 classifier 1140-1, the 1-2 classifier 1140-2, and the 1-3 classifier 1140-3 are intermediate layers of the neural networks 1110 and 1130. It can operate by receiving scores based on embedding vectors extracted from the first level (level 1). The 1-1 classifier 1140-1 uses the 1-1 score of the 1-1 embedding vector extracted from the first level among the intermediate layers of the first neural network 1110 according to the acceptance threshold and the rejection threshold, as actual information or theft. information can be classified. The 1-2 classifier 1140-2 uses the 1-2 score of the 1-2 embedding vector extracted from the intermediate layer of the first level of the second neural network 1130 according to the acceptance threshold and the rejection threshold to steal actual information or theft. information can be classified.

The detection apparatus 1100 may generate the 1-3 th embedding vector based on the 1-1 th embedding vector and the 1-2 th embedding vector. The detection device 1100 may, for example, simply combine the 1-1 embedding vector and the 1-2 embedding vector, or add the 1-1 embedding vector and the 1-2 embedding vector element by element to perform the first- 3 You can create an embedding vector. The 1-3 classifier 1140-3 may classify the 1-3 score of the 1-3 embedding vector as real information or stolen information according to the acceptance threshold and rejection threshold.

The detection apparatus 1100 may detect whether the biometric information is first stolen at the first level by synthesizing the classification results for the 1-1 score, the 1-2 score, and the 1-3 score. The detection apparatus 1100 may detect, for example, whether the first robbery has been stolen according to a classification result corresponding to a plurality of classification results for the 1-1 score, the 1-2 score, and the 1-3 score. .

When whether the first stealing is detected in the first level, the detection apparatus 1100 may prematurely end the operation without performing a subsequent process. Contrary to this, when it is not detected whether the first theft is at the first level, the detection apparatus 1100 may detect the first theft at the second level.

The 2-1 classifier 1150-1, the 2-2 classifier 1150-2, and the 2-3 classifier 1150-3 are the second level ( It can operate by receiving scores of embedding vectors extracted from level 2). The 2-1 classifier 1150-1, the 2-2 classifier 1150-2, and the 2-3 classifier 1150-3 are the aforementioned 1-1 classifiers 1140-1 and first classifiers, respectively. Like the -2 classifier 1140-2 and the 1-3 classifier 1140-3, the 2-1 embedding extracted from the second level among the intermediate layers of the first neural network 1110 and the second neural network 1130 to classify the 2-1 score of the vector, the 2-2 score of the 2-2 embedding vector, and the 2-3 score of the 2-3 embedding vector as real information or spoof information according to the acceptance threshold and rejection threshold. can The detection apparatus 1100 may detect whether the biometric information is first stolen at the second level by synthesizing the classification results for each of the 2-1 score, the 2-2 score, and the 2-3 score. When the first stealing is detected in the second level, the detection apparatus 1100 may prematurely end the operation without performing a subsequent process. Contrary to this, when the first stealing is not detected in the second level, the detection apparatus 1100 may then detect the first stealing in the third level.

The 3-1 classifier 1160-1, the 3-2 classifier 1160-2, and the 2-3 classifier 1160-3 are the third level ( It can operate by receiving scores of embedding vectors extracted from level 3).

The 3-1 classifier 1160-1, the 3-2 classifier 1160-2, and the 2-3 classifier 1160-3 are the aforementioned 2-1 classifier 1150-1 and second classifier, respectively. Like the -2 classifier 1150 - 2 and the 2-3 classifier 1150 - 3 , the 3-1 embedding extracted from the third level among the intermediate layers of the first neural network 1110 and the second neural network 1130 . to classify the 3-1 score of the vector, the 3-2 score of the 3-2 embedding vector, and the 3-3 score of the 3-3 embedding vector as real information or spoof information according to the acceptance threshold and rejection threshold. can The detection apparatus 1100 may detect whether the biometric information is first stolen at the third level by synthesizing the classification results for each of the 3-1 score, the 3-2 score, and the 3-3 score. When whether the first stealing is detected in the third level, the detection apparatus 1100 may prematurely end the operation without performing a subsequent process.

Classifiers 1140-1, 1140-2, 1140-3, 1150-1, 1150-2 for classifying scores of embedding vectors extracted from intermediate layers of each neural network 1110 and 1130 in the detection apparatus 1100 , 1150-3, 1160-1, 1160-2, 1160-3) can be designed as a simple network, for example, as the shallow deep neural network described above. The detection device 1100 is configured by the classifiers 1140-1, 1140-2, 1140-3, 1150-1, 1150-2, 1150-3, 1160-1, 1160-2, and 1160-3 of each level. When it is certain whether the information is actual information or the stolen information, it is possible to quickly determine whether the information is stolen ('the first stolen information') to improve the usability of the detection apparatus 1100 . The detection apparatus 1100 can operate without slowing down due to a small overhead due to an Early Decision (ED) in the intermediate layer even when the intermediate layer fails to determine whether the first stealing is made.

When whether the first stealing is not detected even in the third level, the detection apparatus 1100 outputs the first output vector output from the output layer of the first neural network 1110 and outputted from the output layer of the second neural network 1130 . Whether the biometric information is second stolen may be detected using the second output vector. The detection apparatus 1100 may detect whether the biometric information is second stolen by classifying a score based on the first output vector and the second output vector using the fourth classifier 1170 .

According to an embodiment, classifiers 1140-1, 1140-2, 1140-3, and 1150- for detecting whether or not the first stealing from an intermediate layer is performed in the first neural network 1110 and the second neural network 1130 1, 1150-2, 1150-3, 1160-1, 1160-2, 1160-3) and the classifier 1170 that detects whether the second steal is performed may be trained together or separately.

According to an embodiment, the first neural network 1110 and the second neural network 1130 do not use the embedding vector of the intermediate layer, but use the detection result of the second theft using the output vector of the output layer to perform a baseline (baseline) training can be performed. Thereafter, the baseline is fixed so that it is not trained any more, and an embedding vector is extracted from the fixed baseline, and classifiers 1140-1 and 1140- that detect whether the first stealing from the intermediate layer using the extracted embedding vector. 2, 1140-3, 1150-1, 1150-2, 1150-3, 1160-1, 1160-2, 1160-3) can be trained. This means that the classifiers 1140-1, 1140-2, 1140-3, 1150-1, 1150-2, 1150-3, 1160-1, 1160-2, and 1160-3 that detect whether the first steal in order not to affect the results of

12 is a block diagram of an apparatus for detecting whether biometric information is stolen, according to an exemplary embodiment. Referring to FIG. 12 , a device (“detection device”) 1200 for detecting whether biometric information is stolen according to an embodiment includes a sensor 1210 , a processor 1230 , a communication interface 1250 , and an output device 1270 . ), and a memory 1290 . The sensor 1210 , the processor 1230 , the communication interface 1250 , the output device 1270 , and the memory 1290 may be connected to each other through the communication bus 1205 .

The sensor 1210 captures an image including the user's biometric information. The sensor 1210 may include, for example, at least one of an ultrasonic fingerprint sensor, an optical fingerprint sensor, an electrostatic fingerprint sensor, a depth sensor, and an image sensor, but is not limited thereto. The biometric information may include, for example, any one of a user's fingerprint, iris, and face, but is not limited thereto.

The processor 1230 extracts an embedding vector from an intermediate layer of a neural network that detects whether biometric information is stolen from an image captured by the sensor 1210 . The processor 1230 detects whether the biometric information is first stolen based on the embedding vector. The processor 1230 detects whether the biometric information is second stolen by using an output vector output from the output layer of the neural network according to whether the first theft is detected. However, the operation of the processor 1230 is not limited as described above, and the processor 1230 may perform the above-described operation together with at least one of the operations described above with reference to FIGS. 1 to 11 .

The processor 1230 may be a hardware-implemented neural network or a detection device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, the hardware-implemented detection device may include a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, and a multi-core processor. It may include a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural processing unit (NPU), and the like.

The processor 1230 may execute a program and control the detection device 1200 . The program code executed by the processor 1230 may be stored in the memory 1290 .

The communication interface 1250 may output whether the biometric information detected by the processor 1230 is stolen (eg, whether the first theft and/or the second is stolen) to the outside of the detection device 1200 . Alternatively, the communication interface 1250 may transmit whether the biometric information detected by the processor 1230 has been stolen to another device together with the biometric information. In this case, whether the biometric information has been stolen and the corresponding biometric information may be in a form that matches each other.

Alternatively, according to an embodiment, the communication interface 1250 may receive an image captured by the user's biometric information from the outside of the detection device 1200 and transmit it to the processor 1230 .

The output device 1270 may output whether the biometric information detected by the processor 1230 has been stolen. The output device 1270 may include, for example, a display, an alarm, a speaker, or other output devices capable of notifying a user of whether biometric information is stolen.

The memory 1290 may store the user's biometric information sensed by the sensors 1210 , that is, static features related to the user's biometric information, and images obtained by photographing the biometric information. Also, the memory 1290 may store the first characteristic information and/or the second characteristic information obtained by the processor 1230 . The memory 1290 may store the first score, the second score, and the fused score calculated by the processor 1230 . Also, the memory 1290 may match whether the biometric information detected by the processor 1230 has been stolen with the corresponding biometric information and store it.

Also, the memory 1290 may store various pieces of information generated in the process of the processor 1230 described above. In addition, the memory 1290 may store various data and programs. Memory 1290 may include volatile memory or non-volatile memory. The memory 1290 may include a mass storage medium such as a hard disk to store various data.

13 is a block diagram of an electronic device according to an exemplary embodiment. Referring to FIG. 13 , an electronic device 1300 according to an embodiment may include a fingerprint sensor 1310 , a processor 1330 , an output device 1350 , and a memory 1370 . The fingerprint sensor 1310 , the processor 1330 , the output device 1350 , and the memory 1370 may be connected to each other through the communication bus 1305 .

The fingerprint sensor 1310 captures an image including user's fingerprint information. The fingerprint sensor 1310 may include, for example, at least one of an ultrasonic fingerprint sensor, an optical fingerprint sensor, and an electrostatic fingerprint sensor. There may be a single fingerprint sensor 1310 or a plurality of fingerprint sensors 1310 .

The processor 1330 extracts one or more embedding vectors from an intermediate layer of a neural network that detects whether fingerprint information is stolen from an image captured by the fingerprint sensor 1310 . The processor 1330 detects whether the fingerprint information is first stolen based on one or more embedding vectors. The processor 1330 detects whether the fingerprint information is second stolen by using an output vector output from an output layer of the neural network according to whether the first theft is detected. However, the operation of the processor 1330 is not limited thereto, and the processor 1330 may perform the above-described operation together with at least one of the operations described above with reference to FIGS. 1 to 12 .

The output device 1350 outputs at least one of whether the processor 1330 detects the first stolen or the second stolen. The output device 1350 may output whether fingerprint information is stolen to the inside or outside of the detection device. The output device 1350 may include, for example, a display, an alarm, a speaker, or other various types of output devices capable of notifying a user of whether biometric information has been stolen.

The memory 1370 may store images obtained by photographing the user's fingerprint information captured by the fingerprint sensor 1310 . Also, the memory 1370 may store an embedding vector and/or an output vector extracted by the processor 1330 . The memory 1370 may match whether or not the fingerprint information detected by the processor 1330 has been stolen with the corresponding fingerprint information and store it.

Also, the memory 1370 may store various pieces of information generated in the process of the above-described processor 1330 . In addition, the memory 1370 may store various data and programs. The memory 1370 may include a volatile memory or a non-volatile memory. The memory 1370 may include a mass storage medium such as a hard disk to store various data.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using a general purpose computer or special purpose computer. The processing device may execute an operating system (OS) and a software application running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. may be Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, a person skilled in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (46)

extracting an embedding vector from an intermediate layer of a neural network that detects whether the biometric information has been stolen from an image including the user's biometric information;
detecting whether the biometric information is first stolen based on the embedding vector; and
detecting whether the biometric information is second stolen using an output vector output from an output layer of the neural network according to whether the first theft is detected
A method of detecting whether biometric information has been stolen, including. According to claim 1,
The step of detecting whether the first theft is
calculating a first score based on the embedding vector; and
Detecting whether the first theft is based on the first score
A method of detecting whether biometric information has been stolen, including. 3. The method of claim 2,
The step of detecting whether the first theft is based on the first score
determining whether the first score is a score that falls within a first threshold range for determining whether the first stealing; and
detecting whether the first stealing is based on the first score when the first score is determined to be a score falling within the first threshold range;
A method of detecting whether biometric information has been stolen, including. 4. The method of claim 3,
The first threshold range is
A first threshold corresponding to a maximum probability that the first score can be determined to be a score corresponding to stolen information and a second threshold corresponding to a minimum probability that the first score can be determined to be a score corresponding to actual information A method of detecting whether biometric information is stolen, determined based on the method. 3. The method of claim 2,
The step of detecting whether the first theft is based on the first score
Classifying whether the first score is a score belonging to a range determined as stolen information or a score belonging to a range determined as actual information using a pre-trained classifier
A method of detecting whether biometric information has been stolen, including. 6. The method of claim 5,
outputting a classification result of the classifier when whether the first stealing is detected by the first score
Further comprising, a method of detecting whether biometric information is stolen. 3. The method of claim 2,
If the first theft is not detected by the first score,
extracting a 1-2 embedding vector from a next intermediate layer after the intermediate layer;
calculating a 1-2 score based on the 1-2 embedding vector; and
Detecting whether the first stealing is performed based on the 1-2 score
Further comprising, a method of detecting whether biometric information is stolen. 8. The method of claim 7,
The step of detecting whether the first stealing is performed by the 1-2 score
Classifying whether the 1-2 score is a score that belongs to a range determined as stolen information or a score that belongs to a range determined as real information using a pre-trained 1-2 classifier
A method of detecting whether biometric information has been stolen, including. 3. The method of claim 2,
If the first theft is not detected by the first score,
extracting a 1-2 embedding vector from a next intermediate layer after the intermediate layer;
calculating a 1-2 score based on the 1-2 embedding vector;
fusing the first score and the 1-2 score; and
Detecting whether the first theft is performed by classifying whether the fused score is a score belonging to a range determined as the stolen information or a score belonging to a range determined as the actual information
Further comprising, a method of detecting whether biometric information is stolen. According to claim 1,
The step of detecting whether the second theft is
detecting whether the second stealing is based on a second score based on the output vector when the first stealing is not detected;
A method of detecting whether biometric information has been stolen, including. According to claim 1,
the video is
and a first image and a second image including the user's biometric information,
The step of extracting the embedding vector is
extracting one or more 1-1 embedding vectors from one or more first intermediate layers of a first neural network for detecting whether the biometric information is stolen from the first image; and
extracting one or more 1-2 embedding vectors from one or more second intermediate layers of a second neural network for detecting whether the biometric information is stolen from the second image
A method for detecting whether biometric information is stolen, including. 12. The method of claim 11,
The second image is
An image obtained by capturing the user's biometric information at a different time from the first image by a sensor, or an image generated through image processing on the first image
A method of detecting whether biometric information is stolen, including. 12. The method of claim 11,
The step of detecting whether the first theft is
calculating a 1-3 score based on the 1-1 embedding vector and the 1-2 embedding vector; and
Detecting whether the biometric information is first stolen based on the 1-3 score
A method of detecting whether biometric information has been stolen, including. 14. The method of claim 13,
Detecting whether the biometric information is first stolen based on the 1-3 score includes:
determining whether the 1-3 score is a score that falls within a first threshold range for determining whether the first stealing is performed; and
detecting whether or not the first stealing is performed based on the 1-3 score when the 1-3 score is determined to be a score that falls within the first threshold range;
A method of detecting whether biometric information has been stolen, including. 14. The method of claim 13,
The step of detecting whether the second theft is
If the first theft is not detected by the 1-3 score,
obtaining a first output vector from a first output layer of the first neural network;
obtaining a second output vector from a second output layer of the second neural network; and
detecting whether the second stealing is based on the first output vector and the second output vector
A method of detecting whether biometric information has been stolen, including. 16. The method of claim 15,
Detecting whether the second stealing based on the first output vector and the second output vector includes:
detecting whether the biometric information is stolen or not based on a third score based on the first output vector and the second output vector
A method of detecting whether biometric information has been stolen, including. 12. The method of claim 11,
generating a 1-3 embedding vector based on the 1-1 embedding vector and the 1-2 embedding vector; and
Calculating a 1-3 score based on the 1-3 embedding vector
further comprising,
The step of detecting whether the first theft is
The biometric information is obtained based on the 1-1 score calculated based on the 1-1 embedding vector, the 1-2 score calculated based on the 1-2 embedding vector, and the 1-3 score. Detecting whether the first theft
A method of detecting whether biometric information has been stolen, including. 18. The method of claim 17,
The step of generating the 1-3 embedding vector is
generating the 1-3 embedding vector by combining the 1-1 embedding vector and the 1-2 embedding vector; and
generating the 1-3 embedding vector by element-wise summing the 1-1 embedding vector and the 1-2 embedding vector
A method of detecting whether biometric information is stolen, including any one of. 18. The method of claim 17,
The step of detecting whether the first theft is
classifying the 1-1 score by using a pre-trained 1-1 classifier to determine whether the 1-1 score is a score that belongs to a range determined as stolen information or a score that belongs to a range determined as real information;
performing a 1-2 classifying step using a pre-trained 1-2 classifier to determine whether the 1-2 score is a score within a range determined as the stolen information or a score that belongs to a range determined as the real information;
classifying the 1-3 score as a score belonging to a range determined as the stolen information or a score belonging to a range determined as the actual information using a pre-trained 1-3 classifier; and
detecting whether or not the first stealing is performed using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result
A method of detecting whether biometric information has been stolen, including. 20. The method of claim 19,
The step of detecting whether the first stealing is performed using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result includes:
detecting whether the first stealing is performed by a weighted sum of the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result;
A method of detecting whether biometric information has been stolen, including. 20. The method of claim 19,
The step of detecting whether the first stealing is performed using the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result includes:
detecting whether or not the first robbery is performed according to a majority vote based on the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result;
A method of detecting whether biometric information has been stolen, including. According to claim 1,
The biometric information is
A method of detecting whether biometric information is stolen, including any one of the user's fingerprint, iris, and face. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 22. a sensor for capturing an image including the user's biometric information; and
Extracting an embedding vector from an intermediate layer of a neural network that detects whether the biometric information is stolen from the image, detects whether the first theft of the biometric information is detected based on the embedding vector, and whether the first theft is detected Accordingly, a processor that detects whether the biometric information is second stolen using an output vector output from an output layer of the neural network
A device for detecting whether biometric information is stolen. 25. The method of claim 24,
the processor
An apparatus for detecting whether biometric information is stolen, calculating a first score based on the embedding vector, and detecting whether the first steal is performed based on the first score. 26. The method of claim 25,
the processor
determine whether the first score is a score that falls within a first threshold range for determining whether the first stealing is performed, and when the first score is determined as a score that falls within the first threshold range, based on the first score to detect whether or not the first stolen biometric information has been stolen. 27. The method of claim 26,
The first threshold range is
A first threshold corresponding to a maximum probability that the first score can be determined to be a score corresponding to stolen information and a second threshold corresponding to a minimum probability that the first score can be determined to be a score corresponding to actual information A device for detecting whether biometric information is stolen, determined based on the device. 26. The method of claim 25,
the processor
An apparatus for detecting whether biometric information is stolen by classifying whether the first score is a score belonging to a range determined as stolen information or a score belonging to a range determined as actual information using a pre-trained classifier. 29. The method of claim 28,
the processor
and outputting a classification result of the classifier when the first stealing is detected based on the first score. 26. The method of claim 25,
If the first theft is not detected by the first score, the processor
Extracting a 1-2 embedding vector from the next intermediate layer after the intermediate layer, calculating a 1-2 score based on the 1-2 embedding vector, and determining whether the first stealing is based on the 1-2 score A device for detecting whether biometric information has been stolen. 31. The method of claim 30,
the processor
Using a pre-trained 1-2 classifier to classify whether the 1-2 score is a score within a range determined as the stolen information or a score that belongs to a range determined as actual information, detecting whether biometric information is stolen device to do. 26. The method of claim 25,
If the first theft is not detected by the first score, the processor
Extracting a 1-2 embedding vector from the next intermediate layer after the intermediate layer, calculating a 1-2 score based on the 1-2 embedding vector, and fusing the first score and the 1-2 score and detecting whether the first theft is detected by classifying whether the fused score is a score belonging to a range determined as stolen information or a score belonging to a range determined as actual information. 25. The method of claim 24,
the processor
an apparatus for detecting whether biometric information has been stolen, wherein if the first stealing is not detected, detecting whether the second stealing is performed based on a second score based on the output vector. 25. The method of claim 24,
the video is
and a first image and a second image including the user's biometric information,
the processor
Extracting one or more 1-1 embedding vectors from one or more first intermediate layers of a first neural network for detecting whether the biometric information is stolen from the first image, and determining whether the biometric information is stolen from the second image An apparatus for detecting whether biometric information is stolen by extracting one or more 1-2 embedding vectors from one or more second intermediate layers of a second neural network to be detected. 25. The method of claim 24,
The second image is
An image obtained by capturing the user's biometric information at a different time from the first image by a sensor, or an image generated through image processing on the first image
A device for detecting whether biometric information has been stolen. 35. The method of claim 34,
the processor
calculating a 1-3 score based on the 1-1 embedding vector and the 1-2 embedding vector, and detecting whether the biometric information is first stolen based on the 1-3 score. A device that detects theft. 37. The method of claim 36,
the processor
It is determined whether the 1-3 score is a score that falls within a first threshold range for determining whether the first robbery is determined, and when the 1-3 score is determined to be a score that falls within the first threshold range, the second An apparatus for detecting whether biometric information has been stolen, detecting whether or not the first theft is made based on the 1-3 score. 37. The method of claim 36,
When whether the first theft is not detected by the 1-3 score, the processor
obtain a first output vector from a first output layer of the first neural network, obtain a second output vector from a second output layer of the second neural network, An apparatus for detecting whether the biometric information has been stolen based on the second theft detection. 39. The method of claim 38,
the processor
An apparatus for detecting whether biometric information is stolen or not, detecting whether or not the biometric information has been second stolen based on a third score based on the first output vector and the second output vector. 35. The method of claim 34,
the processor
A 1-3 embedding vector is generated based on the 1-1 embedding vector and the 1-2 embedding vector, and a 1-3 score is calculated based on the 1-3 embedding vector, and the first The first theft of the biometric information based on the 1-1 score calculated based on the -1 embedding vector, the 1-2 score calculated based on the 1-2 embedding vector, and the 1-3 score A device for detecting whether biometric information has been stolen. 41. The method of claim 40,
the processor
By combining the 1-1 embedding vector and the 1-2 embedding vector to generate the 1-3 embedding vector, or by adding the 1-1 embedding vector and the 1-2 embedding vector for each element An apparatus for generating the 1-3 embedding vector and detecting whether biometric information is stolen. 41. The method of claim 40,
the processor
Using the pre-trained 1-1 classifier, the 1-1 classifier classifies whether the 1-1 score is a score that belongs to a range determined as stolen information or a score that belongs to a range determined as real information, and the pre-trained 1-1 classifier is used. Using the 1-2 classifier, the 1-2 classifies whether the 1-2 score is within the range determined by the stolen information or the score belongs to the range determined by the real information, and the pre-trained Classifying 1-3 as whether the score 1-3 is a score belonging to a range determined as the stolen information or a score belonging to a range determined as the real information by using a 1-3 classifier, and the 1-1 An apparatus for detecting whether or not biometric information has been stolen by detecting whether or not the first use has been made using a classification result, the 1-2 classification result, and the 1-3 classification result. 43. The method of claim 42,
the processor
The apparatus for detecting whether biometric information has been stolen is configured to detect whether the first theft is performed based on a weighted sum of the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result. 43. The method of claim 42,
the processor
An apparatus for detecting whether or not biometric information has been stolen according to a majority vote based on the 1-1 classification result, the 1-2 classification result, and the 1-3 classification result. a fingerprint sensor for capturing an image including the user's fingerprint information; and
extracting one or more embedding vectors from an intermediate layer of a neural network that detects whether the fingerprint information has been stolen from the image, and detecting whether the fingerprint information is first stolen based on the one or more embedding vectors; a processor configured to detect whether the fingerprint information is second stolen using an output vector output from an output layer of the neural network according to whether the fingerprint information is detected; and
an output device for outputting at least one of whether the first stealing and the second stealing
comprising, an electronic device. 46. The method of claim 45,
the fingerprint sensor
An electronic device comprising at least one of an ultrasonic fingerprint sensor, an optical fingerprint sensor, and an electrostatic fingerprint sensor.

Images