KR102138652B1 - Apparatus and method for biometrics using finger vein feature and finger shape based on deep learning - Google Patents

Abstract

The present invention relates to a biometric technology, and more particularly, to a deep learning-based finger vein feature and a biometric technology through finger shape. According to an embodiment of the present invention, robust biometric recognition may be performed on factors such as lighting change, position change, shading, misalignment, and finger pressure.

Description

Deep learning-based finger vein feature and biometric device and method using finger shape{APPARATUS AND METHOD FOR BIOMETRICS USING FINGER VEIN FEATURE AND FINGER SHAPE BASED ON DEEP LEARNING}

The present invention relates to a biometric technology, and more particularly, to a deep learning-based finger vein feature and a biometric technology through finger shape.

Biometric recognition is a method of identifying a user by analyzing characteristics of data in a quantitative manner in consideration of unique behavioral or biological factors that can effectively distinguish a user using a fingerprint, iris, or vein recognition method. Biometrics are used in many applications, including system security, financial electronic payments, and access control. Recently, with the popularization of smart phones, biometric authentication technology is emerging in fields other than the PC environment. Among the existing biometric recognition technologies, finger-based recognition technology having various advantages exists. For example, among the biometric techniques, there are a finger vein recognition method through a pattern feature in a blood vessel, a method for recognizing wrinkled fine texture of skin, and a finger shape recognition method based on geometric features. However, in the conventional method using a single recognition, it is difficult to extract patterns for various reasons due to changes in quality due to lighting change, position change, shading, misalignment, finger pressure, etc., and thus, recognition performance is reduced. . In addition, recent advances in hardware and algorithms have attempted to improve biometrics technology with a convolutional neural network (CNN)-based algorithm, but existing CNN-based studies have not shown the expected performance.

Background art of the present invention is disclosed in Republic of Korea Patent Publication No. 2017-0017588.

The problem to be solved by the present invention is to provide a deep learning based biometric recognition apparatus and method using finger vein features and finger shape.

According to an aspect of the present invention, a biometric recognition device using deep learning-based vein characteristics and finger shape is provided.

The biometric recognition device using deep learning-based finger vein features and finger shape according to an embodiment of the present invention is a pre-processing unit that performs pre-processing on an input image to generate a pre-processed image, and an interest to generate an image of interest from the pre-processed image Image extraction unit, generating a spectrogram image for the image of interest, applying the spectrogram image to a convolutional neural network, a shape score calculation unit for calculating a shape score, generating a difference image for the image of interest, and generating the difference A vein score calculating unit that applies an image to a convolutional neural network to calculate a vein score, and a recognition unit that fuses the shape score and the vein score to calculate a final score and generates biometric information according to the final score It can contain.

The shape score calculating unit may calculate the spectrogram image by applying a short time Fourier transform (STFT) to the image of interest.

The finger vein score calculating unit may generate the difference image including a difference in pixel values between the image of interest and an image of the same class and another class.

The convolutional neural network includes 15 convolutional layers, 4 shortcut layers, and 1 fully connected layer, wherein the kernel size used in each convolutional layer is one of 1×1, 3×3 and 7×7, The number of padding may be 0 or 1, and the number of strides may be 1 or 2.

The short cut layer may include a feature map corresponding to any one of 56×56×256, 28×28×512, 14×14×1024 and 7×7×2048.

The pre-processing unit applies binarization to the input image, detects a finger region by restoring a deteriorated portion of the binarized input image, and detects the pre-processed image including an image corresponding to the finger region among the input images. Can be created.

The interest image extracting unit extracts a region of interest (ROI) from the pre-processed image, measures the angle of the boundary surface in the region of interest, rotates the pre-processed image according to the angle of the boundary surface, and rotates the region of interest. The video of interest including a background may be generated.

According to another aspect of the present invention, there is provided a deep learning-based finger vein feature and a biometric recognition method using a finger shape and a recording medium on which a computer program executing the same is recorded.

Generating a pre-processed image by performing pre-processing on an input image in a biometric recognition method using a deep learning-based vein feature and a finger shape according to an embodiment of the present invention and a computer program executing the biometric recognition method, Generating an image of interest from the pre-processed image, generating a spectrogram image for the image of interest, applying the spectrogram image to a convolutional neural network to calculate a shape score, and generating a difference image for the image of interest And calculating a finger vein score by applying the difference image to a convolutional neural network; And fusion of the shape score and the finger vein score to calculate a final score, and generating biometric information according to the final score.

According to an embodiment of the present invention, robust biometric recognition may be performed on factors such as lighting change, position change, shading, misalignment, and finger pressure.

1 is a diagram illustrating a biometric device according to an embodiment of the present invention.
2 is a view for explaining a pre-processing process applied to a biometric device according to an embodiment of the present invention.
3 is a view for explaining a process of generating an image of interest by the biometric device according to an embodiment of the present invention.
4 is a view for explaining a short-time Fourier transform used by the biometric device according to an embodiment of the present invention.
5 is a diagram schematically illustrating the structure of a convolutional neural network used by a biometric recognition device according to an embodiment of the present invention.
6 is a diagram illustrating a specific structure of a convolutional neural network used by a biometric device according to an embodiment of the present invention.
7 is a flowchart illustrating a process in which the biometric device performs biometric recognition according to an embodiment of the present invention.
8 is a diagram illustrating test results of a test image of a biometric device according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, the singular expressions used in the specification and claims should be construed to mean “one or more” in general unless stated otherwise.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, identical or corresponding components are assigned the same reference numbers, and redundant description thereof will be omitted. Shall be

1 is a view illustrating a biometric device according to an embodiment of the present invention, Figure 2 is a view for explaining a pre-processing process applied to the biometric device according to an embodiment of the present invention, Figure 3 is the present invention FIG. 4 is a diagram for explaining a process in which a biometric device according to an embodiment generates an image of interest, and FIG. 4 is a view for explaining a short-time Fourier transform used by the biometric device according to an embodiment of the present invention.

Referring to FIG. 1, the biometric device according to an embodiment of the present invention includes an input unit 110, a pre-processing unit 120, a region of interest extraction unit 130, a shape score calculation unit 140, and a finger vein score calculation unit It includes 150 and the recognition unit 160.

The input unit 110 receives an image through a network or a predetermined input terminal. Hereinafter, an image received by the input unit 110 will be referred to as an input image. The input unit 110 transmits the input image to the pre-processing unit 120.

The pre-processing unit 120 generates a pre-processed image by performing a pre-processing process on the input image. For example, the pre-processing unit 120 performs binarization processing on the input image (eg, 200 in FIG. 2 ), and performs a convex hull algorithm on the binarized input image (eg, 210 in FIG. 2 ). By applying it, a deteriorated portion such as a hole is restored to detect a finger region (eg, 220 in FIG. 2 ), and a pre-processed image including an image corresponding to the finger region of the input image is generated. The pre-processing unit 120 transmits the pre-processed image to the region of interest extraction unit 130.

The region of interest extractor 130 extracts a region of interest (ROI) from the pre-processed image, and generates an image of interest including the region of interest. For example, the region-of-interest extractor 130 may detect an upper boundary surface and a lower boundary surface of a vein using a 4x20 mask to extract a region of interest including thickness information and a vein feature. When it is assumed that the region of interest extracting unit 130 obtains an image such as 310 by performing binarization on an input image such as 300 in FIG. 3, the angle of the interface is measured at a designated ROI, and the angle of the interface is determined as 320. Accordingly, the pre-processed image may be rotationally transformed, and an interest image including a region of interest and a background may be generated as shown in 330. The region of interest extraction unit 130 transmits the image of interest to the shape score calculation unit 140 and the finger vein score calculation unit 150.

The shape score calculating unit 140 measures the vertical height of the thickness of the vein region with respect to the image of interest as shown in 400 of FIG. 4, and performs a short-time Fourier transform (DFT) by performing a binary Fourier transform (DFT) by a specified width. Through Fourier Transform (STFT), a spectogram image that expresses an image of interest in the frequency domain, such as 420, is generated. For example, the shape score calculating unit 140 may use a short-time Fourier transform such as 410 of FIG. 4 and Equation (1).

[Equation 1]

Here, n is a horizontal position of a vein, w is a frequency axis, f(m) is a signal to be analyzed, and W(m) is a window according to a pixel length R.

The shape score calculator 140 calculates feature values for the spectrogram image using a convolutional neural network. The shape score calculating unit 140 is used to calculate the Euclidean distance between the feature values corresponding to the spectrogram image corresponding to the same class as the calculated feature values and the spectrogram images corresponding to classes different from the calculated feature values. The shape score is calculated according to the Euclidean distance between the corresponding feature values. In this case, the shape score calculating unit 140 may use a convolutional neural network that has learned the spectrogram image for each class. The shape score calculation unit 140 transmits the shape score to the recognition unit 160.

The finger vein score calculator 150 extracts a region corresponding to a finger vein (hereinafter referred to as a finger vein region) from the video of interest, and converts the size of the image including the region to a specified size. The vein score calculating unit 150 includes a difference image representing a difference between a pixel value of a vein region included in an image of the same class and a vein region of an image of interest, and a vein region and an interest image included in another class of images. Generates a difference image representing the difference in pixel values in the vein region. The finger vein score calculator 150 calculates a finger vein score for a difference image using a convolutional neural network. In this case, the finger vein score calculating unit 150 may use a convolutional neural network in which a difference image between each same class and a difference image between different classes are learned. The finger vein score calculator 150 transmits the finger vein score to the recognition unit 160.

은 수학식 2에 따라서 계산되고 Perceptron

은 수학식 3에 따라서 계산되는 방식이다. The recognition unit 160 calculates the final score by fusing the shape score and the finger vein score through a predetermined method (eg, weighted sum, weighted product, perception, bayesian, etc.). In this case, the weighted sum is a method in which a sum is obtained by assigning weights to each single method, and a weighted product is a method in which the weight product of each single method is used as a score. Bayesian(

Is calculated according to Equation 2 and Perceptron

Is a method calculated according to equation (3).

[Equation 2]

[Equation 3]

At this time, s1 is a shape score, s2 is a finger vein score, and w0, w1 and w2 are predetermined weights.

At this time, the recognition unit 160 may calculate each final score through various fusion methods, and select the final score through the fusion method having the lowest error rate.

The recognition unit 160 generates biometric information indicating that the user of the corresponding input image is recognized as a legitimate user when the final score is equal to or less than a predetermined threshold. Conversely, when the final score exceeds a predetermined threshold, the recognition unit 160 generates biometric recognition information that recognizes a user of the corresponding input image as an inappropriate user.

5 is a diagram schematically illustrating the structure of a convolutional neural network used by a biometric device according to an embodiment of the present invention, and FIG. 6 is a detailed view of a convolutional neural network used by a biometric device according to an embodiment of the present invention It is a diagram illustrating the structure.

The convolutional neural network used by the biometric device according to an embodiment of the present invention may include the structures of Groups 1 to 3 as shown in FIG. 5.

Specifically, as shown in FIG. 6, the convolutional neural network may include 15 convolutional layers, 4 shortcut layers, and 1 fully connected layer.

The feature map size output based on the resized image to 224x224 pixels is calculated as in Equation 4 below.

[Equation 4]

Feature map size = ((Input size-size of kernel + size of padding *2)/number of stride) +1

Each convolutional layer undergoes batch normalization and Rectified linear unit (ReLU) to optimize the structure. Batch normalization is a process of finding the mean and covariance of features in a mini-batch unit with little correlation between data and going through normalization.

In the first convolutional layer, 3x3 max pooling performs subsampling using the largest value in the 3x3 window area. Group2, Group4, Group 6 and Group8 use a bottleneck structure. In the first 1x1 size convolution layer of Group2, Group4, Group 6, and Group8, the filter size is reduced, and features are extracted from the 3x3 size convolution layer. For connection with the shortcut, the next convolutional layer was again used with a window of size 1x1 to make it the same as the filter of the shortcut. Due to this, the parameters to be learned are reduced and the amount of computation to be processed is less than a single 3x3 size convolution process. In addition, a short cut structure was used in Group3, Group6, and Group9. The shortcut structure connects the learned filter and the skipped filter without learning the previous layers. The convolutional neural network recognizes a class according to a label through a fully connected layer after average grouping that processes an average value in a window after all groups of layers. Therefore, the degradation problem that occurs when the convolutional neural network deepens can be solved and learning can be optimized.

That is, as an example of CNN, Group 1 generates a feature map of 112×112×64 through a convolution layer using 64 filters. The number of strides used in the convolutional layer of Group 1 may be 2, and the number of padding may be 3.

Also, as an example of CNN, Group 2 generates a feature map of 56×56×64 or 56×56×256 through a convolution layer using 64 or 256 filters. The number of strides used in the convolutional layer of Group 2 may be 1, and the number of padding may be 0 or 1.

In addition, for example of CNN, Group 3 generates a feature map of 56×56×256 through a shortcut layer.

Further, as an example of CNN, Group 4 generates a feature map of 28×28×128 or 28×28×512 through a convolution layer using 128 or 512 filters. The number of strides used in the convolution layer of Group 4 may be 1 or 2, and the number of padding may be 0 or 1.

Also, as an example of CNN, Group 5 generates a feature map of 28×28×512 through a shortcut layer.

Further, as an example of CNN, Group 6 generates a feature map of 14×14×256 or 14×14×1024 through a convolution layer using 256 or 1024 filters. The number of strides used in the convolutional layer of Group 6 may be 1 or 2, and the number of padding may be 0 or 1.

Also, as an example of CNN, Group 7 generates a feature map of 14×14×1024 through a shortcut layer.

Further, as an example of CNN, Group 8 generates a feature map of 7×7×512 or 7×7×2048 through a convolution layer using 512 or 2048 filters. The number of strides used in the convolutional layer of Group 6 may be 1 or 2, and the number of padding may be 0 or 1.

2/1* in FIG. 6 means that the value in the first operation is 1, and in the subsequent operations, 2 means, ** is the first group (2,3,4,5,6,7,8) ,9) is the number of repetitions, followed by the number in the second group (2-1, 3-1, 4-1, 5-1, 6-1, 7-1, 8-1, 9-1) This is the number of repetitions. *** is the number of repetitions in resnet101. The remaining number of repetitions without a mark is equal to 50 steps.

7 is a flowchart illustrating a process in which the biometric device performs biometric recognition according to an embodiment of the present invention. Each step described below is a process performed through each functional unit constituting the biometric device, or the subject of each step is collectively referred to as a biometric device for concise and clear description of the invention.

In step S710, the biometric device receives an input image.

In step S720, the biometric device preprocesses the input image. For example, the biometric device performs binarization processing on an input image, restores a deteriorated portion such as a hole by applying a convex hull algorithm to the binarized input image, detects a finger region, and finger region of the input image A pre-processed image including an image corresponding to may be generated.

In step S730, the biometric device extracts a region of interest (ROI) from the pre-processed image and generates an image of interest including the region of interest. The biometric device may measure the angle of the interface at a designated ROI, rotate transform the pre-processed image according to the angle of the interface, and generate an image of interest including a region of interest and a background.

In step S740, the biometric device calculates a spectrogram image by applying a short time Fourier transform (STFT) to the image of interest.

In step S750, the biometric device calculates a feature value for the shape of the finger according to the spectrogram image through the convolutional neural network.

In step S760, the biometric device calculates a shape score according to the feature value. For example, the biometric device may use the Euclidean distance between the feature values corresponding to the spectrogram image corresponding to the same class as the calculated feature values, and the spectrogram images corresponding to classes different from the calculated feature values. The shape score can be calculated according to the Euclidean distance between the corresponding feature values.

In step S770, the biometric device adjusts the size of the image of interest to a predetermined size. For example, the biometric device extracts a vein region from the image of interest and a size of an image including the vein region is specified. Convert as much as possible.

In step S780, the biometric recognition device designates a difference image indicating a difference between a pixel value of a vein region included in an image of the same class and a vein region of the image of interest, and designation of a vein region and an image of interest included in another class of images. A difference image representing differences in pixel values in the Mac area is generated.

In step S790, the biometric device calculates the finger vein score according to the difference image through the convolutional neural network.

In step S792, the biometric device calculates the final score by fusing the shape score and the finger vein score through a predetermined method (eg, weighted sum, weighted product, perception, bayesian, etc.).

In step S794, the biometric device generates biometric information according to the final score. For example, the biometric device generates biometric information indicating that the user of the input image is recognized as a legitimate user when the final score is less than or equal to a predetermined threshold, and when the final score exceeds the predetermined threshold, Biometric information may be generated to recognize a user of the corresponding input image as an inappropriate user.

8 is a diagram illustrating a test result for a test image of the biometric device according to an embodiment of the present invention.

Referring to FIG. 8, there are False Acceptance Ratio (FAR) and False Rejection Rate (FRR) as means for measuring the error rate in biometric recognition. FAR occurs when different classes are judged to be authentic, and FRR is an error number generated when the same classes are judged not to be recognized (imposter). The FAR and FRR curves together are called ROC curves and are graphs showing the error rate according to sensitivity. In general, as sensitivity increases, FRR increases and FAR decreases. Conversely, when sensitivity decreases, FRR decreases and FAR increases. The point where FAR meets FRR according to the graph curve is called Equal Error Rate (EER) and is an indicator of performance.

As a result of calculating EER for two databases (SDUMLA-HMT, PolyU), the perception method shows the lowest error rate.

The biometric recognition method according to an embodiment of the present invention may be implemented in a form of program instructions that can be executed through various computer means and may be recorded in a computer-readable medium. Computer-readable media may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. In addition, the above-described medium may be a transmission medium such as an optical or metal line, a waveguide including a carrier wave that transmits a signal specifying a program command, a data structure, or the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been focused on the embodiments. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Claims (15)

In the biometric recognition device using finger type and finger vein features based on deep learning,
A pre-processing unit to generate a pre-processed image by performing pre-processing on the input image;
An interest image extraction unit generating an interest image from the pre-processed image;
A shape score calculation unit generating a spectrogram image for the image of interest and applying the spectrogram image to a convolutional neural network to calculate a shape score;
A vein score calculator configured to generate a difference image for the image of interest and apply the difference image to a convolutional neural network to calculate a vein score; And
Including the recognition unit for generating a final score by fusing the shape score and the finger vein score, and generates biometric information according to the final score,
The pre-processing unit,
Applying binarization to the input image, restoring a deteriorated portion of the binarized input image to detect a finger region, and generating the pre-processed image including an image corresponding to the finger region among the input images,
The deteriorated portion is restored by applying a convex hull algorithm to the binarized input image,
The form score calculation unit,
Characterized in that the spectrogram image is calculated by applying a short time Fourier transform (STFT) to the image of interest,
Calculate the feature values of the spectrogram image to use the distance between the feature values of the spectrogram image corresponding to the same class and different classes,
The recognition unit,
A biometric device that selects the final score through a fusion method having the lowest error rate.
According to claim 1,
The form score calculation unit,
A biometric device that calculates a shape score according to the Euclidean distance.
According to claim 1,
The finger vein score calculation unit,
And generating the difference image including a pixel value difference between the image of interest and the image of the same class and another class.
According to claim 1,
The convolutional neural network
Includes 15 convolutional layers, 4 shortcut layers and 1 fully connected layer,
The kernel size used in each of the convolutional layers is one of 1×1, 3×3 and 7×7, the number of paddings is 0 or 1, and the number of strides is 1 or 2.
According to claim 4,
The shortcut layer includes a feature map corresponding to any one size of 56×56×256, 28×28×512, 14×14×1024 and 7×7×2048.
delete According to claim 1,
The video extraction unit of interest,
Region of Interest (ROI) is extracted from the pre-processed image,
Measure the angle of the interface in the region of interest,
The pre-processed image is rotated and transformed according to the angle of the boundary surface,
A biometric device characterized by generating the image of interest including a region of interest and a background.
In the biometric device is a method of performing biometric recognition using deep learning based finger vein features and finger shape,
Generating a pre-processed image by performing pre-processing on the input image;
Generating an image of interest from the preprocessed image;
Generating a spectrogram image of the image of interest and applying the spectrogram image to a convolutional neural network to calculate a shape score;
Generating a difference image for the image of interest and applying the difference image to a convolutional neural network to calculate a vein score; And
Comprising the step of fusion of the shape score and the finger vein score to calculate a final score, and generating biometric information according to the final score,
Generating a spectrogram image for the image of interest, and applying the spectrogram image to a convolutional neural network to calculate a shape score,
Comprising the step of calculating the spectrogram image by applying a short time Fourier Transform (STFT) to the image of interest,
Calculate the feature values of the spectrogram image to use the distance between the feature values of the spectrogram image corresponding to the same class and different classes,
Step of generating a pre-processed image by performing pre-processing on the input image,
Applying binarization to the input image, restoring a deteriorated portion of the binarized input image, detecting a finger region, and generating the pre-processed image including an image corresponding to the finger region among the input images Characterized as,
The deteriorated portion is restored by applying a convex hull algorithm to the binarized input image,
The final score is a biometric recognition method selected through a fusion method having the lowest error rate.
delete The method of claim 8,
The step of generating a difference image for the image of interest and applying the difference image to a convolutional neural network to calculate a vein score,
And generating the difference image including a pixel value difference between the image of interest and the image of the same class and another class.
The method of claim 8,
The convolutional neural network
Includes 15 convolutional layers, 4 shortcut layers and 1 fully connected layer,
The kernel size used in each of the convolutional layers is one of 1×1, 3×3 and 7×7, the number of paddings is 0 or 1, and the number of strides is 1 or 2.
The method of claim 11,
The shortcut layer includes a feature map corresponding to any one of 56×56×256, 28×28×512, 14×14×1024 and 7×7×2048.
delete The method of claim 8,
Generating an image of interest from the pre-processed image,
Extracting a region of interest (ROI) from the pre-processed image;
Measuring an angle of the boundary surface in the region of interest;
Rotating transforming the pre-processed image according to the angle of the boundary surface; And
And generating the image of interest including a region of interest and a background.
A computer-readable recording medium recording a computer program that executes the biometric recognition method according to any one of claims 8, 10 to 12, and 14.

Images