KR102161476B1 - Apparatus and method for identifying user using user body based on deep learning - Google Patents

Abstract

The present invention relates to an identity recognition technology, and more particularly, to an identity recognition technology using a user's body based on deep learning. According to an embodiment of the present invention, even when an image photographed by the user is blurred or contains noise, identification of the user may be accurately performed.

Description

Identity recognition device and method using a user's body based on deep learning {APPARATUS AND METHOD FOR IDENTIFYING USER USING USER BODY BASED ON DEEP LEARNING}

The present invention relates to an identity recognition technology, and more particularly, to an identity recognition technology using a user's body based on deep learning.

The identity recognition method identifies the user by analyzing the characteristics of data according to a quantitative method in consideration of unique behavioral or biological factors that can effectively distinguish the user, such as a recognition method using fingerprint, iris, vein, face, etc. That's the way. Identity recognition is used in many applications, including system security, financial electronic payments, and access control. In recent years, with the popularization of smartphones, identity authentication technology has emerged in fields other than the PC environment. Among the existing identity recognition technologies, there are face recognition technologies with various advantages. Face recognition technology has a point that can accurately identify a person, but when an image captured from a distance such as a CCTV camera is used, the image is often blurred or contains noise, making it difficult to recognize an identity using only the face recognition technology.

Background art of the present invention is disclosed in Korean Patent Application Publication No. 2017-0017588.

The present invention provides an apparatus and method for identifying identity using a user's body based on deep learning.

According to an aspect of the present invention, an identity recognition device using a user's body based on deep learning is provided.

An identity recognition apparatus using a user's body based on deep learning according to an embodiment of the present invention includes an image generator that generates a face image and a body image for an input image, and extracts facial feature information from the face image through a facial neural network. According to a facial feature extraction unit, a body feature extraction unit for extracting body feature information from the body image through a body neural network, a fusion score calculation unit for calculating a fusion score according to the facial feature information and the body feature information, and the fusion score It may include an identity recognition unit that generates identity recognition information.

The fusion score calculator may calculate a face matching distance according to the facial feature information and a body matching distance according to the body feature information, and apply a weight to the face matching distance and the body matching distance to calculate the fusion score. .

The facial neural network includes 13 convolutional layers, 5 pooling layers, and 3 fully connected layers, and the kernel size used in each of the convolutional layers is 3×3, the number of paddings is 1, and the number of strides is May be 1.

The body neural network includes five convolutional layers, two pooling layers, and one fully connected layer, and four of the convolutional layers include three branches, and a first of the branches. The kernel size used in the branch and the third branch is 1 × 1, the number of strides is 1 × 1, the number of padding is 0 × 0, the kernel size used in the second branch is 3 × 3, and the number of strides is 1 ×1, and the number of padding may be 1×1.

The image generator may calculate a focus score for the face image, and when the focus score is less than a predetermined threshold value, regenerate a face image for a new input image.

According to another aspect of the present invention, an identity recognition method using a user's body based on deep learning is provided.

An identity recognition method using a user's body based on deep learning according to an embodiment of the present invention includes generating a face image and a body image for an input image, extracting facial feature information from the face image through a facial neural network, Extracting body feature information from the body image through a body neural network, calculating a fusion score according to the facial feature information and the body feature information, and generating identity recognition information according to the fusion score. have.

In the calculating of the fusion score according to the facial feature information and the body feature information, a face matching distance according to the facial feature information and a body matching distance according to the body feature information are calculated, and the face matching distance and the body matching distance It may be a step of calculating the fusion score by applying a weight to.

The facial neural network includes 13 convolutional layers, 5 pooling layers, and 3 fully connected layers, and the kernel size used in each of the convolutional layers is 3×3, the number of paddings is 1, and the number of strides is May be 1.

The body neural network includes five convolutional layers, two pooling layers, and one fully connected layer, wherein four of the convolutional layers include three branches, and the first among the branches. The kernel size used in branch 1 and branch 3 is 1 × 1, the number of strides is 1 × 1, the number of padding is 0 × 0, the kernel size used in the second branch is 3 × 3, and the number of strides is 1×1, and the number of padding may be 1×1.

Generating a face image and a body image for the input image may include calculating a focus score for the face image; If the focus score is less than a predetermined threshold value, regenerating a face image for a new input image may be included.

According to an embodiment of the present invention, it is possible to accurately recognize the identity of a user in an indoor environment by using a visible ray camera based on deep learning.

According to an embodiment of the present invention, even when an image photographed by the user is blurred or contains noise, identification of the user may be accurately performed.

1 is a diagram illustrating an identity recognition apparatus according to an embodiment of the present invention.
2 is a diagram illustrating a face image generated by an identity recognition apparatus according to an embodiment of the present invention.
3 is a diagram illustrating a simplified structure of a facial neural network used by an identity recognition apparatus according to an embodiment of the present invention.
4 is a diagram illustrating a specific initial structure of a facial neural network used by an identity recognition apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a detailed structure of a rear face of a facial neural network used by an identity recognition apparatus according to an embodiment of the present invention.
6 is a diagram illustrating a simplified structure of a body neural network used by an identity recognition apparatus according to an embodiment of the present invention.
7 is a diagram illustrating a specific structure of a body neural network used by an identity recognition apparatus according to an embodiment of the present invention.
8 is a flow chart illustrating a method of recognizing an identity by an identity recognition apparatus according to an embodiment of the present invention.
9 is a diagram illustrating a test result of a test image of a first database of an identity recognition apparatus according to an embodiment of the present invention.
10 is a diagram illustrating a test result of a test image of a first database of an identity recognition apparatus according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided. Specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, the singular expressions used in the specification and claims are to be construed as meaning “one or more” in general, unless otherwise stated.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numbers, and redundant descriptions thereof will be omitted. To

1 is a diagram illustrating an identity recognition apparatus according to an embodiment of the present invention, FIG. 2 is a diagram illustrating a face image generated by an identity recognition apparatus according to an embodiment of the present invention, and FIG. 3 is a view illustrating the present invention. Is a diagram illustrating a simple structure of a facial neural network used by an identity recognition apparatus according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating a specific initial structure of a facial neural network used by an identity recognition apparatus according to an embodiment of the present invention 5 is a diagram illustrating a specific late structure of a facial neural network used by an identity recognition apparatus according to an embodiment of the present invention, and FIG. 6 is a body neural network used by an identity recognition apparatus according to an embodiment of the present invention. Is a diagram illustrating a simple structure of, and FIG. 7 is a diagram illustrating a specific structure of a body neural network used by an identity recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the identity recognition apparatus according to an embodiment of the present invention includes an input unit 110, an image generation unit 120, a face focus detection unit 130, a face feature extraction unit 140, and a body feature extraction unit ( 150), and a fusion score calculation unit 160 and an identity recognition unit 170.

The input unit 110 receives an image through a network or a predetermined input terminal. Hereinafter, the image received by the input unit 110 will be referred to as an input image. The input unit 110 transmits the input image to the image generation unit 120.

The image generator 120 generates a face image including a face region from among the input images according to a predetermined algorithm.

The face focus detection unit 130 calculates a focus score from the generated face image. Here, the focus score is a score indicating to what extent the focus can be identified when a face image is captured. When a convolutional filter is applied to a face image, the face focus detector 130 may calculate a focus score corresponding to a change in an offset value. In this case, when the focus score is high, the face image is clearer than when the focus score is low as shown in 210 of FIG. 2 (220 of FIG. 2). The face focus detection unit 130 transmits the face image to the face feature extraction unit 140 when the focus score for the face image is equal to or greater than a predetermined threshold value as shown in 210 of FIG. 2. When the focus score for the face image is less than the threshold value as shown in 220 of FIG. 2, the face focus detection unit 130 may receive the input image again from the input unit 110 and calculate the focus score for the face image again. Accordingly, when it is difficult to extract effective feature information through a face image having a focus, the face focus detection unit 130 may increase accuracy of identity recognition by extracting a face image through a new input image.

The image generation unit 120 also generates a body image excluding the face region among the input images, and transmits the body image to the body feature extraction unit 150.

The facial feature extraction unit 140 extracts facial feature information from a face image. In this case, the facial feature extractor 140 may extract facial feature information by inputting a face image to the facial neural network. For example, the facial feature extractor 140 may extract facial feature information through a facial neural network having a structure as shown in FIG. 3.

That is, the facial neural network may be a neural network including 13 convolutional layers, 5 pooling layers, and 3 fully connected layers as shown in FIGS. 4 and 5.

The first convolution layer and the second convolution layer generate a 224×224×64 feature map. The kernel size used in the first convolution layer and the second convolution layer may be 3×3, the number of strides may be 1, and the number of paddings may be 1.

The third convolution layer and the fourth convolution layer generate a 112×112×128 feature map. The kernel size used in the third convolution layer and the fourth convolution layer may be 3×3, the number of strides may be 1, and the number of paddings may be 1.

The fifth convolution layer, the sixth convolution layer, and the seventh convolution layer generate a 56×56×256 feature map. The kernel size used in the fifth convolution layer, the sixth convolution layer, and the seventh convolution layer may be 3×3, the number of strides may be 1, and the number of paddings may be 1.

The eighth convolution layer, the ninth convolution layer, and the tenth convolution layer generate a feature map of 28×28×256. The kernel size used in the eighth convolution layer, the ninth convolution layer, and the tenth convolution layer may be 3×3, the number of strides may be 1, and the number of padding may be 1.

The eleventh convolution layer, the twelfth convolution layer, and the thirteenth convolution layer generate a feature map of 14×14×512. The kernel size used in the 11th convolution layer, the twelfth convolution layer, and the 13th convolution layer may be 3×3, the number of strides may be 1, and the number of paddings may be 1.

The facial feature extraction unit 140 transmits the facial feature information to the fusion score calculation unit 160.

The body feature extraction unit 150 calculates body feature information by inputting a body image into a body neural network. For example, the body feature extractor 150 may calculate body feature information through a body neural network having a structure as shown in FIG. 6.

That is, the body neural network may include 5 types of convolutional layers, 2 pooling layers, and 1 fully connected layer as shown in FIG. 7.

Referring to FIG. 7, a first convolution layer conv1 generates a 112×112×64 feature map. The kernel size used in the first convolution layer may be 7×7, the number of strides may be 2×2, and the number of padding may be 3×3.

The second convolutional layer (conv2) includes three branches. The first branch (Branch2a) generates a 56 × 56 × 64 feature map. The kernel size used in the first branch may be 1×1, the number of strides may be 1×1, and the number of paddings may be 0×0. The second branch (Branch2b) generates a 56 × 56 × 64 feature map. The kernel size used in the second branch may be 3×3, the number of strides may be 1×1, and the number of padding may be 1×1. The third branch (Branch2c) generates a 56×56×256 feature map. The kernel size used in the third branch may be 1×1, the number of strides may be 1×1, and the number of padding may be 0×0.

The third convolutional layer (conv3) includes three branches. The first branch (Branch2a) generates a feature map of 28×28×128. The kernel size used in the first branch may be 1×1, the number of strides may be 1×1, and the number of paddings may be 0×0. The second branch (Branch2b) generates a feature map of 28×28×128. The kernel size used in the second branch may be 3×3, the number of strides may be 1×1, and the number of padding may be 1×1. The third branch (Branch2c) generates a feature map of 28 × 28 × 512. The kernel size used in the third branch may be 1×1, the number of strides may be 1×1, and the number of padding may be 0×0.

The fourth convolutional layer (conv4) includes three branches. The first branch (Branch2a) generates a feature map of 14 × 14 × 256. The kernel size used in the first branch may be 1×1, the number of strides may be 1×1, and the number of paddings may be 0×0. The second branch (Branch2b) generates a 14×14×256 feature map. The kernel size used in the second branch may be 3×3, the number of strides may be 1×1, and the number of padding may be 1×1. The third branch (Branch2c) generates a feature map of 14 × 14 × 1024. The kernel size used in the third branch may be 1×1, the number of strides may be 1×1, and the number of padding may be 0×0.

The fifth convolutional layer (conv5) includes three branches. The first branch (Branch2a) generates a 7×7×512 feature map. The kernel size used in the first branch may be 1×1, the number of strides may be 1×1, and the number of paddings may be 0×0. The second branch (Branch2b) generates a 7x7x512 feature map. The kernel size used in the second branch may be 3×3, the number of strides may be 1×1, and the number of padding may be 1×1. The third branch (Branch2c) generates a 7×7×2048 feature map. The kernel size used in the third branch may be 1×1, the number of strides may be 1×1, and the number of padding may be 0×0.

The body feature extraction unit 150 generates a body feature vector and transmits it to the fusion score calculation unit 160.

The fusion score calculation unit 160 calculates a fusion score by combining the facial feature vector and the body feature vector. For example, the fusion score calculator 160 may calculate a face matching distance between a facial feature vector and a feature vector previously specified in the database, and a body matching distance between a body feature vector and a feature vector previously stored in the database. The fusion score calculator 160 may calculate a fusion score by multiplying each of the face matching distance and the body matching distance by a predetermined weight, and adding or multiplying the face matching distance multiplied by the weight and the body matching distance. That is, the fusion score calculation unit 160 may calculate a weight sum or a weight product for the face matching distance and the body matching distance as a fusion score. The fusion score calculation unit 160 transmits the fusion score to the identity recognition unit 170.

The identity recognition unit 170 generates identity recognition information indicating the identity of the user according to the fusion score. For example, when the fusion score is less than or equal to a predetermined threshold, the identity recognition unit 170 generates identity recognition information indicating the identity of the user, and when the fusion score exceeds a predetermined threshold, the user is not a designated number. It is possible to generate identification information indicating

8 is a flow chart illustrating a method of recognizing an identity by an identity recognition apparatus according to an embodiment of the present invention. Each of the steps to be described below is a step performed by each functional unit included in the identity recognition device described above with reference to FIG. 1, but for a concise and clear description of the invention, the subject of each step is collectively referred to as an identity recognition device.

Referring to FIG. 8, in step S810, the identity recognition apparatus receives an input image. For example, the identity recognition apparatus may receive an input image through a network or a predetermined input terminal.

In step S820, the identity recognition apparatus detects a face image from the input image.

In step S830, the identity recognition apparatus calculates a focus score for the face image, and when the focus score is less than a specified threshold value, the process from step S810 is performed again.

When the focus score is greater than or equal to the threshold value in step S830, the identity recognition apparatus calculates facial feature information through a facial neural network in step S840. The facial neural network was described above with reference to FIGS. 3 to 5.

In step S850, the identity recognition apparatus calculates a face matching distance corresponding to the facial feature information.

In step S860, the identity recognition apparatus detects a body image excluding a face area from the input image.

In step S870, the identity recognition device generates body feature information through a body neural network. The body neural network was described above with reference to FIGS. 6 and 7.

In step S880, the identity recognition apparatus calculates a body matching distance corresponding to the body feature information.

In step S890, the identity recognition apparatus calculates a fusion score by combining facial feature information and body feature information. For example, the identity recognition apparatus may calculate a face matching distance corresponding to facial feature information and a body matching distance corresponding to body feature information, and apply a weight to the face matching distance and body matching distance to calculate a fusion score. .

In step S900, the identity recognition apparatus generates identity recognition information corresponding to the fusion score. For example, when the fusion score is less than a predetermined threshold, the identity recognition device generates identity recognition information indicating the identity of the user, and when the fusion score exceeds a predetermined threshold, the identity indicating that the user is not a designated number Can generate recognition information.

9 is a diagram illustrating a test result of a test image of a first database of an identity recognition apparatus according to an embodiment of the present invention, and FIG. 10 is a diagram illustrating a first database of an identity recognition apparatus according to an embodiment of the present invention. This is a diagram illustrating a test result for a test image.

There is a false acceptance ratio (FAR) as a measure of the error rate in identity recognition. FAR refers to a probability that occurs when different classes are determined to be authentic. The general acceptance ratio (GAR) refers to the probability that the same class is recognized as recognized. Therefore, a method with high GAR and low FAR means an identity recognition method with high accuracy.

As shown in the graph of FIG. 9, in the test image of the first database, the identity recognition method according to an embodiment of the present invention, which calculates the weighted sum method fusion score, is the most accurate. In addition, as shown in the graph of FIG. 10, the identity recognition method according to an embodiment of the present invention, which calculates the weighted sum fusion score in the test image of the second database, is the most accurate. The first database and the second database are databases holding images of a user moving indoors, and include different images that are somewhat blurred and include noise. Accordingly, the identity recognition apparatus according to an embodiment of the present invention guarantees higher identity recognition accuracy than a method of recognizing an identity through only a face in an input image or a method of recognizing an identity through only the entire body.

An identity recognition method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. Program instructions recorded on a computer-readable medium may be specially designed and configured for the present invention, or may be known to and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Further, the above-described medium may be a transmission medium such as an optical or metal wire, a waveguide including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (11)

An image generator for generating a face image and a body image for the input image;
A facial feature extraction unit that extracts facial feature information from the face image through a facial neural network;
A body feature extraction unit for extracting body feature information from the body image through a body neural network;
A fusion score calculator configured to calculate a fusion score according to the facial feature information and the body feature information; And
Including an identity recognition unit for generating identity recognition information according to the fusion score,
Further comprising a face focus detector for calculating a focus score for the face image,
The image generator
Generates a face image including a face region according to a predetermined algorithm among the input images,
A body image excluding the face region is generated according to a predetermined algorithm among the input images,
The face focus detection unit
When the focus score is less than a predetermined threshold, a face image for a new input image is re-divided and generated,
The focus score is
When a convolutional filter is applied to the face image, it is calculated as a value corresponding to a change in an offset value,
The fusion score calculation unit
Calculating a face matching distance according to the facial feature information and a body matching distance according to the body feature information, and calculating the fusion score by applying a weight to the face matching distance and the body matching distance,
The facial neural network,
Includes 13 convolutional layers, 5 pooling layers and 3 fully connected layers,
The kernel size used in each of the convolutional layers is 3×3, the number of padding is 1, and the number of strides is 1,
The body neural network is
Including 5 convolutional layers, 2 pooling layers and 1 fully connected layer,
Four of the convolution layers include three branches,
Among the branches, the kernel size used in the first branch and the third branch is 1×1, the number of strides is 1×1, the number of padding is 0×0, and the kernel size used in the second branch is 3×3. And, the number of strides is 1×1, and the number of paddings is 1×1.
delete delete delete delete In a method for an identity recognition device using a user's body based on deep learning to recognize an identity,
Generating a face image and a body image for the input image;
Extracting facial feature information from the face image through a facial neural network;
Extracting body feature information from the body image through a body neural network;
Calculating a fusion score according to the facial feature information and the body feature information; And
Including the step of generating identity recognition information according to the fusion score,
Further comprising the step of calculating a focus score for the face image,
Generating a face image and a body image for the input image
Generates a face image including a face region according to a predetermined algorithm for the input image,
A body image excluding a face region is generated for the input image,
The step of calculating a focus score for the face image
When the focus score is less than a predetermined threshold value, a face image for a new input image is re-divided and generated,
The focus score is
When a convolutional filter is applied to the face image, it is calculated as a value corresponding to a change in an offset value,
The step of calculating a fusion score according to the facial feature information and the body feature information
A step of calculating the face matching distance according to the facial feature information and the body matching distance according to the body feature information, respectively, and calculating the fusion score by applying a weight to the face matching distance and the body matching distance. ,
The facial neural network,
Includes 13 convolutional layers, 5 pooling layers and 3 fully connected layers,
The kernel size used in each of the convolutional layers is 3×3, the number of padding is 1, and the number of strides is 1,
The body neural network is
Including 5 convolutional layers, 2 pooling layers and 1 fully connected layer,
Four of the convolution layers include three branches,
Among the branches, the kernel size used in the first branch and the third branch is 1×1, the number of strides is 1×1, the number of padding is 0×0, and the kernel size used in the second branch is 3×3. , The number of strides is 1×1, and the number of paddings is 1×1.
delete delete delete delete A computer program recorded on a recording medium readable by a computer that executes the identification method of claim 6.

Images