KR102184490B1 - Edge Device for Face Recognition - Google Patents

Abstract

The edge device for face recognition according to an aspect of the present invention capable of performing face recognition and authentication processing to the edge device by distributing the face recognition model generated in the central server to each edge device is a photographed image of a target user to be authenticated. A first photographing unit to obtain a signal; A face recognition unit that inputs the input images generated from the captured image into a face recognition model, extracts a target face image from the input image, and generates a target feature vector from the extracted target face image; An authentication unit for authenticating the target user by comparing the target feature vector with an array file consisting of a plurality of arrays having user identification information and feature vectors extracted from the user's face image; And an interface unit for receiving the array file and the face recognition model from a central server.

Description

Edge Device for Face Recognition

The present invention relates to face recognition, and more specifically, to a device for face recognition.

Face Recognition technology is one of the fields of biometrics, and refers to a technology in which a machine automatically identifies and authenticates a person using unique feature information contained in each person's face. It has superior security compared to the authentication method of, and is widely used in various fields recently.

In a general face recognition system, a face image captured by a device installed at an entrance gate is transmitted to the server, and the server performs user authentication according to face recognition and face recognition, and the authentication result is transmitted to the device to determine whether the entrance gate is open. .

In the case of the general face recognition system as described above, not only the face recognition function but also the authentication function according to the face recognition result are all implemented on the server. Therefore, when applying the face recognition system to a place with a large number of users entering and exiting, high performance and expensive There is a problem that the cost of building a face recognition system increases due to the need for a server of

In addition, in the case of a general face recognition system, since the face recognition function is concentrated on the server, there is a problem that it becomes impossible to provide the face recognition service itself when a failure occurs in the server or the network.

In addition, in the case of a general face recognition system, since the face image photographed by the device needs to be transmitted to the server through the network, the face image may be leaked to the outside through hacking, and thus there is a problem that personal information security is vulnerable.

In addition, in the case of a general face recognition system, there is a problem that even though the person is the same person, when a face is photographed in a different environment or a face is photographed in a different illuminance, it cannot be distinguished as the same person.

The present invention provides an edge device for face recognition capable of performing face recognition and authentication processing to edge devices by distributing a face recognition model generated in a central server to each edge device. Make it a technical task.

In addition, another technical object of the present invention is to provide an edge device for face recognition capable of performing face recognition and authentication processing without storing a user's face image and personal information.

In addition, another technical object of the present invention is to provide an edge device for face recognition that can easily update an array file corresponding to a face image of a new user.

An edge device for face recognition according to an aspect of the present invention for achieving the above object comprises: a first photographing unit for obtaining a photographed image of a target user to be authenticated; A face recognition unit that inputs the input images generated from the captured image into a face recognition model, extracts a target face image from the input image, and generates a target feature vector from the extracted target face image; An authentication unit for authenticating the target user by comparing the target feature vector with an array file consisting of a plurality of arrays having user identification information and feature vectors extracted from the user's face image; And an interface unit for receiving the array file and the face recognition model from a central server.

According to the present invention, face recognition and authentication processing is performed on edge devices by distributing the face recognition model generated in the central server to each edge device, so even if the face recognition system is applied to a place with a large number of users entering and exiting, high performance and expensive Since a server is not required, there is an effect of reducing the cost of building a face recognition system.

In addition, according to the present invention, since face recognition and authentication processing is performed on the edge device by being distributed to each edge device, the face recognition service can be continuously provided even if a server or network failure occurs, thereby improving the reliability of service provision. Since the face image captured by the edge device is not transmitted to the server, the possibility of leakage of the face image to the outside is prevented in advance, thereby improving the security of personal information.

In addition, according to the present invention, since only the face recognition model and array file for face recognition are stored in the edge device, the user's face image or personal information is not stored, so even if the edge device is hacked, there is no fear of leakage of the user's personal information. It has the effect of enhancing security.

In addition, according to the present invention, when a new user is added, only the array file corresponding to the face image of the new user needs to be updated on the edge device without changing hardware or changing the face recognition model, so that it is easy to add new users.

1 is a block diagram schematically showing the configuration of a face recognition system according to an embodiment of the present invention
2 is a block diagram schematically showing the configuration of a central server according to an embodiment of the present invention.
3A is a diagram illustrating a method of down-sampling a user image to obtain a plurality of user images having different resolutions.
3B is a diagram illustrating landmark coordinates in a face image as an example.
4A to 4D are block diagrams showing the configuration of a face image extracting unit constituting a face recognition model.
5 is a block diagram schematically showing the configuration of a feature vector extraction unit according to the present invention.
6 is a block diagram showing the configuration of a first unit included in a face image processing unit.
7 is a block diagram showing the configuration of a second unit included in a face image processing unit.
8 is a diagram illustrating an example in which a general face recognition model does not recognize the same person.
9 is a diagram illustrating an example in which overlapping regions are generated when a learning image is arranged in a vector space according to a distance between learning images.
10 is a diagram showing an example of arranging a learning image on a two-dimensional angular plane.
FIG. 11 is a diagram exemplarily showing that training images are spaced apart by giving margin angles between training images on a 2D angular plane.
12 is a diagram illustrating an example in which images of the same person photographed in different environments are accurately classified when error reduction is performed by the error reduction unit.
13 is a block diagram schematically showing a configuration of an edge device according to a first embodiment of the present invention
14 is a diagram illustrating a method of authenticating a target user by an authentication unit.
15 is a block diagram schematically showing a configuration of an edge device according to a second embodiment of the present invention.
16A and 16B are diagrams illustrating an example of a depth image generated by a second photographing unit.

The meaning of the terms described in this specification should be understood as follows.

Singular expressions should be understood as including plural expressions unless clearly defined differently in context, and terms such as “first” and “second” are used to distinguish one element from other elements, The scope of rights should not be limited by these terms.

It is to be understood that terms such as "comprise" or "have" do not preclude the presence or addition of one or more other features or numbers, steps, actions, components, parts, or combinations thereof.

The term “at least one” is to be understood as including all possible combinations from one or more related items. For example, the meaning of “at least one of the first item, the second item, and the third item” means 2 among the first item, the second item, and the third item, as well as the first item, the second item, and the third item. It means a combination of all items that can be presented from more than one.

Hereinafter, the configuration of the face recognition system according to the present invention will be described in more detail with reference to FIG. 1.

1 is a block diagram schematically showing the configuration of a face recognition system according to an embodiment of the present invention. As shown in FIG. 1, the face recognition system 100 according to an embodiment of the present invention includes a central server 110 and a plurality of edge devices 120.

The central server 110 generates a face recognition model, and an array file for authentication of the target user using the feature vector extracted from the user's face information input from the user terminal 130 using the generated face recognition model ( Array File). The central server 110 transmits the generated array file to the edge device 120 so that the edge device 120 can authenticate the target user.

To this end, the central server 110 according to the present invention, as shown in Fig. 2, the user registration unit 210, the input image generation unit 215, the face recognition unit 220, the face recognition model 225, an array It includes a file generation unit 230, a face recognition model training unit 240, and an interface unit 250.

The user registration unit 210 receives one or more user images from the user terminal 130 of a user who wishes to register. When the user image is received, the user registration unit 210 checks whether the user is the same person as the user image, and if it is determined that the user is the same person, the user registration unit 210 acquires the access permission information granted to the user, and the user database 212 together with the user image Register at

In an embodiment, the user registration unit 210 may receive identification information of a corresponding user from the user terminal 130 together with a user image. For example, the user registration unit 210 may receive the user's identification information, such as the user's ID, name, phone number, or the user's employee number, together with a corresponding user image. According to this embodiment, the user registration unit 210 may register the user's identification information and the user's access point information together with a corresponding user image in the user database 212.

Meanwhile, when a plurality of user images are input from the user terminal 130, the user registration unit 210 may induce different user images to be input. For example, the user registration unit 210 may induce a user to input a user image photographed in a different environment or a user image photographed in a different illuminance through the user terminal 130. In this way, the user registration unit 210 may improve the accuracy of face recognition by receiving a plurality of user images photographed in different environments or different illuminances from one user.

The input image generation unit 215 generates an input image to be used for face recognition from the user image input by the user registration unit 210. Specifically, the input image generation unit 215 generates a plurality of user images having different resolutions from one user image by down-sampling or up-sampling one user image to a predetermined level. For example, the input image generator 215 may generate a plurality of user images 400a to 400n having different resolutions by down-sampling one user image 400 as illustrated in FIG. 3A.

In one embodiment, the input image generator 215 generates a downsampled user image by applying a Gaussian Pyramid to the user image, or up-sampled by applying a Laplacian Pyramid to the user image. User images can be created.

When a plurality of user images having different resolutions are generated, the input image generator 215 creates a window 405 of a predetermined pixel size on the user image 400 for each user image as shown in FIG. 3B. A plurality of images acquired while moving are generated as input images. The input image generating unit 215 inputs a plurality of generated input images to the face recognition unit 220.

The face recognition unit 220 inputs a plurality of input images generated by the input image generation unit 215 to the face recognition model 225 trained by the face recognition model training unit 250 to provide a face including a face region. An image is acquired, and a feature vector is extracted from the acquired face image.

In one embodiment, the face recognition model 225 may include a face image extracting unit 227 for extracting a face image from an input image and a feature vector extracting unit 229 for extracting a feature vector from the face image.

Hereinafter, with reference to FIGS. 4 to 7, the face recognition unit 220 uses the face image extraction unit 227 and the feature vector extraction unit 229 included in the face recognition model 225 to obtain a face image from the input image. Details of extracting feature vectors will be described in detail.

4A to 4D are block diagrams showing the configuration of a face image extracting unit constituting a face recognition model. The face image extracting unit 227 according to the present invention is configured based on a convolutional neural network (CNN) to extract a face image including a face region from an input image. As shown in FIG. 4A, the face image extracting unit 227 includes a first face detection unit 310, a second face detection unit 320, a third face detection unit 330, and a face image alignment unit ( 340).

The first face detection unit 310 extracts features of each input image by applying a convolution operation to the input image inputted by the face recognition unit 220, and detects a face region on the input image based on the extracted feature. It is extracted primarily.

To this end, the first face detection unit 310 includes n convolution calculation units 311a to 311c, a sampling unit 313, first and second dimension reduction units 315a and 315b, as shown in FIG. 4B, and And a first probability value calculating unit 317.

In an embodiment, the first face detection unit 310 may include three convolution calculation units 311a to 311c. In FIG. 4B, for convenience of explanation, the first face detection unit 310 is illustrated as including three convolutional calculation units 311a to 311c, but this is only an example, and the first face detection unit 310 includes four or more It may include a convolution operation unit or may include one or two convolution operation units.

Each of the first to third convolution calculation units 311a to 311c generates a feature map by applying a convolution filter to the input image, and applies an activation function to the generated feature map to reflect nonlinear characteristics to the feature map. In this case, the convolution filters applied to the first to third convolution calculation units 311a to 311c may be different filters.

In one embodiment, the activation function used in the first to third convolution calculation units 311a to 311c outputs a positive value among pixel values of the feature map as it is, and a negative value decreases by a predetermined size. It may be an output activation function. Here, the activation function refers to a function that gives a weight to a plurality of input information and combines it to output a completed result value.

The sampling unit 313 extracts a feature value from the feature map by applying a sampling filter to the feature map output from the first convolution operation unit 311a. In an embodiment, the sampling unit 313 may extract a maximum value of pixel values included in an area corresponding to the sampling filter on the feature map as a feature value of the feature map. According to this embodiment, the sampling unit 313 may be implemented as a Max Pooling layer, and the dimension of the feature map is reduced through the Max Pooling layer. The sampling unit 313 inputs the feature map with a reduced dimension to the second convolution operation unit 311b.

The first dimension reduction unit 315a reduces the dimension of the feature map output from the third convolution operation unit 311c by applying a first dimension reduction filter to the feature map output from the third convolution operation unit 311c. In an embodiment, the first dimensionality reduction filter may be set as a filter capable of reducing the feature map in two dimensions.

The first probability value calculation unit 317 applies a predetermined classification function to the two-dimensional output data output by the first dimension reduction unit 315a to calculate a first probability value for whether a face region is included in the corresponding input image. Calculate. In an embodiment, the first probability value calculating unit 317 may determine that the face region is included in the input image when the calculated first probability value is equal to or greater than the first threshold value.

The second dimensionality reduction unit 315b applies a second dimensionality reduction filter to the feature map output from the third convolution calculation unit 311c when the first probability value calculated by the first probability value calculation unit 317 is greater than or equal to the first threshold value. Thus, the dimension of the feature map output from the third convolution operation unit 311c is reduced. In one embodiment, the second dimensionality reduction filter may be set as a filter capable of reducing the feature map to four dimensions, and the second dimensionality reduction unit 315b stores four values output in four dimensions on a corresponding input image. It is determined by the coordinates of the face area. At this time, the coordinates of the face area are defined as the coordinates of the upper left vertex and the coordinates of the lower right vertex when the area containing the face is displayed in a rectangular bounding box, or the coordinates of the upper right vertex and the lower left vertex Can be defined by the coordinates of

The second face detection unit 320 receives input images determined to include a face area by the first face detection unit 310 and coordinates of the face area on the input images, and Extracting features of the first sub-input images by applying a convolution operation to the first sub-input images corresponding to the coordinates of the face region, and secondly extracting the face region from the first sub-input images based on the extracted features do.

To this end, as shown in FIG. 4C, the second face detection unit 320 includes n convolution calculation units 321a to 321c, second to third sampling units 323a and 323b, and a first dimension increasing unit 325. ), third and fourth dimensional reduction units 327a and 327b, and a second probability value calculating unit 328.

In one embodiment, the second face detection unit 320 may include three convolution calculation units 321a to 321c. In FIG. 4C, for convenience of explanation, the second face detection unit 320 is shown to include three convolution calculation units 321a to 321c, but this is only an example, and the second face detection unit 320 is The number of convolution calculation units 311a to 312c included in the detection unit 320 or more may be included.

Each of the fourth to sixth convolution calculation units 321a to 312c generates a feature map by applying a convolution filter to the input image, and applies an activation function to the generated feature map to reflect nonlinear characteristics to the feature map. In one embodiment, the activation function used by the fourth to sixth convolution calculation units 321a to 312c outputs a positive value among pixel values of a feature map as it is and a negative value as a value reduced by a predetermined size. It may be an activation function that does.

The second sampling unit 323a extracts feature values from the feature map by applying a sampling filter to the feature map output from the fourth convolution operation unit 321a, and the third sampling unit 323b extracts a feature value from the feature map. ), a sampling filter is applied to the feature map output from () and feature values are extracted from the feature map. In an embodiment, the second and third sampling units 323a and 323b may extract a maximum value among pixel values included in an area corresponding to a sampling filter on each feature map as a feature value of the feature map. . According to this embodiment, the second and third sampling units 323a and 323b may be implemented as a maxpooling layer, and the dimension of each feature map is reduced through the maxpooling layer.

The first dimension increasing unit 325 increases the dimension of the feature map by using a plurality of nodes so that the feature map output from the sixth convolution operation unit 321c has a dimension of a predetermined size. In an embodiment, the first dimension increase unit 325 may increase the dimension so that the feature map output from the sixth convolution operation unit 321c has a size of 128*128 or 256*256.

The first dimension increasing unit 325 applies the activation function to the feature map with an increased dimension, thereby reflecting nonlinear characteristics in the feature map with an increased dimension. In one embodiment, the first dimension increasing unit 325 applies an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a value reduced by a predetermined size. Can reflect non-linear characteristics.

The third dimension reduction unit 327a reduces the dimension of the feature map output from the first dimension increase unit 325 by applying a third dimension reduction filter to the feature map output from the first dimension increase unit 325. In an embodiment, the third dimension reduction filter may be set as a filter capable of reducing the feature map to two dimensions.

The second probability value calculating unit 329 applies a predetermined classification function to the two-dimensional output data output by the third dimension reduction unit 327a to determine whether or not the face region is included in the first sub-input image. 2 Calculate the probability value. In an embodiment, the second probability value calculator 328 may determine that the face region is included in the first sub-input image if the calculated second probability value is greater than or equal to the second threshold value greater than the first threshold value.

The fourth dimensionality reduction unit 327b applies a fourth dimensionality reduction filter to the feature map output from the first dimensionality increase unit 325 when the second probability value calculated by the second probability value calculation unit 329 is equal to or greater than the second threshold value. By applying it, the dimension of the feature map output from the first dimension increasing unit 325 is reduced. In an embodiment, the fourth dimensionality reduction filter may be set as a filter capable of reducing the feature map to four dimensions, and the fourth dimensionality reduction unit 327b converts four values output in four dimensions into a corresponding first sub. It is determined by the coordinates of the face area on the input image. At this time, the coordinates of the face area are defined as the coordinates of the upper left vertex and the coordinates of the lower right vertex when the area including the face is displayed in a rectangular bounding box, or the coordinates of the upper right vertex and the coordinates of the lower left vertex. Can be.

The third face detection unit 330 receives the first sub-input images determined to include the face region by the second face detection unit 310 and the coordinates of the face region on the first sub-input images. , Extracting features of the second sub-input images by applying a convolution operation to second sub-input images corresponding to the coordinates of the face area on the corresponding first sub-input images, and a second sub-input image based on the extracted features The face area is extracted thirdly from the fields.

To this end, the third face detection unit 330 includes n+1 convolution calculation units 331a to 331d, fourth to sixth sampling units 333a to 333c, and a second dimension increasing unit as shown in FIG. 4D. 327), fifth to sixth dimensional reduction units 337a to 337c, and a third probability value calculating unit 339.

In one embodiment, the third face detection unit 330 may include four convolution calculation units 331a to 331d. In FIG. 4D, for convenience of explanation, the third face detection unit 330 is shown to include four convolution calculation units 331a to 331d, but this is only an example, and the third face detection unit 330 is a second face. If more than the number of convolution calculation units 321a to 322c included in the detection unit 320 is included, there may be no limit to the number of convolution calculation units.

Each of the seventh to tenth convolution operation units 331a to 331d generates a feature map by applying a convolution filter to the input image, and applies an activation function to the generated feature map to reflect nonlinear characteristics to the feature map. In one embodiment, the activation function used by the seventh to tenth convolutional units 331a to 331d outputs a positive value among pixel values of a feature map as it is and a negative value as a value reduced by a predetermined size. It may be an activation function that does.

The fourth sampling unit 333a extracts feature values from the feature map by applying a sampling filter to the feature map output from the seventh convolution operation unit 331a, and the fifth sampling unit 333b is the eighth convolution operation unit 331b. ), a sampling filter is applied to the feature map output from the feature map and feature values are extracted from the feature map, and the sixth sampling unit 333c applies the sampling filter to the feature map output from the ninth convolution operation unit 331c Extract feature values from the map. In an embodiment, the fourth to sixth sampling units 333a to 333c may extract a maximum value among pixel values included in an area corresponding to the sampling filter on each feature map as a feature value of the feature map. . According to this embodiment, the fourth to sixth sampling units 333a to 333c may be implemented as a max pooling layer, and the dimension of each feature map is reduced through the max pooling layer.

The second dimension increasing unit 335 increases the dimension of the feature map by using a plurality of nodes so that the feature map output from the tenth convolution operation unit 331d has a dimension of a predetermined size. In an embodiment, the second dimension increase unit 335 may increase the dimension so that the feature map output from the tenth convolution operation unit 331d has a size of 128*128 or 256*256.

The second dimension increasing unit 335 applies the activation function to the feature map with the increased dimension, thereby reflecting the nonlinear characterization on the feature map with the increased dimension. In one embodiment, the second dimensional increase unit 335 applies an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a value reduced by a predetermined size. Can reflect non-linear characteristics.

The fifth dimension reduction unit 337a reduces the dimension of the feature map output from the second dimension increase unit 335 by applying the fifth dimension reduction filter to the feature map output from the second dimension increase unit 335. In an embodiment, the fifth dimensionality reduction filter may be set as a filter capable of reducing the feature map in two dimensions.

The third probability value calculation unit 339 applies a predetermined classification function to the two-dimensional output data output by the fifth dimension reduction unit 337a to determine whether or not the face region is included in the second sub-input image. 3 Calculate the probability value. In an embodiment, if the calculated third probability value is greater than or equal to a third threshold value greater than the second threshold value, the third probability value calculator 339 determines that the face region is included in the second sub-input image.

The sixth dimensionality reduction unit 337b applies a sixth dimensionality reduction filter to the feature map output from the second dimensionality increase unit 335 when the third probability value calculated by the third probability value operation unit 339 is equal to or greater than the third threshold value. By applying, the dimension of the feature map output from the second dimension increasing unit 335 is reduced. In one embodiment, the sixth dimensionality reduction filter may be set as a filter capable of reducing the feature map to four dimensions, and the sixth dimensionality reduction unit 337b converts four values output in four dimensions into a corresponding second sub It is determined by the coordinates of the face area on the input image. At this time, the coordinates of the face area are defined as the coordinates of the upper left vertex and the coordinates of the lower right vertex when the area including the face is displayed in a rectangular bounding box, or the coordinates of the upper right vertex and the coordinates of the lower left vertex Can be.

The sixth dimension reduction unit 337b extracts a face image from the second sub-input image determined to include the face region by using the calculated face region coordinates.

The seventh dimension reduction unit 337c applies a seventh dimension reduction filter to the feature map output from the second dimension increase unit 335 when the third probability value calculated by the third probability value calculator 339 is equal to or greater than the third threshold value. By applying, the dimension of the feature map output from the second dimension increasing unit 335 is reduced. In one embodiment, the seventh dimensionality reduction filter may be set as a filter capable of reducing the feature map to ten dimensions, and the seventh dimensionality reduction unit 337c applies ten values output in ten dimensions to the corresponding second sub It is determined by the coordinates of the landmark on the input image. At this time, the landmark coordinates are the coordinates of the two eyes 420 and 425 on the second sub-input image 350, the coordinates of the nose 430, and the coordinates of the two mouths 440 and 450 as shown in FIG. 3C. ), and the coordinates of the two mouths 440 and 450 refer to the coordinates 440 for the left tail of the mouth and the coordinates 450 for the right tail of the mouth.

As described above, according to the present invention, the face image extraction unit 300 is implemented as the first to third face detection units 310 to 330, and the first face detection unit 310 is in the second face detection unit 320 It is composed of a network of shallow depth, and the second face detection unit 320 is configured as a network of a shallower depth than the third face detection unit 330, so that the face image extracting unit 300 is overall Shallow-to -It is formed in a stepwise form with a deep structure. Through this, it is possible to take advantage in terms of face recognition speed by improving the accuracy of facial image extraction and reducing the complexity at the same time.

The face image alignment unit 340 aligns the face image using the landmark coordinates output from the third face detection unit 330. Specifically, the face image alignment unit 340 aligns the face image by performing at least one of rotation, translation, enlargement and reduction on the face image by using landmark coordinates for the extracted face image. In the present invention, the reason for aligning face images using the face image alignment unit 340 is to improve face recognition performance by giving consistency to a face image to be provided as an input to the feature vector extraction unit 229.

In one embodiment, the face image alignment unit 340 resizes the face image extracted by the third face detection unit 330 to the same resolution as the face image used in the feature vector extraction unit 229. , By moving the landmark coordinates calculated by the third face detection unit 330 based on the reference landmark coordinates for the resolution face image used in the feature vector extraction unit 229, the face image is rotated and translated, You can enlarge or reduce it.

Referring back to FIG. 2, when a face image is input from the face image extracting unit 227, the feature vector extractor 229 extracts a feature vector from the face included in the face image. Hereinafter, a feature vector extraction unit according to the present invention will be described in more detail with reference to FIG. 5.

5 is a block diagram schematically showing the configuration of a feature vector extraction unit according to the present invention. As shown in FIG. 5, the feature vector extraction unit according to an embodiment of the present invention includes a plurality of face image processing units 510a to 510n and a feature vector generation unit 520.

The plurality of face image processing units 510a to 510n process input data to generate output data. At this time, a face image output from the face image extraction unit 227 as an input image is input to the first face image processing unit 510a among the plurality of face image processing units 510a to 510n, and the n+1th face image processing unit 510n Output data of the n-th face image processing unit 510n is input to +1) as an input image.

For example, the first face image processing unit 510a generates output data by image-processing the face image, and generates output data twice?? It is input to the face image processing unit 510b. The first face image processing unit 510b generates new output data by image processing the output data output from the first face image processing unit 510a, and generates new output data 3 times?? It is input to the face image processing unit 510c.

Since the functions and detailed configurations of the plurality of face image processing units 510a to 510n shown in FIG. 5 are the same, hereinafter, the first face image processing unit 510a among the plurality of face image processing units 510a to 510n is exemplary. Explain. Hereinafter, for convenience of description, the first face image processing unit 510a will be referred to as the face image processing unit 510.

As shown in FIG. 5, the face image processing unit 510 performs a convolution operation on input data (a face image or output data of a previous face image processing unit) to generate a feature map. It is composed of a second unit 514 that assigns weight to the feature map generated by the unit 512, and an operation unit 516.

Hereinafter, the configuration of the first unit 512 will be described in more detail with reference to FIG. 6. 6 is a block diagram showing the configuration of a first unit included in a face image processing unit. As shown in FIG. 6, the first unit 512 includes a normalization unit 600, a first convolution operation unit 610, and a non-linearization unit 620.

The normalization unit 600 normalizes face images input from the face image extracting unit 227 in batch units. Arrangement means the number of face images to be processed at once. The reason why the normalizer according to the present invention normalizes in batch units is that when normalization is performed in batch units, the average and variance for each face image may be different from the average and variance for the entire batch. This feature acts as a kind of noise. This is because the overall performance can be improved.

In addition, the complexity of learning increases and gradient vanishing or exploding occurs due to the internal covariance shift in which the distribution of inputs for each layer of the network changes inconsistently through batch normalization. This is because it can be prevented.

The first convolution operation unit 610 generates a first feature map by applying a first convolution filter to the face image normalized by the normalization unit 600. In an embodiment, the first convolution operation unit 610 may generate a first feature map by applying a first convolution filter having a 3*3 pixel size and a stride value of 1 to the face image. As described above, since the first convolution operation unit 610 according to the present invention applies the first convolution filter having a 3*3 pixel size and a stride value of 1 to the face image, the resolution of the first feature map can be highly preserved. do.

The nonlinearization unit 620 imparts nonlinear characteristics to the first feature map by applying the activation function to the first feature map. In one embodiment, the nonlinearization unit 620 outputs the same positive pixel value among the pixel values of the first feature map, and reduces the size of the negative pixel value and outputs an activation function to the first feature map. By applying it, nonlinear characteristics can be given to the first feature map.

In FIG. 6, it has been described that the nonlinearization unit 620 imparts nonlinear characteristics to the first feature map generated by the first convolution operation unit 610. However, in a modified embodiment, the normalization unit 600 may additionally normalize the first feature maps generated by the first convolution operation unit 610 in an arrangement unit. According to this embodiment, the normalization unit 600 provides the normalized first feature map to the non-linearization unit 620, and the non-linearization unit 620 applies an activation function to the normalized first feature map. Nonlinear characteristics can be given to feature maps.

Meanwhile, in the above-described embodiment, it has been described that the first unit 512 includes only the first convolution operation unit 610. However, in a modified embodiment, the first unit 512 may further include a second convolution operation unit 630 as shown in FIG. 6.

Specifically, the second convolution operation unit 630 generates a second feature map by applying a second convolution filter to the first feature map to which nonlinear characteristics are given by the nonlinearization unit 620. In an embodiment, the second convolutional filter may be a filter different from the first convolutional filter. The second convolutional filter may be a filter having the same size of the first convolutional filter but having different stride values. As an example, the second convolution filter may be a filter having a 3*3 pixel size and a stride value of 2.

According to this embodiment, the normalization unit 600 may additionally normalize the second feature maps generated by the second convolution operation unit 630 in batch units.

Meanwhile, although not shown in FIG. 6, the first unit 512 may further include a pre-normalization unit. The pre-normalization unit may normalize pixel values of pixels included in the face image input from the face image extraction unit. As an example, the pre-normalization unit may normalize the face image by subtracting 127.5 from each pixel value of the face image and then dividing the subtraction result value by 128. The pre-normalization unit may provide a pre-normalized input face image to the normalization unit 600.

Referring back to FIG. 5, the second unit 514 assigns a weight to the second feature map generated by the first unit 512. In the present invention, the reason why the second feature map is weighted through the second unit 514 is that in the case of a convolution operation, all channels of the input image are multiplied by a convolution filter and then summed. Since the data becomes entangled, the sensitivity of the data is lowered, so that weights are assigned to the second feature map according to the importance of each channel.

Hereinafter, the configuration of the second unit 514 will be described in more detail with reference to FIG. 7. 7 is a block diagram showing the configuration of a second unit included in a face image processing unit. As illustrated in FIG. 7, the second unit 514 includes a sampling unit 710, a weight reflecting unit 720, and an upscaling unit 730.

First, the sampling unit 710 subsamples the second feature map input from the first unit 512 to reduce the dimension. In an embodiment, the sampling unit 710 may reduce the dimension of the second feature map by applying a global pooling filter to the second feature map. For example, when the dimension of the second feature map is H*W*C, the sampling unit 710 may reduce the dimension of the second feature map to 1*1*C through sub-sampling of the second feature map.

The weight reflecting unit 720 assigns a weight to the second feature map whose dimension has been reduced by the sampling unit 710. To this end, as shown in FIG. 7, the weight reflecting unit 720 includes a dimension reducing unit 722, a first nonlinearizing unit 724, a dimension increasing unit 726, and a second nonlinearizing unit 728. I can.

The dimension reduction unit 722 reduces the dimension of the subsampled second feature map by connecting the subsampled second feature map into one layer. For example, when the dimension of the second feature map output from the sampling unit 710 is 1*1*C, the dimension reduction unit 722 reduces the dimension of the second feature map to 1*1*C/r. Here, r denotes a reduction rate and may be determined according to the number of feature vectors desired to be extracted.

The first non-linearization unit 724 applies the first activation function to the second feature map whose dimension has been reduced by the dimension reduction unit 722, thereby imparting a nonlinear characteristic to the second feature map with a reduced dimension. In one embodiment, the first non-linearization unit 724 reduces the dimension by applying a first activation function that outputs a positive pixel value as it is and outputs a negative pixel value as 0 among the pixel values of the second feature map. Nonlinear characteristics can be given to the second feature map.

The dimension increasing unit 726 increases the dimension of the second feature map to which the nonlinear characteristic is given by the first nonlinear unit 724. As an example, when the dimension of the second feature map to which the nonlinear characteristic is assigned is 1*1*c/r, the dimension increasing unit 726 increases the dimension of the second feature map to 1*1*C again.

The second nonlinearization unit 728 applies the second activation function to the second feature map whose dimension is increased by the dimension increase unit 726 to give the nonlinear characteristic to the second feature map whose dimension is increased again. In one embodiment, the second activation function may be a different function from the first activation function. For example, the second non-linearization unit 728 applies a second activation function in which positive pixel values converge to a predetermined value among pixel values of the second feature map with an increased dimension and outputs a negative pixel value as 0. By doing so, it is possible to impart nonlinear characteristics to the second feature map with an increased dimension.

As described above, the weight reflecting unit 720 according to the present invention provides a second feature map through the dimension reduction unit 722, the first non-linearization unit 724, the dimensionality increase unit 726, and the second non-linearization unit 728). A weight is assigned to and a bottleneck is created through the dimension reduction unit 722 and the dimension increase unit 726 to limit the gating mechanism, thereby limiting model complexity and supporting generalization.

The upscaling unit 730 upscales the second feature map weighted by the weight reflecting unit 720 to the same dimension as the face image input to the second unit 514. In one embodiment, when the dimension of the face image input to the second unit 514 is H*W*C, the upscaling unit 720 sets the dimension of the second feature map to which the weight is assigned to H*W*C. Upscale with.

Referring back to FIG. 5, the operation unit 516 adds the upscaled second feature map output through the second unit 514 with the face image input to the first unit 512. In the present invention, the reason why the upscaled second feature map output from the second unit 514 through the operation unit 516 is added to the face image input to the first unit 512 is a feature when the depth increases in the convolutional neural network. This is to prevent this Vanish Problem.

The feature vector generation unit 520 merges the feature map output from the last face image processing unit 510n among the plurality of face image processing units 510a to 510n into one layer to reduce the dimension, thereby generating a predetermined number of feature vectors. Generate. In an embodiment, the feature vector generation unit 520 may output 128 or more feature vectors from a feature map output from the face image processing unit 510n. For example, the feature vector generation unit 520 may output 512 feature vectors from a feature map output from the face image processing unit 510n.

Referring back to FIG. 2, the array file generation unit 230 generates an array for each user using the feature vector generated by the face recognition unit 220, and merges the generated arrays into one file. Create an array file. The array file generation unit 230 may store the generated array file in an array file database (not shown).

In an embodiment, the array generated by the array file generator 230 may be composed of a plurality of feature vectors obtained from a face image of each user and a key value of each user. In this case, the user's key value includes identification information of each user and access permission information of each user. As described above, the identification information of each user can be defined as the ID, name, phone number, or employee number of each user, and the access permission information of each user includes information on each floor to which each user can access. Can include.

In an embodiment, the array file generator 230 may generate an array file for each location where the edge device 120 is installed. For example, the first array file may be composed of arrays of users who have been granted access rights to the first layer, and the second array file may be composed of arrays of users who have been granted access rights to the second layer. have. To this end, the array file generation unit 230 may generate arrays of each user by dividing each user's arrays for each area to which each user can access. For example, when a first user has permission to access the first and third floors, the array file generation unit 230 may provide a first array and a first array containing access permission information for the first floor for the first user. A second array including access authority information for the third floor can be separately created.

The reason that the array file generation unit 230 according to the present invention generates the array file for each location where the edge device 120 is installed is, when the edge device 120 for authenticating the user's face is installed for each location, a specific location This is because only the array file including the access authority information for the location needs to be transmitted to the edge device 120 installed in the device 120, so that the transmission of the array file and management of the array file in the edge device 120 are facilitated.

In the above-described embodiment, it has been described that the array file generation unit 230 generates an array file for each location, but in the modified embodiment, the array file generation unit 230 is located at all locations where the edge device 120 is installed. It is also possible to generate one array file including the rights information for the generated array file and transmit the generated array file to all edge devices 120.

The edge device registration unit 240 registers information on a plurality of edge devices 120 installed at each location in the edge device information database 242. In an embodiment, the edge device registration unit 240 may map identification information of each edge device 120 to a location where each edge device is installed, and store it in the edge device information database 242. Here, the identification information of the edge device 120 may include a manufacturer and a serial number of the edge device 120.

Meanwhile, the edge device registration unit 240 may receive an authentication record from the edge device 120 at each predetermined period through the interface unit 270 and store the received access record in the edge device information database 242.

The access permission information management unit 250 changes the access permission information assigned to each user or adds new access permission information. In an embodiment, the access permission information management unit 250 may separately grant access permission information for each user, or may grant access permission information in an organizational unit to which each user belongs.

Meanwhile, the central server 110 according to the present invention may further include a scheduler 260. The scheduler 260 performs a function of collectively registering a new user whenever a predetermined period arrives or a predetermined event occurs. For example, in the above-described embodiment, it has been described that the user registration unit 210 performs a registration procedure for a new user when a registration request from a user occurs, but when the central server 110 includes the scheduler 260, a predetermined The scheduler 260 initiates the operation of the user registration unit 210, the input image generation unit 215, and the face recognition unit 220 when a time unit or a predetermined event occurs, so that the new user registration process is automatically performed. can do.

The interface unit 270 encrypts the array file generated by the array file generation unit 230 in a predetermined manner and transmits it to each edge device 120. In one embodiment, the interface unit 270 may encrypt the array file using an encryption algorithm based on a public key and transmit it to each edge device 120.

Meanwhile, the interface unit 270 may transmit the encrypted array file to the edge device 120 according to a protocol promised with the edge device 120.

In addition, the interface unit 270 may receive the authentication record from each edge device 120 every predetermined period and provide it to the edge device 120.

The face recognition model training unit 280 generates a face recognition model 225 based on a convolutional neural network, and trains the generated face recognition model 225. Specifically, the face recognition model training unit 280 generates an optimal face recognition model by continuously training the convolutional neural network constituting the face recognition model 225.

To this end, the face recognition model training unit 280 includes a face image extraction training unit 282 for training the face image extraction unit 227, a feature vector extraction training unit 284 for training the feature vector extraction unit 229, And an error reduction unit 286 for reducing an error of the feature vector extraction unit 229.

The face image extraction training unit 282 trains the first to third face detection units 310 to 330 constituting the face image extraction unit 227 by using the training image. Specifically, the face image extraction training unit 282 inputs a plurality of training images having a predetermined size to the first face detection unit 310 having a structure as shown in FIG. 4B, and the face region is included in the training image. The first probability value and the face region coordinates are calculated, and the calculated first probability value and the face region coordinates are fed back to the first face detection unit 310 according to a back propagation algorithm to the first face detection unit 310. The filter coefficients and weights of the applied convolution filters are updated.

In FIG. 4B, since it has been described that the first face detection unit 310 calculates only the first probability value that the image will include the face region and the coordinates of the face region on the image, the face image extraction training unit 282 It has been described that the probability value and the face region coordinates are fed back to the first face detection unit 310 using a backpropagation algorithm to update the filter coefficients and weights of the convolutional filters applied to the first face detection unit 310.

However, in another embodiment, the face image extraction training unit 282 receives the landmark coordinates from the first face detection unit 310 during training of the first face detection unit 310 in order to improve the accuracy of the landmark coordinate extraction. Filter of convolutional filters applied to the first face detection unit 310 by additionally calculating and feeding the calculated landmark coordinates together with the first probability value and face region coordinates to the first face detection unit 310 through a backpropagation algorithm It may be possible to update the coefficients and weights.

According to this embodiment, the first face detection unit 310 applies a dimension reduction filter to the feature map output from the third convolution operation unit 311c to obtain the landmark coordinates, thereby outputting it from the third convolution operation unit 311c. It may further include a dimension reduction unit (not shown) for reducing the dimension of the feature map to 10 dimensions. At this time, 10 values output in 10 dimensions are determined as the coordinates of the two eyes, which are landmarks, the coordinates of the nose, the coordinates of the left mouth tail, and the coordinates of the right mouth tail.

In addition, the face image extraction training unit 282 inputs a plurality of training images having a predetermined size to the second face detection unit 320 having a structure as shown in FIG. 4C, and the face region is included in the training image. 2 The probability value and the face region coordinates are calculated, and the calculated second probability value and the face region coordinates are fed back to the second face detection unit 320 using a backpropagation algorithm, so that the convolutional filters applied to the second face detection unit 320 are Update the filter coefficient and weight. In this case, the learning image input to the second face detection unit 320 may be selected as a learning image determined by the first face detection unit 310 to include a face region.

In FIG. 4C, since it has been described that the second face detection unit 320 calculates the probability that the image includes the face region and only the face region coordinates on the image, the face image extraction training unit 282 calculates the calculated second probability value and It has been described that the face region coordinates are fed back to the second face detection unit 320 using a backpropagation algorithm to update the filter coefficients and weights of the convolutional filters applied to the second face detection unit 320.

However, in another embodiment, in order to improve the accuracy of the landmark coordinate extraction, the face image extraction training unit 282 may also receive the landmark coordinates from the second face detection unit 320 during training of the second face detection unit 320. Is additionally calculated, and the calculated landmark coordinates are fed back to the second face detection unit 320 through a backpropagation algorithm together with the second probability value and the face region coordinates, so that the convolutional filters applied to the second face detection unit 320 are The filter coefficients and weights could be updated.

According to this embodiment, the second face detection unit 320 applies a dimension reduction filter to the feature map output from the first dimension increase unit 325 in order to obtain the landmark coordinates, so that the first dimension increase unit 325 A dimension reduction unit (not shown) for reducing the dimension of the feature map output from to 10 dimensions may be additionally included. At this time, 10 values output in 10 dimensions are determined as the coordinates of the two eyes, which are landmarks, the coordinates of the nose, the coordinates of the left mouth tail, and the coordinates of the right mouth tail.

In addition, the facial image extraction training unit 282 inputs a plurality of training images having a predetermined size to the third face detection unit 330 having a structure as shown in FIG. 4D, so that the face region is included in the training image. 3 The probability value, the face region coordinates, and the coordinates of the landmark are calculated, and the calculated third probability value, the face region coordinates, and the coordinates of the landmark are fed back to the third face detection unit 330 using a backpropagation algorithm. 3 The filter coefficients and weights of the convolutional filters applied to the face detection unit 330 are updated. In this case, the learning image input to the third face detection unit 330 may be selected as a learning image determined by the second face detection unit 320 to include a face region.

The feature vector extraction training unit 284 trains the feature vector extraction unit 229 having the configuration as shown in FIGS. 5 to 7 by using the training image. Specifically, the feature vector extraction training unit 284 inputs a plurality of training images in a predetermined arrangement unit to the feature vector extraction unit 229 having a structure as shown in FIGS. 5 to 7 to obtain features from each training image. Extract the vector.

The feature vector extraction training unit 284 predicts a probability value to be included in a specific class by applying the extracted feature vectors to a predetermined classification function, and between the predicted probability value (hereinafter referred to as'predicted value') and the actual value. The filter coefficients and weights of the convolutional filters applied to the feature vector extractor 229 are updated by calculating the error and feeding the result back to the feature vector extractor 229 using a backpropagation algorithm.

Meanwhile, the face recognition model training unit 280 according to the present invention may further improve the performance of the feature vector extracting unit 229 by further reducing an error generated when extracting a feature vector through the error reducing unit 286.

Specifically, the error reduction unit 286 includes predicted values and actual values based on the feature vectors extracted through the feature vector extraction unit 229 in the process of the feature vector extraction training unit 284 training the feature vector extraction unit 229. It reduces the error between values. Specifically, the error reduction unit 286 calculates an error between the predicted value and the actual value based on the feature vectors extracted by the feature vector extraction unit 229 from the training image, and converts each training image into a two-dimensional angle so that the error can be reduced. Arranged on a plane and using a probability value according to the placement result, the feature vector extraction training unit 284 can train the feature vector extraction unit 229.

The reason why the face recognition model training unit 280 according to the present invention trains the feature vector extraction unit 279 to reduce errors through the error reduction unit 286 is in the case of a general face recognition model as shown in FIG. In spite of being the same person, when the lighting or environment in which the face was photographed changes, an error such as not being able to distinguish that person is the same person occurs.Thus, a feature vector capable of reducing the error in face recognition through the error reduction unit 286 is This is to train the feature vector extraction unit 229 to be extracted.

The error reducing unit 286 according to an embodiment of the present invention includes a face image arranging unit 287 and a probability calculating unit 289 as shown in FIG. 2.

The face image arranging unit 287 arranges each training image on a 2D angular plane based on a plurality of feature vectors extracted by the feature vector extraction unit 229 for the training image. Specifically, the face image arranging unit 287 calculates the cosine similarity between the training images included in different classes, and calculates the reference angle, which is the separation angle between the training images according to the cosine similarity. It is placed on a plane.

In the present invention, when the face image arranging unit 287 arranges the training images in the vector space according to the distances between the training images calculated based on the feature vector of each training image, as shown in FIG. There is a limitation in that it is difficult to clearly distinguish between the same person and the other person during learning because the overlapping region 900 is inevitable.

Therefore, in the present invention, the face image arranging unit 287 calculates the angle between learning images included in different classes through cosine similarity, and arranges each learning image on a two-dimensional angular plane based on the calculated angle. Is to do.

The probability calculation unit 289 varies a margin angle to be added to the reference angle calculated by the face image arranging unit 287 on a two-dimensional angular plane, and calculates the probability that each learning image will be included in the corresponding class for each variable margin angle. Calculate.

Specifically, as shown in FIG. 10, the probability calculation unit 289 varies the margin angles (P1, P2) added to the reference angles (θ ₁ , θ ₂ ) between each training image, while learning having characteristics overlapping with each other. Images are spaced apart on a two-dimensional angular plane. In an embodiment, the margin angles P1 and P2 may be determined according to a learning rate within a range greater than 0 and less than 90 degrees.

For example, when the learning rate increases, the margin angle may increase accordingly, and when the learning rate decreases, the margin angle may be set to decrease accordingly. In this case, the probability calculator 288 may vary the margin angle by a predetermined reference unit.

When the margin angle is added to the reference angle by the probability calculation unit 289, it can be seen that, as shown in FIG. 11, training images having features that overlap each other in the vector space are arranged to be spaced apart from each other.

The probability calculation unit 289 calculates the probability that each training image will be included in the corresponding class for each margin angle added to the reference angle, and provides the calculated probability value to the feature vector extraction training unit 284, thereby extracting the feature vector and training unit 284. ) Can train the feature vector extraction unit 229 based on the probability value calculated by the probability calculation unit 289. That is, the feature vector extraction training unit 284 updates at least one of the coefficients and weights of the convolutional filters applied to the feature vector extraction unit 229 based on the probability value calculated by the probability calculation unit 288. You will learn (279).

In an embodiment, the probability calculator 289 may calculate a probability that each training image will be included in a corresponding class for each margin angle using Equation 1 below.

In Equation 1, x represents a reference angle, p represents the margin angle, and n represents the number of classes.

In one embodiment, the probability calculation unit 289 is a predetermined test in the feature vector extraction unit 229 trained by the feature vector extraction training unit 284 based on the probability value calculated by the probability calculation unit 289. When the face image is applied, the margin angle can be continuously varied until the error between the predicted value and the actual value becomes less than the reference value.

That is, the probability calculation unit 289 determines the margin angle at the time when the error between the predicted value and the actual value calculated when a predetermined test face image is applied to the trained feature vector extraction unit 229 is less than the reference value, as the final margin angle. Decide. In this case, the error between the predicted value and the actual value may be calculated using a cross entropy function.

When error reduction is performed through the error reduction unit 286 as described above, as illustrated in FIG. 12, even when photographed in different environments or under different lighting conditions, the same person can be accurately classified.

In the above-described embodiment, the face image extraction training unit 282, the feature vector extraction training unit 284, and the error reduction unit 286 constituting the face recognition model training unit 280 are implemented in algorithm-type software. It can be mounted on the central server 110.

Referring back to FIG. 1, the edge device 120 recognizes the face of a target user who wishes to enter the corresponding place using a face recognition model 225 that is arranged in each specific place and distributed by the central server 110. And, based on the recognition result, it performs the function of authenticating the access of the target user.

In the present invention, the reason why the central server 110 does not perform face recognition and authentication of the target user and the edge device 120 performs face recognition and authentication of the target user is the central server for face recognition and authentication of the target user. This is because, when a failure occurs in the central server 110 or the network, face recognition and authentication cannot be performed, and as the number of users increases, an expensive central server 110 is required to be expanded.

Accordingly, the present invention applies an edge computing method to allow the edge device 120 to perform face recognition and authentication of the target user, so that even if a failure occurs in the central server 110 or the network, the face recognition service can be provided. It is possible to improve the reliability of service provision, and even if the number of users increases, it is possible to reduce the cost of building the face recognition system 100 since there is no need to increase the expensive central server 110.

Hereinafter, the configuration of the edge device 120 according to the present invention will be described in more detail with reference to FIG. 13.

13 is a block diagram schematically showing the configuration of an edge device according to the first embodiment of the present invention. 13, the edge device 120 according to the first embodiment of the present invention includes a first photographing unit 1210, an input image generating unit 1250, a face recognition unit 1300, and an authentication unit 1310. ), a face recognition model 1320, an array file update unit 1330, a memory 1340, and an interface unit 1350.

When a target user to be authenticated approaches, the first photographing unit 1210 photographs the target user and generates a photographed image. The first photographing unit 1210 transmits the generated photographed image to the input image generating unit 1250.

The input image generating unit 1250 generates an input image to be used for face recognition from the photographed image of the target user transmitted from the first photographing unit 1210. Specifically, the input image generator 1250 generates images of a plurality of target users having different resolutions from the captured image of one target user by down-sampling or up-sampling the captured image of one target user to a predetermined level.

For example, the input image generator 1250 may generate a down-sampled target user image by applying a Gaussian pyramid to an image of the target user, or may generate an upsampled target user image by applying a Laplacian pyramid to the target user image. .

When a plurality of target user images having different resolutions are generated, the input image generating unit 1250 inputs a plurality of images obtained by moving a window of a predetermined pixel size on the target user image for each target user image. It is created as an image. The input image generation unit 1250 inputs the generated input image to the face recognition unit 1300.

When the input image of the target user is received from the input image generation unit 1250, the face recognition unit 1300 inputs the received input image of the target user into the face recognition model 1320 distributed from the central server 110 to provide a target face. An image is extracted, and a target feature vector is extracted from the extracted target face image. In particular, the face recognition model 1320 distributed from the central server 110 may have an error reduced by learning through the error reduction unit 286 described above.

In addition, the face recognition model 1320 may be updated every predetermined period. For example, the edge device 120 receives a new face recognition model 1320 from the central server 110 whenever the face recognition model 1320 is updated by the central server 110, thereby distributing a previously deployed face recognition model. The 1320 may be updated with a new face recognition model 1320.

Since the face recognition model 1320 used for extracting the target face image and extracting the target feature vector is the same as the face recognition model 225 shown in FIGS. 4 to 8, a detailed description thereof will be omitted.

In addition, the method of extracting the target face image and the target feature vector from the input image of the target user by the face recognition unit 1300 using the face recognition model 1320 is the face recognition unit 220 included in the central server 110 Since is the same as extracting a face image and a feature vector using the face recognition model 225, a detailed description thereof will be omitted.

The authentication unit 1310 compares the target feature vector obtained by the face recognition unit 1300 with the feature vectors included in the array file received from the central server 110 to determine whether the target user is a legitimate user who can access the location. Whether to verify. Hereinafter, a method of authenticating the target user by the authentication unit 1310 will be described in detail.

First, for each array included in the array file, the authentication unit 1310 calculates a first result value obtained by subtracting the target feature vector by the same index from the feature vector included in the corresponding array for each array included in the array file. The authentication unit 1310 calculates a second result value by summing the first result value calculated for each index, and calculates a third result value obtained by subtracting the second result value from a predetermined reference value.

The authentication unit 1310 determines that the user mapped to the array having the largest third result value among the arrays included in the array file is the user most similar to the target user. In this case, when the third result value is greater than or equal to a predetermined threshold value, the authentication unit 1310 authenticates the target user as a user with legitimate authority, and accordingly, the target user may be permitted to enter the corresponding place.

In one embodiment, the threshold value may be set differently according to the security level of the place where the edge device 120 is installed. For example, when the edge device 120 is installed in an area to which a high security level is applied, the threshold value may be set high, and when the edge device 120 is installed in an area to which a low security level is applied, the threshold value may be set low. I can.

Hereinafter, the authentication unit 1310 compares the target feature vector acquired by the face recognition unit 1300 with the feature vectors included in the array file to verify whether the target user is a legitimate user who can access the floor. Let me explain with an example.

14 is a diagram illustrating a method of authenticating a target user by an authentication unit. As shown in FIG. 14, in the array file, an array including feature vectors of each user is arranged for each row. For example, feature vectors for a first user are sequentially arranged in a first row, and feature vectors for a second user are sequentially arranged in a second row. In this case, feature vectors of each user are arranged in one row according to the index order.

As shown in FIG. 14A, the authentication unit 1310 calculates a difference value between the feature vectors of each array of the array file 1410 and the target feature vectors for each index, and the difference value calculated as shown in FIG. 14B A first result value is calculated by squaring them, and a second result value is calculated by summing all the first result values for each array, as shown in FIG.

Thereafter, as shown in FIG. 14D, the authentication unit 1310 calculates a third result value by subtracting the second result value from a predetermined reference value (eg, 1), and the largest among the calculated third result values. The user mapped to the array corresponding to the value of 0.310528 is determined as the user most similar to the target user. Further, since the third result value determined as the largest value is a value greater than a predetermined threshold value (eg, 0.15), the authentication unit 1310 finally authenticates the target user as a user mapped to the corresponding array.

When the authentication unit 1310 authenticates the target user by an equation, it can be expressed as in equation 2 below.

In Equation 2, Z represents a third result value, R represents a predetermined reference value, Xi represents a feature vector corresponding to the i-th index among n feature vectors, and Yi is the i-th index among n feature vectors. Represents a target feature vector corresponding to.

The face recognition model 1320 is generated and distributed by the central server 110, and the face recognition model 1320 is an updated face whenever the face recognition model 225 is trained and updated by the central server 110. It is replaced by the recognition model 225. In this case, the face recognition model 1320 may be received from the central server 110 through the interface unit 1350.

When an array file is received from the central server 110 through the interface unit 1350, the array file update unit 1330 uploads it to the first memory 1342 so that the authentication unit 1310 authenticates the target user using it. Make it possible. In particular, the array file update unit 1330 according to the present invention may dynamically load an array file.

Specifically, the array file update unit 1330 stores the new array file in the second memory 1344 when a new array file is received from the central server 110 when an array file is loaded in the first memory 1342. After loading, and when loading of the new ray file in the second memory 1344 is completed, the array file loaded in the first memory 1342 is replaced with a new array file loaded in the second memory 1344.

The reason why the array file update unit 1330 according to the present invention dynamically loads the array file as described above is that the authentication unit 1310 performs authentication processing for the target user and the array file update unit 1330 This is to enable the edge device 120 to perform face recognition in real time based on the newly updated array file by enabling the file to be updated.

An array file used by the authentication unit 1310 is loaded into the first memory 1342, and a new array file newly received is loaded into the second memory 1344. When the loading of the new array file in the second memory 1344 is completed, the array file recorded in the first memory 1342 is replaced by the new array file by the array file update unit 1330.

The interface unit 1350 mediates data transmission/reception between the edge device 120 and the central server 110. Specifically, the interface unit 1350 receives the face recognition model 1320 from the central server 110, receives the array file from the central server 110, and receives the first memory 1342 through the array file update unit 1330. ) Or the second memory 1344. In addition, the interface unit 1350 periodically transmits the authentication record by the authentication unit 1330 to the central server 110.

In an embodiment, the array file and the face recognition model 1320 may be updated at predetermined cycles through the interface unit 1350.

As described above, according to the present invention, only the face recognition model 1320 and array file for face recognition are stored in the edge device 120, and the user's face image or personal information is not stored. Even if it is hacked, there is no fear that the user's personal information will be leaked, thus enhancing security.

In the case of the edge device 120 according to the first embodiment described above, it has been described that the photographed image generated by the first photographing unit 1210 is directly input to the input image generating unit 1250. However, according to this embodiment, even when a picture including the face of the target user is taken by the first photographing unit 1210, the first photographing unit 1210 generates a normal photographed image and the input image generating unit 1250 By transmitting to, there is a problem that a user other than the target user may perform authentication using the photo of the target user.

Hereinafter, the edge device 120 according to the second embodiment capable of solving the above-described problem will be described in detail.

15 is a block diagram showing a configuration of an edge device according to a second embodiment of the present invention. The edge device according to the second embodiment shown in FIG. 15 is first compared to the edge device according to the first embodiment shown in FIG. 13 in that it further includes a second photographing unit 1370 and an authenticity determining unit 1380. It is distinguished from the edge device according to the first embodiment. Hereinafter, for convenience of explanation, a description of a configuration having the same function as the edge device according to the first embodiment is omitted, and the newly added second photographing unit 1510 and the authenticity determining unit 1520 and the newly added It will be described only for the first photographing unit 1210 whose function has been changed due to the configuration.

The first photographing unit 1210 photographs a photographing object and generates a photographed image. The first photographing unit 1210 transmits the generated photographed image to the authenticity determining unit 1520.

The second photographing unit 1510 photographs an object to be photographed and generates a depth image. The second photographing unit 1510 may be performed at the same time point as when the object to be photographed is photographed by the first photographing unit 1210 or before or after a predetermined time from the time the object is photographed by the first photographing unit 1210. You can shoot the subject.

In one embodiment, the second photographing unit 1510 may be implemented as an IR camera capable of generating a depth image by photographing a photographing object.

As described above, the reason that the edge device 120 according to the second embodiment photographs a photographing object through the second photographing unit 1510 to generate a depth image is that the actual face of the photographing object is determined by the second photographing unit 1510. This is because different types of depth images are generated when a picture is taken and a picture including a face of the subject is taken.

For example, when a picture including the face of the subject to be photographed is taken by the second photographing unit 1510, a depth image in the form as shown in FIG. 16A is generated, whereas the photographing subject by the second photographing unit 1510 When the actual face of is captured, a second depth image of the shape as shown in FIG. 16B is generated.

The second photographing unit 1510 transmits the generated depth image to the authenticity determining unit 1520.

The authenticity determining unit 1520 determines whether the photographed object photographed by the second photographing unit 1510 is a photograph or whether the actual photographing object is a face using the depth image transmitted from the second photographing unit 1510.

Specifically, the authenticity determination unit 1520 extracts depth data from the depth image received from the second photographing unit 1510, and determines whether the depth image is a real face of the subject to be photographed through binary classification. . In an embodiment, the authenticity determination unit 1520 may improve classification accuracy for real faces and photos through training based on a deep learning algorithm.

The authenticity determining unit 1520 transmits the photographed image received from the first photographing unit 1210 to the input image generating unit 1250 when it is determined that the photographing target photographed by the second photographing unit 1510 is a real face. . On the other hand, the authenticity determination unit 1520, if it is determined that the photographed object photographed by the second photographing unit 1510 is not a real face, the photographed image received from the first photographing unit 1210 is input to the input image generating unit 1250. Without sending to, the authentication process can be output using an alarm message in text or voice format, or an abnormal access attempt can be notified to the system operator.

As described above, according to the present invention, by accurately determining whether the photographed subject is a real face or a photograph, the authentication process is not performed when the photograph is taken, so that a user without a legitimate authority attempts to obtain authentication using another person's photograph. Attempts can be fundamentally blocked, and security can be improved through this.

Referring back to FIG. 1, the user terminal 130 transmits a user image for newly registering a user together with the user's identification information to the central server 110. In one embodiment, the user terminal 130 is equipped with a face registration agent (not shown) capable of interworking with the central server 110, and the user executes the face registration agent on the user terminal 130 An image of a face or a previously photographed image may be transmitted to the central server 110 together with user identification information.

In one embodiment, the user terminal 130 may request to register a plurality of user images for each user. In this case, the plurality of images requested for registration for each user may be photos taken in different environments or photos taken under different lighting.

The user terminal 130 may use any type without limitation as long as it can request user registration by transmitting a user image to the central server 110. For example, the user terminal 130 may be implemented as a smartphone, notebook, desktop, or tablet PC.

Those skilled in the art to which the present invention pertains will appreciate that the above-described present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof.

For example, the configuration of the central server illustrated in FIG. 2 and the configuration of the edge device illustrated in FIG. 13 may be implemented in a program form. When the configuration of the central server and the edge device according to the present invention are implemented as programs, each of the configurations shown in FIGS. 2 and 13 are implemented as codes, and codes for implementing specific functions are implemented as one program, or In addition, a plurality of programs may be divided and implemented.

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: face recognition system 110: central server
120: edge device 130: user terminal
210: user register 215: input image generation unit
220: face recognition unit 225: face recognition model
230: array file generation unit 240: face recognition model training unit
250: interface unit

Claims (18)

A first photographing unit that acquires a photographed image of a target user to be authenticated;
A face recognition unit that inputs the input images generated from the captured image into a face recognition model, extracts a target face image from the input image, and generates a target feature vector from the extracted target face image;
An authentication unit for authenticating the target user by comparing the target feature vector with an array file consisting of a plurality of arrays having feature vectors extracted from face images of each of a plurality of users and identification information of each user; And
Including an interface unit for receiving the array file and the face recognition model from the central server,
The face recognition model is trained through an error reduction algorithm,
The error reduction algorithm arranges the training images on a two-dimensional angular plane based on a plurality of feature vectors obtained by inputting a training image to the face recognition model, and adds them to a reference angle between training images included in different classes. The edge device for face recognition, characterized in that, by varying the margin angle, the probability that each learning image will be included in each class for each of the variable margin angles is calculated, and the face recognition model is trained based on the calculated probability. The method of claim 1,
The authentication unit,
A first result value obtained by subtracting the target feature vector by the same index from the feature vector included in each array and squared is calculated, and a second result value obtained by summing the first result values calculated for each index in the array is used. Edge device for face recognition, characterized in that to authenticate the target user. The method of claim 2,
The authentication unit,
A user whose third result value is mapped to the largest array among the plurality of arrays included in the array file by subtracting the second result value from a predetermined reference value for each array Edge device for face recognition, characterized in that it is determined to be the user most similar to the target user. The method of claim 2,
The authentication unit,
A face recognition edge, characterized in that: subtracting the second result value from a predetermined reference value for each array to calculate a third result value, and authenticating the target user when the calculated third result value is greater than or equal to a predetermined threshold value device. The method of claim 4,
The edge device for face recognition, characterized in that the threshold value is differentially set according to the security level of a place where the edge device is installed. The method of claim 1,
When a new array file is received from the central server through the interface unit when the array file is loaded into the first memory, the new array file is loaded into the second memory, and the new array file is loaded into the second memory. The edge device for face recognition, further comprising an array file update unit for replacing the array file loaded in the first memory with a new array file loaded in the second memory upon completion. The method of claim 1,
And an input image generator configured to generate a plurality of input images having different resolutions by up-sampling or down-sampling the photographed image captured by the first photographing unit. The method of claim 1,
The face recognition model,
A face image extraction unit for extracting the target face image from the input image;
A plurality of face image processing units that image-process the input data to generate output data; And
A feature vector generator for generating a predetermined number of feature vectors by merging the output data output from the last face image processor among the plurality of face image processors into one layer,
The target face image extracted by the face image extracting unit as the input image is input to a first face image processing unit among the plurality of face image processing units, and the nth face as the input image is input to the n+1th face image processing unit. Edge device for face recognition, characterized in that the output data of the image processing unit is input. The method of claim 8,
The face image extracting unit extracts the target face image from the input images using two or more face detection units having neural network networks of different depths,
The two or more face detection units are sequentially arranged in the order of deepening of the depth, so that the number of input images input to the n-th face detection unit is reduced than the number of input images input to the n-1th face detection unit. Edge device for face recognition. The method of claim 8,
The face image extraction unit,
A feature map of a plurality of input images is generated using a first neural network network having a first depth, and a plurality of first sub-input images including a face region among the plurality of input images based on the feature map A first face detection unit that primarily selects them;
Using a second neural network network having a second depth deeper than the first depth, feature maps of the plurality of first sub-input images are generated, and the plurality of feature maps are generated based on the feature maps generated through the second neural network network. A second face detection unit that secondarily selects a plurality of second sub-input images including a face region from among the first sub-input images; And
A feature map of the plurality of second sub-input images is generated using a third neural network network having a third depth deeper than the second depth, and the plurality of feature maps are generated based on the feature map generated through the third neural network network. And a third face detection unit configured to select the target face image including a face region among second sub-input images. The method of claim 8,
The face image extraction unit,
Obtaining landmark coordinates from the target face image, and performing at least one of rotation, translation, enlargement, and reduction on the target face image so that the obtained landmark coordinates match a predetermined reference landmark coordinate Face recognition edge device, characterized in that it further comprises a face image alignment unit for aligning the target face image. The method of claim 8,
The face image processing unit,
A normalization unit normalizing the input data;
A first convolution operation unit generating a first feature map by applying a first convolution filter to the normalized input data; And
And a non-linearization unit that outputs a positive value among pixel values of the first feature map as it is and outputs a negative value by reducing its size to give the first feature map a non-linear characteristic. Edge device for recognition. The method of claim 12,
The face image processing unit,
Further comprising a second convolution operation unit for generating a second feature map by applying a second convolution filter to the first feature map to which the nonlinear characteristic is assigned,
The first convolutional filter and the second convolutional filter are filters having the same size and different stride values. The method of claim 8,
The face image processing unit,
A sampling unit for sub-sampling the generated feature map to reduce the dimension of the feature map;
A weight reflecting unit that reflects a weight on the sub-sampled feature map; And
And an upscaling unit for upscaling the feature map in which the weight is reflected to the same dimension as the feature map input to the sampling unit. The method of claim 14,
The weight reflecting unit,
A dimension reduction unit for reducing a dimension by connecting the sub-sampled feature maps into one layer;
A first non-linearizer configured to provide a nonlinear characteristic to the reduced-dimensional feature map by outputting a positive value as it is and a negative value as 0 among pixel values of the feature map whose dimension has been reduced;
A dimension increasing unit for increasing the dimension of the feature map to which the nonlinear characteristic is assigned; And
Includes a second non-linearization unit that gives non-linear characteristics to the feature map with the increased dimension by converging a positive value to a predetermined value and outputting a negative value as 0 among pixel values of the feature map with an increased dimension. Edge device for face recognition, characterized in that. delete The method of claim 1,
The face recognition edge device, characterized in that the array file and the face recognition model are updated every predetermined period. The method of claim 1,
A second photographing unit that photographs an object to be photographed and generates a depth image; And
Based on the depth image generated by the second photographing unit, it is determined whether the photographing object is a face or a photograph, and when it is determined that the photographing object is a face, the photographing image generated by the first photographing unit is input Edge device for face recognition, characterized in that it further comprises an authenticity determination unit to be generated as an image.

Images