KR102312152B1 - Face Recognition Server For Reflecting Space-Time Environment and Face Recognition System Having The Same - Google Patents

Abstract

A face recognition server and a face recognition system for face recognition according to a change in a spatio-temporal environment that can change a reference image registered in advance to perform face recognition according to a change in a spatio-temporal environment is a system that measures the similarity between the user's photographed image and the user's reference image. an authentication result collection unit that collects authentication results for authenticating access by comparison with a reference threshold; Among the authentication results, second authentication results normally approved as a registered user are extracted, the second authentication results are classified for each user, and the second authentication result having the maximum value among the similarities included in the classified second authentication result. an image determination unit for determining a photographed image corresponding to , as an optimal reference image of a corresponding user; and a reference image changing unit for changing the reference image of users into an optimal reference image determined for each user.

Description

Face Recognition Server For Reflecting Space-Time Environment and Face Recognition System Having The Same}

The present invention relates to facial recognition technology.

Face Recognition technology is one of the fields of biometrics. It refers to a technology in which a machine automatically identifies and authenticates a person using the unique feature information contained in each person's face. It has superior security compared to the authentication method, and has been widely used in various fields recently.

A general face recognition system transmits a face image taken from a device installed in an access gate, etc. to the server, and the server performs face recognition and user authentication according to face recognition, and transmits the authentication result to the device to determine whether the access gate is open. .

A general facial recognition system performs authentication for a corresponding target user by comparing it with a pre-registered reference image and a target user image captured by the device. At this time, since the pre-registered reference image is taken for registration, it is different from the user image captured by the device, even for a registered user, depending on the location of the device performing authentication, camera information of the device, change of time, etc. There is a problem that can be judged.

The present invention provides a face recognition server and a face recognition system for face recognition according to a change in a space-time environment that can change a reference image registered in advance to perform face recognition according to a change in a space-time environment. make it a technical task.

In addition, the present invention is to provide a face recognition server and a face recognition system for face recognition according to changes in the spatiotemporal environment that can change the reference image based on the authentication results that normally approve the user to be authenticated as a registered user make it a task

In order to achieve the above object, a face recognition server and a face recognition system for face recognition according to a change in a spatiotemporal environment according to an aspect of the present invention compare the similarity between the user's photographed image and the user's reference image with a reference threshold to enter and exit. an authentication result collection unit that collects authentication results that have authenticated the ; Among the authentication results, second authentication results normally approved as a registered user are extracted, the second authentication results are classified for each user, and the second authentication result having the maximum value among the similarities included in the classified second authentication result. an image determination unit for determining a photographed image corresponding to , as an optimal reference image of a corresponding user; and a reference image changing unit for changing the reference image of users into an optimal reference image determined for each user.

According to the present invention, since the reference image registered in advance for face recognition can be changed according to the change of the spatio-temporal environment, there is no limitation on the user's aging according to the change of time or the resolution of the user's photographed image according to the characteristics of the place. Since the user can be authenticated, the authentication accuracy is improved.

In addition, according to the present invention, since the reference image is changed based on the authentication results that normally approve the user to be authenticated as a registered user, it is possible not only to ensure the reliability of the changed reference image, but also to improve the security performance. It has the effect that it can be

1 is a block diagram schematically showing the configuration of a face recognition system including a face recognition server according to an embodiment of the present invention.
2 is a block diagram schematically showing the configuration of a face recognition server according to an embodiment of the present invention.
3 is a diagram showing the configuration of the edge device manager 210 according to an embodiment.
FIG. 4A is a view showing an example of a photographed image of a target user photographed at a first place.
As shown in FIG. 4B , it is a view showing an example of a photographed image of a target user photographed at a second place.
5 is a diagram illustrating the arrangement of an optimal reference image according to a time interval.
6A is a diagram illustrating an example in which a reference image changing unit changes a reference image to an optimal reference image for each location.
6B is a diagram illustrating an example in which the reference image changing unit 340 changes the reference image to an optimal reference image for each location and entry/exit time.
7 is a block diagram schematically showing the configuration of an edge device according to the first embodiment of the present invention.
8 is a diagram exemplarily illustrating a method in which an authenticator authenticates a target user.
9 is a block diagram schematically showing the configuration of an edge device according to a second embodiment of the present invention.

The meaning of the terms described herein should be understood as follows.

The singular expression is to be understood as including the plural expression unless the context clearly defines otherwise, and the terms "first", "second", etc. are used to distinguish one element from another, The scope of rights should not be limited by these terms.

It should be understood that terms such as “comprise” or “have” do not preclude the possibility of addition or existence of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

The term “at least one” should be understood to include all possible combinations from one or more related items. For example, the meaning of “at least one of the first, second, and third items” means 2 of the first, second, and third items as well as each of the first, second, or third items. It means a combination of all items that can be presented from more than one.

Hereinafter, the configuration of a face recognition system including a face recognition server (hereinafter referred to as 'facial recognition server') and a face recognition system for face recognition according to a change in the spatiotemporal environment according to the present invention will be described in more detail with reference to FIG. 1 .

1 is a block diagram schematically showing the configuration of a face recognition system including a face recognition server according to an embodiment of the present invention. As shown in Figure 1, the face recognition system (100, hereinafter, referred to as 'face recognition system') including a face recognition server according to an embodiment of the present invention is a face recognition server 110 and a plurality of edge devices (120) are included.

The face recognition server 110 according to the present invention receives authentication results from a plurality of edge devices 120 that perform authentication of a target user who is an authentication target, and based on this, the edge device 120 according to a change in the spatiotemporal environment. Change the reference image of the registered user so that authentication can be performed. In this case, the registered user means a user registered in the face recognition server 110, and the reference image is an image compared with the target user's photographed image when the edge device 120 performs authentication of the target user.

The facial recognition server 110 transmits the changed reference image to the plurality of edge devices 120 .

On the other hand, the face recognition server 110 generates a face recognition model, and uses the generated face recognition model to authenticate the target user using a feature vector extracted from the user's face information input from the user terminal 130 . Create an Array File. In addition, the facial recognition server 110 may generate an array file for authentication of the target user by using the feature vector extracted from the changed user's reference image.

The face recognition server 110 transmits the generated array file to the edge device 120 so that the edge device 120 can authenticate the target user.

As shown in FIG. 2, the face recognition server 110 according to the present invention includes an edge device manager 210, a user register 220, a user database 222, an input image generator 225, and a face recognition unit. 230 , a face recognition model 235 , an array file generation unit 240 , an access right information management unit 250 , an interface unit 270 , and a face recognition model training unit 280 .

The edge device manager 210 according to the present invention registers information on a plurality of edge devices 120 installed in each location in the edge device information database 219 . In an embodiment, the edge device manager 210 may store identification information of each edge device 120 in the edge device information database 219 by mapping the identification information of each edge device to a place where each edge device is installed. Here, the identification information of the edge device 120 may include a manufacturer and a serial number.

Meanwhile, the edge device manager 210 may receive a plurality of authentication results from the edge device 120 every predetermined period through the interface 270 and store the received authentication results in the edge device information database 219 . . In this case, the plurality of authentication results are obtained by the edge device 120 performing authentication on target users.

The edge device management unit 210 according to the present invention may change a reference image previously registered for a registered user to an optimal reference image reflecting the authentication environment based on a plurality of authentication results. In more detail, the edge device manager 210 may determine an optimal reference image according to a change in a spatio-temporal environment based on a plurality of authentication results, and may change the reference image of each user into an optimal reference image.

In addition, the edge device manager 210 according to the present invention may collect a plurality of authentication results and change the reference threshold based on the collected authentication results. In this case, the reference threshold is a reference value that determines whether the target user is authenticated when the edge device 120 authenticates the target user. Specifically, when the edge device 120 authenticates the target user, the edge device 120 calculates a degree of similarity between the captured image of the target user and the reference image of the registered user. In this case, the edge device 120 compares the similarity with a reference threshold to determine whether the target user is authenticated.

Hereinafter, the edge device manager 210 according to the present invention will be described in more detail with reference to FIG. 3 .

3 is a diagram showing the configuration of the edge device manager 210 according to an embodiment.

The edge device manager 210 includes an authentication result collecting unit 310 , an image determining unit 330 , and a reference image changing unit 340 .

The authentication result collection unit 310 collects a plurality of authentication results from each edge device 120 . In more detail, the authentication result collection unit 310 may collect a plurality of authentication results from each edge device 120 through the interface unit 270 for a predetermined period. In this case, the authentication result may include a first authentication result in which an unregistered user is incorrectly approved as a registered user, a second authentication result in which the user is normally approved as a registered user, and a third authentication result in which the registered user is incorrectly approved as an unregistered user. can Here, the second authentication result may be described as a normal authentication result.

The authentication result collection unit 310 stores a plurality of collected authentication results in the edge device database 219 .

The image determining unit 330 determines an optimal reference image based on a plurality of authentication results.

The image determination unit 330 extracts a plurality of second authentication results that normally approve the user as a registered user from the plurality of authentication results. Accordingly, the present invention can ensure the reliability of the optimal reference image determined based on the second authentication result by excluding the predetermined erroneously approved authentication result.

In an embodiment, the plurality of authentication results include a photographed image of a target user, a degree of similarity between a photographed image of the target user and a reference image of a registered user, device information of the edge device 120, information on a place where the edge device is installed, information on access time , and registered user information. In this case, the captured image of the target user means an image captured by the edge device 120 . The similarity between the photographed image of the target user and the reference image of the registered user means the similarity calculated when the edge device 120 approves the target user. The device information of the edge device 120 means information of the edge device 120 from which the corresponding approved access data is generated and camera information for photographing the target user.

The image determining unit 330 selects an optimal reference image reflecting the authentication environment by using the second authentication result. In this case, the authentication environment means an environment determined according to at least one of place information, access time information, and device information.

The reason that the image determining unit 330 according to the present invention selects the optimal reference image reflecting the authentication environment is the target user's photographed image photographed in the first place as shown in Fig. 4a according to the authentication environment, and Fig. 4b As shown in Fig., since the shot image of the target user photographed at the second location changes each time authentication is performed, the edge device 120 cannot quickly authenticate the target user, but also is a registered user. This is because you can authenticate as an unregistered user.

As shown in FIGS. 4A and 4B , different photographed images are generated even in the same place for the same target user, and the difference in image quality between the photographed image of the target user in FIG. 4A and the photographed image of the target user in FIG. 4B is also Occurs. In addition, as shown in FIGS. 4A and 4B , the brightness of the photographed images taken at different time periods is also different.

Accordingly, the present invention has an effect that by selecting an optimal reference image reflecting the authentication environment, not only can authentication be performed quickly, but also errors in judging a registered user as an unregistered user can be reduced.

The image determiner 330 classifies the second authentication results for each user. In detail, the image determining unit 330 classifies the second authentication results of the normal approval of the user as a registered user for each user included in the second authentication result.

An example will be described. In the plurality of second authentication results, if the 2-a authentication result, the 2-b authentication result, the 2-c authentication result, and the 2-d authentication result exist, and the first and second users exist, the image is determined The unit 330 classifies the second authentication results generated by the first and second users for each user.

If the 2-a authentication result and the 2-b authentication result exist for the first user, and the 2-c authentication result and the 2-d authentication result exist for the second user, the image determination unit 330 . classifies the plurality of second authentication results into 2-a authentication results and 2-b authentication results for the first user, and classifies them into 2-c authentication results and 2-d authentication results for the second user do.

In an embodiment, the image determiner 330 may classify the second authentication result classified for each user by access time at which the second authentication result occurred. In this case, the access time may be divided into a daily time section and a weekly, monthly, quarterly, or yearly time section, respectively.

An example will be described.

It is classified into 2-a authentication result, 2-b, 2-c authentication result for the first user, 2-d authentication result, 2-e authentication result, 2-f for the second user Classified as authentication results, authentication results 2-a, 2-b, and 2-f occur in the morning, and authentication results 2-c, 2-d, and 2-e occur in the afternoon. assumed to have occurred in

In this case, the image determination unit 330 classifies the second authentication result of the first user into the 2-a and 2-b authentication results that occurred in the morning time, and classifies the 2-c authentication result that occurred in the afternoon time. can do. The image determiner 330 may classify the second authentication result of the second user as the 2-f authentication result that occurred in the morning time, and classify the 2-d and 2-e authentication result that occurred in the afternoon time. .

In an embodiment, the image determiner 330 may classify the second authentication result classified for each user by location where the second authentication result occurs.

In an embodiment, the image determiner 330 may classify the second authentication result classified by place for each edge device 120 in which the second authentication result is generated. According to this embodiment, a plurality of edge devices 120 may exist for each location.

In an embodiment, the image determiner 330 may classify the second authentication result classified by place by access time at which the second authentication result occurs. In addition, the second authentication result classified for each edge device 120 may be classified by access time at which the second authentication result occurred.

Meanwhile, the image determiner 330 may determine a photographed image corresponding to the second authentication result having the maximum value among the similarities included in the classified second authentication result as the optimal reference image of the user.

In an embodiment, when the second authentication result classified for each user is classified by access time at which the second authentication result occurred, the image determiner 330 determines whether the similarity is determined by the second authentication result classified for each access time. The photographed image corresponding to the second authentication result having the maximum value may be determined as the optimal reference image of the corresponding access time.

For example, if the first user is classified as the 2-a and 2-b authentication results in the morning time and classified as the 2-c and 2-d authentication results in the afternoon time, the image determining unit is the first user in the morning time zone. The photographed image corresponding to the second authentication result having the maximum similarity among the 2 authentication results (the 2-a and 2-b authentication results) is determined as the optimal reference image for the corresponding access time, and the second authentication is performed in the afternoon. A photographed image corresponding to the second authentication result having the maximum similarity among the results (the authentication results 2-c and 2-b) is determined as the optimal reference image for the corresponding access time.

In an embodiment, when the second authentication result classified for each user is classified for each place where the second authentication result occurs, the image determiner 330 may have a maximum similarity among the second authentication results classified for each place. It is possible to determine the photographed images corresponding to the second authentication result having

In an embodiment, when the second authentication result classified for each location is classified for each edge device 120 in which the second authentication result has occurred, the image determiner 330 may be configured to A photographed image corresponding to the second authentication result having the maximum similarity among the two authentication results may be determined as the optimal reference image of the corresponding edge device 120 .

As described above, in the present invention, even if the plurality of edge devices 120 are located in the same place, the position of the camera and the performance of the camera may be different for each edge device, and thus the optimal reference image may be determined accordingly. In this case, the performance of the camera may include a pixel, an angle of view, a sensitivity, and the like.

In addition, the image determination unit 330 classifies the second authentication result classified for each location by access time at which the second authentication result occurred or the second authentication result classified for each edge device 120 when the second authentication result occurred. Even when classified by access time, each optimal reference image can be determined according to each classified criterion. Conversely, the image determiner 330 may determine each optimal reference image according to the classified criteria.

The reference image changing unit 340 changes the reference images of pre-registered users to the optimal reference image selected by the image determining unit 330 . Specifically, the reference image changing unit 340 changes the reference images of users into the optimal reference images determined for each user.

In an embodiment, the reference image changing unit 340 may change the reference image to an optimal reference image determined by the image determining unit 330 for each entry/exit time, respectively.

Also, the reference image changing unit 340 may change the reference image to the optimal reference image determined for each location by the image determining unit 330 . Also, the reference image changing unit 340 may change the reference image to an optimal reference image determined for each edge device 120 by the image determining unit 330 .

For example, the image determining unit 330 selects the first photographed image of the first user in the place A as the optimal reference image, the second photographed image of the first user in the place B is selected as the optimal reference image, and the location C When the third photographed image of the first user is selected as the optimal reference image, the reference image changing unit 340 changes the reference image in the place A to the first photographed image, and the second photographing the reference image in the place B The image is changed, and the reference image at location C is changed to the third photographed image.

In an embodiment, when the reference image changing unit 340 changes the optimal reference image for each access time, as shown in FIG. 5 , the time period T1 per day and the time per week, month, quarter, or year The optimal reference image may be changed according to the section T2. The reference image change unit 340 registers an optimal reference image according to the time period T1 per day in the row, and registers the optimal reference image according to the time period T2 in units of weeks, months, quarters, or years in the column. . This is only an example, and the rows and columns of the time interval T1 per day and the time interval T2 per week, month, quarter, or year may be reversed.

This is because the user's photographed image may be photographed differently during the day according to changes in external sunlight, and may be photographed differently depending on climate change according to season, aging of each user, weight change, wearing glasses or not.

6A is a diagram illustrating an example in which the reference image changing unit 340 changes the reference image to the optimal reference image for each location. 6B is a diagram illustrating an example in which the reference image changing unit 340 changes the reference image to the optimal reference image for each location and entry/exit time.

As shown in FIG. 6A , there is a pre-registered reference image 410, and the optimal reference image 420 is registered for each location. The first photographed image and the second photographed image are registered in the A place 421, the third photographed image and the fourth photographed image are registered in the B place 423, and the fifth photographed image and the second photographed image are registered in the C place 425. 6 Register the shot image.

In addition, as shown in FIG. 6B , the reference image changing unit 340 may register the optimal reference image 420 according to each location and entry/exit time. The reference image change unit 340 registers the first captured image 427a for the place A 421 at the first access time, and the second captured image 429a for the place A 421 at the second access time. register In addition, the reference image change unit 340 registers the third photographed image 427b for the place B 423 at the first access time, and registers the fourth photographed image 429b for the place C 423 . . In addition, the reference image change unit 340 registers the fifth photographed image 427c at the first access time for the place C 425, and the sixth photographed image ( 429c) is registered.

The reference image change unit 340 stores the optimal reference image in the user database 222 , and uses the input image generation unit 225 , the face recognition unit 230 , and the array file generation unit 240 of the registered user. A new array file for the optimal reference image is created. The generated new array file is distributed to each edge device 120 through the interface unit 270 .

Meanwhile, the edge device manager 210 may further include an optimal threshold value calculator 360 and a reference threshold value changer 370 .

The optimal threshold calculation unit 360 extracts first authentication results from a plurality of authentication results and calculates an optimal threshold value based on the first authentication results. In this case, the first authentication result means an authentication result generated by erroneously authorizing an unregistered user as a registered user.

The optimum threshold calculation unit 360 calculates the first optimum threshold value based on the maximum value among the similarities included in the first authentication results. Specifically, the optimal threshold value calculator 360 calculates a maximum value among the similarities included in the first authentication result as the first optimal threshold value. In this case, the optimum threshold value calculator 360 may calculate a result value obtained by adding the margin value to the similarity level as the first optimum threshold value.

In an embodiment, the optimal threshold calculation unit 214 may classify the first authentication results for each second type user corresponding to the first authentication result, and calculate the first optimal threshold value as a different value for each second type user. have.

In an embodiment, the optimal threshold calculation unit 214 additionally calculates a second optimal threshold based on a minimum value among similarities included in third authentication results for erroneously authorizing a registered user as an unregistered user among authentication results. can do.

In an embodiment, the optimal threshold calculation unit 214 classifies the first authentication results for each edge device 120 that has transmitted the first authentication result, and calculates the first optimal threshold value as a different value for each edge device 120 . can do. In this case, the optimal threshold value calculator 214 may also calculate the second optimal threshold value as a different value for each edge device 120 for the third authentication results.

In an embodiment, the optimal threshold calculation unit 214 may classify the first authentication result by access time, and may calculate the first optimal threshold value as a different value for each time. In this case, the optimal threshold value calculating unit 214 may calculate the second optimal threshold value as a different value for each time even for the third authentication results.

The reference threshold value change unit 370 changes the predetermined reference threshold value to the optimum threshold value calculated by the optimum threshold value calculation unit 360 .

In an embodiment, the reference threshold change unit 218 maintains a reference threshold value for a first type user corresponding to a second authentication result normally approved as a registered user among authentication results, and corresponds to the first authentication result. For the second type user, the reference threshold may be changed to the first optimal threshold.

The reference threshold value change unit 218 changes the reference threshold value to the second optimum threshold value calculated by the optimum threshold value calculation unit 214 for the third type user corresponding to the third authentication result that incorrectly approves the registered user as an unregistered user. can

The reference threshold value changing unit 218 may change the reference threshold value of each second type user to the first optimal threshold value calculated as a different value for each second type user, and set the reference threshold value of each third type user to the third type The second optimal threshold values calculated as different values for each user may be respectively changed.

Also, the reference threshold change unit 218 may change the reference threshold of each edge device to first optimal thresholds calculated for each edge device of the second type users, and the second optimum calculated for each edge device of the third type users. Thresholds can be changed.

Also, the reference threshold change unit 218 may change the reference threshold of each access time to the first optimal threshold values calculated for each access time of the second type users, and the second optimal value calculated for each access time of the third type users. Thresholds can be changed.

However, in the above-described embodiment, management difficulties may occur because the reference thresholds are set differently in units of information. Therefore, in a modified embodiment, the reference threshold value changing unit 218 collectively changes the reference threshold value to the first optimal threshold value. If the authentication speed of the edge device 120 is lower than the predetermined speed, each user, the edge The reference threshold may be changed to the first optimal threshold determined for each device or access time.

Meanwhile, the reference threshold change unit 370 distributes the changed reference threshold to each edge device 120 . In more detail, the reference threshold change unit 370 may distribute the changed reference threshold value to each edge device 120 through the interface unit 270 . Accordingly, each edge device 120 performs authentication with the changed reference threshold.

Referring back to FIG. 2 , the user registration unit 220 receives one or more user images from the user terminal 130 of a user who wishes to be registered. When the user image is received, the user registration unit 220 checks whether the corresponding user is the same person as the user image, and when it is determined that the user is the same person, acquires the access right information granted to the user and enters the user database 222 together with the user image. ) to register In this case, the user image means a reference image.

The input image generating unit 225 generates an input image to be used for face recognition from the user image input by the user registration unit 220 . Specifically, the input image generator 225 generates a plurality of user images having different resolutions from one user image by downsampling or upsampling one user image to a predetermined step.

In an embodiment, the input image generator 225 generates a downsampled user image by applying a Gaussian Pyramid to the user image, or upsampled by applying a Laplacian Pyramid to the user image. You can create custom images.

When a plurality of user images having different resolutions are generated, the input image generating unit 225 generates, as an input image, a plurality of images obtained by moving a window having a predetermined pixel size on the user image for each user image. do. The input image generating unit 225 inputs the generated plurality of input images to the face recognition unit 230 .

The face recognition unit 230 inputs a plurality of input images generated by the input image generation unit 225 to the face recognition model 235 trained by the face recognition model training unit 250, so that the face including the face region is An image is acquired, it is determined whether the acquired face image is a real image of a person, and a feature vector is extracted from the acquired face image.

In an embodiment, the face recognition model 230 includes a face image extracting unit 237 for extracting a face image from an input image, a real image determining unit 238 for determining whether to be a real image from the extracted face image, and a face image. It may include a feature vector extractor 239 for extracting the feature vector.

The face image extraction unit 237 according to the present invention is configured based on a convolutional neural network (CNN) and extracts a face image including a face region from an input image. The face image extraction unit 237 includes a first face detection unit, a second face detection unit, a third face detection unit, and a face image alignment unit.

The first face detection unit extracts features of each input image by applying a convolution operation to the input image input by the face recognition unit 230, and primarily extracts a face region from the input image based on the extracted features. .

To this end, the first face detection unit includes n convolution operation units, a sampling unit, first and second dimension reduction units, and a first probability value calculation unit.

In an embodiment, the first face detection unit may include three convolutional operation units. For convenience of explanation, it is described that the first face detection unit includes three convolutional operation units, but this is only an example. There will be.

Each of the first to third convolution operation units applies a convolution filter to an input image to generate a feature map, and applies an activation function to the generated feature map to reflect nonlinear characteristics in the feature map. In this case, the convolution filters applied to the first to third convolution operators may be different filters.

In one embodiment, the activation function used in the first to third convolution operation units may be an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a reduced value by a predetermined size. have. Here, the activation function refers to a function that gives a weight to a plurality of input information and outputs a completed result value by combining them.

The sampling unit extracts a feature value from the feature map by applying a sampling filter to the feature map output from the first convolution operation unit. In an embodiment, the sampling unit may extract a maximum value among pixel values included in a region corresponding to the sampling filter on the feature map as a feature value of the feature map. According to this embodiment, the sampling unit may be implemented as a max pooling layer, and the dimension of the feature map is reduced through the max pooling layer. The sampling unit inputs the reduced-dimensional feature map to the second convolution operation unit.

The first dimension reduction unit reduces the dimension of the feature map output from the third convolution operation unit by applying the first dimension reduction filter to the feature map output from the third convolution operation unit. In an embodiment, the first dimension reduction filter may be set as a filter capable of reducing the feature map in two dimensions.

The first probability value calculating unit calculates a first probability value of whether a face region is included in the input image by applying a predetermined classification function to the two-dimensional output data output by the first dimension reduction unit. In an embodiment, when the calculated first probability value is equal to or greater than a first threshold value, the first probability value calculator may determine that the input image includes the face region.

The second dimensionality reduction unit applies the second dimension reduction filter to the feature map output from the third convolution operation unit when the first probability value calculated by the first probability value operation unit is equal to or greater than the first threshold, thereby applying the second dimension reduction filter to the feature map output from the third convolution operation unit. reduce the dimension of In an embodiment, the second dimension reduction filter may be set as a filter capable of reducing the feature map in four dimensions, and the second dimension reduction unit converts four values output in four dimensions to the coordinates of the face region on the input image. decide with At this time, the coordinates of the face area are defined as the coordinates of the upper left vertex and the lower right vertex when the area including the face is displayed as a rectangular bounding box, or the coordinates of the upper right vertex and the lower left vertex can be defined as the coordinates of

The second face detector receives input images determined by the first face detector to include the face region and the coordinates of the face region on the input images, and receives the coordinates of the face region on the input images. A feature of the first sub-input images is extracted by applying a convolution operation to the first sub-input images, and a face region is secondarily extracted from the first sub-input images based on the extracted features.

To this end, the second face detection unit includes n convolutional operation units, second to third sampling units, a first dimensional increase unit, third and fourth dimensionality decrease units, and a second probability value calculation unit.

In an embodiment, the second face detection unit may include three convolution operation units. For convenience of explanation, it is described that the second face detection unit includes three convolution operation units, but this is only an example. The second face detection unit may include more than the number of convolution operation units included in the first face detection unit. have.

Each of the fourth to sixth convolution operation units applies a convolution filter to an input image to generate a feature map, and applies an activation function to the generated feature map to reflect nonlinear characteristics in the feature map. In an embodiment, the activation function used by the fourth to sixth convolution operation units may be an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a reduced value by a predetermined size. .

The second sampling unit applies a sampling filter to the feature map output from the fourth convolution operation unit to extract a feature value from the corresponding feature map, and the third sampling unit applies the sampling filter to the feature map output from the fifth convolution operation unit and applies the sampling filter to the corresponding feature. Extract feature values from the map. In an embodiment, the second and third sampling units may extract a maximum value among pixel values included in a region corresponding to a sampling filter on each feature map as a feature value of the feature map. According to this embodiment, the second and third sampling units may be implemented as a maxpooling layer, and the dimension of each feature map is reduced through the maxpooling layer.

The first dimension increase unit increases the dimension of the feature map by using a plurality of nodes so that the feature map output from the sixth convolution operation unit has a dimension of a predetermined size. In an embodiment, the first dimension increase unit may increase the dimension so that the feature map output from the sixth convolution operation unit has a size of 128*128 or 256*256.

The first dimensional increase unit reflects non-linear characteristics in the dimensionally increased feature map by applying the activation function to the dimensionally increased feature map. In an embodiment, the first dimensional increase unit applies an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a value reduced by a predetermined size to apply a non-linear characteristic to the feature map. can reflect

The third dimension reduction unit reduces the dimension of the feature map output from the first dimension increase unit by applying the third dimension reduction filter to the feature map output from the first dimension increase unit. In an embodiment, the third dimension reduction filter may be set as a filter capable of reducing the feature map in two dimensions.

The second probability value calculating unit calculates a second probability value of whether the face region is included in the first sub-input image by applying a predetermined classification function to the two-dimensional output data output by the third dimension reduction unit. In an embodiment, the second probability value calculator may determine that the face region is included in the corresponding first sub-input image when the calculated second probability value is greater than or equal to a second threshold greater than the first threshold.

The fourth dimensionality reduction unit applies the fourth dimension reduction filter to the feature map output from the first dimension increase unit when the second probability value calculated by the second probability value calculating unit is equal to or greater than the second threshold value. Reduce the dimension of the feature map. In an embodiment, the fourth dimension reduction filter may be set as a filter capable of reducing the feature map in four dimensions, and the fourth dimension reduction unit converts the four values output in four dimensions to the corresponding first sub-input image. Determined by face area coordinates. At this time, the coordinates of the face area are defined as the coordinates of the upper left vertex and the lower right vertex when the area containing the face is displayed in a rectangular bounding box, or defined as the coordinates of the upper right vertex and the coordinates of the lower left vertex. can be

The third face detector receives the first sub-input images determined by the second face detector to include the face region and the coordinates of the face region on the first sub-input images, and receives the first sub-input image The features of the second sub-input images are extracted by applying a convolution operation to the second sub-input images corresponding to the coordinates of the face region in the fields, and the facial region is tertiary on the second sub-input images based on the extracted features. extracted with

To this end, the third face detection unit includes n+1 convolution operation units, fourth to sixth sampling units, a second dimensional increase unit, a fifth to sixth dimension decrease unit, and a third probability value calculating unit.

In an embodiment, the third face detection unit may include four convolution operation units. For convenience of explanation, it is described that the third face detection unit includes four convolution operation units, but this is only an example. If the third face detection unit includes more than the number of convolution operation units included in the second face detection unit, the The number may be unlimited.

Each of the seventh to tenth convolution operation units applies a convolution filter to an input image to generate a feature map, and applies an activation function to the generated feature map to reflect non-linear characteristics in the feature map. In an embodiment, the activation function used by the seventh to tenth convolution operation units may be an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a reduced value by a predetermined size. .

The fourth sampling unit applies a sampling filter to the feature map output from the seventh convolution operation unit to extract a feature value from the corresponding feature map, and the fifth sampling unit applies the sampling filter to the feature map output from the eighth convolution operation unit to apply the sampling filter to the corresponding feature. The feature value is extracted from the map, and the sixth sampling unit applies a sampling filter to the feature map output from the ninth convolution operation unit to extract the feature value from the corresponding feature map. In an embodiment, the fourth to sixth sampling units may extract a maximum value among pixel values included in an area corresponding to the sampling filter on each feature map as a feature value of the feature map. According to this embodiment, the fourth to sixth sampling units may be implemented as a maxpooling layer, and the dimension of each feature map is reduced through the maxpooling layer.

The second dimension increase unit increases the dimension of the feature map by using a plurality of nodes so that the feature map output from the tenth convolution operation unit has a dimension of a predetermined size. In an embodiment, the second dimension increase unit may increase the dimension so that the feature map output from the tenth convolution operation unit has a size of 128*128 or a size of 256*256.

The second dimensional increase unit reflects non-linear characteristics in the dimensionally increased feature map by applying the activation function to the dimensionally increased feature map. In an embodiment, the second dimensional increase unit applies an activation function that outputs a positive value among pixel values of the feature map as it is and outputs a negative value as a value reduced by a predetermined size to apply a non-linear characteristic to the feature map. can reflect

The fifth dimension reduction unit reduces the dimension of the feature map output from the second dimension increase unit by applying the fifth dimension reduction filter to the feature map output from the second dimension increase unit. In an embodiment, the fifth dimension reduction filter may be set as a filter capable of reducing the feature map in two dimensions.

The third probability value calculating unit calculates a third probability value of whether a face region is included in the corresponding second sub-input image by applying a predetermined classification function to the two-dimensional output data output by the fifth dimension reduction unit. In an embodiment, the third probability value calculating unit determines that the face region is included in the corresponding second sub-input image when the calculated third probability value is greater than or equal to a third threshold greater than the second threshold.

The sixth dimension reduction unit applies the sixth dimension reduction filter to the feature map output from the second dimension increase unit when the third probability value calculated by the third probability value calculating unit is equal to or greater than the third threshold value, thereby outputting from the second dimension increase unit. Reduce the dimension of the feature map. In an embodiment, the sixth dimension reduction filter may be set as a filter capable of reducing the feature map in four dimensions, and the sixth dimension reduction unit converts four values output in four dimensions to the corresponding second sub-input image. Determined by face area coordinates. In this case, the coordinates of the face area are defined as the coordinates of the upper left vertex and the lower right vertex when the area including the face is displayed in a rectangular bounding box, or defined as the coordinates of the upper right vertex and the coordinates of the lower left vertex. can be

The sixth dimension reduction unit extracts a face image from the second sub-input image determined to include the face region by using the calculated face region coordinates.

The seventh dimension reduction unit applies the seventh dimension reduction filter to the feature map output from the second dimension increase unit when the third probability value calculated by the third probability value calculating unit is equal to or greater than the third threshold value, thereby outputting from the second dimension increase unit. Reduce the dimension of the feature map. In an embodiment, the seventh dimensionality reduction filter may be set as a filter capable of reducing the feature map in ten dimensions, and the seventh dimension reduction unit converts ten values output in ten dimensions to the corresponding second sub-input image. Determined by landmark coordinates. In this case, the landmark coordinates mean the coordinates of two eyes, the coordinates of the nose, and the coordinates of the two mouths on the second sub-input image, and the coordinates of the two mouths are the coordinates for the left tail of the mouth and the right tail of the mouth. means the coordinates for

As described above, according to the present invention, the face image extracting unit 237 is implemented as first to third face detecting units, and the first face detecting unit is composed of a network of shallower depth than the second face detecting unit, 2 The face detection unit is composed of a network with a shallower depth than the third face detection unit, so that the face image extraction unit is formed in a step-by-step form with a shallow-to-deep structure as a whole. Through this, it is possible to take advantage in terms of face recognition speed by improving the accuracy of face image extraction and reducing the complexity at the same time.

The face image alignment unit aligns the face images by using the landmark coordinates output from the third face detection unit. Specifically, the face image aligning unit aligns the face image by performing at least one of rotation, translation, enlargement, and reduction on the face image by using the landmark coordinates of the extracted face image.

The reason for aligning the face images using the face image alignment unit in the present invention is to improve the face recognition performance by giving consistency to the face image to be provided as an input to the real image determining unit 238 and the feature vector extracting unit 239 . am.

In an embodiment, the face image alignment unit resizes the face image extracted by the third face detection unit to the same resolution as the face image used in the real image determination unit 238 and the feature vector extraction unit 239. and rotate, parallel move, enlarge or can be reduced

The real image determining unit 238 determines whether the face image extracted from the face image extracting unit 237 is a real image obtained by photographing a person. Specifically, the real image determination unit 238 uses a neural network to determine the RGB feature vector expressing the RGB of the face image, the depth feature vector expressing the depth of the face image, and the light reflection of the face image from the face image of the target user. The reflection feature vector to be expressed is extracted, and it is determined whether the face image is a real image with each extracted feature vector.

The reason that the real image determining unit 238 according to the present invention determines whether or not the real image is a real image with the face image extracted by the face image extracting unit 237 is to minimize the information about the surrounding environment, thereby overcharging caused by unnecessary information. This is to prevent overfitting and generalization performance degradation.

The real image determination unit 237 according to the present invention includes an actual feature vector extraction unit, a feature vector fusion unit, and a determination unit.

The real feature vector extraction unit uses a neural network to determine whether the face image is a real image from the target user's face image, among a depth feature vector expressing the depth of the face image, and a reflection feature vector expressing light reflection of the face image. At least one and an RGB feature vector representing the RGB of the face image are extracted.

The RGB feature vector extractor extracts the RGB feature vector from the face image using the RGB neural network. At this time, the RGB neural network network is trained with a two-dimensional real image of a person and a two-dimensional fake image of a photograph.

To this end, the RGB feature vector extractor includes a first sub-RGB feature vector extractor, a second sub-RGB feature vector extractor, a third sub-RGB feature vector extractor, a fourth sub-RGB feature vector extractor 240 , and an RGB feature. It includes a vector generator. Here, it has been described that the RGB feature vector extractor includes four sub-RGB feature vector extractors, but this is for convenience of explanation. The RGB feature vector extractor 210 includes three or less sub-RGB feature vector extractors or 5 It may include more than one sub-RGB feature vector extractor.

The first sub-RGB feature vector extractor generates a feature map of the face image with the first RGB neural network, and extracts the first sub-RGB feature vector from the face image based on the feature map. The first sub-RGB feature vector extractor transmits the extracted first sub-RGB feature vector to the second sub-RGB feature vector extractor. In addition, the first sub-RGB feature vector extractor transmits the first sub-RGB feature vector to the depth feature vector extractor and the reflection feature vector extractor.

The second sub-RGB feature vector extractor generates a feature map of the first sub-RGB feature vector transmitted from the first sub-RGB feature vector extractor to the second sub-RGB neural network network, and based on the feature map, generates the first sub-RGB feature vector. A second sub-RGB feature vector is extracted from The second sub-RGB feature vector extractor transmits the extracted second sub-RGB feature vector to the third sub-RGB feature vector extractor. In addition, the second sub-RGB feature vector extractor transmits the second sub-RGB feature vector to the depth feature vector extractor and the reflection feature vector extractor.

The third sub-RGB feature vector extractor generates a feature map of the second RGB sub feature vector transmitted from the second sub-RGB feature vector extractor to the third sub-RGB neural network network, and based on the feature map, generates the second sub-RGB feature vector. A third sub-RGB feature vector is extracted from The third sub-RGB feature vector extractor transmits the extracted third sub-RGB feature vector to the fourth sub-RGB feature vector extractor. In addition, the third sub-RGB feature vector extractor 235 transmits the third sub-RGB feature vector to the depth feature vector extractor and the reflection feature vector extractor.

The fourth sub-RGB feature vector extractor generates a feature map of the third sub-RGB feature vector transmitted from the third sub-RGB feature vector extractor to the fourth sub-RGB neural network network, and based on the feature map, generates the third sub-RGB feature vector. A fourth sub-RGB feature vector is extracted from The fourth sub-RGB feature vector extractor transmits the extracted fourth sub-RGB feature vector to the RGB feature vector generator.

In this way, the sub-RGB feature vector extractor extracts n sub-RGB feature vectors by using the n RGB neural network. The sub-RGB feature vector extractor generates a feature map of the face image for each RGB neural network, and extracts n sub-RGB feature vectors from the face image based on the feature map.

The reason that the sub-RGB feature vector extractor according to the present invention extracts n sub-RGB feature vectors is that generating RGB feature vectors, depth feature vectors, and reflection feature vectors with sub-RGB feature vectors having different sizes is a fixed method. This is because the expressive power and discriminative power of the corresponding feature vector are richer than when the feature vector is generated with the specified size.

The RGB feature vector generator combines the first sub-RGB feature vector, the second sub-RGB feature vector, the third sub-RGB feature vector, and the fourth sub-RGB feature vector generated by the first to fourth sub-RGB feature vector extractor. to generate an RGB feature vector. Specifically, the RGB feature vector generator generates the RGB feature vector by summing the first sub-RGB feature vector, the second sub-RGB feature vector, the third sub-RGB feature vector, and the fourth sub-RGB feature vector.

The reason that the RGB feature vector generator according to the present invention generates the RGB feature vector by summing the first to fourth sub-RGB feature vectors is compared to determining whether the face image is a real image with one fixed size feature vector. , it is because performance is improved to determine whether a face image is a real image using feature vectors of various sizes.

The depth feature vector extractor generates a depth feature vector (b) based on the first sub-RGB feature vector, the second sub-RGB feature vector, and the third sub-RGB feature vector input from the RGB feature vector extractor.

A depth value cannot exist in a fake image in which a photo containing another person's face is taken. Accordingly, the present invention can accurately determine the real image based on the depth feature vector extracted by the depth feature vector extractor.

The depth feature vector extractor adjusts the sizes of the first sub-RGB feature vector, the second sub-RGB feature vector, and the third sub-RGB feature vector, and adds them to generate an input feature vector. The depth feature vector extractor generates a feature map of an input feature vector generated using a depth neural network and extracts a depth feature vector based on the feature map. In this case, the depth feature vector extractor may perform padding to adjust the size. The depth neural network network is trained with a first depth image extracted from a real image of a person and a second depth image extracted from a fake image of a photograph.

The reflection feature vector extractor generates the reflection feature vector c based on the first sub-RGB feature vector, the second sub-RGB feature vector, and the third sub-RGB feature vector input from the RGB feature vector extractor.

The reason why the present invention extracts the reflection feature vector from the reflection feature vector extractor is that when photographing a person's face with a camera, there is no light reflection, but when an image of another person is photographed, a minute but light reflection occurs.

Accordingly, the present invention has the effect of improving the accuracy of face recognition by determining whether a face image is a real image based on the reflection feature vector.

The reflection feature vector extractor adjusts the sizes of the first sub-RGB feature vector, the second sub-RGB feature vector, and the third sub-RGB feature vector, and adds them to generate an input feature vector. The reflection feature vector extractor generates a feature map of the input feature vectors summed with the reflex neural network and extracts the reflection feature vector based on the feature map. In this case, the reflection feature vector extractor may perform padding to adjust the size. The reflex neural network is trained with a first reflected image extracted from a real image of a person and a second reflected image extracted from a fake image of a photograph.

The above-mentioned neural network network includes an RGB neural network network, a depth neural network network, and a reflex neural network network, and each neural network network has a plurality of convolution operation units and a plurality of sampling units sequentially arranged to extract each RGB, depth, and reflection feature vectors. do.

The feature vector fusion unit generates a fusion feature vector by fusing at least one of a depth feature vector and a reflection feature vector with an RGB feature vector. Specifically, the feature vector fusion unit fuses at least one of the depth feature vector extracted from the depth feature vector extractor and the reflective feature vector extracted from the reflection feature vector extractor with the RGB feature vector extracted from the RGB feature vector extractor to obtain a fusion feature. create a vector

The reason why the feature vector fusion unit according to the present invention fuses each feature vector is a fake image obtained by taking a picture with a feature vector having rich expressive power by appropriately reflecting the characteristics of the RGB image, the characteristics of the reflection image, and the characteristics of the depth image. This is in order to effectively block authentication with a fake image by making it possible to discriminate.

The feature vector fusion unit includes a first output feature vector generator, a second output feature vector generator, and a fusion feature vector generator.

The first output feature vector generating unit generates a first output feature vector by passing the RGB feature vectors through k fusion convolution operation units.

The second output feature vector generator sequentially passes the result obtained by subtracting the reflection feature vector from the depth feature vector through the k fusion convolution operator and the fusion sampling section disposed at the output of the k-th convolution operator to generate a second output feature vector. do.

The fusion feature vector generator generates a fusion feature vector by adding the first output feature vector and the second output feature vector.

The determination unit determines whether the face image is a real image of a person by using the fusion feature vector. Specifically, the determination unit determines whether the face image is a real image by inputting the fusion feature vector into the pre-learned binary classifier.

In this case, the binary classifier can learn the first depth image extracted from the real image and the second depth image extracted from the fake image, and the first reflected image extracted from the real image and the second reflected image extracted from the fake image. . In the case of simply learning a binary classifier with a two-dimensional real image or a two-dimensional fake image, it is difficult to discriminate images from other environments as it overfits to a specific external environment such as camera quality, location, and lighting. There was a problem.

Accordingly, the present invention can prevent overfitting to a specific environment by learning not only the two-dimensional real image and the two-dimensional fake image, but also the depth image and the reflection image extracted from the real image and the fake image in the binary classifier. Even if it is a face image taken in the environment, it is possible to accurately judge the real image.

The binary classifier learns the first depth image extracted from the real image and learns the second depth image extracted from the fake image. At this time, the first depth image extracted from the real image is learned as a standard correct answer, and the second depth image extracted from the fake image is learned by using RGB as black. RGB becomes (0,0,0).

In addition, the binary classifier learns the first reflection image extracted from the real image, and learns the second reflection image extracted from the fake image. At this time, the first reflection image extracted from the real image is learned by using RGB as black. RGB becomes (0,0,0). The second reflection image extracted from the fake image is learned as a reference incorrect answer.

If the probability value output from the binary classifier is less than a predetermined threshold, the determination unit determines the corresponding face image as a fake image. In addition, if the probability value output from the binary classifier is equal to or greater than a predetermined threshold, the determination unit determines the corresponding face image as a real image.

At this time, when at least one of the plurality of face images is determined to be a fake image, the determination unit removes the plurality of face images so that the corresponding face image is not input to the feature vector extraction unit 239 .

When the determination unit determines that the face image is a fake image, it generates an actual face photographing request message and transmits it to a display (not shown).

In addition, when all of the plurality of face images are determined to be real images, the determination unit transmits the face image including the largest face among the plurality of face images to the feature vector extraction unit 239 .

Referring back to FIG. 2 , when a face image determined as a real image is input from the real image determining unit 238 , the feature vector extracting unit 239 extracts a feature vector from the face included in the corresponding face image.

The feature vector extracting unit according to an embodiment of the present invention includes a plurality of face image processing units and a feature vector generating unit.

The plurality of face image processing units generate output data by image processing the input data. At this time, the face image output from the face image extraction unit 237 as an input image is input to the first face image processing unit among the plurality of face image processing units, and the g-th face image processing unit is inputted to the g+1th face image processing unit as an input image. Output data is input.

Since the functions and detailed configurations of the plurality of face image processing units are the same, the first face image processing unit among the plurality of face image processing units will be exemplarily described below.

The face image processing unit 305 is a first unit that generates a feature map by performing a convolution operation on input data (a face image or output data of a previous face image processing unit), and weights the feature map generated by the first unit. It is composed of a second unit for giving, and an arithmetic unit.

The first unit includes a normalization unit, a first convolution operation unit, and a non-linearization unit.

The normalization unit normalizes the face images input from the face image extraction unit in batch units. Batch means the unit of number of face images to be processed at once. The reason that the regularizer according to the present invention performs normalization in batches is that when normalization is performed in batches, the mean and variance for each face image may be different from the mean and variance for the entire batch. This feature acts as a kind of noise. This is because overall performance can be improved.

In addition, through batch normalization, the complexity of learning increases and gradient vanishing or exploding occurs due to the internal covariance shift phenomenon, in which the distribution of inputs for each layer of the network changes inconsistently. because it can be prevented.

The first convolution operation unit generates a first feature map by applying a first convolution filter to the face image normalized by the normalization unit. In an embodiment, the first convolution operation unit may generate a first feature map by applying a first convolution filter having a size of 3*3 pixels and having a stride value of 1 to the face image. As described above, since the first convolution operation unit according to the present invention applies the first convolution filter having a size of 3*3 pixels and a stride value of 1 to the face image, it is possible to preserve the resolution of the first feature map high.

The non-linearization unit applies an activation function to the first feature map, thereby imparting a non-linear characteristic to the first feature map. In one embodiment, the non-linearization unit outputs the same positive pixel value among the pixel values of the first feature map and reduces the size of the negative pixel value to the first feature map by applying an activation function to the first feature map. 1 Non-linear characteristics can be given to the feature map.

It has been described that the non-linearization unit gives a non-linear characteristic to the first feature map generated by the first convolution operation unit. However, in a modified embodiment, the normalization unit may additionally normalize the first feature maps generated by the first convolution operation unit in a batch unit. According to this embodiment, the normalization unit provides the normalized first feature map to the non-linearization unit, and the non-linearization unit applies an activation function to the normalized first feature map, thereby imparting a non-linear characteristic to the normalized first feature map. have.

Meanwhile, in the above-described embodiment, it has been described that the first unit includes only the first convolution operation unit. However, in a modified embodiment, the first unit may further include a second convolution operation unit.

Specifically, the second convolution operation unit generates the second feature map by applying the second convolution filter to the first feature map to which the non-linear characteristic is given by the non-linearization unit. In an embodiment, the second convolutional filter may be a different filter from the first convolutional filter. The second convolutional filter may be a filter having the same size as the first convolutional filter but different stride values. As an example, the second convolutional filter may be a filter having a size of 3*3 pixels and having a stride value of 2.

According to this embodiment, the normalization unit may additionally normalize the second feature maps generated by the second convolution operation unit in a batch unit.

Meanwhile, the first unit may further include a pre-normalization unit. The pre-normalizer may normalize a pixel value of each pixel included in the face image input from the face image extractor. For example, the pre-normalizer may normalize the face image by subtracting 127.5 from each pixel value of the face image and then dividing the subtraction result by 128. The pre-normalizer may provide the pre-normalized input face image to the normalizer.

The second unit gives weight to the second feature map generated by the first unit. The reason why the second feature map is weighted through the second unit in the present invention is that, in the convolution operation, all channels of the input image are multiplied by the convolution filter and then summed, both important and insignificant channels are entangled. Since the sensitivity of is lowered, a weight is given to the second feature map according to its importance for each channel.

The second unit includes a sampling unit, a weight reflection unit, and an upscaling unit.

First, the sampling unit reduces the dimension by subsampling the second feature map input from the first unit. In an embodiment, the sampling unit may reduce the dimension of the second feature map by applying a global pooling filter to the second feature map. For example, when the dimension of the second feature map is H*W*C, the sampling unit may reduce the dimension of the second feature map to 1*1*C through subsampling of the second feature map.

The weight reflection unit gives weight to the second feature map whose dimension has been reduced by the sampling unit. To this end, the weight reflecting unit may include a dimension reduction unit, a first non-linearization unit, a dimension increase unit, and a second non-linearization unit.

The dimension reduction unit reduces the dimension of the sub-sampled second feature map by connecting the sub-sampled second feature map into one layer. For example, when the dimension of the second feature map output from the sampling unit is 1*1*C, the dimension reduction unit reduces the dimension of the second feature map to 1*1*C/r. Here, r means a reduction rate, and may be determined according to the number of feature vectors to be extracted.

The first non-linearization unit applies a first activation function to the second feature map whose dimension has been reduced by the dimension reduction unit, thereby imparting a non-linear characteristic to the second feature map with a reduced dimension. In an embodiment, the first non-linearization unit applies a first activation function that outputs a positive pixel value among pixel values of the second feature map as it is and outputs a negative pixel value as 0, thereby reducing the second dimension. Non-linear characteristics can be given to the feature map.

The dimension increase unit increases the dimension of the second feature map to which the non-linear characteristic is given by the first non-linearization unit. For example, when the dimension of the second feature map to which the non-linear characteristic is given is 1*1*c/r, the dimension increasing unit increases the dimension of the second feature map back to 1*1*C.

The second non-linearization unit applies the second activation function to the second feature map whose dimension has been increased by the dimension increase unit, thereby re-applying the non-linear characteristic to the second feature map whose dimension has been increased. In an embodiment, the second activation function may be a different function from the first activation function. For example, the second non-linearization unit applies a second activation function that causes positive pixel values among the pixel values of the second feature map with increased dimensions to converge to a predetermined value and outputs negative pixel values to 0, thereby increasing the dimension. A non-linear characteristic may be imparted to the increased second feature map.

As described above, the weight reflecting unit according to the present invention applies a weight to the second feature map through the dimension reduction unit, the first non-linearization unit, the dimension increase unit, and the second non-linearization unit, and the bottleneck section through the dimension reduction unit and the dimension increase unit , limiting the model complexity and supporting generalization by limiting the gating mechanism.

The upscaling unit upscales the second feature map weighted by the weight reflection unit to the same dimension as the face image input to the second unit. In an embodiment, when the dimension of the face image input to the second unit is H*W*C, the upscaling unit upscales the weighted dimension of the second feature map to H*W*C.

The calculator adds the upscaled second feature map output through the second unit with the face image input to the first unit. In the present invention, the reason for adding the upscaled second feature map output from the second unit to the face image input to the first unit through the operation unit in the present invention is to avoid the problem of feature blurring when the depth is increased in the convolutional neural network (Vanish Problem). it is to do

The feature vector generator generates a predetermined number of feature vectors by merging the feature map output from the last face image processor among the plurality of face image processors into one layer to reduce the dimension. In an embodiment, the feature vector generator may output 128 or more feature vectors from the feature map output from the face image processor. For example, the feature vector generator may output 512 feature vectors from the feature map output from the face image processor.

The array file generation unit 240 generates an array for each user by using the feature vector generated by the face recognition unit 230 , and merges the generated arrays into one file to generate an array file. In this case, the array file includes an existing array file and a new array file. The new array file may include an array for the optimal reference image changed by the edge device manager 210 . The array file generator 240 may store the generated array file in an array file database (not shown).

In one embodiment, the array generated by the array file generator 240 may be composed of a plurality of feature vectors obtained from each user's face image and each user's key value. In this case, the user's key value includes each user's identification information and each user's access right information. Each user's identification information may be defined as each user's ID, name, phone number, or employee number as described above, and each user's access right information includes information on each floor to which each user can access. may include

In an embodiment, the array file generator 240 may generate an array file for each location where the edge device 120 is installed. For example, the first array file may consist of arrays of users granted access to the first floor, and the second array file may consist of arrays of users granted access to the second floor. have. To this end, the array file generating unit 240 may generate each user's arrays by dividing them by regions to which each user can access. For example, when the first user has permission to access the first floor and the third floor, the array file generating unit 240 generates a first array including access permission information for the first floor for the first user and a second A second array including access permission information for the third floor can be separately created.

The reason that the array file generating unit 240 according to the present invention generates an array file for each place where the edge device 120 is installed is that when the edge device 120 for authenticating the user's face is installed for each place, a specific place This is because it is necessary to transmit only the array file including the access permission information for the corresponding place to the edge device 120 installed in the .

In the above-described embodiment, it has been described that the array file generating unit 240 generates an array file for each location, but in a modified embodiment, the array file generating unit 240 is installed in all places where the edge device 120 is installed. It is also possible to generate one array file including permission information for the data and transmit the generated array file to all edge devices 120 .

The access authority information management unit 250 changes access authority information assigned to each user or adds new access authority information. In an embodiment, the access right information management unit 250 may separately grant access right information to each user or grant access right information to an organizational unit to which each user belongs.

Meanwhile, the face recognition server 110 according to the present invention may further include a scheduler 260 . The scheduler 260 performs a function of collectively registering new users whenever a predetermined period arrives or a predetermined event occurs. For example, in the above-described embodiment, it has been described that the user registration unit 220 performs the registration procedure of a new user when a registration request occurs from the user, but when the face recognition server 110 includes the scheduler 260, When a predetermined time unit or a predetermined event occurs, the scheduler 260 initiates the operations of the user registration unit 220 , the input image generation unit 225 , and the face recognition unit 230 to automatically perform a new user registration procedure. can make it happen Also, when the reference image is changed to the optimal reference image by the edge device manager 210 , the scheduler 260 starts the operations of the input image generator 225 and the face recognition unit 230 .

The interface unit 270 encrypts the reference threshold value changed by the edge device management unit 210 and the array file generated by the array file generation unit 240 in a predetermined manner and transmits it to each edge device 120 . In this case, the array file includes an existing array file and a new array file. The new array file may contain an array for the optimal reference image.

In an embodiment, the interface unit 270 may encrypt the reference threshold value and the array file using a public key-based encryption algorithm, and transmit it to each edge device 120 .

Meanwhile, the interface unit 270 may transmit the encrypted reference threshold and the array file to the edge device 120 according to a protocol agreed with the edge device 120 .

In addition, the interface unit 270 may receive an authentication result from each edge device 120 every predetermined period and provide it to the edge device manager 210 .

The face recognition model training unit 280 generates a face recognition model 235 based on the convolutional neural network, and trains the generated face recognition model 235 . Specifically, the face recognition model training unit 280 generates an optimal face recognition model by continuously training the convolutional neural network constituting the face recognition model 235 .

To this end, the face recognition model training unit 280 includes a face image extraction training unit 282 for training the face image extraction unit 227, a feature vector extraction training unit 284 for training the feature vector extraction unit 229, It includes an actual image determination training unit 285 for training the real image determination unit 238 and an error reduction unit 286 for reducing the error of the feature vector extraction unit 229 .

The face image extraction training unit 282 trains the face image extraction unit 227 using the learning image. The feature vector extraction training unit 284 trains the feature vector extraction unit 229 using the training image. The real image judgment training unit 285 trains the real image judgment unit 238 .

The face image extraction training unit 282 trains the first to third face detection units constituting the face image extraction unit 237 using the learning image. Specifically, the face image extraction training unit 282 inputs a plurality of training images having a predetermined size to the first to third face detection units, and calculates and calculates a probability value and facial region coordinates to include a face region in the training image. By feeding back the obtained probability values and face region coordinates to the first to third face detectors according to a back propagation algorithm, the filter coefficients and weights of the convolutional filters applied to the first to third face detectors are updated.

The feature vector extraction training unit 284 trains the feature vector extraction unit 239 using the training image. Specifically, the feature vector extraction training unit 505 extracts a feature vector from each training image by inputting a plurality of training images to the feature vector extraction unit 239 in a predetermined batch unit.

The feature vector extraction training unit 284 predicts a probability value that the corresponding learning image will be included in a specific class by applying the extracted feature vectors to a predetermined classification function, and between the predicted probability value (hereinafter referred to as a 'prediction value') and the actual value. The filter coefficients and weights of the convolutional filters applied to the feature vector extractor 239 are updated by calculating the error and feeding the result back to the feature vector extractor 284 using a backpropagation algorithm.

In an embodiment, the feature vector extraction training unit 284 may train the real feature vector extraction unit of the real image determination unit 284 using the learning image. Hereinafter, since the feature vector extraction training unit 284 trains the feature vector extractor 239 and training the real feature vector extractor is the same, a detailed description of training the real feature vector extractor will be omitted.

The real image judgment training unit 285 trains the real image judgment unit 238 . Specifically, the real image judgment training unit 238 trains the RGB neural network network, the depth neural network network, the reflex neural network network, and the binary classifier used by the judgment unit of the real image judgment unit 238 .

The real image judgment training unit learns the real image and the fake image in the binary classifier, and additionally passes through the depth neural network network to create a first depth image separated from the real image, a second depth image separated from the fake image, and a reflex neural network network. The first reflected image separated from the real image by passing through and the second reflected image separated from the fake image are learned. At this time, the second depth image extracted from the fake image is trained with RGB as (0,0,0), and the first reflection image extracted from the real image is learned with RGB as (0,0,0).

On the other hand, the face recognition model training unit 280 according to the present invention can further improve the performance of the feature vector extraction unit 239 by further reducing the error generated when the feature vector is extracted through the error reducing unit 286 . The real feature vector extracting unit of the real image determining unit 238 may also improve performance through the error reducing unit 286 in the same manner as the feature vector extracting unit 239 .

The error reducing unit 286 is an error between the predicted value and the actual value based on the feature vectors extracted through the feature vector extracting unit 239 in the process where the feature vector extraction training unit 284 trains the feature vector extracting unit 239 . reduces the Specifically, the error reduction unit 286 calculates an error between the predicted value and the actual value based on the feature vectors extracted by the feature vector extraction unit 239 from the training image, and converts each training image to a two-dimensional angle so that the error can be reduced. The feature vector extraction training unit 284 can train the feature vector extraction unit 239 by placing it on a plane and using a probability value according to the arrangement result.

The reason that the face recognition model training unit 280 according to the present invention trains the feature vector extraction unit 239 to reduce the error through the error reduction unit 286 is that the lighting or environment in which the face is photographed changes despite the same person. In order to train the feature vector extracting unit 239 to extract a feature vector that can reduce the error of face recognition through this error reducing unit 286, since an error occurs such as not being able to distinguish the same person when will be.

The error reduction unit 286 according to an embodiment of the present invention includes a face image arrangement unit 287 and a probability calculation unit 289 .

The face image arrangement unit 287 arranges each training image on a two-dimensional angular plane based on the plurality of feature vectors extracted by the feature vector extraction unit 239 with respect to the training image. Specifically, the face image arrangement unit 287 calculates the cosine similarity between training images included in different classes, and calculates a reference angle that is a separation angle between each training image according to the cosine similarity, thereby setting the training images as two-dimensional angles. placed on a flat surface.

In the present invention, when the face image arrangement unit 287 arranges the training images in the vector space according to the distance between each training image calculated based on the feature vector of each training image, an overlapping area between the training images may occur. There is a limit in that it is difficult to clearly distinguish the same person from others during learning.

Therefore, in the present invention, the face image arrangement unit 287 calculates the angle between the learning images included in different classes through the cosine similarity, and arranges each learning image on a two-dimensional angle plane based on the calculated angle. will do

The probability calculation unit 289 varies the margin angle to be added to the reference angle calculated by the face image arrangement unit 287 on the two-dimensional angular plane, and determines the probability that each learning image is included in the corresponding class for each variable margin angle. Calculate.

Specifically, the probability calculating unit 525 changes _{the margin angles P1 and P2 added to the reference angles θ 1} , θ ₂ between each training image while the training images having overlapping characteristics are displayed on a two-dimensional angular plane. to be spaced apart. In an embodiment, the margin angles P1 and P2 may be determined according to a learning rate within a range greater than 0 and less than 90 degrees.

For example, when the learning rate increases, the margin angle increases accordingly, and when the learning rate decreases, the margin angle may be set to decrease accordingly. In this case, the probability calculator 525 may vary the margin angle by a predetermined reference unit.

When the margin angle is added to the reference angle by the probability calculator 289, it can be seen that the training images having overlapping features in the vector space are arranged to be spaced apart from each other.

The probability calculating unit 289 calculates the probability that each training image is included in the corresponding class for each margin angle added to the reference angle, and provides the calculated probability value to the feature vector extraction training unit 284, whereby the feature vector extraction training unit 284 ) to allow the feature vector extractor 239 to learn based on the probability value calculated by the probability calculator 289 . That is, the feature vector extraction training unit 284 updates at least one of the coefficients and weights of the convolutional filters applied to the feature vector extraction unit 239 based on the probability value calculated by the probability calculation unit 289, thereby extracting the feature vector extraction unit. (239) will be learned.

In an embodiment, the probability calculator 289 may calculate a probability that each training image is included in a corresponding class for each margin angle using Equation 1 below.

In Equation 1, x denotes a reference angle, p denotes the margin angle, and n denotes the number of classes.

In an embodiment, the probability calculating unit 289 performs a predetermined test in the feature vector extracting unit 239 trained by the feature vector extraction training unit 284 based on the probability value calculated by the probability calculating unit 289 . When a face image is applied, the margin angle can be continuously varied until the error between the predicted value and the actual value is less than or equal to the reference value.

That is, the probability calculating unit 289 converts the margin angle at the point in time when the error between the predicted value and the actual value calculated when a predetermined test face image is applied to the trained feature vector extraction unit 239 becomes less than or equal to the reference value as the final margin angle. decide In this case, the error between the predicted value and the actual value may be calculated using a cross entropy function.

If the error reduction is performed through the error reduction unit 286 as described above, the same person can be accurately classified even when the pictures are taken in different environments or under different lighting conditions.

In the above-described embodiment, the face recognition model training unit 280 may be implemented as software in the form of an algorithm and mounted on the face recognition server 110 .

Referring back to FIG. 1 , the edge device 120 uses the face recognition model 235 that is disposed at each specific place and distributed by the face recognition server 110 to detect the face of the target user who wants to enter the place. It recognizes and performs a function of authenticating the target user's access based on the recognition result.

In the present invention, the reason that the edge device 120 performs the face recognition and authentication of the target user without the face recognition server 110 performing the face recognition and authentication of the target user is the face recognition and authentication of the target user. In the case of performing in the recognition server 110, when a failure occurs in the face recognition server 110 or the network, not only face recognition and authentication cannot be performed, but also, as the number of users increases, the expansion of the expensive face recognition server 110 is required. because it becomes

Accordingly, the present invention applies an edge computing method to perform face recognition and authentication of a target user in the edge device 120, thereby providing a face recognition service normally even if a failure occurs in the face recognition server 110 or the network. It is possible to improve the reliability of service provision, and even if the number of users increases, there is no need to expand the expensive face recognition server 110 , so that it is possible to reduce the cost of constructing the face recognition system 100 .

Hereinafter, the configuration of the edge device 120 according to the present invention will be described in more detail with reference to FIG. 4 .

7 is a block diagram schematically showing the configuration of an edge device according to the first embodiment of the present invention. 7 , the edge device 120 according to the first embodiment of the present invention includes a first capturing unit 1210 , an input image generating unit 1250 , a face recognition unit 1300 , and an authenticating unit 1310 . ), a face recognition model 1320 , an update unit 1330 , a memory 1340 , and an interface unit 1350 .

When the target user to be authenticated approaches, the first photographing unit 1210 creates a photographed image by photographing the target user. The first photographing unit 1210 transmits the generated photographed image to the input image generation unit 1250 .

The input image generating unit 1250 generates an input image to be used for face recognition from the captured image of the target user transmitted from the first capturing unit 1210 . Specifically, the input image generator 1250 generates images of a plurality of target users having different resolutions from the captured images of one target user by downsampling or upsampling the captured images of one target user to a predetermined stage.

When a plurality of target user images having different resolutions are generated, the input image generator 1250 inputs a plurality of images obtained by moving a window having a predetermined pixel size on the target user image for each target user image. create as an image The input image generating unit 1250 inputs the generated input image to the face recognition unit 1300 .

When the input image of the target user is received from the input image generator 1250, the face recognition unit 1300 inputs the received input image of the target user into the face recognition model 1320 distributed from the face recognition server 110 to target the target. A face image is extracted, and a target feature vector is extracted from the extracted target face image.

Also, the face recognition model 1320 may be updated every predetermined period. As an example, the edge device 120 receives a new face recognition model 1320 from the face recognition server 110 whenever the face recognition model 1320 is updated by the face recognition server 110, thereby pre-distributed face. The recognition model 1320 may be updated with a new face recognition model 1320 .

The authenticator 1310 calculates a similarity by comparing the target feature vector obtained by the face recognition unit 1300 with the feature vectors included in the array file received from the face recognition server 110, and sets the calculated similarity as a reference threshold value. It is compared with and authenticates whether the target user is a registered user who can access the corresponding place. At this time, the authenticator 1310 authenticates the target user with the array file including the optimal reference image changed by the updater 1330 . The authenticator 1310 receives the reference threshold changed to the optimal threshold from the facial recognition server 110 according to a predetermined period, compares it with the similarity, and authenticates the target user.

Hereinafter, a method in which the authenticator 1310 authenticates the target user will be described in detail.

First, for each array included in the array file, the authenticator 1310 calculates a first result value obtained by subtracting a target feature vector by the same index from a feature vector included in the corresponding array and squaring it. The authenticator 1310 calculates a second result value by summing the first result value calculated for each index, and calculates a third result value obtained by subtracting the second result value from a predetermined reference value as a degree of similarity.

The authenticator 1310 determines that the user mapped to the array having the greatest similarity among the arrays included in the array file is the user most similar to the target user. In this case, when the degree of similarity is equal to or greater than the reference threshold, the authenticator 1310 authenticates the target user as a user having a legitimate authority, and accordingly, the target user may be permitted to enter the corresponding place.

Hereinafter, the authentication unit 1310 compares the target feature vector obtained by the face recognition unit 1300 with the feature vectors included in the array file to authenticate whether the target user is a legitimate user who can access the floor. Let me explain with an example.

8 is a diagram exemplarily illustrating a method for an authenticator to authenticate a target user. As shown in FIG. 8 , in the array file, an array including each user's feature vectors is arranged for each row. For example, feature vectors for the first user are sequentially arranged in the first row, and feature vectors for the second user are sequentially arranged in the second row. At this time, the feature vectors of each user are arranged in one row according to the index order.

As shown in FIG. 8A , the authenticator 1310 calculates a difference value between the feature vectors of each array of the array file 1410 and the target feature vectors for each index, and the calculated difference value as shown in FIG. 8B . A first result value is calculated by squaring them, and a second result value is calculated by summing all the first result values for each array as shown in FIG. 8C .

Thereafter, as shown in FIG. 8D , the authenticator 1310 calculates a third result similarity by subtracting the second result value from a predetermined reference value (eg, 1), and calculates a similarity value, which is the largest value among the calculated similarities. The user mapped to the array corresponding to 0.310528 is determined as the user most similar to the target user. Also, the authenticator 1310 finally authenticates the target user as a user mapped to the corresponding array because the similarity determined as the largest value is greater than the reference threshold.

When the authentication unit 1310 expresses the method of authenticating the target user with an equation, it can be expressed as Equation 2 below.

In Equation 2, Z denotes a third result similarity, R denotes a predetermined reference value, Xi denotes a feature vector corresponding to the i-th index among n feature vectors, and Yi denotes i among n feature vectors. It indicates the target feature vector corresponding to the th index.

The authenticator 1310 receives and updates the changed reference threshold from the face recognition server 110 according to a predetermined cycle, and authenticates the target user using the updated reference threshold.

The face recognition model 1320 is generated and distributed by the face recognition server 110, and the face recognition model 1320 is updated every time the face recognition model 235 is trained and updated by the face recognition server 110. It is replaced with a face recognition model 235 that has been used. In this case, the face recognition model 1320 may be received from the face recognition server 110 through the interface unit 1350 .

The update unit 1330 receives a new array file composed of a plurality of arrays including feature vectors extracted from the optimal reference image every predetermined update period from the Anmen recognition server 110, and changes the existing array file to a new array file. do.

In one embodiment, the update unit 1330 calculates a ratio of the number of approval times to recognition attempts for each user based on the second authentication results obtained for a predetermined time period, and if the calculated ratio is greater than or equal to a predetermined reference value, the update period is If it is lower than the reference value, the update period is increased.

Also, the update unit 1330 may vary the update period according to a change rate of the number of approvals compared to recognition attempts. The update unit 1330 may decrease the update period when the change rate increases by more than a predetermined reference value, and increase the update period when the change rate decreases below the predetermined reference value.

When the update unit 1330 receives an array file including an optimal reference image or a reference threshold value from the face recognition server 110 through the interface unit 1350, it uploads it to the first memory 1342, and the authentication unit 1310 performs it. to authenticate the target user. In particular, the update unit 1330 according to the present invention may dynamically load an array file or a reference threshold.

Specifically, when an existing array file or a reference threshold is loaded in the first memory 1342 , the update unit 1330 receives a new array file or a new reference threshold from the face recognition server 110 when receiving a new array file or The new reference threshold is loaded into the second memory 1344 , and when the loading of the new reference threshold or the new array file into the second memory 1344 is completed, the existing array file or reference threshold loaded in the first memory 1342 is removed. 2 It is replaced with a new array file or a new reference threshold loaded in the memory 1344 .

The reason that the update unit 1330 according to the present invention dynamically loads the reference threshold or the array file as described above is that the authenticator 1310 performs authentication processing on the target user and the update unit 1330 simultaneously loads the new array file. Alternatively, by allowing the new reference threshold to be updated, the edge device 120 may perform face recognition in real time based on the newly updated new array file or the new reference threshold.

An existing array file or an existing reference threshold used by the authenticator 1310 is loaded into the first memory 1342 , and a newly received new array file or a new reference threshold is loaded into the second memory 1344 . When the loading of the new array file or the new reference threshold into the second memory 1344 is completed, the existing array file or the existing reference threshold recorded in the first memory 1342 by the update unit 1330 is the new array file or the new reference threshold. will be replaced with

The interface unit 1350 mediates data transmission/reception between the edge device 120 and the face recognition server 110 . Specifically, the interface unit 1350 receives the face recognition model 1320 from the face recognition server 110, and receives an array file including an optimal reference image or a reference threshold from the face recognition server 110, and receives the update unit ( 1330 is loaded into the first memory 1342 or the second memory 1344 . In addition, the interface unit 1350 periodically transmits the access data by the authentication unit 1330 to the face recognition server 110 .

In an embodiment, the array file, the reference threshold, and the face recognition model 1320 may be updated at predetermined intervals through the interface unit 1350 .

As described above, according to the present invention, only the face recognition model 1320 for face recognition, the array file, and the reference threshold are stored in the edge device 120 according to the present invention, but the user's face image or personal information is not stored. 120) is hacked, there is no fear of leakage of user's personal information, so security is strengthened.

In the above-described embodiment, the real image determination unit included in the face recognition model 1320 determines whether the photographed image of the target user is a real image of a person or a fake image of a photograph of a registered user. Unlike the above-described embodiment, the present invention may further determine whether the image is an actual image by using the second photographing unit 1510 which is an IR camera.

9 is a block diagram showing the configuration of an edge device according to a second embodiment of the present invention. The edge device according to the second embodiment shown in FIG. 9 is second in that it further includes a second photographing unit 1510 and a truth/false determination unit 1520 compared to the edge device according to the first embodiment shown in FIG. 7 . It is distinguished from the edge device according to the first embodiment. Hereinafter, for convenience of explanation, a description of a configuration having the same function as that of the edge device according to the first embodiment will be omitted, and the newly added second photographing unit 1510 and authenticity determining unit 1520 and newly added Only the first photographing unit 1210 whose function is changed due to the configuration will be described.

The first photographing unit 1210 creates a photographed image by photographing an object to be photographed. The first photographing unit 1210 transmits the generated photographed image to the authenticity determining unit 1520 .

The second photographing unit 1510 generates a depth image by photographing an object to be photographed. The second photographing unit 1510 is configured to be captured by the first photographing unit 1210 at the same time as when the photographing target is photographed, or after a predetermined time before or after a predetermined time from the time when the photographing object is photographed by the first photographing unit 1210 . You can take a picture of the subject.

In an embodiment, the second photographing unit 1510 may be implemented as an IR camera capable of generating a depth image by photographing an object to be photographed.

As described above, the reason that the edge device 120 according to the second embodiment generates a depth image by photographing an object to be photographed through the second photographing unit 1510 is that the real face of the photographing target is captured by the second photographing unit 1510 . This is because different types of depth images are generated when a photograph is taken and when a photograph including the face of a photographing target is photographed.

The second photographing unit 1510 transmits the generated depth image to the authenticity determining unit 1520 .

The authenticity determination unit 1520 determines whether the photographing target photographed by the second photographing unit 1510 is a photograph or an actual photographing target's face using the depth image transmitted from the second photographing unit 1510 .

Specifically, the authenticity determination unit 1520 extracts depth data from the depth image received from the second photographing unit 1510 and determines whether the depth image is a real face of the photographing target through binary classification. . In an embodiment, the authenticity determination unit 1520 may improve classification accuracy for real faces and photos through deep learning algorithm-based training.

When it is determined that the photographing target photographed by the second photographing unit 1510 is a real face, the authenticity determining unit 1520 transmits the photographed image received from the first photographing unit 1210 to the input image generating unit 1250 . .

On the other hand, when it is determined that the photographing target photographed by the second photographing unit 1510 is not a real face, the authenticity determining unit 1520 uses the photographed image received from the first photographing unit 1210 as the input image generating unit 1250 . Instead of sending it to , it is possible to output that authentication processing has failed using an alarm message in text or voice format, or notify the system operator that an abnormal access attempt has been made.

As described above, according to the present invention, the authenticity determining unit 1520 first determines whether a real image is present, and secondly, the real image determining unit of the face recognition model 1320 secondarily determines whether or not a real image is a photographed target. It can accurately determine whether it is a real face or a photograph. Accordingly, the present invention can fundamentally block an attempt by a user without legitimate authority to obtain authentication using another person's photo by preventing the authentication process from being performed when a photo is taken, thereby improving security. .

Referring back to FIG. 1 , the user terminal 130 transmits a user image for newly registering a user to the face recognition server 110 together with the user's identification information. In an embodiment, the user terminal 130 is equipped with a face registration agent (not shown) capable of interworking with the face recognition server 110 , and the user executes the face registration agent on the user terminal 130 . It is possible to transmit an image of the face of the photographed image or a pre-photographed image to the face recognition server 110 together with user identification information.

In an embodiment, the user terminal 130 may request to register a plurality of user images for each user. In this case, the plurality of images requested for registration by each user may be photos taken in different environments or photos taken under different lighting conditions.

The user terminal 130 may use any type without limitation as long as it can request user registration by transmitting a user image to the face recognition server 110 . For example, the user terminal 130 may be implemented as a smart phone, a laptop computer, a desktop computer, or a tablet PC.

Those skilled in the art to which the present invention pertains will understand that the above-described present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof.

For example, the configuration of the face recognition server shown in FIG. 2 and the configuration of the edge device shown in FIGS. 4 and 6 may be implemented in the form of a program. When the configuration of the face recognition server and the configuration of the edge device according to the present invention are implemented as a program, each configuration shown in FIGS. 2, 4, and 6 is implemented as a code, and the codes for implementing a specific function are one It may be implemented as a program of , or may be implemented by dividing a plurality of programs.

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: face recognition system 110: face recognition server
120: edge device 130: user terminal
210: edge device management unit 310: authentication result collection unit
330: image determining unit 340: reference image changing unit
360: optimum threshold value calculation unit 370: reference threshold value change unit

Claims (17)

an authentication result collecting unit that compares the similarity between the user's photographed image and the corresponding user's reference image with a reference threshold and collects authentication results for authenticating access from a plurality of edge devices every predetermined period;
An image for extracting normal authentication results normally approved as a registered user from among the authentication results, classifying the normal authentication results for each user, and determining an optimal reference image for each user based on the normal authentication results classified for each user decision part; and
and a reference image changing unit for changing the reference image of users into an optimal reference image determined for each user.
The degree of similarity included in the normal authentication result is the degree of similarity between the user's photographed image and the reference image of the user, calculated when the user normally approves as a registered user,
The edge device receives a new array file composed of a plurality of arrays including feature vectors extracted from the optimal reference image every predetermined update period from the face recognition server, and changes the existing array file to the new array file,
The image determination unit,
The normal authentication result classified for each user is classified by access time zone in which the normal authentication result occurred, and the photographed image corresponding to the normal authentication result having the maximum value among the similarities included in the normal authentication result in the same user and access time zone is corresponding It is determined as the optimal reference image for the corresponding access time zone for the user, or
The normal authentication result classified for each user is classified by the edge device where the normal authentication result occurred, and the captured image corresponding to the normal authentication result having the maximum value among the similarities included in the normal authentication result in which the user and the edge device are the same A face recognition system for face recognition according to a change in a spatiotemporal environment, characterized in that it is determined as an optimal reference image of the corresponding edge device for the user. According to claim 1,
The reference image change unit,
A face recognition system for face recognition according to a change in a spatiotemporal environment, characterized in that each of the reference images is changed to an optimal reference image determined for each access time for each user. delete According to claim 1,
The reference image change unit,
A face recognition system for face recognition according to a change in a spatiotemporal environment, characterized in that the reference image is changed to an optimal reference image determined for each edge device for each user. According to claim 1,
The edge device compares the user's photographed image with the changed optimal reference image using a face recognition model to perform authentication of the user. 6. The method of claim 5,
The edge device generates a target feature vector from a photographed image of a target user, and compares the target feature vector with a new array file comprising a plurality of arrays including feature vectors extracted from the optimal reference image to authenticate the user. A face recognition system for face recognition according to a change in the spatiotemporal environment, characterized in that. 7. The method of claim 6,
The edge device calculates a first result value obtained by squaring the target feature vector by subtracting the same index from the feature vector included in each array, and summing the first result values calculated for each index in the array. A face recognition system for face recognition according to a change in a spatiotemporal environment, characterized in that it comprises an authenticator for authenticating the target user by using a second result value. 8. The method of claim 7,
The authenticator calculates a third result value obtained by subtracting the second result value from a predetermined reference value among the arrays included in the array file with the similarity degree, and the user mapped to the array having the greatest similarity is the target user and the target user. It is determined that the user is the most similar, and if the similarity is greater than or equal to a predetermined reference threshold, the target user is normally approved. delete According to claim 1,
The edge device is
Based on the normal authentication results obtained for a predetermined time, the change rate of the number of approvals compared to the recognition attempt is calculated for each user, and if the calculated rate of change is greater than or equal to a predetermined reference value, the update period is decreased, and if it is less than the reference value, the update period is increased. A face recognition system for face recognition according to a change in the spatiotemporal environment, characterized in that. According to claim 1,
The edge device is
The existing array file is loaded into a first memory, and when a new array file is received from the facial recognition server, the new array file is loaded into a second memory, and when the loading of the new array file into the second memory is completed, the A face recognition system for face recognition according to a change in a spatiotemporal environment, characterized in that the existing array file loaded in the first memory is replaced with a new array file loaded in the second memory. According to claim 1,
Further comprising a face recognition unit that extracts a face image by inputting an input image of a user requested for registration into a face recognition model, and generates a plurality of feature vectors from the extracted face image,
The face recognition model is
a face image extraction unit for extracting the face image from the input image;
a real image determination unit for determining whether the face image is a real image of a person; and
and a feature vector extraction unit for extracting a feature vector from a face included in the face image when the face image is determined to be a real image. 13. The method of claim 12,
The face image extraction unit,
Extracting the face image from input images using two or more face detectors having neural networks of different depths,
The two or more face detectors are sequentially arranged in the order of depth in depth, so that the number of input images input to the n-th face detector is reduced than the number of input images input to the n-1 face detector, characterized in that A face recognition system for face recognition according to changes in the space-time environment. 13. The method of claim 12,
The real image determination unit,
Real feature vector extraction for extracting from the face image at least one of a depth feature vector expressing the depth of the face image and a reflection feature vector expressing light reflection of the face image and an RGB feature vector expressing RGB of the face image wealth;
a feature vector fusion unit generating a fusion feature vector by fusing at least one of the depth feature vector and the reflection feature vector with the RGB feature vector; and
and a determination unit for determining whether the face image is a real image of a person photographed by using the fusion feature vector. 13. The method of claim 12,
The feature vector extraction unit,
a plurality of face image processing units for image processing input data to generate output data; and
a feature vector generator for generating a predetermined number of feature vectors by merging the output data output from the last face image processor among the plurality of face image processing units into one layer;
Among the plurality of face image processing units, a face image is input as the input image to the first face image processing unit, and output data from the nth face image processing unit is input as the input image to the n+1th face image processing unit. A face recognition system for face recognition according to changes in the space-time environment. 13. The method of claim 12,
an array file generator for generating an array composed of the plurality of feature vectors and user identification information for each user, and merging the generated arrays to generate an array file; and
A face recognition system for face recognition according to a change in a space-time environment, characterized in that it further comprises an interface unit for transmitting the generated array file to an edge device. 13. The method of claim 12,
The face recognition model is trained through an error reduction algorithm,
The error reduction algorithm arranges the training images on a two-dimensional angular plane based on a plurality of feature vectors obtained by inputting training images to the face recognition model, and adds them to a reference angle between training images included in different classes. Face recognition according to a change in a space-time environment, characterized in that by varying the margin angle to calculate the probability that each training image is included in each class for each variable margin angle, and learning the face recognition model based on the calculated probability face recognition system for

Images