KR102393543B1 - Method for collecting and processing face data for training face authentication deep learning model in smartphone device and device performing thereof - Google Patents

Abstract

The present invention relates to a device for federated learning for a face authentication deep learning model to train a face authentication deep learning model without collecting face images According to the present invention, the device comprises a memory storing instructions and a processor executing the instructions. The processor is configured to receive a last updated state of weights of a facial authentication deep learning model from a server managing the facial authentication deep learning model; take a picture of a user of the device to acquire a facial image of the user; extract query facial features from the facial image on the basis of the weights of the last updated state; calculate a facial feature similarity on the basis of a comparison result of the query facial features and registered facial features registered in the device to identify the user; compare the facial feature similarity with a threshold similarity to determine whether to unlock the device's lock screen; perform data suitability verification on the facial image to set an image determined to be suitable as a suitable facial image; calculate model parameters corresponding to changes in weights for updating the face authentication deep learning model by performing on-device learning with the suitable facial image; and transmit the model parameters to a server.

Description

A facial data collection and processing method for learning a facial authentication deep learning model within a smartphone device, and a device performing the method

The present invention relates to a facial data collection and processing method for learning a facial authentication deep learning model in a smartphone device, and a device for performing the method. More specifically, the present invention performs federated learning in which only the results of on-device learning are collected by the server after facial image data corresponding to personal information is not collected by the server but used for on-device learning in individual devices It relates to a device for, and a method in the device for.

As AI technologies such as artificial neural networks develop, facial authentication is widely used as in Patent Document 1. For facial authentication, it is necessary to train a deep learning model based on a large amount of facial data, and it may be determined whether or not the facial image inputted through the trained model matches the facial image of the registered user.

In the process of training a deep learning model, securing a large amount of facial data can be a problem. In order to train the deep learning model according to the service domain, that is, to learn according to the characteristics of users who are the target of providing the facial authentication service, it may be desirable to utilize the facial data in the corresponding service domain rather than the accessible open data.

However, in the service domain, the method of preparing facial data by collecting user images generated by users on their devices with a deep learning server corresponds to the collection of personal information, which may cause security problems. Therefore, an improved learning method that can train a deep learning model without problems regarding personal information security may be required.

Patent Document 1: Unexamined Patent Publication No. 10-2020-0116138

The technical problem to be solved by the present invention is to provide a federated learning method that can solve the problem of personal information security by preventing image data from being collected to a server in the process of learning a deep learning model.

As a means for solving the above-described technical problem, a device for federated learning for a facial authentication deep learning model according to an aspect of the present invention, a memory for storing instructions; and by executing the instructions: receiving the final updated status of the weights of the facial authentication deep learning model from the server managing the facial authentication deep learning model, photographing the user of the device to obtain a facial image of the user, extracting query facial features from the facial image based on the weights of the last updated status, and based on a comparison result for the query facial features and registered facial features pre-registered in the device to identify the user calculates the facial feature similarity, compares the facial feature similarity with a threshold similarity to determine whether to unlock the device lock screen, and performs data suitability verification on the facial image to select a suitable image set as an image, calculate model parameters corresponding to the amount of change of the weights for updating the facial authentication deep learning model by performing on-device learning by the suitable facial image, and transmit the model parameters to the server configured processor; includes

A method in a device for federated learning for a facial authentication deep learning model, performed by a processor executing instructions stored in a memory according to another aspect of the present invention, from the server managing the facial authentication deep learning model receiving a last updated state of weights of the facial authentication deep learning model; acquiring a facial image of the user by photographing the user of the device; extracting query facial features from the facial image based on the weights of the last updated state; calculating a degree of facial feature similarity based on a comparison result of the query facial features and registered facial features pre-registered in the device to identify the user; determining whether to unlock the lock screen of the device by comparing the facial feature similarity with a threshold similarity; performing data suitability verification on the face image to set an image determined to be suitable as a suitable facial image; calculating a model parameter corresponding to a change amount of the weights for updating the facial authentication deep learning model by performing on-device learning based on the fit face image; and sending the model parameters to the server. includes

According to the device and method according to the present invention, the facial image of the user acquired for unlocking the lock screen is not directly transmitted to the server, but only model parameters generated by on-device learning can be transmitted to the server, A facial authentication deep learning model can be trained by federated learning without collecting facial images on the server.

1 is a diagram for explaining a federated learning system of a facial authentication deep learning model according to some embodiments.
2 is a block diagram illustrating elements constituting a device for federated learning for a facial authentication deep learning model according to some embodiments.
3 is a diagram for explaining a method in which on-device learning is performed in the federated learning system according to some embodiments.
4 is a view for explaining the difference between the federated learning system and the existing central learning system according to some embodiments.
5 is a diagram for describing a specific process in which a device for federated learning operates according to some embodiments.
6 is a flowchart illustrating steps of configuring a method in a device for federated learning for a facial authentication deep learning model according to some embodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The description below is only for specifying the embodiments, and is not intended to limit or limit the scope of rights according to the present invention. What a person of ordinary skill in the art related to the present invention can easily infer from the detailed description and embodiments of the invention should be construed as belonging to the scope of the present invention.

The terms used in the present invention have been described as general terms widely used in the technical field related to the present invention, but the meaning of the terms used in the present invention is the intention of a technician in the relevant field, the emergence of new technology, examination standards or precedents. It may vary depending on Some terms may be arbitrarily selected by the applicant, and in this case, the meaning of the arbitrarily selected terms will be described in detail. Terms used in the present invention should be interpreted as meanings reflecting the overall context of the specification, not just dictionary meanings.

Terms such as 'consisting of' or 'comprising' used in the present invention should not be construed as necessarily including all of the components or steps described in the specification, and if some components or steps are not included, And when additional components or steps are further included, it should also be construed as intended from the term.

Terms including an ordinal number such as 'first' or 'second' used in the present invention may be used to describe various components or steps, but the components or steps should not be limited by the ordinal number. . Terms containing an ordinal number should only be interpreted for the purpose of distinguishing one element or step from other elements or steps.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Detailed descriptions of matters widely known to those of ordinary skill in the art related to the present invention will be omitted.

1 is a diagram for explaining a federated learning system of a facial authentication deep learning model according to some embodiments.

Referring to FIG. 1 , the federated learning system 10 of the facial authentication deep learning model may include a server 100 managing the facial authentication deep learning model and a device 200 for federated learning. As shown, in the federated learning system 10 , a single number of servers 100 may interact with a plurality of devices 200 .

The facial authentication deep learning model operated in the federated learning system 10 may mean a Convolutional Neural Network (CNN) trained by deep learning, and the weights constituting the CNN are a plurality of devices 200 . It can be continuously updated by federated learning by , and the plurality of devices 200 can provide a facial authentication function to the user by driving the facial authentication deep learning model inside the device by using the updated weights again.

Federated Learning for the facial authentication deep learning model, the learning data generated in the plurality of devices 200, that is, the user's facial image of the device 200 is collected by the server 100 and is not concentrated, Instead, it may refer to a learning method in which learning of the user's face image is performed in each device 200 in the form of on-device learning, and only the results are collected by the server 100 .

When the server 100 does not collect the facial images of users as learning data from a plurality of devices 200, and cooperative learning through on-device learning in each device 200 is performed, corresponding to personal information Since facial images that require sensitive handling may not be exposed to the outside of each device 200, learning data suitable for the service domain of the federated learning system 10, that is, suitable for users of a plurality of devices 200, is personal information. can be secured without fear of leakage of

On the other hand, the federated learning system 10 may be installed in each device 200 and operated by an application for performing federated learning. Each device 200 generates an incentive for the user to provide his or her face image by an application, for example, a tentative name cashFace, and the user executes a cashFace on the device 200 as will be described later. It is possible to provide training data for training the facial authentication deep learning model while receiving rewards such as mileage.

2 is a block diagram illustrating elements constituting a device for federated learning for a facial authentication deep learning model according to some embodiments.

Referring to FIG. 2 , the device 200 for federated learning for the facial authentication deep learning model may include a memory 210 and a processor 220 . However, the present invention is not limited thereto, and other general-purpose elements other than those shown in FIG. 2 may be further included in the device 200 . For example, the device 200 may further include a communication means for communicating with the server 100 or a photographing means for photographing a user's face image.

The device 200 may be a computing device for federated learning for a facial authentication deep learning model. The device 200 may execute a mobile/web application or computer program implementing the federated learning system 10 . For example, the device 200 may be any one of a PC, a smartphone, and a tablet PC, but is not limited thereto, and the device 200 may be implemented in the form of various electronic devices having processing capabilities.

With respect to a lock unlocking app such as a provisional name CashFace, as described above, the device 200 may include a photographing means, and may also include a touch screen in connection with unlocking the lock screen by a user's slide. can

The device 200 may include a memory 210 as a means for storing various data, instructions, and at least one program or software, and performs processing on various data by executing the instructions or at least one program. The processor 220 may be included as a means for doing so.

The memory 210 may store various instructions for the federated learning system 10 in the form of at least one program. For example, the memory 210 may store instructions constituting software such as a computer program or mobile/web application. In addition, the memory 210 may store various data required for execution of an application or program.

The memory 210 may be implemented as a non-volatile memory such as ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, or the like, or a volatile memory such as DRAM, SRAM, SDRAM, PRAM, RRAM, FeRAM, etc. can be implemented as In addition, the memory 210 may be implemented as HDD, SSD, SD, Micro-SD, or the like.

The processor 220 may operate the federated learning system 10 by executing instructions stored in the memory 210 . The processor 220 may perform a series of processing processes for implementing the federated learning system 10 . In addition, the processor 220 may perform an overall function for controlling the device 200 , and may process various operations inside the device 200 .

The processor 220 may be implemented as an array of multiple logic gates or a general-purpose microprocessor. The processor 220 may be configured as a single processor or a plurality of processors. The processor 220 may be integrally configured with the memory 210 , rather than as a separate configuration from the memory 210 for storing instructions. For example, the processor 220 may be implemented in the form of at least one of a CPU, a GPU, and an AP provided in the device 200 .

The processor 220 may receive the final updated state of the weights of the facial authentication deep learning model from the server 100 managing the facial authentication deep learning model by executing instructions stored in the memory 210 .

The server 100 may continuously update the weights of the facial authentication deep learning model based on the result of on-device learning in the device 200 in the federated learning system 10, and then back to the device 200. can provide

The processor 220 may acquire a face image of the user by photographing the user of the device 200 by executing instructions stored in the memory 210 .

A lock screen may be set on the device 200 by using a cashFace app that implements the federated learning system 10 , and the user of the device 200 unlocks the lock screen through the cashFace app When doing so, a facial image of the user may be acquired. To this end, a photographing means such as a camera capable of being provided in the device 200 may be utilized.

The processor 220 may extract query facial features from the facial image based on the weights of the last updated state by executing instructions stored in the memory 210 .

Since the weights of the last updated state constitute the facial authentication deep learning model stored in the server 100 , the facial authentication deep learning model may be utilized in the device 200 according to the weights. Query facial features may be extracted from the facial image by the deep learning model using the corresponding weights. The query facial features may then be compared for comparison with registered facial features.

The processor 220 executes instructions stored in the memory 210 to query facial features and facial features based on the comparison result for registered facial features pre-registered with the device 200 to identify the user. The similarity can be calculated.

The registered facial features are pre-registered for the user of the device 200 , and may be generated, for example, in a process of registering a user in order to perform facial authentication in a cashFace app. The closer the query facial features are to the registered facial features, the closer the facial image may be determined to the user's face.

The processor 220 may determine whether to unlock the lock screen of the device 200 by executing instructions stored in the memory 210 by comparing the facial feature similarity with a threshold similarity.

The query facial features and the registered facial features may be expressed numerically, and facial feature similarity may also be numerically calculated based thereon. As the query facial features and registered facial features are similar, the numerical value of the facial feature similarity may increase, and when the numerical value exceeds the threshold, the lock screen of the device 200 is released, and the user may be authenticated by facial authentication.

The processor 220 may perform data suitability verification on the face image by executing instructions stored in the memory 210 , and may set an image determined to be suitable as a suitable facial image.

Apart from unlocking the lock screen of the device 200 , whether the facial image is an image suitable for on-device learning in the federated learning system 10 may be determined by data suitability verification. On-device learning is not performed on the image determined to be inappropriate, and only the image determined to be suitable is set as a suitable face image, and then on-device learning may be performed thereon.

The processor 220, by executing the instructions stored in the memory 210, performs on-device learning by a suitable facial image to calculate a model parameter corresponding to the amount of change in weights for updating the facial authentication deep learning model. can

On-device learning means that the process of learning the facial authentication deep learning model in the federated learning system 10 is performed directly in each device 200 of the federated learning system 10, not by the server 100. can According to on-device learning, model parameters indicating how to update the weights constituting the CNN of the facial authentication deep learning model, that is, whether to increase, decrease, or maintain each weight, can be calculated.

The processor 220 may transmit the model parameter to the server 100 by executing instructions stored in the memory 210 .

When the model parameters are transmitted from each device 200 to the server 100 in the federated learning system 10, the server 100 may update the weights of the facial authentication deep learning model, that is, CNN based on the model parameters. there is. Thereafter, the updated weights may be transmitted back to each device 200 and utilized for facial authentication of the user.

3 is a diagram for explaining a method in which on-device learning is performed in the federated learning system according to some embodiments.

Referring to FIG. 3 , a scenario 300 for explaining how on-device learning is performed in the federated learning system 10 is shown. The scenario 300 may mean, for example, a scenario of a cashFace app implementing the federated learning system 10 .

In addition to the server 100 and the device 200 , the scenario 300 shows a user 310 and an advertisement provider 320 . In the scenario 300, the device 200 may obtain a facial image of the user 310 by providing rewards such as advertisements and mileage to the user 310 through the cashFace app, and through this, the on- Device learning can be performed.

As shown in the scenario 300 , the device 200 may execute and release the lock screen, expose and remove the advertisement screen, and acquire the face image of the user 310 in the process. In particular, as the server 100 does not intervene in the process of acquiring the user's 310's face image and performing on-device learning through it, as shown, personal information security problem due to data collection to the server 100 It can be confirmed that this does not occur.

4 is a diagram for explaining the difference between the federated learning system and the existing central learning system according to some embodiments.

Referring to FIG. 4 , an existing centralized learning system 20 and a federated learning system 10 are illustrated. The central learning system 20 and the federated learning system 10 may differ in terms of where learning based on facial images is performed.

In the central learning system 20 , the facial image generated by the device 22 is transmitted to the server 21 , so that model learning by the facial image can be performed in the server 21 . That is, the learning module could exist in the server 21 rather than the device 22 . Alternatively, in the federated learning system 10 , the learning model may exist in the device 200 rather than the server 100 as described above.

Also, the central learning system 20 and the federated learning system 10 may differ from each other as to where the data cleaning process is performed. In the central learning system 20, the server 21 is provided with a purification unit for data purification, but in contrast to that, in the federated learning system 10, the facial information extraction data collection unit, ie, data purification, is provided in the device 200 as shown in the figure. A configuration for the may be provided.

That is, in the federated learning system 10 , a data purification process for a facial image may be performed in the device 200 , and the data purification process is a variety of information accessible in the device 200 , for example, a user's touch/slide This may be performed based on a pattern, a location or time of taking a picture, and the brightness of a background. Accordingly, since data purification may be performed as a preprocessing of model parameter generation in the device 200 , filtering of inappropriate data may be performed to improve the learning efficiency of federated learning.

5 is a diagram for explaining a specific process in which a device for federated learning operates according to some embodiments.

Referring to FIG. 5 , the operation process in the device 200 and the server 100 in the federated learning system 10 is presented as steps ① to 28. Meanwhile, '...unit' described in FIG. 5 with respect to steps ① to 28 may mean a set of instructions, a program, or a software configuration executed by the processor 220 to perform a corresponding function.

In step ①, a camera view is executed by a photographing means of the device 200, for example, a camera, and a facial image may be acquired. In step ②, the facial data capturing unit may transfer the image of the camera view to the face detecting unit. In step ③, the face detection unit can use a CNN-based deep learning model to detect the position of the face and landmarks (a total of five points such as eyes, nose, and corners of the mouth) and pass it on to the facial feature extraction unit.

With respect to step ③, the processor 220, when acquiring the facial image, may acquire the facial image by extracting the facial location and facial landmarks from the photographed image for the user based on the facial detection CNN model.

In step ④, the facial feature extraction unit may extract the features of the user's face as an N-dimensional vector by using the facial authentication deep learning model, that is, the weights transmitted from the server 100 . In this case, N may be 128, or may be various other values. In step ⑤, the authenticity determining unit can calculate the cosine similarity of the registered user's facial features and the features extracted by the facial feature extraction unit, and when the cosine similarity exceeds a threshold, it is transmitted to the advertisement exposure unit to advertise the cashFace app can be turned off.

With respect to step ④ and step ⑤, each of the query facial features and the registered facial features may be configured in the form of a query vector consisting of 128 elements and a registration vector consisting of 128 elements, and the processor ( 200), when calculating the facial feature similarity, may calculate a comparison result based on the cosine similarity with respect to the query vector and the registration vector.

If the result of the authenticity judgment in step ⑥ is False, the number of failures may increase by 1. In step ⑦, if the number of failures is less than the reference value, the value is passed to the facial data capturing unit, and facial authentication may be restarted. If the number of failures exceeds another threshold in step ⑧, it is determined that the person attempting authentication is not a registered user in the device 200, the algorithm is stopped in the camera view, and facial authentication may not proceed any more.

If the determination value of the authenticity determination unit is True in step ⑨, the facial feature of the N-dimensional vector may be passed to the identity verification unit. In step ⑩, the photo taken by the facial data capturing unit may be handed over to the identity verification unit, and at this time, the time of photo taking and GPS coordinates may be handed over together. In step ⑪, the slide unit may execute the slide algorithm unit, and in step ⑫, the slide algorithm unit may determine whether the user has performed the slide, and transmit the corresponding value to the advertisement exposure unit to release the advertisement. In step ⑬, the slide pattern may be transmitted to the authenticity verification module. In step ⑭, the slide pattern may be identified by a deep learning algorithm and transmitted to the identity verification unit. In step ⑮, identity verification may be performed based on a slide pattern, a user's face photo, a photo taking time, GPS coordinates, an N-dimensional facial feature vector, and whether or not the face authentication is successful, and when the identity is verified, the data suitability determining unit may be executed.

In relation to steps ⑨ to ⑮, the processor 220, when set as a suitable facial image, performs identity verification on the facial image to verify that the facial image was acquired by the user, and then performs data suitability verification. there is.

In addition, in relation to steps ⑨ to ⑮, the lock screen may be released by a user's slide, and the processor 220, when performing data suitability verification, includes a slide pattern in which the user slides the lock screen, lock Identity verification may be performed based on whether to release the screen and capturing data of the face image, and the capturing data may include a time at which the face image was captured and a GPS location.

In addition, in relation to steps ⑨ to ⑮, mileage may be accumulated to the user whenever the lock screen is released by the user's slide, and an advertisement image may be displayed on the lock screen.

In step 16, image learning environment information stored in the memory 210 may be requested. Data suitability may be determined by comparing the image learning environment information in step 17 with the time, GPS, and N-dimensional facial features in step ⑮.

In relation to steps 16 and 17 , when performing data suitability verification, the processor 220 compares the photographed data and query facial features with image learning environment information stored in advance in the device 200 to determine whether the facial image is suitable. It can be determined whether or not

If the data suitability is high in step 18, the facial feature extraction model learning unit is executed to start learning, and as a result of the learning, a model parameter corresponding to the differential value of the weight of the deep learning model may be derived.

In relation to step 18, the processor 220 performs on-device learning by determining the amount of change in weights in a direction in which query facial features extracted from the fitted facial image approach the registered facial features when calculating the model parameters. can do. That is, the amount of change of the weights may be determined in a direction to reduce the difference between the query facial features and the pre-stored registered facial features. In this case, the model parameter may change the weights in a direction to better recognize the face of the user of the device 200 .

In step 19 , the value of the model parameter may be stored in the memory 210 . When communication with the server 200 is connected in step 20 , the value of the model parameter stored in the memory 210 may be transferred to the learned model collecting unit of the server 200 and stored in the buffer of the server 200 . In step 21, it may be determined whether the number of model parameters collected as the reception counter is greater than or less than the total size of the buffer. If the number of model parameters collected in step 22 is smaller than the buffer size, the trained model collector may collect more model parameters while waiting. If the number of model parameters collected in step 23 is not less than the buffer size, the reception counter may be set to zero. If the number of model parameters collected in step 24 is not smaller than the buffer size, the values of the model parameters stored in the buffer may be passed to the model updater.

In step 25, the model update unit may generate a new parameter value by averaging the model parameter values and transmit it to the model verification unit. In step 26, data may be requested from the verification DB. In step 27, performance evaluation may be performed based on the data received from the verification DB and the updated model. If the best performance has been updated in step 28, the updated model may be delivered to the model distribution unit. If communication is established with the device 200 in step 29, the updated model may be newly distributed and the facial feature model may be changed. In operation 30 , the model with the highest performance may be transmitted to the facial feature extraction unit.

In relation to steps 25 to 30, the server 100, when the highest performance of the facial authentication deep learning model is updated as a result of updating the weights based on a plurality of model parameters, the final updated state of the weights updated results may be transmitted to the device 200 as

On the other hand, in the federated learning system 10 as shown, on-device learning can be performed not only by the device 200 but also by other devices, and the model parameters for learning the facial authentication deep learning model based on the result are can be created That is, federated learning may be performed through a plurality of devices including the device 200 , and the server 100 is transmitted from the plurality of devices based on a plurality of model parameters including the model parameter. You can update the weights.

6 is a flowchart illustrating steps of configuring a method in a device for federated learning for a facial authentication deep learning model according to some embodiments.

Referring to FIG. 6 , the method in the device 200 for federated learning for the facial authentication deep learning model may include steps 610 to 680 . However, the present invention is not limited thereto, and general steps other than the steps shown in FIG. 6 may be further included in the method of FIG. 6 .

The method in the device 200 for federated learning for the facial authentication deep learning model of FIG. 6 may consist of steps processed in time series in the device 200 described through FIGS. 1 to 5 . Therefore, even if the contents of the method of FIG. 6 are omitted below, the contents described above with respect to the device 200 of FIGS. 1 to 5 may be equally applied to the method of FIG. 6 .

The method in the device 200 for federated learning for the facial authentication deep learning model of FIG. 6 may be performed by the processor 220 executing instructions stored in the memory 210 .

In step 610 , the device 200 may receive a final updated state of weights of the facial authentication deep learning model from the server 100 managing the facial authentication deep learning model.

In operation 620 , the device 200 may acquire a face image of the user by photographing the user of the device 200 .

When acquiring the facial image, the device 200 may acquire the facial image by extracting the facial location and facial landmarks from the captured image for the user based on the facial detection CNN model.

In operation 630 , the device 200 may extract query facial features from the facial image based on the weights of the last updated state.

In operation 640 , the device 200 may calculate the facial feature similarity based on the query facial features and the comparison result for the registered facial features registered in advance in the device 200 to identify the user.

Each of the query facial features and the registered facial features may be configured in the form of a query vector consisting of 128 elements and a registration vector consisting of 128 elements, and the device 200 calculates the facial feature similarity In this case, a comparison result may be calculated based on the cosine similarity of the query vector and the registration vector.

In operation 650 , the device 200 may determine whether to unlock the lock screen of the device 200 by comparing the facial feature similarity with a threshold similarity level.

In operation 660 , the device 200 may set the image determined to be suitable by performing data suitability verification on the facial image as a suitable facial image.

The device 200 may perform data suitability validation after it is verified that the facial image was acquired by the user by performing identity validation on the face image when setting it as a suitable facial image.

The lock screen may be released by a user's slide, and when the device 200 performs data suitability verification, a slide pattern in which the user slides the lock screen, whether to release the lock screen, and photographing data of a face image Identity verification may be performed based on , and the captured data may include a time at which the facial image was captured and a GPS location.

When performing the data suitability verification, the device 200 may determine whether the facial image is suitable by comparing the photographed data and the query facial features with image learning environment information previously stored in the device 200 .

Each time the lock screen is released by the user's slide, the user may be credited with mileage, and an advertisement image may be displayed on the lock screen.

In operation 670, the device 200 may calculate a model parameter corresponding to a change amount of weights for updating the facial authentication deep learning model by performing on-device learning based on the fit face image.

When calculating the model parameter, the device 200 may perform on-device learning by determining the amount of change in weights in a direction in which query facial features extracted from the fit facial image approach the registered facial features.

In step 680 , the device 200 may transmit the model parameter to the server 100 .

The federated learning may be performed through a plurality of devices including the device 200, and the server 100 calculates weights based on the plurality of model parameters, including the model parameters, transmitted from the plurality of devices. can be updated.

Server 100, when the highest performance of the facial authentication deep learning model is updated as a result of updating the weights based on a plurality of model parameters, the weights can be transmitted to the device 200 as a final updated state. there is.

On the other hand, the method in the device 200 for federated learning for the facial authentication deep learning model of FIG. 6 is a computer-readable recording medium in which at least one program or software including instructions for executing the method is recorded. can be recorded.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and floppy disks. Magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like may be included. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments of the present invention have been described in detail above, the scope of the rights according to the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention described in the following claims are also provided. It should be interpreted as being included in the scope of rights according to the

Claims (20)

In a device for federated learning for a facial authentication deep learning model,
a memory storing instructions; and
By executing the above commands:
Receive the final updated state of the weights of the facial authentication deep learning model from the server managing the facial authentication deep learning model,
To obtain a facial image of the user by photographing the user of the device,
extracting query facial features from the facial image based on the weights of the last updated state;
calculating a facial feature similarity based on a comparison result for the query facial features and registered facial features pre-registered in the device to identify the user,
determining whether to unlock the lock screen of the device by comparing the facial feature similarity with a threshold similarity;
Performing data suitability verification on the face image to set the image determined to be suitable as a suitable facial image,
Calculating a model parameter corresponding to the amount of change of the weights for updating the facial authentication deep learning model by performing on-device learning based on the fit face image,
a processor configured to send the model parameters to the server; including,
The processor, when setting as the suitable facial image, performs identity verification on the facial image to perform the data suitability verification after verifying that the facial image has been obtained by the user, facial authentication deep learning model A device for federated learning about
delete The method of claim 1,
The lock screen is released by the user's slide,
When performing the data suitability verification, the processor performs the identity verification based on a slide pattern in which the user slides the lock screen, whether to release the lock screen, and photographing data of the face image,
The photographing data, including the time and GPS location at which the facial image was captured, a device for federated learning for a facial authentication deep learning model.
4. The method of claim 3,
The processor, when performing the data suitability verification, compares the photographed data and the query facial features with image learning environment information stored in advance in the device to determine whether the facial image is suitable, facial authentication deep A device for federated learning on a learning model.
4. The method of claim 3,
Each time the lock screen is released by the user's slide, mileage is accumulated to the user, and an advertisement image is displayed on the lock screen.
The method of claim 1,
The processor performs the on-device learning by determining the amount of change of the weights in a direction in which the query facial features extracted from the fitted facial image approach the registered facial features when calculating the model parameter. A device for federated learning of facial authentication deep learning models.
The method of claim 1,
The processor, when acquiring the facial image, acquires the facial image by extracting facial positions and facial landmarks from the photographed image for the user based on a facial detection CNN model A device for learning.
The method of claim 1,
Each of the query facial features and the registered facial features is configured in the form of a query vector consisting of 128 elements and a registration vector consisting of 128 elements,
The processor, when calculating the facial feature similarity, calculates the comparison result based on the cosine similarity with respect to the query vector and the registration vector, a device for federated learning for a facial authentication deep learning model.
The method of claim 1,
The federated learning is performed through a plurality of devices including the device,
The server updates the weights based on a plurality of model parameters, including the model parameter, transmitted from the plurality of devices, a device for federated learning for a facial authentication deep learning model.
10. The method of claim 9,
The server, when the highest performance of the facial authentication deep learning model is updated as a result of updating the weights based on the plurality of model parameters, transmits the updated result of the weights to the device as the final updated state , a device for federated learning on facial authentication deep learning models.
A method in a device for federated learning for a facial authentication deep learning model, performed by a processor executing instructions stored in a memory, the method comprising:
Receiving a final updated state of the weights of the facial authentication deep learning model from the server managing the facial authentication deep learning model;
acquiring a facial image of the user by photographing the user of the device;
extracting query facial features from the facial image based on the weights of the last updated state;
calculating a degree of facial feature similarity based on a comparison result of the query facial features and registered facial features pre-registered in the device to identify the user;
determining whether to unlock the lock screen of the device by comparing the facial feature similarity with a threshold similarity;
performing data suitability verification on the face image to set an image determined to be suitable as a suitable facial image;
calculating a model parameter corresponding to a change amount of the weights for updating the facial authentication deep learning model by performing on-device learning based on the fit face image; and
sending the model parameters to the server; including,
The step of setting as the suitable facial image includes performing identity verification on the facial image to verify that the facial image is obtained by the user and then performing the data suitability verification, facial authentication deep learning An on-device method for federated learning of a model.
delete 12. The method of claim 11,
The lock screen is released by the user's slide,
The performing of the data suitability verification includes performing the identity verification based on a slide pattern in which the user slides the lock screen, whether to release the lock screen, and photographing data of the face image,
The photographing data is a method in a device for federated learning for a facial authentication deep learning model, including the time and GPS location at which the facial image was photographed.
14. The method of claim 13,
The step of performing the data suitability verification comprises determining whether the facial image is suitable by comparing the photographed data and the query facial features with image learning environment information stored in advance in the device. An on-device method for federated learning of deep learning models.
14. The method of claim 13,
Each time the lock screen is released by the user's slide, mileage is accumulated to the user, and an advertisement image is displayed on the lock screen. Method in a device for federated learning for a facial authentication deep learning model.
12. The method of claim 11,
Calculating the model parameter includes performing the on-device learning by determining the amount of change of the weights in a direction in which the query facial features extracted from the fitted facial image approach the registered facial features , a method on the device for federated learning of facial authentication deep learning models.
12. The method of claim 11,
The step of acquiring the facial image includes acquiring the facial image by extracting the facial location and facial landmarks from the photographed image for the user based on the facial detection CNN model, facial authentication for a deep learning model A method on a device for federated learning.
12. The method of claim 11,
Each of the query facial features and the registered facial features is configured in the form of a query vector consisting of 128 elements and a registration vector consisting of 128 elements,
The calculating of the facial feature similarity includes calculating the comparison result based on the cosine similarity for the query vector and the registration vector. Method in a device for federated learning for a facial authentication deep learning model .
12. The method of claim 11,
The federated learning is performed through a plurality of devices including the device,
wherein the server updates the weights based on a plurality of model parameters including the model parameter transmitted from the plurality of devices.
20. The method of claim 19,
The server, when the highest performance of the facial authentication deep learning model is updated as a result of updating the weights based on the plurality of model parameters, transmits the updated result of the weights to the device as the final updated state , a method on the device for federated learning of facial authentication deep learning models.

Images