WO2024111814A1

WO2024111814A1 - Device and method for generating de-identified face image capable of face authentication

Info

Publication number: WO2024111814A1
Application number: PCT/KR2023/013223
Authority: WO
Inventors: 최대선; 나현식; 이정재
Original assignee: 숭실대학교 산학협력단
Priority date: 2022-11-25
Filing date: 2023-09-21
Publication date: 2024-05-30
Also published as: KR20240077829A

Abstract

The present invention provides a device and a method for generating a de-identified face image capable of face authentication. The device for generating a de-identified face image capable of face authentication comprises: an encoder which extracts a plurality of first style vectors of an original face image and a plurality of second style vectors of a synthesized target face image; a style vector synthesizer that generates a plurality of synthesized style vectors by combining the plurality of first style vectors and the plurality of second style vectors with a plurality of synthesis parameters that control a synthesis ratio; and a face image generator that generates a de-identified final face image of the original face image using the plurality of synthesized style vectors.

Description

Device and method for generating de-identified face image capable of face authentication

The present invention relates to an apparatus and method for generating facial images, and more specifically, to an apparatus and method for generating facial images that enable face authentication using a deep learning-based facial authentication model but are non-identifiable to the naked eye.

Deep learning-based face authentication models are being commercialized in many fields to authenticate users. At this time, while the user is inputting his or her face image into the model, data may be leaked due to a packet attack by an external attacker or deception by the administrator. Facial images are sensitive personal information of users and will cause serious problems if appropriate security measures are not in place. Therefore, an effective method is needed to prevent the possibility of violating the user's personal information when using the face authentication model.

Even if the input image for face authentication is leaked, the user's personal information can be protected if the attacker cannot identify the person in question. At this time, de-identification can be effectively used to minimize exposure of the user's face. However, existing de-identification technologies only focused on high-level anonymization to minimize exposure of actual faces. Therefore, there are limits to applying existing methods in situations where both de-identification and face authentication are required, and there is a demand for methods to complement this.

Although face authentication artificial intelligence models show high performance and are used in many fields, there is a possibility that the user's face image may be leaked during the process of inputting the input image into the model. Although de-identification technology exists as a method to minimize exposure of face images, there is a problem that authentication performance is greatly reduced when applying existing technology in the special situation of face authentication.

When a user attempts to authenticate himself through a face authentication model, he must input his face image into the model. At this time, the user's facial image may be exposed due to external attacks such as packet theft or hacking, which may lead to privacy infringement issues.

As a countermeasure against this, a de-identification method that can minimize exposure of the user's face can be effectively used. However, existing de-identification methods have a limitation in that they focus only on high-level anonymization and cannot maintain face authentication performance. On the other hand, if the de-identification strength is lowered to maintain face authentication performance, the amount of user face modification is reduced and does not help protect privacy. In other words, an appropriate countermeasure that can satisfy both de-identification performance and face authentication performance is required.

[Prior art literature]

[Patent Document]

(Patent Document 1) Registered Patent Publication 10-2192235 (2020.12.10)

The problem to be solved by the present invention is to provide an image generation device and method capable of sufficiently de-identifying input images while maintaining the authentication performance of a face authentication model.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A de-identified facial image generating device capable of face authentication according to an embodiment of the present invention,

an encoder for extracting a plurality of first style vectors of the original face image and a plurality of second style vectors of the synthesized target face image;

a style vector synthesizer for generating a plurality of composite style vectors by combining the plurality of first style vectors and the plurality of second style vectors with a plurality of synthesis parameters that adjust a synthesis ratio; and

and a face image generator that generates a de-identified final face image of the original face image using the plurality of composite style vectors.

Preferably,

The face image generator further generates at least one intermediate face image,

It further includes a deep learning-based face authentication model that determines whether the intermediate face image is authenticated as the original face image.

Preferably,

When the intermediate face image is authenticated as the original face image using the deep learning-based face authentication model, the style vector synthesizer generates a new synthesis style vector using the increased synthesis parameter value, and the face image generator Generate a new intermediate face image using the new composite style vector,

Generation of a new composite style vector using increased synthesis parameter values in the style vector synthesizer and generation of a new intermediate face image in the face image generator are repeated until the new intermediate face image is not authenticated as the original face image. do.

Preferably,

The face image generator generates the final face image using a synthesis style vector having synthesis parameters of a final intermediate face image that is authenticated as the original face image by the deep learning-based face authentication model.

Preferably,

The range of the synthesis parameter value is 0 to 1,

The face image generator sets some of the synthesis parameters to zero (0) so that the final face image maintains the shape of the original face image.

A method for generating a de-identified face image capable of face authentication according to an embodiment of the present invention,

Extracting a plurality of first style vectors of the original face image using an encoder;

extracting a plurality of second style vectors of a target face image synthesized with the original face image using the encoder;

generating a plurality of composite style vectors by combining the plurality of first style vectors and the plurality of second style vectors with a plurality of synthesis parameters that adjust a synthesis ratio using a style vector synthesizer;

generating an intermediate face image using the plurality of composite style vectors using a face image generator;

determining whether the intermediate face image is authenticated as the original face image using a deep learning-based face authentication model;

When the intermediate face image is authenticated as the original face image, the synthesis parameter value is increased, a plurality of new synthesis style vectors are generated using the increased synthesis parameter value, and the new synthesis style vectors are used to generate a plurality of new synthesis style vectors. generating a new intermediate face image; and

The steps of generating a new intermediate face image and determining whether it is authenticated as the original face image are repeated to have synthesis parameters of a final intermediate face image that is authenticated as the original face image by the deep learning-based face authentication model. and generating a de-identified final face image of the original face image using a composite style vector.

Preferably,

The range of the synthesis parameter value is 0 to 1,

When the synthesis parameter value is 0, the intermediate face image is identical to the original face image.

Preferably,

In the face image generator, each of the plurality of first style vectors and the plurality of second style vectors is classified into a first set, a second set, and a third set,

A first set of each of the plurality of first style vectors and the plurality of second style vectors is associated with a shape of the face image, a second set is associated with eyes, nose and mouth, a third set is associated with hair color and It is related to skin color.

Preferably,

The face image generator sets a synthesis parameter to zero (0) when combining each first set of the plurality of first style vectors and the plurality of second style vectors so that the final facial image is in the form of the original facial image. to maintain.

Specific details of other embodiments are included in the detailed description and drawings.

The device and method for generating a de-identified face image capable of face authentication according to the present invention can perform face authentication using the generated de-identified image, and at the same time prevent leakage of facial personal information.

By visualizing images generated using the device and method for generating a de-identified face image capable of face authentication according to the present invention, there is an effect of maintaining de-identification performance while maintaining authentication performance.

The device and method for generating a de-identified face image capable of face authentication according to the present invention can be used in the process of inputting a face image into a face authentication model in a situation where both face authentication and de-identification are required, so there is a possibility of violating the user's facial personal information. can be prevented.

However, the effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram showing a schematic diagram of a de-identified facial image generating device capable of face authentication according to the present invention.

Figure 2 is a diagram showing a schematic diagram of the encoder of the de-identified face image generating device capable of face authentication according to the present invention.

Figure 3a is a diagram showing a schematic diagram of the face image generator of the de-identified face image generating device capable of face authentication according to the present invention.

FIG. 3B is a diagram schematically showing an operation flowchart of the facial image generator of FIG. 3A.

Figure 4 is a diagram for explaining the concept of adjusting the synthesis parameters of the synthesis parameter adjuster of the de-identified face image generating device capable of face authentication according to the present invention.

Figure 5 is a diagram illustrating a deep learning-based face authentication model for verifying the authentication performance of the de-identified face image generating device capable of face authentication according to the present invention.

Figure 6 is a flowchart of a method for generating a de-identified face image capable of face authentication according to the present invention.

Figure 7 is a diagram showing an example of the final de-identified image generation result using the de-identified face image generating device and method capable of face authentication according to the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to be understood by those skilled in the art in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Embodiments described herein will be explained with reference to cross-sectional views and/or plan views, which are ideal illustrations of the present invention. In the drawings, the thickness of the components is exaggerated for effective explanation of technical content. Accordingly, the configurations illustrated in the drawings have schematic properties, and the shapes of the configurations illustrated in the drawings are intended to illustrate specific forms of the configuration and are not intended to limit the scope of the invention. In various embodiments of the present specification, terms such as first, second, and third are used to describe various components, but these components should not be limited by these terms. These terms are merely used to distinguish one component from another. Embodiments described and illustrated herein also include complementary embodiments thereof.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” means the presence of one or more other components, steps, operations and/or elements in a mentioned element, step, operation and/or element. or does not rule out addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, with reference to the drawings, the concept of the present invention and embodiments thereof will be described in detail.

The present invention relates to a new de-identification method using StyleGAN, an image generation model that can generate images by injecting various styles into face images.

The present invention combines the user's facial feature information with the facial feature information of a specific person (virtual person) to create a new facial image, maintaining the authentication performance of the face authentication model while maximizing the strength of de-identification. .

The Pixel2Style2Pixel (PSP) encoder of the present invention extracts the features of the target face image that is synthesized with the original face image.

The image generation model of the present invention - StyleGAN combines the facial characteristics of the target person synthesized with the original face image and then generates a de-identified face image.

The present invention proposes a method of optimizing the combination ratio of two facial features according to the face authentication model using HopSkipJumpAttack.

The de-identified face image generating device 100 capable of face authentication of the present invention includes an encoder 110, a style vector synthesizer 120, a face image generator 130, a deep learning-based face authentication model 140, and a synthesis parameter adjuster. Includes (150).

The encoder 110 extracts a plurality of first style vectors of the original face image and a plurality of second style vectors of the synthesized target face image.

The style vector synthesizer 120 generates a plurality of composite style vectors by combining a plurality of first style vectors and a plurality of second style vectors using a plurality of synthesis parameters that control the synthesis ratio.

The face image generator 130 generates a de-identified final face image of at least one intermediate face image and an original face image using a plurality of composite style vectors.

The deep learning-based face authentication model 140 compares the intermediate face image with the original face image to determine whether it is authenticated as the original face.

The synthesis parameter controller 150 adjusts the value of the synthesis parameter (λ) that controls the synthesis ratio. The range of synthesis parameter values is 0 to 1, and the face image generator 130 sets some of the synthesis parameters to zero (0) so that the final face image maintains the shape of the original face image.

The style vector synthesizer 120, the face image generator 130, the deep learning-based face authentication model 140, and the synthesis parameter adjuster 150 form a loop.

When the intermediate face image is authenticated as the original face image using the deep learning-based face authentication model 140, the synthesis parameter adjuster 150 increases the synthesis parameter value to increase the de-identification strength.

The style vector synthesizer 120 generates a new synthesis style vector using the increased synthesis parameter value.

The face image generator 130 generates a new intermediate face image using the new composite style vector.

until the new intermediate face image is not authenticated as the original face image, increasing the compositing parameter values in the compositing parameter adjuster 150, generating a new compositing style vector using the increased compositing parameter values in the style vector compositor 120, and Generation of a new intermediate face image in the face image generator 130 is repeated.

The face image generator 130 generates a final face image using a synthesis style vector having synthesis parameters of the final intermediate face image that is authenticated as the original face image by the deep learning-based face authentication model 140.

Using FIG. 1, the operation of the de-identified facial image generating device capable of face authentication according to the present invention will be described in more detail.

The encoder 110 uses style vectors of X _source (original face image) and X _target (face image of a virtual person)

and

Extract . style vector

and

is input to the StyleGAN model of the face image generator 130 and synthesized using synthesis parameters of size 18 x 512 before generating the image.

synthesis parameters

The range is between 0 and 1.

Equation (1) is the equation for generating a composite style vector.

If is 0, the face image generator 130

Since the image is generated through , the final face image is created the same as _the on the other way

If is 1, the face image generator 130

_Since the image is generated through

If is between 0 and 1, the style vector synthesizer 120

creates .

Through this, the face will be authenticated, but a final face image that can be sufficiently de-identified will be created.

The face image generator 130 images the shape and appearance of the face from the input style in the upper 4 layers of the 18 layers of the StyleGAN model, and the present invention

All are fixed to the value 0. top 4 tiers

By fixing all to the value 0, the final generated face image can maintain the same facial shape as the original face image.

This setting was selected because the present invention is far from the intended goal of de-identification when even the shape of the face changes.

In order to optimize for the purpose, a type of attack is performed on the face authentication model using the HSJA (HopSkipJumpAttack) method of the synthetic parameter adjuster 150.

λ _start = [[0, 0, … which can generate an image close to the X _source (original face image). ] … ] is set as the initial value, and this is set to λ _target = [[1, 1, … ] … Adjust towards ].

With the updated λ, the style vector synthesizer 120 synthesizes the style vector as Equation (1):

creates .

The face image generator 130 is a composite style vector

Use this to create _an intermediate face image, X _mix , which combines the facial features of the

The deep learning face authentication model 140 determines whether to authenticate the de-identified image X _mix .

Depending on whether authentication is successful, if authentication fails, the composite parameter value in the parameter adjuster 150 is decreased, and if authentication is successful, the composite parameter value in the parameter adjuster 150 is increased. Intermediate image creation is repeated while adjusting the synthesis parameter values depending on whether authentication is successful or not.

The encoder 110 of the present invention uses Pixel2Style2Pixel (PSP) encoding technique.

The encoder 110 receives the original face image and inputs it as a style to the face image generator 130.

Output vector. The facial image generator 130 corresponds to

The final face image generated using the vector may generate a face image very similar to the original image received as input by the encoder 110.

The encoder 110 of the present invention can obtain a vector with a size of 18 x 512 tailored to the 18th layer, rather than the 1 x 512 size used conventionally. By acquiring vectors of expanded size, the facial image generating device of the present invention outperforms conventional image generating devices.

The face image generator 130 of the present invention uses the StyleGAN image generation model. StyleGAN is a Generative Adversarial Networks (GAN) model that generates an image by copying the style W ⁺ vector of size 1 .

The StyleGAN model of the face image generator 130 of the present invention is an adversarial generation model that can generate images of various styles. The StyleGAN model of the present invention consists of 18 layers, and the W ⁺ (1 x 512) vector representing the style of the face image is copied and input into each layer. The face image generator 130 generates a face image using an upsampling method with reference to each style. Of the 18 layers, 4 layers are involved in creating the shape of the face. The next four layers are involved in the positioning of the eyes, nose, and mouth. The remaining 10 layers are involved in hair color, skin color, etc.

The synthesis parameters of the synthesis parameter adjuster 150 of the present invention are adjusted to maximize the amount of facial modulation within the range in which the original face image is authenticated.

The present invention performs style vector synthesis to maintain the user's face shape in order to maintain authentication of the original face image.

The synthesis parameter adjuster 150 uses the HSJA (HopSkipJumpAttack) method.

HSJA (HopSkipJumpAttack) transforms the input image and inputs it into the classification model. This is an attack method that uses the classification results of the process to induce misclassification of existing input images. Unlike other attacks that deform the image using the gradient calculated through backpropagation of the model, the attack randomly adjusts the deformation.

HopSkipJumpAttack attempts to query input data in a black box environment that does not have access to the internal information (weight layer, activation layer, etc.) of the classification model to be attacked, and then uses the output value of the classification model to produce the same predicted value even though it looks like a different image. This is a method of generating adversarial examples to print.

As shown in Figure 4, a query is attempted using binary classification from the original image (λ_start) to the target image (a virtual image, which is an image synthesized with the original image) (λ_target), and λ_start is performed using the criteria that does not change the classification result. Adjust.

Next, several queries are performed to slightly adjust movement within the classification results to obtain a direction vector. After adjusting the direction vector within the criteria that does not change the classification result, it again approaches λ_target through binary classification.

As a result, it is classified into the same class as λ_start, that is, it is authenticated as the original image, but it is possible to generate an adversarial example that looks like λ_target.

HopSkipJumpAttack is performed by calculating the median value from the source to the target using a binary query method, checking whether the median value is in a range that can be recognized as the original, and calculating the λ value to get as close to the target as possible.

The deep learning-based face authentication model 140 of the present invention uses deepface's VGG-FACE.

The deep learning-based face authentication model 140 extracts feature vectors from two face input images through a Conv layer and compares the similarity.

The deep learning-based face authentication model 140 operates by returning True if the compared similarity is less than a threshold, and False if it is more than the threshold.

The deep learning-based face authentication model 140 of the present invention sets the threshold to 0.4, which is the default value.

The method of generating a de-identified face image capable of face authentication of the present invention includes the step of extracting a plurality of first style vectors of the original face image using the encoder 110 (S610).

The method of generating a de-identified face image capable of face authentication of the present invention includes the step of extracting a plurality of second style vectors of the target face image synthesized with the original face image using the encoder 110 (S620).

The method of generating a de-identified face image capable of face authentication of the present invention uses a style vector synthesizer 120 to combine the plurality of first style vectors and the plurality of second style vectors with a plurality of synthesis parameters that control the synthesis ratio. It includes a step (S630) of generating a plurality of composite style vectors by combining them.

The method of generating a de-identified face image capable of face authentication of the present invention includes the step of generating an intermediate face image using a plurality of composite style vectors using the face image generator 130 (S640).

The method of generating a de-identified face image capable of face authentication of the present invention includes a step (S650) of determining whether an intermediate face image is authenticated as the original face image using a deep learning-based face authentication model 140.

The method of generating a de-identified face image capable of face authentication of the present invention increases the synthesis parameter value when the intermediate face image is authenticated as the original face image, and creates a plurality of new synthesis style vectors using the increased synthesis parameter value. and generating a new intermediate face image using a plurality of new composite style vectors (S660).

The method of generating a de-identified face image capable of face authentication of the present invention repeats the step of generating a new intermediate face image and the step of determining whether it is authenticated as the original face image, and converts it to the original face image by a deep learning-based face authentication model. It includes generating a de-identified final face image of the original face image using a synthesis style vector having synthesis parameters of the final intermediate face image to be authenticated (S670).

The range of the synthesis parameter value is 0 to 1, and when the synthesis parameter value is 0, the intermediate face image is the same as the original face image.

In the facial image generator 130, each of the plurality of first style vectors and the plurality of second style vectors is classified into a first set, a second set, and a third set, and the plurality of first style vectors and the plurality of second style vectors are respectively classified into a first set, a second set, and a third set. The first set of each is associated with the shape of the facial image, the second set is associated with the eyes, nose and mouth, and the third set is associated with hair color and skin color.

The face image generator 130 sets the synthesis parameter to zero (0) when combining each first set of the plurality of first style vectors and the plurality of second style vectors so that the final facial image maintains the shape of the original facial image. Let's do it.

Figure 7 shows the results generated by attempting 5000 queries on the deep learning face authentication model 140. X _source is the original user face image, and X _mix (λ= 0) is an intermediate face image similar to the original. X _mix (λ=1) is the target face image.

X _{mix_last} is the final de-identified face image. This is the final facial image de-identified through the method of the present invention. Visually, that is, when viewed with the naked eye, the original face image is de-identified, and the original face is authenticated with respect to the deep learning face authentication model 140.

As can be seen in Figure 7, it can be confirmed that the style of the target image has been injected and sufficient de-identification has been achieved.

모델이름model name	평균 코사인 거리average cosine distance	인증 성공Authentication successful
VGG-FaceVGG-Face	0.36670.3667	100%100%

Table 1 shows the results of facial similarity measurement.

In the case of the face recognition model, two face images are vectorized through a learned model, then the cosine distance (Equation 2) of the two vectors is calculated, and if measured lower than a certain threshold, they are recognized as the same person. In the present invention, the threshold value set for the experiment is 0.4. Authentication performance was calculated by comparing the final result image and the original unknown cosine distance.

When all cosine distances are less than the set threshold of 0.4, the authentication success rate becomes 100%, so that the image generated by the de-identified face image generation method capable of face authentication of the present invention is sufficiently de-identified, but the authentication of the face authentication model It was confirmed that performance was maintained.

In the above, preferred embodiments of the present invention have been shown and described, but the present invention is not limited to the specific embodiments described above, and can be used in the technical field to which the invention pertains without departing from the gist of the present invention as claimed in the patent claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

Claims

an encoder for extracting a plurality of first style vectors of the original face image and a plurality of second style vectors of the synthesized target face image;

a style vector synthesizer for generating a plurality of composite style vectors by combining the plurality of first style vectors and the plurality of second style vectors with a plurality of synthesis parameters that adjust a synthesis ratio; and

A face image generator that generates a final de-identified face image of the original face image using the plurality of composite style vectors. A de-identified face image generating device capable of face authentication, including a.
In claim 1,

The face image generator further generates at least one intermediate face image,

A de-identified face image generating device capable of face authentication, further comprising a deep learning-based face authentication model that determines whether the intermediate face image is authenticated as the original face image.
In claim 2,

When the intermediate face image is authenticated as the original face image using the deep learning-based face authentication model, the style vector synthesizer generates a new synthesis style vector using the increased synthesis parameter value, and the face image generator Generate a new intermediate face image using the new composite style vector,

Generation of a new composite style vector using increased synthesis parameter values in the style vector synthesizer and generation of a new intermediate face image in the face image generator are repeated until the new intermediate face image is not authenticated as the original face image. A non-identifiable facial image generating device capable of face authentication.
In claim 3,

The face image generator generates the final face image using a synthesis style vector having a synthesis parameter of a final intermediate face image that is authenticated as the original face image by the deep learning-based face authentication model. A device for generating facial images.
In claim 3,

The range of the synthesis parameter value is 0 to 1,

The face image generator sets some of the synthesis parameters to zero (0) so that the final face image maintains the shape of the original face image.
Extracting a plurality of first style vectors of the original face image using an encoder;

extracting a plurality of second style vectors of a target face image synthesized with the original face image using the encoder;

generating a plurality of composite style vectors by combining the plurality of first style vectors and the plurality of second style vectors with a plurality of synthesis parameters that adjust a synthesis ratio using a style vector synthesizer;

generating an intermediate face image using the plurality of composite style vectors using a face image generator;

determining whether the intermediate face image is authenticated as the original face image using a deep learning-based face authentication model;

When the intermediate face image is authenticated as the original face image, the synthesis parameter value is increased, a plurality of new synthesis style vectors are generated using the increased synthesis parameter value, and the new synthesis style vectors are used to generate a plurality of new synthesis style vectors. generating a new intermediate face image; and

The steps of generating a new intermediate face image and determining whether it is authenticated as the original face image are repeated to have synthesis parameters of a final intermediate face image that is authenticated as the original face image by the deep learning-based face authentication model. A method of generating a de-identified face image capable of face authentication, comprising: generating a de-identified final face image of the original face image using a composite style vector.
In claim 6,

The range of the synthesis parameter value is 0 to 1,

When the synthesis parameter value is 0, the intermediate face image is the same as the original face image. A method of generating a de-identified face image capable of face authentication.
In claim 7,

In the face image generator, each of the plurality of first style vectors and the plurality of second style vectors is classified into a first set, a second set, and a third set,

A first set of each of the plurality of first style vectors and the plurality of second style vectors is associated with a shape of the face image, a second set is associated with eyes, nose and mouth, a third set is associated with hair color and A method for generating de-identified facial images that are related to skin color and enable face authentication.
In claim 6,

The face image generator sets a synthesis parameter to zero (0) when combining each first set of the plurality of first style vectors and the plurality of second style vectors so that the final facial image is in the form of the original facial image. A method of generating a de-identified face image capable of face authentication that maintains .