KR20230030251A - Privacy preserved monitoring system based on re-identification of anonymized individual face - Google Patents

Abstract

Provided is a surveillance or observation system for personal information protection and personal identification that can store an anonymized image by synthesizing or converting the face of an observer into a third face through artificial intelligence technology rather than storing an original image in a process of obtaining an observation image, extract identifier feature information that contains the features of the face of the observer from the original image before the anonymization of the observer and cannot be restored to the original face, and re-identify the anonymized identity of the observer when necessary through a method of simultaneously storing the anonymized image. An image-based monitoring device includes: a processor; and a communication interface.

Description

Surveillance or observation system that provides personal information protection based on face information anonymization and re-identification

The technical field to which the present invention belongs relates to a surveillance or observation system that provides personal information protection and personal identification functions by applying face information anonymization and re-recognition technologies.

Existing camera-based surveillance or monitoring systems continuously photograph an individual's face with a camera and store or transmit the image as it is, so there is a problem in that information on the individual's face of the subject is exposed. Various methods have been developed to de-identify individual facial information by blurring or mosaicking facial parts when collecting a database to be used in machine learning, but these methods lose a lot of information such as expression information expressed on an individual's face. do. With the development of artificial intelligence technology, the adversarial generative network (GAN) technology that creates a new face by synthesizing the faces of different people has developed, and face swap technology that changes one person's face image to another face This has allowed the development of deep fake technology.

Korea Patent Registration No. 10-1911900 (2018.10.19)

Embodiments of the present invention exist while preserving the facial expression information of the monitoring subject through face swap or deep fake artificial intelligence technology in the image acquired by the monitoring or surveillance camera. The face information of the subject to be monitored is stored as an anonymized image by synthesizing or converting to a third virtual face that does not exist, and the characteristics of the subject's face are embedded from the original image of the subject to be observed. The feature information to be extracted and stored separately, and if the identity of the anonymized object of observation is to be identified, the main purpose is to re-identify the object of observation from the feature information and anonymized image information of the face.

Other non-specified objects of the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

According to an aspect of the present embodiment, in the image-based monitoring device including personal information protection and personal identification functions, a first anonymized face is generated by synthesizing an original face and a selected source face, and identifier characteristic information of the original face and a processor generating an identifier feature information pair by matching the identifier feature information of the first anonymized face; and a communication interface for transmitting the identifier characteristic information pair to a personal information protection and personal identification server.

The processor may obtain an observation image of a subject to be observed through an imaging device, detect the original face from the observation image, and extract the identifier feature information from the detected original face.

The imaging device may be installed in a self-driving vehicle or a general vehicle, or in a place where monitoring is required.

The processor determines (i) periodic time per frame, (ii) one-time time after boarding, (iii) when a specific action or a specific situation occurs, (iv) when a specific facial expression is detected or the reference size of the face area is determined. At a satisfied time point or a combination time point, the original face may be detected and identifier feature information may be extracted from the detected original face.

The processor compares a plurality of identifier feature information extracted from a plurality of face sources according to gender, age, race, or combined attributes of the original face with identifier feature information of the original face, and selects a suitable source face according to similarity. can

The source face may include a face of a non-existent person virtually created through artificial intelligence.

The processor may generate the first anonymized face using a Generative Adversarial Network (GAN) model having a generator and discriminator.

The processor may generate an anonymized image by outputting the first anonymized face at a corresponding position in the observation image.

The communication interface may transmit the anonymized image to the personal information protection and personal identification server.

According to another aspect of this embodiment, in the personal information protection and personal identification server, a first anonymized image including a first anonymized face is received, and the identifier characteristic information of the original face and the identifier characteristic information of the first anonymized face are matched. a communication interface for receiving the identified identifier characteristic information pair; a storage medium for storing the first anonymized image and the identifier characteristic information pair; and a processor for transmitting a signal to the communication interface and the storage medium.

The communication interface may receive a second anonymized image including a second anonymized face in a situation where identification of an observer is required.

The processor extracts a second anonymized face from the second anonymized image, extracts identifier characteristic information of the first anonymized face from the second anonymized face, and identifies characteristic information of the second anonymized face and the first anonymized face. The identifier characteristic information of the face may be compared, and the identifier characteristic information of the second likely anonymized face may be output according to the degree of similarity.

The processor may detect the identifier characteristic information of the original face that matches the identifier characteristic information of the likely first anonymized face.

The communication interface may receive a picture or video of the suspect.

The processor may extract identifier characteristic information of a candidate face from a photo or video of the suspect, compare the identifier characteristic information of the candidate face with the detected identifier characteristic information of the original face, and output a similarity.

As described above, according to the embodiments of the present invention, in the process of acquiring the observation camera image, the face of the observer (or the subject of observation) is changed to a third face that does not exist through artificial intelligence technology rather than storing the original image. Anonymized images are stored by synthesis or conversion, and anonymized in necessary situations by extracting identifier information that contains the characteristics of the observer's face from the original image before anonymization of the observer and extracts identifier information that cannot be restored to the original face and stores it together with the anonymized image. It has the effect of identifying the identity of the observed observer.

Even if the effects are not explicitly mentioned here, the effects described in the following specification expected by the technical features of the present invention and their provisional effects are treated as described in the specification of the present invention.

Although the embodiment of the present invention suggests an internal camera of a vehicle, the present invention is not limited to an in-cabin monitoring camera of a vehicle, and can also be used for anonymization and re-identification of an observer's facial image of a general observation or surveillance camera. Applicable.

1 is a diagram illustrating an abnormal or unexpected situation occurring inside a vehicle.
2 is a diagram illustrating the dilemma between privacy and personal identification.
3 is a block diagram illustrating an observer monitoring system according to embodiments of the present invention.
4 is a diagram illustrating a device implementing a video-based monitoring device and a personal information protection and personal identification server according to embodiments of the present invention.
5 is a flowchart illustrating an operation of an image-based monitoring device according to embodiments of the present invention.
6 is a flowchart illustrating the operation of a personal information protection and personal identification server according to embodiments of the present invention.
7 is a diagram illustrating an identifier characteristic information matching operation of an observer monitoring system according to embodiments of the present invention.
8 is a diagram illustrating an operation of an observer monitoring system according to embodiments of the present invention.

Hereinafter, in the description of the present invention, if it is determined that the subject matter of the present invention may be unnecessarily obscured as it is obvious to those skilled in the art for related known functions, the detailed description thereof will be omitted, and some embodiments of the present invention will be described. It will be described in detail through exemplary drawings.

1 is a diagram illustrating an abnormal or unexpected situation occurring inside a vehicle.

Observers of self-driving cars consist of passengers, not drivers. Observers of self-driving cars do not take responsibility for the traffic situation as passengers, so it is important to ensure security and safety not only during operation but also before and after operation. Malicious behaviors or threats may be generated by some observers inside the vehicle. Abnormal situations also require the protection of the observer.

Observers can be protected by monitoring the inside of the car. When monitoring observers, there are things to consider in terms of privacy and security.

2 is a diagram illustrating the dilemma between privacy and personal identification.

First, the monitoring method using the masked face in Case 1 has a problem in that important face information is lost. Next, the monitoring method using the real face in Case 2 has a problem in that personal facial information is exposed as it is. Finally, in Case 3, the conventional monitoring method using the anonymized face has a problem in that it is difficult to identify the individual of the observer after the fact.

An observer monitoring system according to an embodiment of the technology of the present invention, "surveillance or observation system providing personal information protection function based on face information anonymization and re-identification function", performs anonymization operation for personal information protection and feature matching for individual identification. It satisfies both information protection and personal identification requirements.

3 is a block diagram illustrating an observer monitoring system according to embodiments of the present invention.

The observer monitoring system does not store the original image in the process of acquiring the observation image, but synthesizes or converts the face of the observer into a third face through artificial intelligence technology to store anonymized images, and anonymized images in necessary situations. In order to solve the problem of identification of the subject, the feature of the face of the subject is included from the original image before anonymization of the subject, and the identifier feature information that cannot be restored with the original face is extracted and stored together with the anonymized image.

The observer monitoring system includes an image-based monitoring device 10 and a personal information protection and personal identification server 20 .

4 is a diagram illustrating a device for implementing a video-based monitoring device and a personal information protection and personal identification server according to embodiments of the present invention.

The device 110 includes at least one processor 120 , a computer readable storage medium 130 and a communication bus 170 .

The processor 120 may control the optimization device 110 to operate. For example, the processor 120 may execute one or more programs stored in the computer readable storage medium 130 . The one or more programs may include one or more computer executable instructions, which when executed by processor 120 may cause device 110 to perform operations in accordance with an example embodiment. .

Computer-readable storage medium 130 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. Computer executable instructions or program codes, program data and/or other suitable forms of information may also be provided via input/output interface 150 or communication interface 160. The program 140 stored in the computer readable storage medium 130 includes a set of instructions executable by the processor 120 . In one embodiment, computer readable storage medium 130 may include memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, other types of storage media that can be accessed by the optimization device 110 and store desired information, or a suitable combination thereof.

The communication bus 170 interconnects various other components of the optimization device 110, including the processor 120 and the computer readable storage medium 130.

Device 110 may also include one or more input/output interfaces 150 and one or more communication interfaces 160 that provide interfaces for one or more input/output devices. The input/output interface 150 and the communication interface 160 are connected to the communication bus 170 . An input/output device (not shown) may be connected to other components of the optimization device 110 through the input/output interface 150 .

5 is a flowchart illustrating an operation of an image-based monitoring device according to embodiments of the present invention.

In step S210, the processor of the video-based monitoring device observes through the imaging device or detects the original face of one or more observers using a surveillance camera. An observation image of an observer is acquired, an original face is detected on the observation, and identifier feature information is extracted from the detected original face. The imaging device can be installed in an autonomous vehicle or a general vehicle, or in a place where monitoring is required.

(i) at periodic times per frame, (ii) at one time after boarding, (iii) when a specific action or situation occurs, (iv) when a specific expression is detected or the standard size of the face area is satisfied At the time of doing this, or when they are combined, an original face may be detected and identifier feature information may be extracted from the detected original face.

In step S220, the processor of the image-based monitoring device compares a plurality of identifier characteristic information extracted from a plurality of source faces according to gender, age, race, or a combination of these attributes with the identifier characteristic information of the original face, and selects an appropriate one according to the degree of similarity. Select the source face. The source face may include a face of a non-existent, virtually created person through artificial intelligence.

In step S230, the processor of the video-based monitoring device synthesizes the original face and the selected source face to generate a first anonymized face, matches the identifier feature information of the original face and the identifier feature information of the first anonymized face, and matches the identifier feature information pair. generate The communication interface of the video-based monitoring device transmits the identifier characteristic information pair to the personal information protection and personal identification server.

The processor may generate the first anonymized face using a Generative Adversarial Network (GAN) model having a generator and a discriminator.

In step S240, the processor of the image-based monitoring device generates an anonymized image by outputting a first anonymized face at a corresponding position in the observation image. The communication interface of the video-based monitoring device may transmit anonymized video to a personal information protection and personal identification server.

6 is a flowchart illustrating the operation of a personal information protection and personal identification server according to embodiments of the present invention.

In step S310, the communication interface of the personal information protection and personal identification server receives the anonymized image including the first anonymized face, and generates an identifier feature information pair in which the identifier feature information of the original face and the identifier feature information of the first anonymized face are matched. receive The storage medium of the personal information protection and personal identification server stores the first anonymized image and identifier characteristic information pair. The processor transmits signals to the communication interface and the storage medium.

In step S320, the communication interface of the personal information protection and personal identification server receives the anonymized image including the first anonymized face in a situation where identification of the observer is required. The processor of the personal information protection and personal identification server extracts a second anonymized face from the second anonymized image, extracts identifier feature information of the second anonymized face from the second anonymized face, and identifies the identifier feature information of the second anonymized face and the second anonymized face. 1 The identifier characteristic information of the anonymized face is compared, and the identifier characteristic information of the first anonymized face that is influential according to the degree of similarity is output.

In step S330, the processor of the personal information protection and personal identification server detects the identifier characteristic information of the original face matching the identifier characteristic information of the first likely anonymized face.

In step S340, the communication interface of the personal information protection and personal identification server receives the suspect's photo or video. The processor of the personal information protection and personal identification server extracts the identifier characteristic information of the candidate face from the suspect's photo or video, compares the identifier characteristic information of the candidate face with the identifier characteristic information of the detected original face, and outputs a degree of similarity. .

FIG. 7 is a diagram illustrating an identifier characteristic information matching operation of the observed monitoring system according to embodiments of the present invention, and FIG. 8 is a diagram illustrating the operation of the observed monitoring system according to embodiments of the present invention.

An observer monitoring system including an image-based monitoring device and a personal information protection and personal identification server extracts a face from an image. The identifier characteristic information extracted from the face may be implemented in N dimensions. For example, it can be implemented as a 128-dimensional vector. If the vector is less than 128-dimensional, face comparison is difficult, and if it is greater than 128-dimensional, the parameter is unnecessarily increased.

The observer monitoring system applies a facial feature detection model.

The facial feature detection model extracts features and processes the features into data. In the face detection model, a plurality of layers are connected through a network and include a hidden layer. A layer can contain parameters, and the parameters of a layer contain a set of learnable filters. Parameters include weights and/or biases between nodes.

Facial feature detection models can use deep and wide neural networks to maximize extraction. Residual operation that receives an input value and adds the input value and a convolution operation may be applied.

The facial feature detection model applies a triple loss function used to measure errors related to person recognition. The model learns parameters to minimize the loss function. The triple loss function is expressed as Equation 1.

The anonymized anchor image (A) represents a person identified in an unusual situation, the anonymized positive image (P) represents an image of the same person stored in the cloud, and the anonymized negative image (N) represents another observer. Euclidean distance can be applied between data pairs. A margin that minimizes misclassification can be applied.

The similarity of identifier characteristic information can be measured based on the length of a distance. The similarity of the vector space is expressed as Equation 2.

An image R represents a person, an image C represents an image of the same person stored in the cloud, and an image D represents an image of another observer.

The observer monitoring system applies a source face generation model.

The source face generation model generates a face similar to the target face required for exchange. The source face generating model may generate a plurality of source faces according to attributes such as gender, age, race, or a combination thereof, and may generate virtual faces expressing various identifier characteristic information. The source face set corresponds to the identifier feature information set. The source face generation model effectively renders the behavior or original emotion by the observer. Similarity matching between a source face and a target face is useful for mutually identifying similar emotions and behavioral intentions required for monitoring. A face for synthesis may be selected by comparing the identifier characteristic information extracted from the identifier characteristic information set with the identifier characteristic information of the original face.

The observer monitoring system applies a face anonymization model. The model learns parameters to minimize the loss function.

The face anonymization model distinguishes face and hair regions from a target face. Separate the face area and hair area where the location of landmarks on the face is mapped.

)에 대해서 영상 쌍(x,y)의 지각 손실(

)은 수학식 3과 같이 표현된다.ith layer feature map (

), the perceptual loss of the image pair (x,y) (

) is expressed as in Equation 3.

Perceptual loss is used to estimate the error in capturing fine facial details.

)은 수학식 4와 같이 표현된다.Reconstruction loss of image pairs (

) is expressed as in Equation 4.

Reconstruction loss is used to evaluate color inaccuracies on a pixel-by-pixel basis. λ corresponds to a hyperparameter.

)은 수학식 5와 같이 표현된다.pixel loss (

) is expressed as in Equation 5.

The face anonymization model may generate an anonymized face using a Generative Adversarial Network (GAN) model having a generator and discriminator.

)은 수학식 6과 같이 표현된다.Adversarial loss between generator and discriminator (

) is expressed as in Equation 6.

Adversarial loss enhances the generated image to give it a realistic look. E(x,y) is the expected value for every real data instance, and E _x represents the expected value for every random input to the constructor.

)은 수학식 6과 같이 표현된다.production loss (

) is expressed as in Equation 6.

)은 수학식 7과 같이 표현된다.Reference cross entropy loss (

) is expressed as in Equation 7.

For the ith class, it is defined by the validation label t _i and the softmax probability P _i .

)은 수학식 9와 같이 표현된다.area creation loss (

) is expressed as in Equation 9.

)은 수학식 10과 같이 표현된다.The model evaluates and fills incomplete parts based on the face and hair regions of the target image. Loss of inpainting creation (

) is expressed as in Equation 10.

)은 수학식 11과 같이 표현된다.The model blends the fully recreated faces so that the swapped face matches the background environment as the original target face. face blending loss (

) is expressed as in Equation 11.

The observed person monitoring system generates an identification signature corresponding to each observed person. For example, an identification signature pair corresponding to identifier characteristic information for a real face and identifier characteristic information for an anonymized image is generated. After facial anonymization, the video frames are sent to the server via the cloud along with a pair of identifying signatures of the observer.

Algorithm 1 for obtaining identifier characteristic information in abnormal situations is shown in Table 1.

The system obtains an observer monitoring image using a camera or the like.

The system receives the image as an input, exports the features of all passengers in the image as output, and extracts features (IDs) from the face of the input image. A 128-dimensional vector feature can be extracted per detected face.

The system takes an image as input and outputs the facial positions of all passengers in the image as output.

The system receives the acquired image as input and outputs the anonymized image as output. Anonymization is performed through the synthesis of each original face (target face) existing in the original video and the previously stored source face.

Through anonymization, the original face is anonymized into an intermediate face in which the original face and the source face are properly mixed. This may be implemented through adversarial generative neural networks (GANs) or the like.

The system finds the locations of the original faces from the input image

The system extracts the IDs of the original faces. A predetermined number of source faces for anonymization are stored in the device, and these may also be persons virtually created through artificial intelligence. The similarity (or distance) between the ID extracted from the source face and the ID extracted from the original face of the video is measured to find the source face most similar to the original face. The source face most similar to each original face in the video is used to anonymize the original face at each position.

The system transmits the IDs of all passengers extracted from the acquired original video and the acquired anonymized video to the server. At this time, the original ID and the anonymized ID for the same person in the video are paired (personal identification information) and transmitted to the server together with the acquired anonymized video.

Algorithm 2 for finding evidence of a person in an abnormal situation is shown in Table 2.

The system receives and stores the anonymized monitoring video and personal identification information of the corresponding frame of the video from the device.

The system extracts the characteristics of an anonymized individual observer from the anonymized video when an event requiring identification of the observer occurs (ID acquisition). The similarity between the extracted ID and the stored anonymized ID is calculated to find out the most similar anonymized ID. Find out the original ID stored in pairs with the corresponding anonymized ID. The ID is extracted from other photos of the prime suspect in the case and compared with the stored original ID for similarity.

According to the video-based monitoring device and personal information protection and personal identification server according to the present embodiment, personal privacy of others is protected through the anonymized face, and through matching using facial identification feature information of the real face and the anonymized face, related in an abnormal situation. It provides a way to find people.

The image-based monitoring device, personal information protection and personal identification server may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or special-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The image-based monitoring device, personal information protection and personal identification server may be installed in a computing device or server equipped with hardware elements in the form of software, hardware, or a combination thereof. A computing device or server includes all or part of a communication device such as a communication modem for communicating with various devices or wired/wireless communication networks, a memory for storing data for executing a program, and a microprocessor for executing calculations and commands by executing a program. It can mean a variety of devices, including

In FIGS. 5 and 6, it is described that each process is sequentially executed, but this is merely an example, and a person skilled in the art will refer to FIGS. 5 and 6 without departing from the essential characteristics of the embodiment of the present invention. Various modifications and variations may be applied by changing the order described, executing one or more processes in parallel, or adding another process.

Operations according to the present embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. Computer readable medium refers to any medium that participates in providing instructions to a processor for execution. A computer readable medium may include program instructions, data files, data structures, or combinations thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. The computer program may be distributed over networked computer systems so that computer readable codes are stored and executed in a distributed manner. Functional programs, codes, and code segments for implementing this embodiment may be easily inferred by programmers in the art to which this embodiment belongs.

These embodiments are for explaining the technical idea of this embodiment, and the scope of the technical idea of this embodiment is not limited by these embodiments. The scope of protection of this embodiment should be interpreted according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

An embodiment of the present invention presents an internal camera of a vehicle, but the present invention is not limited to an in-cabin monitoring camera of a vehicle, and an observation or surveillance camera such as a general closed-circuit TV (CCTV). It can also be applied to the anonymization and re-identification of the face image of the observed person.

Claims (13)

In the image-based monitoring device including personal information protection and personal identification,
a processor generating a first anonymized face by synthesizing or transforming the original face and the selected source face, and generating an identifier feature information pair by matching the identifier characteristic information of the original face and the identifier characteristic information of the first anonymized face; and
A video-based monitoring device comprising a communication interface for transmitting the identifier characteristic information pair to a personal information protection and personal identification server. According to claim 1,
Wherein the processor acquires an observation image of one or more observers captured through an imaging device, detects the original face from the observation image, and extracts identifier characteristic information from the detected original face. According to claim 2,
The imaging device is an image-based monitoring device, characterized in that installed in an autonomous vehicle or a general vehicle, a place where monitoring is required. According to claim 2,
The processor determines (i) periodic time per frame, (ii) one-time time after boarding, (iii) when a specific action or a specific situation occurs, (iv) when a specific facial expression is detected or the reference size of the face area is determined. An image-based monitoring device characterized in that the original face is detected and identifier characteristic information is extracted from the detected original face at a satisfied time point or a combination time point. According to claim 1,
The processor compares a plurality of identifier characteristic information extracted from a plurality of source faces according to gender, age, race, or a combination thereof with identifier characteristic information of the original face, and selects a suitable source face according to similarity. video-based monitoring device. According to claim 1,
The source face is an image-based monitoring device, characterized in that it comprises a face of a virtually created person who does not exist through artificial intelligence. According to claim 1,
Wherein the processor generates the first anonymized face using a Generative Adversarial Network (GAN) model having a generator and a discriminator. According to claim 2,
The processor generates an anonymized image by outputting the first anonymized face at a corresponding position in the observation image. According to claim 8,
The video-based monitoring device, characterized in that the communication interface transmits the anonymized video to the personal information protection and personal identification server. In the personal information protection and personal identification server,
a communication interface for receiving a first anonymized image including a first anonymized face and receiving a pair of identifier feature information in which the identifier feature information of an original face and the identifier feature information of the first anonymized face are matched;
a storage medium for storing the first anonymized image and the identifier characteristic information pair; and
A personal information protection and personal identification server including a processor for transmitting a signal to the communication interface and the storage medium. According to claim 10,
The communication interface receives a second anonymized image including a second anonymized face in a situation where identification of an observer is required,
The processor extracts a second anonymized face from the second anonymized image, extracts identifier characteristic information of the second anonymized face from the second anonymized face, and identifies characteristic information of the second anonymized face and the first anonymized face. A personal information protection and personal identification server, characterized in that for comparing face identifier feature information and outputting likely first anonymized face identifier feature information according to similarity. According to claim 11,
The personal information protection and personal identification server, characterized in that the processor detects the identifier characteristic information of the original face matching the identifier characteristic information of the first likely anonymized face. According to claim 12,
The communication interface receives a picture or video of a suspect,
The processor extracts the identifier characteristic information of the candidate face from the suspect's photo or video, compares the identifier characteristic information of the candidate face with the detected identifier characteristic information of the original face, and outputs a similarity. Protection and Personal Identification Servers.

Images