KR102633279B1 - Apparatus and method for selective deidentification - Google Patents

Abstract

Pertaining to a selective de-identification device and a selective de-identification method, the selective de-identification device according to one embodiment includes a storage unit that stores a face to be compared, extracts at least one face from a first image frame, and extracts Perform de-identification processing on all the extracted faces, create a first set corresponding to some of the extracted faces, compare the faces corresponding to the first set with the comparison target face, and create the first set. It may include a processor that determines exposed faces and non-exposed faces among the faces corresponding to .

Description

APPARATUS AND METHOD FOR SELECTIVE DEIDENTIFICATION}

It relates to a selective de-identification device and method.

With the improvement of Internet services, expansion of Internet networks, popular distribution of smart phones, development of high-speed data transmission technology, development of high-definition video technology, and rapid growth of social network services and social media platforms, individuals can use them more easily and quickly than before. It became possible to produce media, stream or share the produced media widely, and an unspecified number of people could easily access and view media. However, with the mass production and indiscriminate disclosure of such media, inconsiderate exposure of faces or personal information in the media is also rapidly increasing, causing serious problems such as invasion of individual privacy and face theft. In particular, in cases where personal information has been widely disclosed and distributed by the media, the reality is that it is very difficult to restore the original damage and prevent redistribution.

Therefore, in order to prevent exposure of such individuals' sensitive information and invasion of privacy, it is necessary to remove or shield personal information such as faces from media in advance to make them unidentifiable. In the past, for this purpose, a method was used to search for faces in the video and apply de-identification processing methods such as mosaics to the searched faces to make the faces unidentifiable. However, when the image is a video, the face search and de-identification process for each of multiple frames requires a lot of labor, a large amount of computing resources, and/or a long time of image processing, so this method has convenience, accessibility, speed, and efficiency. It had many problems in various aspects such as economic feasibility and efficiency. In addition, many methods focused on minimizing the loss of facial features in order to use facial features as data, excluding features that could identify individuals. On the other hand, there may also be faces of individuals in the video that require intentional exposure, such as creators or actors. Since these faces are natural or scheduled to be exposed to the media, the application of the de-identification process described above should be excluded. However, conventional methods were unable to perform de-identification processing only on faces other than those of individuals that required intentional exposure.

The problem sought to be solved is to provide a selective de-identification device and method that can quickly process images to enable identification of specific objects in the image while consuming less resources and to make other objects other than the specific object unidentifiable. do.

In order to solve the above-mentioned problems, a selective de-identification device and a selective de-identification method are provided.

The selective de-identification device includes a storage unit that stores faces to be compared, extracts at least one face from a first image frame, performs de-identification processing on all extracted faces, and selects some of all extracted faces. It may include a processor that generates a corresponding first set, compares the face corresponding to the first set with the face to be compared, and determines an exposed face and an unexposed face among the faces corresponding to the first set.

The processor assigns identification numbers to all faces extracted from the first image frame, arranges the identification numbers according to a predefined standard to obtain a first arranged identification number, and obtains a first arranged identification number. The first set can be generated by detecting a standard number of identification numbers.

The predefined criteria may include at least one of the size and position of the face within the first image frame.

The reference number may include the number of faces to be compared or the number of faces to be compared plus an extra value.

The processor generates a second set by combining the remaining identification numbers from the arranged identification numbers, extracts at least one face from the second image frame, and performs de-identification processing on all extracted faces, Obtain a second arranged identification number by arranging the identification numbers of all faces extracted from the second image frame according to the predefined standard, compare the second set with the second arranged identification number, and It is also possible to obtain a first rearranged identification number by rearranging the second arranged identification numbers according to whether they are included in the second set.

If the identification number corresponding to the exposed face is present in the first rearranged identification number, the processor may rearrange the first rearranged identification number to obtain a second rearranged identification number.

The processor generates a third set by detecting the reference number of identification numbers from the first rearranged identification number or the second rearranged identification number, and compares the face corresponding to the third set with the target. By comparing faces, an exposed face and an unexposed face may be determined among the faces corresponding to the third set.

The processor restores the face corresponding to the first set, compares the restored face with the comparison target face, or performs de-identification processing on the comparison target face, and compares the face of the first set. An exposed face may be determined from among the faces in the first set by comparing it with the identified comparison target face.

The processor may generate a result image so that the exposed face can be identified and the non-exposed face cannot be identified.

When the faces of the first set are restored, the processor further performs de-identification on unexposed faces among the faces of the first set, or performs de-identification on all faces of the first set. And by further performing identification on the exposed face, an image frame is generated so that the exposed face can be identified and the non-exposed face cannot be identified, or when de-identification processing is performed on the face to be compared, the By performing restoration on an exposed face among the faces of the first set, an image frame may be generated so that the exposed face can be identified and the non-exposed face cannot be identified.

The selective de-identification method includes obtaining a face to be compared, extracting at least one face from a first image frame, performing de-identification processing on all extracted faces, and some of all extracted faces. Generating a first set corresponding to the first set, comparing the face corresponding to the first set with the face to be compared, and determining an exposed face and an unexposed face among the faces corresponding to the first set. You can.

The step of generating a first set corresponding to some of all the extracted faces includes assigning identification numbers to all the faces extracted from the first image frame, and arranging the identification numbers according to a predefined standard to create a first set. 1. It may include obtaining an array of identification numbers and generating the first set by detecting a standard number of identification numbers from the first array of identification numbers.

The predefined criteria may include at least one of the size and position of a face within the first image frame, and the number of criteria may be the number of faces to be compared or the number of faces to be compared and an extra value. It can include the added value.

The selective de-identification method includes generating a second set by combining the remaining identification numbers from the arranged identification numbers, extracting at least one face from the second image frame, and de-identifying all extracted faces. performing, arranging the identification numbers of each face extracted from the second image frame according to the predefined criteria to obtain a second arranged identification number, identifying the second set and the second arranged identification number. It may further include comparing numbers and obtaining a first rearranged identification number by rearranging the second arranged identification numbers according to whether they are included in the second set among the second arranged identification numbers. there is.

The selective de-identification method is, if an identification number corresponding to the exposed face is present in the first rearranged identification number, rearrange the first rearranged identification number to obtain a second rearranged identification number. It is also possible to include more steps.

The selective de-identification method includes generating a third set by detecting the reference number of identification numbers from the first rearranged identification number or the second rearranged identification number, and the face corresponding to the third set. It may further include comparing the face to be compared and determining an exposed face and an unexposed face among the faces corresponding to the third set.

Comparing the face corresponding to the first set and the face to be compared includes restoring the face corresponding to the first set, comparing the restored face to the face to be compared, and comparing the face to be compared to the face to be compared. It may also include at least one of performing de-identification processing and comparing the first set of faces with de-identified comparison target faces.

The selective de-identification method may further include generating a result image so that the exposed face can be identified and the non-exposed face cannot be identified.

The step of generating a result image so that the exposed face can be identified and the non-exposed face cannot be identified includes further de-identifying the non-exposed face among the faces in the first set when the first set of faces is restored. When the faces of the first set are restored, de-identification is performed on all faces of the first set and further identification is performed on the exposed faces, so that the exposed faces can be identified. generating an image frame so that the non-exposed face cannot be identified; and when de-identification processing is performed on the face to be compared, performing restoration on the exposed face among the faces of the first set, thereby identifying the exposed face. It may also include at least one of generating an image frame so that the unexposed face cannot be identified.

According to the above-described selective de-identification device and method, a specific object(s) among a plurality of objects in an image can be identified within the image, while objects other than the specific object can be made unidentifiable.

According to the above-described selective de-identification device and method, not only can selective identification and de-identification be performed by consuming less resources, but also the processing time is shortened to enable more rapid selective identification and de-identification. You can also get the effect of making it possible to practice fire.

According to the above-described selective de-identification device and method, it is possible to achieve the effect of more appropriately protecting the personal information and privacy of others by preventing others other than specific people from being exposed in the video.

1 is a block diagram of one embodiment of a selective de-identification device.
Figure 2 is a diagram of an example of a target image.
Figure 3 is a block diagram of an embodiment of the de-identification processing unit.
Figure 4 is a diagram of an example of an image on which de-identification processing has been performed as a whole.
Figure 5 is a diagram for explaining an example of the operation of the de-identification processing unit.
Figure 6 is a diagram of an example of an image on which identification processing has been performed.
Figure 7 is a diagram of an example of an image on which additional identification processing has been performed.
Figure 8 is a first flow diagram of one embodiment of a selective de-identification method.
Figure 9 is a second flow diagram of one embodiment of a selective de-identification method.
Figure 10 is a third flowchart of an embodiment of a selective de-identification method.

Throughout the specification below, the same reference signs refer to the same components unless there are special circumstances. Terms with the addition of 'unit' used below may be implemented as software and/or hardware, and depending on the embodiment, one 'unit' may be implemented as one physical or logical part, or a plurality of 'units' may be implemented. It is also possible to be implemented with one physical or logical part, or one 'part' to be implemented with a plurality of physical or logical parts. When a part is said to be connected to another part throughout the specification, this may mean that the part and the other part are physically connected to each other and/or electrically connected. In addition, when a part includes another part, this does not mean excluding another part other than the other part unless specifically stated to the contrary, and means that another part may be included depending on the designer's choice. do. Expressions such as the first to Nth (N is a natural number greater than or equal to 1) are intended to distinguish at least one part(s) from other part(s), and do not necessarily mean that they are sequential unless otherwise specified. Additionally, singular expressions may include plural expressions, unless the context clearly makes an exception.

Hereinafter, a selective de-identification device according to an embodiment will be described with reference to FIGS. 1 to 7.

FIG. 1 is a block diagram of an embodiment of a selective de-identification device, and FIG. 2 is a diagram of an example of a target image.

As shown in FIG. 1, the selective de-identification device 100 according to an embodiment may include an input unit 101, an output unit 103, a storage unit 105, and a processor 110. At least two of the input unit 101, output unit 103, storage unit 105, and processor 110 are provided to transmit data or commands one-way or two-way through a cable, circuit line, or wireless communication network. do. If necessary, at least one of the input unit 101, output unit 103, and storage unit 105 may be omitted.

The input unit 101 may acquire an image 60 (hereinafter referred to as a target image) that is the subject of selective de-identification and transmit the acquired target image 60 to the storage unit 105 or the processor 110. Here, the target image 60 may include at least one of a still image and a moving image. If the target image 60 is a moving image, the target image 60 may include two or more image frames (60-(t-1 in FIG. 3). ) to 60-(t+1), etc. Here, t may include a natural number of 1 or more. Depending on the situation, at least one person's face 70 may appear in the target image 60, as shown in FIG. 2. The face 70 of at least one person includes a face to be processed to be identifiable (71: 71a, 71b, hereinafter exposed face) and a face to be processed to be unidentifiable (72: 72a, 72b, 72c, hereinafter unexposed face). It can be included. Here, the exposed face (71: 71a, 71b) refers to the face of a person (hereinafter referred to as the exposed person) who needs or can be intentionally exposed to the outside, such as a creator, actor, or host, and the non-exposed face (72: 72a) , 72b, 72c) may refer to the face of a person who will not be exposed to the outside for the protection of personal information (hereinafter referred to as a non-exposed person), but is not limited to this. Depending on the choice of the designer or user, the exposed face (71: 71a, 71b) and the non-exposed face (72: 72a, 72b, 72c) may be defined differently. Here, the user may include a person who has requested selective de-identification processing, and may include, for example, a video producer, video intermediary, or video streaming service provider.

Additionally, the input unit 101 may receive an image 50 (hereinafter referred to as a face image) containing a face to be compared. The comparison target face is used as a comparison object to determine whether one or more faces 70 in the target image 60 are exposed faces 71: 71a, 71b or non-exposed faces 72: 72a, 72b, 72c. It means face. Depending on the embodiment, the face to be compared may include the exposed person's face (71: 71a, 71b). In the face image 50, there may be one face to be compared or multiple faces to be compared. If only one face to be compared exists in the face image 50, the face image 50 may be input for each face to be compared. In this case, when there are multiple faces to be compared, multiple face images 50 may be input accordingly. If necessary, it is possible for multiple face images 50 to be input for one face to be compared. The face to be compared may be fully exposed within the face image 50 for more accurate determination of the exposed face 71 (71a, 71b), and may be photographed approximately from the front depending on the situation. Input of the face image 50 may be performed before or simultaneously with selective de-identification processing. The input face image 50 may be transmitted to the processor 110, and, if necessary, may be transmitted and stored in the storage unit 105 and provided to the processor 110 according to a call from the processor 110.

In addition, the input unit 101 may receive instructions regarding the operation of the selective de-identification device 100 from a user, etc., and a program (referred to as an app, application, or software) related to the operation of the selective de-identification device 100. (possible) can also be entered.

The input unit 101 may be provided integrally with the selective de-identification device 100, or may be provided to be physically separable. The input unit 101 includes, for example, a keyboard, mouse, tablet, touch screen, touch pad, track ball, track pad, scanner device, image capture module, ultrasonic scanner, motion detection sensor, vibration sensor, light receiving sensor, and pressure sensor. , it may include a proximity sensor, a microphone data input/output terminal, or a communication module (for example, a LAN card, a short-range communication module, or a mobile communication module, etc.).

The output unit 103 may output the resultant image (eg, 63 in FIGS. 3 and 7 ) obtained according to the operation of the processor 110 to the outside. Here, the result image 63 is an image generated corresponding to the target image 60 and may be a still image or a moving image. In the resulting image 63, the exposed person's face (71: 71a, 71b) is revealed to be identifiable, and the other non-exposed person's face (72: 72a, 72b, 72C) is blocked so as not to be identified. The output unit 103 may include, for example, a display, a printer device, a speaker device, an image output terminal, a data input/output terminal, or a communication module.

The storage unit 105 may temporarily or non-temporarily store at least one data or program related to the operation of the selective de-identification device 100. Here, the at least one data may include a face image 50 or a target image 60 input through the input unit 101, and may be data temporarily generated during the processing of the processor 110 or the processor 110. ) It may also include an image 63 according to the processing results. Additionally, the storage unit 105 may store a face database 90 built based on the exposed person's face (71: 71a, 71b) extracted from the face image 50. The program may include an algorithm for the operation of the processor 110, and may be written or modified directly by a designer such as a programmer and then stored in the storage unit 105, or may be stored in the storage unit 105, or may be stored in another physical recording medium (external memory). It may be received and stored from a device (compact disk (CD), etc.), or it may be acquired or updated through an electronic software distribution network accessible through a wired or wireless communication network. The storage unit 105 may include, for example, at least one of a main memory and an auxiliary memory. These can be implemented using semiconductors or magnetic disks.

The processor 110 may perform selective de-identification processing on the target image 60 and transmit the resulting image 63 resulting from the selective de-identification to the storage unit 105 or the output unit 103. Selective de-identification processing for the target image 60 may be performed by executing a program stored in the storage unit 105. The processor 110 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), and a microcontroller unit (MCU). ), an electronic control unit (ECU), and/or at least one electronic device capable of performing various calculations and control processes. These processing or control devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination. Depending on the embodiment, the processor 110 may be implemented by combining different processing units, for example, a central processing unit and a graphics processing unit.

According to one embodiment, the processor 110 may include a face acquisition unit 111, a face processing unit 120, and a de-identification processing unit 130. At least two of the face acquisition unit 111, the face processing unit 120, and the de-identification processing unit 130 may be logically or physically distinct from each other. When logically separated, at least two of them can be implemented using one processing unit, and when physically separated, at least two of them can be implemented using different processing units (e.g., a central processing unit and a graphics processing unit). ) can also be implemented using. For the convenience of explanation below, the target image 60 is a video, and the face processing unit 120 and the de-identification processing unit 130 extract an image frame 60-k from the target image 60, where k is 0 or 1 or more. A natural number (which may include t) will be described by way of an example, but this is an example and the target image 60 may be a still image as described above.

When the face image 50 is input, the face acquisition unit 111 may obtain a face to be compared from the face image 50. For example, the face acquisition unit 111 detects one or more faces to be compared for at least one exposed person from the face image 50, and then uses the detected at least one face to be compared to the face database 90. The existing face database 90 may be updated by creating a new face or adding a detected face to be compared. In addition, after detecting a face to be compared, the face acquisition unit 111 transmits at least one detected face to be compared to the de-identification processing unit 130 in real time or before processing, so that the de-identification processing unit 130 exposes the face to the target. It is also possible to identify the person's face (71: 71a, 71b). In this case, the face to be compared may be temporarily or non-temporarily stored in the storage unit 105 before being delivered to the de-identification processing unit 130.

According to one embodiment, the face acquisition unit 111 may use a predetermined learning algorithm to detect a face to be compared from the face image 50. Predetermined learning algorithms include previously learned algorithms for face recognition and detection, for example, deep neural network (DNN), convolutional neural network (CNN), and recurrent neural network (RNN). Such as Recurrent Neural Network (CRNN), Convolutional Recurrent Neural Network (CRNN), Deep Belief Network (DBN), Long Short Term Memory (LSTM), or Multi-layer Perceptron. It can be built based on at least one learning model.

When a target image 60 to perform de-identification processing on an unexposed person 72 (72a, 72b, 72c) is input, the face processing unit 120 selects at least one face 70 from the target image 60, that is, At least one of the exposed person's face (71: 71a, 71b) and the non-exposed person's face (72: 72a, 72b, 72c) can be obtained. In one embodiment, the face processing unit 120 may include a pre-processing unit 121, a face extracting unit 123, and a face tracking unit 125.

The preprocessor 121 may perform preprocessing on the target image 60 so that the operations of the face extraction unit 123, face tracking unit 125, and/or de-identification processing unit 130 can be performed effectively. there is. For example, the preprocessor 121 may extract all or part of the image frames 60-k from the target image 60, which is a video. Additionally, the preprocessor 121 may perform correction on the target image 60 or at least one image frame 60-k obtained therefrom. Specifically, for example, the preprocessor 121 crops the surroundings of at least one image frame 60-k to reduce the size of the image frame 60-k, or reduces the size of the image frame 60-k. ), enlarge the entire video frame (60-k), or perform compression conversion on the video frame (60-k) to reduce the data capacity of the video frame (60-k), and/ Alternatively, the brightness or contrast of at least one image frame 60-k may be adjusted. The target image 60 pre-processed by the pre-processing unit 121 may be transmitted to at least one of the face extraction unit 123 and the face tracking unit 125. The pre-processing unit 121 may be omitted depending on the embodiment.

The face extractor 123 may obtain human faces 71 and 72 by extracting them from each image frame 60-k. The face extraction unit 123 may use a previously trained learning algorithm to extract faces. Here, the learning algorithm may be obtained by training a deep learning-based target detection model, such as YOLOv5. Additionally, when the faces 71 and 72 are extracted, the face extraction unit 123 may generate at least one block (box) corresponding to the faces 71 and 72. Here, at least one block corresponding to the corresponding faces 71 and 72 may have the shape of a predetermined shape, such as a square or circle, and the corresponding faces 71 and 72 may be arranged inside the predetermined shape. .

The face tracking unit 125 generates an identification number (ID: Identification number, for example, 70-1 to 70- in FIG. 5) for the human face 71 or 72 acquired by the face extraction unit 123 or a block corresponding thereto. 3) can be assigned to enable tracking of the corresponding faces 71 and 72. Here, the identification numbers (70-1 to 70-3) can be defined in various ways depending on the user or designer. For example, the face tracker 125 may assign one or more different natural numbers as identification numbers 70-1 to 70-3 to each of at least one face 71 or 72. In this case, for each frame (60-k, 60-(k+1), 60-( Identification numbers (70-1 to 70-3) are assigned to all or some of the faces (70: 71, 72) that appear in (k+2), etc.), but different frames (60-t, 60-(t+) 1), 60-(t+2), etc.), the same identification number (e.g., [9], etc.) is assigned to blocks corresponding to the same face (71a, etc.) or the same face (71, 72), and different Different identification numbers are assigned to faces or blocks corresponding thereto. More specifically, for example, if the face tracker 125 designates the value of [9] as the identification number for the specific face 71a within the specific image frame 60-k, the other The value of [9] can also be designated as an identification number for the same face 71a within the video frame (60-(k+1), etc.). At least one face (71, 72) may change its position and/or size or disappear for each image frame (60-k), and in this way, the same face (71) in each image frame (60-k) , 72) or the corresponding block is given the same identification number (70-1 to 70-3), the exposed person's face 71 or the non-exposed person's face 72 is tracked even in different image frames (60-k). It becomes possible. The face tracking unit 125 can obtain tracking results for the same faces 71 and 72 by connecting and combining the faces 71 and 72 with the same identification number within each image frame 60-t. . The tracking results of the faces 71 and 72 may be prepared in the form of trajectories for the corresponding faces 71 and 72. According to one embodiment, the face tracking unit 125 may be implemented using an algorithm such as DeepSort. Deepsort is a deep learning-based real-time multi-target tracking video framework that can prevent multiple trajectories from being created for one target due to occlusion or disappearance of the target.

The de-identification processing unit 130 receives the processing results of the face processing unit 120 (e.g., a face extracted from the target image 60 and a tracking result thereof, etc.), and based on this, the The exposed face (71: 71a, 71b) and the non-exposed face (72: 72a, 72b, 72c) are determined, the exposed face (71: 71a, 71b) is made identifiable, and the non-exposed face (72: 72a, 72b, In 72c), the resulting image 63 corresponding to the target image 60 can be obtained by making it unidentifiable.

FIG. 3 is a block diagram of an embodiment of a de-identification processing unit, and FIG. 4 is a diagram of an embodiment of an image on which de-identification processing has been performed as a whole. Figure 5 is a diagram for explaining an example of the operation of the de-identification processing unit.

According to one embodiment, the de-identification processing unit 130 includes a full de-identification processing unit 131, an arrangement unit 132, a set classification unit 133, and an identification unit 134, as shown in FIG. 3. ), a partial de-identification processing unit 135, and a rearrangement unit 136. Depending on the embodiment, at least one of these (131 to 136) may be omitted.

The entire de-identification processing unit 131 receives one or more image frames (for example, the tth image frame (60-t) to the (t+2)th image frame (60-(t+2)) from the face processing unit 120. ) simultaneously or sequentially, and after receiving information about the faces 71 and 72 from the face extraction unit 131 or the face tracking unit 125, the corresponding image frame, for example, the t image frame 60- De-identification processing is performed on all faces (71, 72) in t). That is, both the exposed face 71 and the non-exposed face 72 are processed to be unidentifiable. Here, the de-identification processing may include, for example, at least one of blurring processing, mosaic application, pixel rate filtering, and inpainting processing (removal and transformation of the faces 71 and 72, etc.). , but is not limited to this, and the user or designer may adopt various de-identification processing methods depending on their choice. By the above-described de-identification process, a new image frame 61-t (hereinafter referred to as the entire de-identification image frame) including the de-identified faces 74 and 75, as shown in FIG. 4, can be obtained. there is. Within the entire de-identified image frame (61-t), there are faces (74: 74a, 74b) corresponding to the exposed face (71) of the existing t image frame (60-t) and a face (74: 74b) corresponding to the non-exposed face (72). The faces (75: 75a, 75b, 75c) are all expressed indistinguishable. Depending on the embodiment, one or more image frames (60-t to 60-(t+2)) may be transmitted from the preprocessor 121.

After de-identification processing is performed on all faces 71 and 72 of a specific image frame, for example, the t image frame 60-t, before or at the same time, the array unit 132 is shown in FIG. 5. As described above, the identification numbers 70-1 for all the faces 71 and 72 of the t image frame 60-t can be arranged according to a predetermined standard. Specifically, the array unit 132 has an identification number (70-1) corresponding to each of the faces (71, 72) in the t image frame (60-t), for example [1, 2, 3, 4, 5, 6. , 7, 8, 9, 10]) from the face processing unit 120, and within the size or image frame 60-t of the faces 71 and 72 corresponding to each identification number 70-1. By arranging the identification number (70-1) according to the position, the newly arranged identification number (70-1r, for example [3, 9, 2, 1, 6, 8, 4, 7, 10, 5], below A first arranged identification number) can be obtained. For example, in the first arranged identification numbers 70-1r, the faces 71a and 72b are relatively large or are located at or around the center of the image frame 60-t, the more the faces 71a and 72b are identified. The identification numbers for the faces (71b, 72a, 72c) are arranged at the front, and the smaller the face (71b, 72a, 72c) is or located at or near the border of the image frame (60-t), the identification numbers for the corresponding faces (71b, 72a, 72c) are arranged at the back. do.

The set classification unit 133 may classify the identification number 70-1r into at least two sets using the first arranged identification number 70-1r. For example, the set classification unit 133 includes an identification number (e.g., [3]) from the frontmost identification number (e.g., [3]) in the first arranged identification number (70-1r) to the reference number (e.g., 3). [2]) is obtained, the obtained identification numbers ([3, 9, 2]) are combined to generate the first set (81-1), and the remaining identification numbers (for example, [1, 6]) are obtained. , 8, 4, 7, 10, 5]) can be combined to create the second set (82-1). That is, the first set 81-1 corresponds to some faces (for example, faces corresponding to identification numbers ([3, 9, 2]) among all the faces extracted from the image frame 60-t, The second set 82-1 corresponds to other remaining faces (e.g., faces corresponding to remaining identification numbers (e.g., [1, 6, 8, 4, 7, 10, 5]). Here, the reference number may be the total number of comparison target faces acquired by the exposed face acquisition unit 111 and/or stored in the face database 90 (for example, 3), or in case the exposed face is misidentified. It may be a value (e.g., 4=3+1) that is the sum of the acquired or stored number of faces to be compared plus a predefined extra value (e.g., 1, etc.). The reference number may be input by a user or the like through the input unit 101 before or during the selective de-identification process. Information about the generated first set (81-1) is transmitted to the identification unit 134, and information about the second set (82-1) is transmitted to the rearrangement unit 136.

The identification unit 134 obtains an identification number (for example, [3, 9, 2]) belonging to the first set 81-1, and identifies the number corresponding to the received identification number ([3, 9, 2]). The face 70 can be identified.

Specifically, for example, the identification unit 134 first identifies face(s) (for example, 74a, 75a, 75b) corresponding to the identification numbers ([3, 9, 2]) belonging to the first set (81-1). etc.) are compared with each other and the comparison target face(s) stored in the face database 90. The face(s) (74a, 75a, 75b) corresponding to the identification numbers ([3, 9, 2]) belonging to the first set (81-1) are de-identified by the entire de-identification processing unit 131. is being carried out. Accordingly, the identification unit 134, in one embodiment, includes face(s) 74a, 75a, 75b corresponding to identification numbers ([3, 9, 2]) belonging to the first set 81-1. is first restored to enable identification, and the face(s) that have become identifiable (for example, 71a, 72a, 72b) and the face(s) to be compared can be compared. According to another embodiment, the identification unit 134 first performs de-identification processing on the face(s) to be compared in the same manner as the entire de-identification processing unit 131, and then performs de-identification processing on the face(s) to be compared. ) and the face(s) (74a, 75a, 75b) corresponding to the identification numbers ([3, 9, 2]) belonging to the first set (81-1) may be compared.

If the face (e.g., 74a) corresponding to at least one identification number (71-1, e.g., [9]) is the same or similar to the comparison target face stored in the face database 90 and corresponds to each other, the identification unit 134 ) determines the face 74a corresponding to the corresponding identification number 71-1 as the face to be identified (i.e., exposed face). Conversely, if the face (e.g., 75b) for at least one identification number (72-1, e.g., [3, 2]) is different from the comparison target face stored in the face database 90 and does not correspond to each other, the identification unit 134 ) determines that the face 75b for the corresponding identification number 72-1 is a face subject to de-identification (i.e., an unexposed face). Depending on the embodiment, the identification unit 134 may use the comparison target face acquired by the above-described exposure face acquisition unit 111 and delivered in advance or in real time, instead of the comparison target face stored in the face database 90. . The identification number (71-1, for example [9]) determined as an exposed face by the identification unit 134 is stored in the storage unit 105, for example, in the form of a main face set 95. It can be. The main face set 95 may be transmitted to the rearrangement unit 136.

Figure 6 is a diagram of an example of an image on which identification processing has been performed.

The partial de-identification processing unit 135 exposes the face (for example, 74a) determined to be an exposed face by the identification unit 134 in the entire de-identified image frame 61-t, and displays a non-exposed face that is not an exposed face. Faces determined to be faces (e.g., 74b, 75a, 75b, 75c) are processed to block their exposure.

According to one embodiment, the partial de-identification processing unit 135 determines the face 74a to be restored using the identification number 71-1, [9], and based on this, determines the face 74a to be restored. Identification and/or de-identification of the non-exposed face 72 may be performed. If the identification unit 143 first performs restoration on the face(s) (74a, 75a, 75b) corresponding to the identification numbers ([3, 9, 2]) belonging to the first set (81-1), If comparison between faces is performed, the partial de-identification processing unit 135 performs de-identification processing on the faces 75a and 75b determined to be non-exposed faces, and the face 74a determined to be exposed is maintained and exposed. Faces 75a and 75b other than the face 74a determined to be a face can be made indistinguishable. In this case, the partial de-identification processing unit 135 de-identifies all of the face(s) corresponding to the identification numbers ([3, 9, 2]) belonging to the first set 81-1 again, as necessary. It is also possible to perform image processing and perform restoration processing again only on the face 74a determined to be an exposed face to enable identification. On the other hand, if the identification unit 134 performs a comparison between faces after de-identifying the face(s) to be compared, the face 74a determined to be an exposed face is restored to the original image and the face 74a is replaced with the corresponding face 74a. Identification processing may be performed on the faces 75a and 75b, and the remaining faces 75a and 75b may be left as is, making identification impossible.

According to another embodiment, the partial de-identification processing unit 135 may selectively perform identification or de-identification using the main face set 95. That is, when the main face set 95 is created by an operation as described later, the partial de-identification processing unit 135 refers to the main face set 95 and de-identifies the exposed faces 71 and 74 to be identified. After determining at least one of the non-exposed faces 72 and 75 to be identified, identification of the exposed faces 71 and 74 appearing in all or some of the frames 60-k based on the tracking results of the face tracking unit 125. You can also perform . In this case, since the entire de-identification processing unit 131 has performed de-identification processing on all faces 71 and 72, the unexposed faces 72 and 75 remain de-identified.

According to the operation of the partially de-identified processing unit 135, as shown in FIG. 6, a new image frame 62-t (hereinafter referred to as a first partially de-identified image frame) or a combination of these image frames 62-t is performed. As a result, the image 63 can be obtained. In this case, the exposed face 71a corresponding to at least one specific identification number 71-1, [9] in the first partially de-identified image frame 62-t or the resulting image 63 is restored and identified. This is possible, but the rest of the face is still de-identified and cannot be identified. The remaining faces are, for example, faces (75b, for example) corresponding to other identification numbers (72-1, [3, 2]) of the first set (81-1) and those of the second set (82-1). It may include a face (for example, 74b, 75a, 75c) corresponding to an identification number ([1, 6, 8, 4, 7, 10, 5]).

After the generation of the first partially de-identified video frame 62-t, before or simultaneously with the generation, the (t+1)th video frame 60-(t+1) is again generated, as shown in FIG. It may be transmitted to the de-identification processing unit 130. In response, the entire de-identification processing unit 131 may perform de-identification processing as described above on all faces in the (t+1)th image frame 60-(t+1).

Subsequently, the rearrangement unit 136 identifies identification numbers 70-2, for example [2, 3, 4, 5, 7, 8, 9, 10, 11, 12]) are received from the face processing unit 120. In this case, the identification number 70-2 corresponding to at least one face 71 or 72 is the same face 71 of the t-th image frame 60-t according to the tracking result of the face tracker 125. , 72) is given the same as the identification number (70-1). The rearrangement unit 136 arranges the identification numbers 70-2 according to the position or size of the faces 71 and 72 corresponding to each identification number 70-2, so that the identification numbers 70-2 are arranged as shown in FIG. 5. A number (70-2r1, for example [3, 9, 2, 8, 11, 4, 7, 10, 12, 5], hereinafter referred to as a second arranged identification number) can be obtained. For example, as described above, the array unit 132 arranges the faces 71a and 72b further forward as the faces 71a and 72b become larger or closer to the center of the image frame 60-t. You can also obtain an identification number (70-2r1).

Subsequently, the rearrangement unit 136 performs rearrangement on the second arranged identification number 70-2r1 using the second set 82-1 received from the set sorting unit 133 to obtain the first rearranged identification number 70-2r1. An identification number (70-2r2, for example [8, 4, 7, 10, 5, 3, 9, 2, 11, 12]) can be obtained. Specifically, for example, the rearrangement unit 136 compares the second arranged identification number 70-2r1 and the second set 82-1, and selects the second arranged identification number 70-2r1. 2 Obtain the identification numbers (e.g. [8, 4, 7, 10, 5]) also included in the set (82-1), and place them relatively in front to obtain the first rearranged identification number (70-2r2). You can also obtain it.

The rearrangement unit 136 further obtains an identification number (71-1, for example [9]) corresponding to the exposed face by referring to the main face set 95, and uses the first rearranged identification number (70-2r2) By further reflecting this, a second rearranged identification number (70-2r3) may be obtained. Specifically, for example, the rearrangement unit 136 checks whether the identification number ([9]) corresponding to the exposed face is present in the first rearranged identification number (70-2r2), and determines whether the identification number ([9]) corresponding to the exposed face is present in the first rearranged identification number (70-2r2). If ([9]) is present in the first rearranged identification number (70-2r2), the identification number ([9]) corresponding to the exposed face is changed to another identification number (e.g. [8, 4, 7, 10, 5, 3, 2, 11, 12], to obtain a second rearranged identification number (70-2r3) in which the identification number ([9]) corresponding to the exposed face is located at the very front.

According to the above-described process, the second rearranged identification number (70-2r3) is an identification number ([9]) corresponding to the exposed face, and an identification number also included in the second set (82-1) (for example, [8] , 4, 7, 10, 5]) and other identification numbers (for example, [3, 2, 11, 12]).

The second rearranged identification number 70-2r3 is transmitted to the set classification unit 133. Depending on the embodiment, the second rearranged identification number (70-2r3) may not be obtained, and in this case, the first rearranged identification number (70-2r2) may be transmitted to the set classification unit 133. there is.

As described above, the set classification unit 133 may classify the second rearranged identification number 70-2r3 into at least two sets, for example, a third set 81-2 and a fourth set. It can be classified as (82-2). In this case, the set classification unit 133 detects and combines the reference number of identification numbers (for example, [9, 8, 4]) from the front in the second rearranged identification number 70-2r3 to create a third set ( 81-2), and combining the remaining identification numbers (e.g. [7, 10, 5, 3, 2, 11, 12]) of the second rearranged identification number (70-2r3) to create a fourth set ( 82-2) can be generated. The third set (81-2) may be transmitted to the identification unit 134, and the fourth set (82-2) may be transmitted to the rearrangement unit 136.

Figure 7 is a diagram of an example of an image on which additional identification processing has been performed.

The identification unit 134 acquires the identification number (for example, [9, 8, 4]) of the third set 81-2 based on the (t+1)th image frame as described above, and 3 The face corresponding to the identification number ([9, 8, 4]) of the set (81-2) can be identified. In this case, the identification unit 134 selects the comparison target face pre-stored in the face database 90 or the comparison target face acquired by the exposed face acquisition unit 111 with an identification number ([ 9, 8, 4]), and a face identical or similar to the comparison target face is detected among the faces corresponding to the identification numbers ([9, 8, 4]) of the third set (81-2). If so, the face is determined as the exposed face. In this case, depending on the embodiment, comparison between the two may be performed after restoration of the face corresponding to the identification number ([9, 8, 4]) of the third set 81-2 is performed, and/or De-identification of the face to be compared may be performed after further de-identification. If necessary, the identification unit 134 may update the main face set 95 using the identification numbers 71-2, [9, 8] for the exposed faces. In this case, as shown in FIG. 7, a new face (i.e., a face corresponding to identification number [8]) that was not determined as an exposed face in the analysis process of the t image frame 60-t is newly exposed face 71b. ) may be determined. Conversely, if the face (e.g., 75a, 75c, etc.) corresponding to at least one identification number (72-2, e.g., [4]) does not correspond to the previously stored comparison target face, the identification unit 134 generates the corresponding identification number ( The faces (75a, 75c, etc.) corresponding to 72-2) are determined as non-exposed faces.

The partially de-identified processing unit 135 determines the face determined to be an exposed face by the identification unit 134 in the first partially de-identified image frame 62-t (i.e., the face corresponding to the identification number [8] ( Reconstruction may be performed on 74b)) and identification processing may be performed, or de-identification processing may be performed on the face determined to be a non-exposed face (i.e., the face 74b corresponding to identification number [4]). If a face that was previously determined to be an exposed face (i.e., the face 71a corresponding to identification number [9]) has already been restored, identification processing for the face is not performed. As a result, as shown in FIG. 7, another new image frame 62-(t+1), hereinafter referred to as a second partially de-identified image frame 63-t, is obtained. In the second partially de-identified image frame 63-t, the exposed face 71b corresponding to at least one specific identification number 71-2, [8] is restored and identifiable, but the remaining faces are not identifiable. . Here, the remaining faces are, for example, faces that correspond to identification numbers (72-2, [4]) belonging to the third set (81-2) but are dissimilar to the previously stored faces, and faces that are dissimilar to the previously stored faces and the fourth set (82-2). 2) may include a face corresponding to an identification number ([7, 10, 5, 3, 2, 11, 12]).

In addition, when the (t+2)th image frame 60-(t+2) is additionally acquired, as described above, the entire de-identification processing unit 131 processes the (t+2)th image frame 60-(t +2)), de-identification processing is performed on all faces in the rearrangement unit 136, and the rearrangement unit 136 generates an identification number 70 corresponding to each face in the (t+2)th image frame (60-(t+2)). -3, for example [2, 3, 5, 7, 8, 9, 10, 12, 13, 14]), and obtain the faces (71, 72) corresponding to each identification number (70-3). By arranging the identification number (70-3) according to the position or size, the arranged identification number (70-3r1, for example [3, 9, 2, 8, 7, 10, 13, 12, 5, 14], etc. A third arranged identification number) can be obtained. In addition, the rearrangement unit 136 arranges the new second set 82-2 and a common identification number (for example, [7, 10, 5, 3, 2, 12]) relatively in the front, thereby arranging the third array. Obtain a third rearranged identification number (70-3r2, for example [7, 10, 5, 3, 2, 12, 9, 8, 13, 14]) by rearranging the rearranged identification number (70-3r1). And, if necessary, use the newly updated main face set 95 to identify the identification number (71-2, for example [9, 8]) corresponding to the exposed face with the new second set (82-2). The fourth rearranged identification number (70-3r3) may be obtained by placing it further forward than the number (for example, [7, 10, 5, 3, 2, 12]).

The set classification unit 133 may classify the third arranged identification number 70-3r1 or the fourth rearranged identification number 70-3r3 into at least two sets 81-3 and 82-3. . As described above, the set classification unit 133 detects and combines the standard number of identification numbers (for example, [9, 8, 7]) from the fourth rearranged identification number 70-3r3 in order from the front. Generating a fifth set (81-3), as well as the remaining identification numbers (e.g. [7, 10, 5, 3, 2, 12, 13, 14]) of the fourth rearranged identification number (70-3r3) The sixth set (82-3) can be created using . The fifth set (81-3) is transmitted to the identification unit 134, and the identification unit 134 uses the previously obtained comparison target face as described above, and the identification number of the fifth set (81-3) ( Among [9, 8, 7]), the identification number corresponding to the exposed face (71-3, for example [9,8]) is determined, and the identification number corresponding to the non-exposed face (72-3, for example [7]) is determined. ) can be determined.

The partial de-identification processing unit 135 performs restoration and identification of the face determined to be an exposed face using the identification number 71-3 corresponding to the exposed face, or performs de-identification of the face determined to be an unexposed face. Perform identification. If all faces determined to be exposed are identifiable, face restoration may not be performed.

When the above-described process is completed, the partial de-identification processing unit 135 sequentially combines all or part of the image frame(s) (62-t to 62-(t+2)) that have been selectively identified, At least one result image 63 can be generated. In addition, as described above, the partial de-identification processing unit 135, depending on the embodiment, generates the main face set 95 including all or some of the exposed faces 71, and then generates the main face set 95 and Using the tracking results of the face tracking unit 125, identification is performed on each of the exposed faces 71 in all or some of the image frame(s) (62-t to 62-(t+2)), and the result is An image 63 can also be generated. The generated result image 63 may be transmitted to the output unit 103 or the storage unit 105 and provided to users, viewers, etc.

In a video produced by a creator, etc., there may also be faces of individuals such as the creator or an actor that require intentional exposure, and exposure of these faces is natural or scheduled. Therefore, if a creator or actor is registered in advance as an exposed face through the exposed face acquisition unit 111 as described above, identification processing is further performed on these faces in addition to de-identification processing, resulting in de-identification. It has the same effect as if the application of the processing was excluded. Additionally, other people other than these are recognized as non-exposed faces and only de-identification processing is performed so that they do not appear in the resulting image 63. That is, in the obtained result image 63, only specific people are exposed and other people are blocked. Accordingly, unnecessary exposure of people in the video is prevented and his/her privacy is protected.

The above-described selective de-identification device 100 is a specially designed device to perform the above-described processes such as exposed face acquisition, pre-processing, face extraction, face tracking, de-identification processing, set classification, and exposed face identification and restoration. It may be implemented using, or it may be implemented by using one or more information processing devices alone or in combination. Here, one or more information processing devices include, for example, desktop computers, laptop computers, server hardware devices, smart phones, tablet PCs, portable game consoles, navigation devices, head mounted display (HMD) devices, digital devices, etc. Televisions, set-top boxes, digital media player devices, media streaming devices, audio playback devices (artificial intelligence speakers, etc.), closed-circuit television (CCTV), home appliances, manned or unmanned vehicles (e.g. vehicles such as cars, buses, or two-wheelers) , mobile robots, wireless model vehicles, robot vacuum cleaners, etc.), manned or unmanned flying vehicles (e.g. aircraft, helicopters, drones, model airplanes, model helicopters, etc.), medical devices (X-ray imaging devices, computed tomography devices (CT: Computed Tomography or Magnetic Resonance Imaging (MRI) devices, etc.), industrial robots, or industrial machines, but are not limited thereto. Depending on the situation or conditions, designers, users, etc. may consider and employ at least one of various devices for computational processing and control of information as the above-described device 100 in addition to the above-described information processing device.

Hereinafter, an embodiment of the selective de-identification method will be described with reference to FIGS. 8 to 10.

Figure 8 is a first flow diagram of one embodiment of a selective de-identification method.

As shown in FIG. 8, first, an image (i.e., face image) having a face to be compared is acquired, and then the face to be compared can be extracted from the acquired face image (200). Here, the face to be compared includes a face that will be processed to be identifiable within the target image and exposed to the outside. The face to be compared may be detected from the face image using a predetermined learning algorithm. Depending on the embodiment, when a face to be compared is detected from a face image, a face database may be newly created or updated using the face to be compared.

A target image to perform selective de-identification processing is acquired (202). Acquisition of the target image may be performed before, after, or simultaneously with the acquisition of the face image. The target image may include a still image or a moving image. A video consists of multiple frames connected sequentially.

When the target image is a video, a face is extracted from at least one video frame of the target image, and a predetermined identification number is assigned to each extracted face (204). In this case, even if the faces are extracted from different frames, if they are identical, the same identification number is given. Depending on the embodiment, at least one block corresponding to the extracted face may be created. Additionally, when an identification number is assigned, a tracking result for at least one face can be obtained by connecting faces with the same identification number in each video frame. Depending on the embodiment, it is possible to further perform preprocessing on the target image before extracting the face and assigning an identification number.

When at least one image frame constituting the target image is acquired, de-identification processing may be performed on all extracted faces within the at least one image frame (206). In other words, both the exposed face that will be processed to be identifiable in the video and the unexposed face that will be processed to be unidentifiable in the video are processed to be unidentifiable. Here, de-identification processing may include blurring processing, mosaic application, pixel rate filtering, or inpainting processing. Accordingly, the entire de-identified image frame corresponding to at least one image is obtained.

Subsequently, all identification numbers corresponding to each face can be classified into one of at least two sets (208). Set classification can be performed using predetermined criteria. For example, identification numbers corresponding to relatively large faces and/or identification numbers corresponding to faces located relatively at or around the center within an image frame are classified into one set (e.g., a first set), Other identification numbers are classified into another set (for example, a second set).

The face corresponding to the identification number belonging to the first set and the face to be compared are compared, and the exposed face may be determined according to the comparison result (210). Specifically, among the faces corresponding to the identification number belonging to the first set, a face that is identical or similar to the face to be compared may be determined as an exposed face, and a face that is not may be determined as an unexposed face.

Once the exposed face is determined, identification processing is performed on the exposed face, and the non-exposed face is maintained as is (212). If identification processing has been performed on the exposed face and the non-exposed face in the exposed face determination process 210, the exposed face may be maintained as is, and de-identification processing may be performed on the non-exposed face. Accordingly, a result image can be obtained in which exposed faces can be identified but non-exposed faces cannot be identified.

Hereinafter, with reference to FIGS. 9 and 10, the above-described overall de-identification processing process and partial de-identification processing process (206 to 212) will be described in more detail.

9 and 10 are second and third flowcharts of an embodiment of a selective de-identification method.

As shown in FIG. 9, when an identification number is assigned to a face in one or more image frames (for example, at least one of the first and second image frames) in the target image and the first image frame is given (300), Identification numbers corresponding to each of at least one face in the first image frame are arranged according to predefined standards (302). Here, the predefined criteria may include the size or location of the face corresponding to each identification number, as described above. For example, the identification number may be arranged relatively in front as the size of the face corresponding to the identification number is larger and the location of the face is at or close to the center. Otherwise, the identification number may be arranged relatively in the back.

Once the identification numbers are arranged according to predetermined criteria, the identification numbers are extracted to create a first set and a second set (304). The extracted identification number may include identification numbers from the front to the standard number among the arrayed identification numbers. For example, if the standard number is 3, a total of 3 identification numbers can be extracted correspondingly, starting from the very front. The reference number may be the total number of faces to be compared, or may be the sum of a predefined extra value to the number of faces to be compared. The extracted identification numbers are treated as the first set, and the remaining identification numbers are treated as the second set.

Subsequently, it may be determined whether the face corresponding to each identification number belonging to the first set is an exposed face (306). Specifically, if the face corresponding to each identification number belonging to the first set and the comparison target face that is already acquired or acquired in real time are compared, and as a result of the comparison, both faces are the same or are approximate within a predefined range, The face in question is judged to be an exposed face. Conversely, if the two faces are different as a result of the comparison, the face is judged to be an unexposed face. The face corresponding to the identification number belonging to the first set is processed to be unidentifiable. Therefore, in order to compare both faces, according to one embodiment, the face corresponding to the identification number belonging to the first set is first restored and can be identified. Afterwards, a comparison is performed between the identifiable face and the face to be compared, and it can be determined whether the face is exposed. According to another embodiment, after de-identification processing is performed on the stored comparison target face, the de-identified comparison target face and the face corresponding to the identification number belonging to the first set are compared, and whether or not the face is exposed is determined. It could be.

Depending on the embodiment, the face determined to be an exposed face may be identified, and the face determined to be an unexposed face may be processed to be non-identifiable, thereby generating a new frame corresponding to the first image frame. In this case, if identification processing has already been performed on all faces for comparison, de-identification processing is performed on the non-exposed faces, and if all faces are still de-identified, identification processing is performed on the exposed faces. Processing may be performed to create a new frame.

The main face set is created based on the identification number corresponding to the face determined to be an exposed face, or if the main face set has already been created, the identification number corresponding to the face determined to be exposed is added to the main face set. The face set is updated (308).

A second image frame may then be acquired (310).

The identification numbers corresponding to each face extracted from the second video frame may be arranged in a predetermined order, and the first arranged identification numbers are newly arranged according to a predefined standard to become the second arranged identification numbers. Can be obtained (312). Here, the predefined criteria may include the size or position of the face corresponding to each identification number, as described above, and the larger the face, or the closer the face is placed to the center, the more likely it is that the face will be identified. The corresponding identification number may be placed relatively forward with respect to other identification numbers.

Referring to FIG. 10, the second arranged identification numbers may be rearranged using the second set generated in relation to the first image frame (314). Specifically, if there is an identification number in common with an identification number belonging to the second set among the second arranged identification numbers, the corresponding identification number is placed relatively forward with respect to the other identification numbers, and rearrangement of the identification numbers is performed accordingly. do. Accordingly, a first rearranged identification number is obtained.

After the acquisition of the first rearranged identification number, before or simultaneously with the acquisition of the first rearranged identification number, the identification number is rearranged using the main face set, and finally the second rearranged identification number is obtained. Can be obtained (316). Specifically, if there is an identical identification number belonging to the main face set among the second arranged identification number or the first rearranged identification number, the corresponding identification number is located relatively in front with respect to the other identification numbers, and as a result, Identification numbers are rearranged.

When the identification numbers are rearranged, at least one identification number is classified into the third set, and the remaining identification numbers are classified into the fourth set (318). The identification numbers classified into the third set may be identification numbers ranging from the first identification number among the second rearranged identification numbers to a predefined standard number. For example, if the reference number is given as 3, the third set will contain a total of 3 identification numbers starting from the very beginning. As described above, the reference number may be the total number of faces to be compared, or may be the sum of the number of faces to be compared plus a predefined extra value.

It is determined whether the face corresponding to each identification number belonging to the third set is an exposed face or an unexposed face (320). In this case, the face corresponding to the identification number belonging to the third set is compared with the comparison target face, and depending on the comparison result, the face is determined to be an exposed face or an unexposed face. If both faces are identical or close within a predefined range, the face is judged to be an exposed face. Conversely, if the two faces are different as a result of the comparison, the face is judged to be an unexposed face. For comparison of the two, identification processing of the face corresponding to the identification number belonging to the third set or de-identification processing of the face to be compared may be performed first.

In this case, if necessary, a new frame corresponding to the second frame may be created. Here, the new frame can be identified for faces determined to be exposed, but cannot be identified for faces determined to be non-exposed. As described above, if identification processing has already been performed on all faces for comparison between faces, de-identification processing is performed on the non-exposed faces, and if all faces are still de-identified, identification is performed on the exposed faces. A new frame may be created by performing image processing.

When a specific face is determined to be an exposed face, an identification number corresponding to the face determined to be an exposed face is added to the main face set, and the main face set is updated accordingly (322).

Depending on the embodiment, a third image frame may be additionally provided after determining whether the main exposed face is present (320) and/or updating the main face set (322). In this case, the above-described identification number arrangement (312), identification number rearrangement (314, 316), generation of the set (318), determination of the exposed face (320), and generation of the main face set (322) are performed repeatedly. It may be possible. Here, rearrangement of identification numbers can be performed based on the fourth set and the main face set.

Finally, the resulting image is generated (324). Generation of the resulting image may be performed by combining image frames generated during the process of performing the above-described process. Additionally, the generation of the resulting image may be performed by combining the identification number of the main face set obtained through the above-described process and the tracking result for the face. That is, the resulting image may be generated by tracking the identification number belonging to the main face set and performing identification processing on the face corresponding to the tracked identification number.

The selective de-identification method according to the above-described embodiment may be implemented in the form of a program that can be driven by a computer device. A program may include instructions, libraries, data files, and/or data structures, etc., singly or in combination, and may be designed and produced using machine code or high-level language code. The program may be specially designed to implement the above-described method, or may be implemented using various functions or definitions that are known and available to those skilled in the art in the computer software field. In addition, here, the computer device may be implemented by including a processor or memory that enables the function of the program, and may further include a communication device if necessary.

A program for implementing the above-described selective de-identification method may be recorded on a recording medium readable by a computer device or the like. Recording media that can be read by a computer device, etc. include, for example, semiconductor storage media such as ROM, RAM, SD cards, or flash memory (eg, solid state drive (SSD), etc.), or magnetic disks such as hard disks or floppy disks. At least a storage medium, an optical recording medium such as a compact disk or DVD, or a magneto-optical recording medium such as a floptical disk, capable of temporarily or non-temporarily storing one or more programs that are executed in response to a call from a device such as a computer. May contain one type of physical storage medium.

Although several embodiments of the selective de-identification device and the selective de-identification method have been described above, the selective de-identification device or the selective de-identification method is not limited to the above-described embodiments. Various other devices or methods that can be implemented by those skilled in the art by modifying and modifying the above-described embodiments can also be an embodiment of the above-described selective de-identification device or selective de-identification method. . For example, the described method(s) may be performed in an order other than as described, and/or component(s) of the described system, structure, device, circuit, etc. may be combined, connected, or otherwise in a form other than as described. Even if it is combined, or replaced or replaced by other components or equivalents, it can be an embodiment of the above-described selective de-identification device and/or selective de-identification method.

50: Face image 60: Target image
90: Face database 95: Main face set
100: Selective de-identification device 101: Input unit
103: output unit 105: storage unit
110: Processor 111: Exposure face acquisition unit
120: face processing unit 121: pre-processing unit
123: face extraction unit 125: face tracking unit
130: De-identification processing unit 131: Full de-identification processing unit
132: Arrangement unit 133: Set classification unit
134: identification unit 135: partial de-identification processing unit
136: Rearrangement unit

Claims (20)

a storage unit that stores faces to be compared;
Extract at least one face from the first image frame, perform de-identification processing on all extracted faces, and generate a first set corresponding to some of all extracted faces and a second set corresponding to the remaining portion. And a processor that compares the face corresponding to the first set and the face to be compared, and determines an exposed face and an unexposed face among the faces corresponding to the first set,
The processor,
Extract at least one face from the second image frame,
Assigning identification numbers to all faces extracted from the second image frame, and arranging the identification numbers of each face extracted from the second image frame to obtain a second arranged identification number,
Obtaining a first rearranged identification number by comparing the second set and the second arranged identification number, and rearranging the second arranged identification number according to whether or not it is included in the second set,
If an identification number corresponding to the exposed face is present in the first rearranged identification number, the first rearranged identification number is rearranged using the identification number corresponding to the exposed face to create a second rearranged identification. Optional de-identification device for obtaining numbers. According to paragraph 1,
The processor assigns identification numbers to all faces extracted from the first image frame, arranges the identification numbers of all faces extracted from the first image frame according to a predefined standard, and generates a first arranged identification number. A selective de-identification device for generating the first set by obtaining and detecting a reference number of identification numbers from the first arranged identification numbers. According to paragraph 2,
The predefined criterion includes at least one of the size and position of the face within the first image frame. According to paragraph 2,
The reference number is a selective de-identification device including the number of faces to be compared or the number of faces to be compared plus an extra value. According to paragraph 2,
The processor is a selective de-identification device for generating the second set by combining the remaining identification numbers from the first arranged identification numbers. delete According to paragraph 1,
The processor generates a third set by detecting a reference number of identification numbers from the first rearranged identification number or the second rearranged identification number, and compares the face corresponding to the third set with the target face. A selective de-identification device that compares and determines exposed faces and non-exposed faces among the faces corresponding to the third set. According to paragraph 1,
The processor,
Restore the face corresponding to the first set and compare the restored face with the comparison target face, or
By performing de-identification processing on the comparison target face and comparing the first set of faces with the de-identified comparison target face,
A selective de-identification device for determining an exposed face among the first set of faces. According to paragraph 1,
The processor is a selective de-identification device for generating a result image so that the exposed face can be identified and the non-exposed face cannot be identified. According to clause 9,
The processor,
When the faces of the first set are restored, de-identification is further performed on non-exposed faces among the faces of the first set, or de-identification is performed on all faces of the first set and the exposed faces By further performing identification, an image frame is generated such that the exposed face is identifiable and the non-exposed face is not identifiable, or
When de-identification processing is performed on the face to be compared, restoration is performed on the exposed face among the faces of the first set, thereby generating an image frame so that the exposed face can be identified and the non-exposed face cannot be identified. Selective de-identification device. Obtaining a face to be compared;
extracting at least one face from a first image frame;
performing de-identification processing on all extracted faces;
generating a first set corresponding to a portion of all extracted faces and a second set corresponding to the remaining portion;
Comparing the face corresponding to the first set with the comparison target face;
determining exposed faces and non-exposed faces among faces corresponding to the first set;
Extracting at least one face from a second image frame, arranging identification numbers of each face extracted from the second image frame, and obtaining a second arranged identification number;
Comparing the second set and the second arranged identification numbers, and rearranging the second arranged identification numbers according to whether they are included in the second set, thereby obtaining a first rearranged identification number; and
If an identification number corresponding to the exposed face is present in the first rearranged identification number, the first rearranged identification number is rearranged using the identification number corresponding to the exposed face to create a second rearranged identification. A selective de-identification method comprising: obtaining a number. According to clause 11,
The step of generating a first set corresponding to some of all the extracted faces includes:
assigning identification numbers to all faces extracted from the first video frame;
Arranging the identification numbers of all faces extracted from the first image frame according to a predefined standard to obtain a first arranged identification number; and
A selective de-identification method comprising: generating the first set by detecting a reference number of identification numbers from the first arranged identification numbers. According to clause 12,
A selective de-identification method wherein the predefined criterion includes at least one of the size and position of a face within the first image frame. According to clause 12,
The reference number is a selective de-identification method including the number of faces of the comparison target or the number of faces of the comparison target plus an extra value. According to clause 12,
Further comprising: performing de-identification processing on all faces extracted from the second image frame,
The second set is generated by combining the remaining identification numbers from the first arranged identification numbers. delete According to clause 11,
generating a third set by detecting a standard number of identification numbers from the first rearranged identification numbers or the second rearranged identification numbers;
Comparing the face corresponding to the third set with the comparison target face;
A selective de-identification method further comprising: determining exposed faces and non-exposed faces among faces corresponding to the third set. According to clause 11,
The step of comparing the face corresponding to the first set and the comparison target face is,
Reconstructing a face corresponding to the first set and comparing the restored face with the comparison target face; and
performing de-identification processing on the comparison target face and comparing the first set of faces with the de-identified comparison target face; A selective de-identification method comprising at least one of: According to clause 11,
Generating a result image so that the exposed face can be identified and the non-exposed face cannot be identified. According to clause 19,
The step of generating a result image so that the exposed face is identifiable and the non-exposed face is not identifiable,
When the faces of the first set are restored, further performing de-identification on unexposed faces among the faces of the first set;
When the first set of faces is restored, de-identification is performed on all faces of the first set and further identification is performed on the exposed faces, so that the exposed faces can be identified and the unexposed faces generating an image frame so that it cannot be identified; and
When de-identification processing is performed on the face to be compared, restoration is performed on the exposed face among the faces of the first set, thereby generating an image frame so that the exposed face can be identified and the non-exposed face cannot be identified. step; A selective de-identification method comprising at least one of: