KR102420890B1 - Service providing apparatus and method for de-identification of objects included in images - Google Patents

Abstract

The present invention relates to a service providing apparatus and method for de-identification of an object included in an image and, more specifically, to a service providing apparatus and method for de-identification of an object included in an image, which change a face of an object included in an original image to be different from the original image, thereby preventing leakage of personal information and supporting the use of images for various purposes. According to the present invention, after a composite image is generated by synthesizing an original image corresponding to a face area extracted from an object corresponding to a person in an image with a preset face model, the corresponding composite image is inserted into the face area of the object, and a face of the object is changed to a face different from an actual face of the person corresponding to the object, and thus, de-identification processing may be performed so that a face of an actual person corresponding to the object cannot be identified in the image. Therefore, images with personal information protected may be provided without worrying about infringement of portrait rights, and images including natural people may be provided compared to existing de-identification processing methods such as mosaic or blind, such that, by increasing the realism of images, satisfaction of users who use or distribute the images can be greatly increased.

Description

Service providing apparatus and method for de-identification of objects included in images

The present invention relates to an apparatus and method for providing a service for de-identification of an object included in an image, and more particularly, by changing the face of an object included in an original image to be different from the original image to prevent leakage of personal information; In addition, the present invention relates to an apparatus and method for providing a service for de-identification of an object included in an image that supports the use of an image for various purposes.

In general, when shooting a video, the face of the person photographed in the video appears, and an individual can be identified through such a face. It has to be post-processed.

To this end, conventionally, an object corresponding to an individual is identified in an image, and a de-identification process such as mosaic or blind is performed on the face area in the object to prevent leakage of personal information due to image distribution.

However, in such a conventional post-processing process, the object created through de-identification processing such as mosaic or blind of the face area is unnatural, and thus the image quality is deteriorated, thereby lowering the satisfaction of viewers watching the video. there is

Korean Patent Registration No. 10-2129031

The present invention identifies an object corresponding to a person from an image, extracts a face image of the object, synthesizes it with a preset basic face model, creates a composite image, and inserts it into the face region of the object, corresponding to the person appearing in the image The purpose of this is to support the protection of personal information by increasing the realism of the object, and by supporting that the individual cannot be identified through the object.

In addition, an object of the present invention is to support the use of an image in which personal information is protected by de-identification processing of an individual's face appearing in an image for various purposes without fear of infringement of the right to portrait.

A service providing apparatus for de-identification of an object included in an image according to an embodiment of the present invention includes an image receiving unit for receiving an image, identifying one or more objects corresponding to a person in the image, and each of the one or more objects a face extractor for generating an original image by extracting a face region from a synthesizing unit for generating a composite image for each object by synthesizing the original image generated for each of the one or more objects with a preset face model; and the one or more objects; and an inserter for inserting the composite image into a face region of an object corresponding to the composite image in the image for each of the one or more composite images generated to correspond to each other.

As an example related to the present invention, the service providing apparatus further includes a detection unit for calculating attribute information including attribute values for each of the one or more attributes corresponding to the original image among one or more attributes distinguishable through the image, , the synthesizing unit compares the attribute information with category information for a plurality of preset categories, selects a preset specific face model to correspond to a specific category corresponding to the attribute information, and synthesizes the specific face model and the original image to generate the composite image.

As an example related to the present invention, the one or more attributes may include at least one of age, gender, nationality, skin color, and race.

As an example related to the present invention, the detection unit applies the original image generated through the face extraction unit to a learning model in which the correlation between the one or more attributes and the image for the face is learned, and the original image through the learning model It may be characterized in that the attribute information corresponding to the image is generated.

As an example related to the present invention, the detection unit identifies an original image in which a body part of a face is not detected, and the synthesizing unit interworks with the detection unit and sets an original image in which a body part of the face is not detected in a preset GAN model. to generate a virtual image obtained by interpolating a region of the body part that is not detected in the original image, and synthesizing the virtual image with the face model to generate a composite image.

As an example related to the present invention, the service providing apparatus further includes a storage unit storing category information for each of the plurality of categories and a plurality of face models each matching the plurality of categories, wherein the category information includes the one or more attributes Each set value is preset, and at least one of the set values for the one or more attributes is set differently among the plurality of category information for each category, and the synthesizing unit is configured to: It may be characterized in that any one of a plurality of face models matched with a category is randomly selected as the specific face model and then combined with the original image.

As an example related to the present invention, the face extractor identifies a mutually identical specific object in each of a plurality of frames constituting the image, and generates an original image corresponding to the specific object for each of the plurality of frames, and the synthesis The unit sets a specific face model selected to correspond to the specific object based on any one frame among the plurality of frames as a reference model in conjunction with the face extractor, and corresponds to the specific object and frame for each of the plurality of frames The original image is synthesized with the reference model to generate a composite image corresponding to the specific object and frame, and the inserting unit interworks with the synthesis unit to generate a composite image corresponding to the frame for each of the plurality of frames of the specific object. It may be characterized in that it is inserted into the face region.

As an example related to the present invention, the detection unit analyzes the original image and determines whether any one of one or more event conditions set in relation to facial features is satisfied, and when the specific event condition is satisfied, the specific event that the original image is satisfied with After generating event information corresponding to the condition, it is provided to the synthesizing unit together with the original image, and the synthesizing unit identifies a specific event type corresponding to the original image based on the event information provided together with the original image and, after removing an event region satisfying the specific event condition from the original image according to an event processing function preset to correspond to the specific event type, synthesized with the face model to generate the composite image or to respond to the specific event type It may be characterized in that after extracting the additional items stored in advance, synthesizing them in an event area of the original image corresponding to the additional items, and then synthesizing them with the face model corresponding to the original image to generate the composite image.

As an example related to the present invention, the synthesizing unit detects the posture of the face in the original image, changes the posture of the face model to correspond to the posture, and then synthesizes it with the original image to generate the synthesized image. can be characterized.

A service providing method for de-identification of an object included in an image of a service providing apparatus according to an embodiment of the present invention includes: receiving an image; identifying one or more objects corresponding to a person in the image; generating an original image by extracting a face region from each of the above objects; generating a composite image for each object by synthesizing the original image generated for each of the one or more objects with a preset face model; The method may include inserting each of the one or more synthetic images generated to correspond to each other into a face region of an object corresponding to the composite image in the image.

The present invention generates a composite image by synthesizing an original image corresponding to a face region extracted from an object corresponding to a person in an image with a preset face model, and then inserting the composite image into the face region of the object to create a person corresponding to the object. By changing the face of the object to a face different from the real face of In addition to being able to provide images, it is possible to provide images with natural people compared to existing de-identification processing methods such as mosaics and blinds. can

In addition, the present invention uses a face model having properties similar to or identical to those of the person corresponding to the object identified from the object corresponding to the person in the image, such as age, gender, nationality, skin color, race, etc. By substituting another face, it is possible to support the faces of people appearing in the image to be displayed without any sense of heterogeneity, so that the quality of the image generated through de-identification processing of individuals appearing in the image can be guaranteed.

In addition, the present invention can support the image to be used as the background of various media contents by providing an image in which the face of an individual appearing in the image is de-identified, and through this, it is possible to display various images that do not infringe on personal information. We can support you to use it for your intended purpose.

1 is a block diagram of a service providing apparatus for de-identification of an object included in an image according to an embodiment of the present invention;
2 and 3 are diagrams illustrating an operation of an apparatus for providing a service according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a synthesis process with a face model in consideration of the posture of the original image of the service providing apparatus according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a synthesis process with a face model in consideration of the properties of the original image of the service providing apparatus according to an embodiment of the present invention.
6 is an exemplary diagram illustrating an interpolation process of an original image in which a body part of a service providing apparatus is covered according to an embodiment of the present invention;
7 is a flowchart of a service providing method for de-identification of an object included in an image according to an embodiment of the present invention;

Hereinafter, detailed embodiments of the present invention will be described with reference to the drawings.

1 is a block diagram of a service providing apparatus (hereinafter, service providing apparatus) for de-identification of an object included in an image according to an embodiment of the present invention.

As illustrated, the service providing apparatus 100 may include a communication unit 110 , a storage unit 120 , and a control unit 130 .

In this case, the service providing apparatus 100 may be implemented by more components than the components shown in FIG. 1 , or the service providing apparatus 100 may be implemented by fewer components than the components shown in FIG. 1 .

First, the communication unit 110 may communicate with an external device through a communication network to receive an image from the external device, and may provide the image to the controller 130 .

In this case, the communication network described in the present invention may include a wired/wireless communication network, and as an example of such a wireless communication network, a wireless LAN (WLAN), a DLNA (Digital Living Network Alliance), a WiBro (Wireless Broadband: Wibro), Wimax (World Interoperability for Microwave Access: Wimax), GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice- Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-A (LTE-A) Advanced), Wireless Mobile Broadband Service (WMBS), 5G mobile communication service, Bluetooth (Bluetooth), LoRa (Long Range), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct may be included. In addition, wired communication networks include wired LAN (Local Area Network), wired WAN (Wide Area Network), Power Line Communication (PLC), USB communication, Ethernet, serial communication, optical/coaxial cables, etc. may be included.

In addition, the external device may be composed of various devices such as a user terminal or an external server.

In addition, the storage unit 120 may store various types of information, and may include a DB or may be configured as a DB.

For example, the storage unit 120 includes an image DB for storing images, an object DB for storing object-related information, a category DB for storing category-related information, and one or more additional item-related information applicable to the face region of the object. It may include a stored feature DB and the like.

In addition, the control unit 130 performs an overall control function of the service providing device 100, and may include a RAM, ROM, CPU, GPU, and bus, and RAM, ROM, CPU, GPU, etc. can be connected to each other through

In this case, an operation configuration of the service providing apparatus 100 described below may be performed by the controller 130 .

In addition, the control unit 130 may be configured to include a plurality of components including an image receiving unit 131 , a face extracting unit 132 , a detecting unit 133 , a synthesizing unit 134 , and an inserting unit 135 . , at least one of the plurality of components constituting the control unit 130 may be included in another component.

In this case, the plurality of components constituting the control unit 130 may be implemented by a processor capable of data processing, etc., and may be separately implemented by different processors, or may be functionally separated within one processor.

Hereinafter, a detailed operation configuration of the service providing apparatus 100 will be described with reference to the following drawings based on the configuration of the control unit 130 that performs the overall control function of the service providing apparatus 100 .

First, FIGS. 2 and 3 are diagrams illustrating an operation of the service providing apparatus 100 according to an embodiment of the present invention.

First, an object identification and analysis process of the service providing apparatus 100 will be described with reference to FIG. 2 .

The image receiving unit 131 may receive an image from an external device, store the image in an image DB included in the storage unit 120 , and provide the image to the face extractor 132 . may be

In addition, the face extractor 132 may extract the image from the image DB or receive the image from the image receiver 131 , identify one or more objects corresponding to a person in the image, and After identifying a face region through image analysis in each of one or more objects, the corresponding face region may be extracted to generate an original image composed of the extracted face region.

For example, the face extractor 132 may use a preset image analysis algorithm to separate a foreground and a background, and identify an object corresponding to the foreground.

At this time, the face extractor 132 may apply various image analysis algorithms for background modeling to the image, for example, a differential image method, a Model of Gaussian (MOG) algorithm using Gaussian Mixture Models (GMM), A codebook algorithm or the like may be applied.

In addition, in order to identify an object corresponding to a person from among one or more objects identified in the image, the face extractor 132 is configured to set a feature extraction algorithm for extracting feature information in advance, Extracting characteristic information for each of one or more objects, and comparing the extracted characteristic information with reference characteristic information preset for identification of an object corresponding to a person to identify an object corresponding to a person among the identified one or more objects can

In this case, as an example of the feature extraction algorithm, an algorithm for extracting feature information such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradient (HOG), Haar feature, Ferns, etc. may be used.

Also, an object used in the following description may mean an object corresponding to the person.

In addition, the face extraction unit 132 extracts a face region from each of one or more objects corresponding to the person identified in the image through a preset algorithm for face identification, and based on the extracted face region, an original image can create

In addition, the detection unit 133 analyzes the original image through a preset attribute analysis algorithm to calculate an attribute value of one or more attributes corresponding to the original image among one or more preset attributes distinguishable through an image, Attribute information including attribute values for one or more attributes may be generated.

In this case, the detection unit 133 may be preset with a learning model in which the correlation between the one or more attributes and the image for the face is learned, and the detection unit 133 may include the face extraction unit ( 132) may be applied to the learning model as an input, and attribute information including attribute values for each of the one or more attributes corresponding to the original image may be output as an output through the learning model.

In the above configuration, the attribute analysis algorithm may be configured as the learning model, and the learning model may be configured as a deep learning algorithm.

In this case, the deep learning algorithm may be composed of one or more neural network models.

In addition, the neural network model (or neural network) described in the present invention may be composed of an input layer, one or more hidden layers and an output layer, and the neural network model includes a Deep Neural Network (DNN). , Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), Support Vector Machine (SVM), and the like may be applied to various types of neural networks.

In addition, the one or more attributes may include age (or age group), gender, nationality, skin color, race, etc., and the detection unit 133 may include an age (or age group), Attribute information including data on each of gender, nationality, skin color, and race may be generated.

In addition, the face extraction unit 132 may generate object information on the object when identifying the object corresponding to the person and store it in the object DB included in the storage unit 120 , and the detection unit 133 . may work with the face extractor 132 to match the original image corresponding to the object information and the attribute information with the object information and store it in the object DB.

Meanwhile, FIG. 3 is an operation example diagram illustrating a de-identification process for an object of the service providing apparatus 100 .

First, the synthesizing unit 134 may generate a composite image for each of the one or more objects by synthesizing the original image generated for each one or more objects included in the image with a preset face model.

Also, the insertion unit 135 may insert the composite image into the face region of the object corresponding to the composite image in the image for each of the one or more composite images generated to correspond to the one or more objects, respectively. .

For example, the three-dimensional face model may be preset in the synthesizing unit 134 as a basic model, and the synthesizing unit 134 synthesizes the original image with the face model according to a preset synthesizing algorithm. The composite image may be generated.

In this case, as an example of the synthesis algorithm, a morphing algorithm may be used.

In addition, as shown in FIG. 4 , the synthesizing unit 134 identifies a plurality of regions respectively corresponding to eyes, nose and mouth in the original image when synthesizing the original image and the face model, and the eyes, The posture of the object included in the original image with respect to the face may be identified (determined) based on the arrangement state of the plurality of regions respectively corresponding to the nose and the mouth.

In this case, the region may be a coordinate.

In addition, the synthesizing unit 134 generates posture-related posture information for the face of the object, and sets the posture of the face model based on the posture information to have the same posture as the posture identified with respect to the original image. After the change, the composite image may be generated by synthesizing the face model whose posture is changed and the original image.

That is, the synthesizing unit 134 changes the posture of the face model to match the posture corresponding to the face of the object shown in the original image and then synthesizes it with the original image to generate the synthesized image, thereby creating a natural face. You can create the composite image shown.

In addition, when the original image and the face model are synthesized, the synthesizing unit 134 includes attributes such as age (or age group), gender, nationality, skin color, race, etc. of a person corresponding to the face of the object included in the original image. When the attributes such as age, gender, nationality, skin color, race, etc. corresponding to the face model are not identical to each other, a composite image of an unnatural face is generated.

A configuration for preventing this will be described with reference to FIG. 5 .

First, the storage unit 120 may store a plurality of different category information for each category, and the category information may include preset values for each of the one or more attributes.

Also, set values of at least one attribute among a plurality of category information stored in the storage unit 120 may be different from each other.

In addition, one or more face models may be stored for each of the plurality of category information in the storage unit 120 , and the plurality of face models matched with the category information may be configured to be different from each other.

For example, in the storage unit 120 , first category information about a Caucasian woman of American nationality in her twenties and one or more face models corresponding to the first category information are matched with the first category information and stored In addition, second category information about a yellow male of Korean nationality who is 30 years of age and one or more face models corresponding to the second category information may be stored by matching the second category information.

In this case, the plurality of category information and one or more face models for each category information may be stored in a separate category DB included in the storage unit 120 .

Accordingly, the synthesizing unit 134 cooperates with the face extracting unit 132 and the detecting unit 133, and based on the attribute information generated through the detecting unit 133 for the original image, the storage unit 120 ), it is possible to identify specific category information including set values for one or more attributes matching the attribute values for one or more attributes according to the attribute information among the plurality of category information.

Also, the synthesizer 134 may select any one of one or more face models matched with the specific category information and then synthesize the selected face model with the original image to generate the synthesized image.

Through the above-described configuration, the synthesizing unit 134 selects a face model having properties matching the properties of the original image and synthesizes the face model with the original image, based on the original image. While de-identifying a face, it is possible to create a composite image including another natural virtual face with minimal difference with the face shown in the original image by transforming it into another virtual person having the same properties as the person concerned.

In addition, the insertion unit 135 may identify a specific object corresponding to the composite image in the image by interworking with at least one of the face extraction unit 132, the detection unit 133, and the synthesis unit 134. , the composite image may be inserted into the face region of the specific object extracted by the face extraction unit 132 to de-identify the face of the specific object included in the image.

That is, the insertion unit 135 may identify a region from which the face region is extracted from the object corresponding to the composite image as the insertion target region of the composite image, and insert the composite image into the insertion target region.

In addition, the controller 130 generates an original image by extracting a face region from the object as described above for one or more objects corresponding to the person identified in the image, and then generates attribute information corresponding to the original image. Then, the face model selected in the storage unit 120 based on the attribute information is synthesized with the original image in consideration of the posture of the original image to generate a composite image, and then the face region of the object corresponding to the composite image. can be inserted into

Meanwhile, in the above configuration, the image may be composed of a plurality of frames.

Accordingly, the controller 130 generates an original image through the face extractor 132 for each object corresponding to a person in the frame as described above for each of the plurality of frames, and then generates the original image and the attribute information. A face model selected on the basis of is synthesized in a state of considering the posture of the original image to generate a composite image, and then the composite image is inserted into the face region of the object to de-identify the face for each object included in the frame. have.

At this time, the face extraction unit 132 of the control unit 130 gives each object identified in the frame a unique identifier for distinguishing it from other objects, generates object information including the unique identifier, and generates the object information in the storage unit 120 . ) may be stored in the object DB, and original image and attribute information for each frame corresponding to the object information may be matched with the object information and stored in the object DB.

On the other hand, when the image is composed of a plurality of frames, the controller 130 identifies the mutually identical specific object in each of a plurality of frames constituting the image, and an original image corresponding to the specific object for each of the plurality of frames. generating a specific face model selected to correspond to the specific object based on any one frame among the plurality of frames as a reference model, and setting an original image corresponding to the specific object and frame for each of the plurality of frames. By synthesizing a reference model to generate a composite image to correspond to the specific object and frame, and inserting the composite image corresponding to the frame for each frame into the face region of the specific object, the same object in different frames has the same face It is possible to de-identify the face of a specific object so that it appears as .

As an example, the face extractor 132 may identify a mutually identical specific object in each of a plurality of frames constituting the image, and generate an original image corresponding to the specific object for each of the plurality of frames. .

To this end, the face extractor 132 may track a specific object identified in the image using an object tracking algorithm, for example, for each of a plurality of frames constituting the image using a Kalman filter. Continuous tracking of the same object as the specific object may be performed.

Also, the synthesizing unit 134 may extract model identification information from a face model selected based on the attribute information of the specific object, match it with object information corresponding to the specific object in the object DB, and store it.

Accordingly, the synthesizing unit 134 may identify the same object in each of the plurality of frames by interworking with the face extractor 132, and in each of the plurality of frames, the same object as the specific object In the case of generating the composite image, the composite image may be generated using only the face model corresponding to the model identification information stored in the object DB by matching with the object information of the specific object.

In addition, the insertion unit 135 identifies the same object as the specific object in the frame corresponding to the composite image in cooperation with the face extraction unit 132 and the synthesis unit 134 , and the identified identical object The composite image may be inserted into the face region of , and the composite image generated for the specific object for each of the plurality of frames may be inserted into the face region of the specific object.

In addition, when the de-identification process for all objects included in the image or one or more frames is completed, the insertion unit 135 performs de-identification of the image or each of the one or more frames constituting the image as described above. A de-identification-processed image may be generated based on an image including one or more processed objects or one or more frames on which de-identification processing has been completed, and the de-identified processed image may be stored in the image DB.

In addition, the inserting unit 135 may transmit the de-identified processed image to the external device through the communication unit 110, and the de-identified processed image is used as a background of various media contents such as movies and dramas. It can be distributed for use.

That is, the controller 130 can maintain the sameness and consistency with respect to the face of the same object by allowing only a specific face model to be used to generate a composite image for all the same objects included in each of the plurality of frames. By preventing the face of the same object from continuously changing by using different face models for the same object between a plurality of frames, an image including a natural object without a sense of heterogeneity can be generated.

As described above, in the present invention, a composite image is generated by synthesizing an original image corresponding to a face region extracted from an object corresponding to a person in an image with a preset face model, and then inserting the synthesized image into the face region of the object. By changing the face of the object to a face different from the real face of the person corresponding to the object, de-identification processing can be performed so that the face of the real person corresponding to the object cannot be identified in the image, and through this, without worrying about infringement of portrait rights Users who use or distribute the video by increasing the realism of the video by providing not only privacy-protected video, but also natural person compared to existing de-identification processing methods such as mosaic or blinds. can significantly increase satisfaction.

In addition, the present invention uses a face model having properties similar to or identical to those of the person corresponding to the object identified from the object corresponding to the person in the image, such as age, gender, nationality, skin color, race, etc. By substituting (changing) other faces, it is possible to support the faces of people appearing in the image to be displayed without a sense of heterogeneity, thereby ensuring the quality of the image generated through de-identification processing of individuals appearing in the image.

In addition, the present invention can support the image to be used as the background of various media contents by providing an image in which the face of an individual appearing in the image is de-identified, and through this, it is possible to display various images that do not infringe on personal information. We can support you to use it for your intended purpose.

On the other hand, if the characteristics of the object corresponding to the person appearing in the image are excessive (strong) compared to other objects, or if the object's face is covered by various wears such as masks, glasses, or accessories, it is synthesized with the face model. A highly distorted composite image can be generated, and when it is inserted into the face of an object appearing in the image as it is, the sense of heterogeneity of the image increases, so a configuration is required to prevent this. do.

First, as shown in FIG. 6 , the detection unit 133 analyzes the original image and generates error information when a body part is not detected in the face included in the original image, and then attributes corresponding to the original image. It may be provided to the synthesizing unit 134 together with the information and the original image.

In addition, the synthesizing unit 134, in conjunction with the detection unit 133, applies the original image corresponding to the error information to a preset GAN (Generative Adversarial Network) model to the body part that is not detected in the original image. A virtual image obtained by interpolating the corresponding region may be generated.

For example, the detection unit 133 may generate error information when the nose and mouth are covered by a mask, which is an article to be worn on the face, in the face shown in the original image, and even in this case, the original image is applied to the learning model. can be applied to generate the attribute information.

In addition, a generator that generates a virtual face image by learning a plurality of face images, and a generator that learns a plurality of face images to determine whether the virtual image generated by the generator is normal, and provides the determination result to the generator A discriminator that trains the generator to generate a natural face image, and a GAN model configured to compete with the generator and the discriminator may be preset in the synthesizing unit 134 .

Accordingly, the synthesis unit 134, in conjunction with the detection unit 133, applies the original image corresponding to the error information as an input to the GAN model, and is trained to generate only a normal virtual image by the discriminator. Through the generator of the GAN model, a virtual image obtained by interpolating a region corresponding to a body part hidden from the face displayed in the original image may be generated (calculated) as an output.

At this time, the synthesizing unit 134 applies the attribute information corresponding to the original image and the original image together to the GAN model, and interpolates the body parts hidden in the original image according to the attribute information through the GAN model. You can also create virtual virtual images.

That is, the synthesizing unit 134 applies the attribute information to the generator of the GAN model together with the original image in which the body part is covered, so that age (or age range), gender, nationality, In consideration of at least one of skin color and race, a virtual image may be generated by interpolating a body part covered in the original image. After generating an interpolated image, a virtual image that is an interpolated original image is generated with the interpolated image, and the virtual image is generated so that the interpolated region has continuity with the remaining regions of the original image. can be obtained.

At this time, in the interpolation described in the present invention, one or more body parts that are not detected by the detector 133 in the image and one or more interpolation target regions included in the image corresponding respectively to the virtual body predicted for the interpolation target region This may mean generating an image of a region and inserting it into a corresponding interpolation target region.

In addition, the synthesizing unit 134 is configured to synthesize a virtual image generated through the GAN model with respect to the original image corresponding to the error information with a face model selected based on attribute information of the original image instead of the original image. A composite image may be generated, and the composite image may be inserted into a face region of a specific object included in an image or frame corresponding to the original image.

In addition, the detection unit 133 may analyze the original image to determine whether any one of one or more preset event conditions related to facial features is satisfied, and event information corresponding to the event condition satisfied by the original image can create

For example, the one or more event conditions may include various event conditions such as a bald face or a beard present in the original image.

In addition, the detection unit 133 may generate event information including an event type corresponding to an event condition satisfied by the original image and provide it to the synthesis unit 134 , and the event condition satisfied by the original image In this case, a plurality of event information corresponding to a plurality of event conditions may be generated to correspond to the original image, and then provided to the synthesizing unit 134 together with the original image and attribute information corresponding to the original image. have.

Accordingly, the synthesizing unit 134 may identify one or more event types corresponding to the specific original image based on one or more event information provided together with the specific original image from the detection unit 133 .

In addition, an event processing function to be performed by the synthesizing unit 134 for each of the one or more event types may be set in the synthesizing unit 134 .

In addition, the storage unit 120 of the service providing apparatus 100 may include a feature DB in which additional items added to the original image are stored for each of one or more preset event types.

Accordingly, the synthesizing unit 134 generates an event region satisfying the event condition according to the event information in the original image according to an event processing function corresponding to the event type according to the event information provided from the detection unit 133 . After removal, an additional item corresponding to an event type according to the event information is extracted from the feature DB by synthesizing with a face model corresponding to the original image, or an event area of the original image corresponding to the extracted additional item in the event area After synthesizing with the face model corresponding to the original image, event processing may be performed.

For example, when the face shown in the original image is bald, the synthesizing unit 134 extracts the additional item related to the hair from the feature DB according to an event processing function identified based on the event type corresponding to the baldness. The composite image may be generated by applying it to the hair region, which is the event region identified in the original image, and then synthesizing it with the face model.

Alternatively, the synthesizing unit 134 may remove an event region including a beard from the original image according to an event processing function identified based on an event type corresponding to the beard when a beard is present on the face shown in the original image. Then, the composite image may be generated by synthesizing it with the face model.

Also, the insertion unit 135 may insert the event-processed composite image into a face region of a specific object corresponding to the composite image in an image or frame.

Through the above-described configuration, the service providing apparatus 100 eases or removes these features from the original image including the face with strong features that satisfy the event condition, and de-identifies the features so that the features are covered, so that the features are easily We can help you protect your personal information.

7 is a flowchart of a service providing method for de-identification of an object included in an image of the service providing apparatus 100 according to an embodiment of the present invention.

As shown, the service providing apparatus 100 may receive an image from an external device (S1).

Also, the service providing apparatus 100 may identify one or more objects corresponding to a person in the image (S2), and extract a face region from each of the one or more objects to generate an original image (S3).

Also, the service providing apparatus 100 may generate a composite image for each object by synthesizing the original image generated for each one or more objects with a preset face model (S4).

Also, the service providing apparatus 100 may insert each of the one or more composite images generated to correspond to the one or more objects in the face region of the object corresponding to the composite image in the image (S5).

The components described in the embodiments of the present invention include, for example, a storage unit 120 such as a memory, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and an FPGA. (Field Programmable Gate Array), hardware such as a programmable logic unit (PLU), a microprocessor, software including an instruction set, or any combination thereof or any other device capable of executing and responding to instructions, one or more It may be implemented using a general purpose computer or a special purpose computer.

Those of ordinary skill in the art to which the present invention pertains may modify and modify the above-described contents without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: service providing device 110: communication unit
120: storage unit 130: control unit
131: image receiving unit 132: face extracting unit
133: detection unit 134: synthesis unit
135: insert

Claims (10)

an image receiving unit for receiving an image;
a face extractor for identifying one or more objects corresponding to a person in the image, and extracting a face region from each of the one or more objects to generate an original image;
a detector for calculating attribute information including attribute values for each of the one or more attributes corresponding to the original image among one or more attributes distinguishable through the image;
Comparing the attribute information corresponding to the original image with category information for a plurality of preset categories, selecting a preset face model to correspond to a specific category corresponding to the attribute information, and synthesizing the selected face model with the original image a synthesis unit generating the composite image for each of the one or more original images respectively corresponding to the one or more objects in a manner of generating a composite image; and
Insertion unit for inserting the composite image into a face region of an object corresponding to the composite image in the image for each of the one or more composite images generated to correspond to the one or more objects, respectively
includes,
The detector analyzes the original image to determine whether any one of one or more preset event conditions related to facial features is satisfied, and generates event information corresponding to a specific event condition satisfied by the original image when the specific event condition is satisfied and then provided to the synthesizing unit together with the original image,
The synthesizing unit identifies a specific event type corresponding to the original image based on the event information provided together with the original image, and selects the specific event type from the original image according to an event processing function preset to correspond to the specific event type. After removing the event area that satisfies the event condition, the composite image is generated by synthesizing with the selected face model, or a pre-stored additional item corresponding to the specific event type is extracted and then synthesized in the event area of the original image corresponding to the additional item and then synthesized with the selected face model corresponding to the original image to generate the synthesized image.
delete The method according to claim 1,
The one or more attributes include at least one of age, gender, nationality, skin color, and race.
The method according to claim 1,
The detection unit applies the original image generated through the face extraction unit to a learning model in which the correlation between the one or more attributes and the image for the face is learned, and the attribute information corresponding to the original image through the learning model A service providing apparatus for de-identification of an object included in an image, characterized in that it is generated.
The method according to claim 1,
The detection unit identifies an original image in which a body part of the face is not detected,
The synthesizing unit generates a virtual image obtained by interpolating an area for the undetected body part from the original image by applying an original image in which a body part of the face is not detected in the original image by interworking with the detection unit to a preset GAN model, A service providing apparatus for de-identification of an object included in an image, characterized in that a virtual image is synthesized with the face model to generate a composite image.
The method according to claim 1,
and a storage unit storing category information for each of the plurality of categories and a plurality of face models each matching the plurality of categories;
In the category information, set values for each of the one or more attributes are preset,
At least one of the set values for each of the one or more attributes is set differently in advance among the plurality of category information for each category,
wherein the synthesizing unit randomly selects any one of a plurality of face models matched to a specific category corresponding to attribute information generated for a specific object and synthesizes it with the original image. service providing device for
7. The method of claim 6,
The face extractor identifies a specific object identical to each other in each of a plurality of frames constituting the image, and generates an original image corresponding to the specific object for each of the plurality of frames,
The synthesizing unit, in conjunction with the face extracting unit, sets a specific face model selected to correspond to the specific object based on any one of the plurality of frames as a reference model, and sets the specific object and frame for each of the plurality of frames. By synthesizing the original image corresponding to the reference model and generating a composite image corresponding to the specific object and frame,
The insertion unit interlocks with the synthesis unit to insert a composite image corresponding to each frame for each of the plurality of frames into the face region of the specific object.
delete The method according to claim 1,
The synthesizing unit detects a posture of the face from the original image, changes the posture of the face model to correspond to the posture, and synthesizes the face model with the original image to generate the synthesized image. A service providing device for the de-identification of
In the service providing method for de-identification of an object included in the image of the service providing device,
receiving an image;
generating an original image by identifying one or more objects corresponding to a person in the image and extracting a face region from each of the one or more objects;
calculating attribute information including attribute values for each of the one or more attributes corresponding to the original image among one or more attributes distinguishable through the image;
Comparing the attribute information corresponding to the original image with category information for a plurality of preset categories, selecting a preset face model to correspond to a specific category corresponding to the attribute information, and synthesizing the selected face model with the original image generating the composite image for each of the one or more original images respectively corresponding to the one or more objects in a manner of generating a composite image; and
Inserting the composite image into a face region of an object corresponding to the composite image in the image for each of the one or more composite images generated to correspond to the one or more objects, respectively
includes,
The calculating of the attribute information may include analyzing the original image to determine whether any one of one or more preset event conditions related to facial features is satisfied, and corresponding to a specific event condition satisfied by the original image when a specific event condition is satisfied After generating the event information to be further comprising the step of providing it together with the original image,
The generating of the composite image may include identifying a specific event type corresponding to the original image based on the event information provided together with the original image, and according to an event processing function preset to correspond to the specific event type. After removing an event region that satisfies the specific event condition from the original image, the composite image is generated by synthesizing with the selected face model, or an original image corresponding to the additional item after extracting a pre-stored additional item corresponding to the specific event type The method for providing a service for de-identification of an object included in an image, characterized in that the composite image is generated by synthesizing it in an event area of , and then synthesizing it with the selected face model corresponding to the original image.

Images