KR20200115706A - Semantic image segmentation method based on deep-learning and apparatus thereof - Google Patents

Abstract

The present invention relates to a method for deep learning-based image segmentation, and a device therefor. According to the present invention, the device comprises: a communication unit for receiving a training data set; a storage unit for storing a deep learning model; and a processor for training the deep learning model by using the training data set. According to the present invention, segmentation of the character faces in a cartoon image can be rapidly and accurately performed.

Description

Deep learning-based image segmentation method and its device {SEMANTIC IMAGE SEGMENTATION METHOD BASED ON DEEP-LEARNING AND APPARATUS THEREOF}

The present invention relates to a deep learning-based image segmentation method and apparatus therefor. In more detail, the present invention relates to a method and apparatus for regionalizing a face in a cartoon image using a deep learning neural network.

As the use of user terminals such as smartphones, tablet PCs, personal digital assistants (PDAs), notebooks, etc. is popular and information processing technology is developed, images or videos are photographed using user terminals, and the captured images are adapted to the needs of users. There is a trend of active research on video editing technology to edit properly.

One of the fields of commercial use of image editing is the field of cartoon image face editing, in which the skin color, hairstyle, and hair color of a character's face is changed or changed to another face within the cartoon image. In order to edit only the character face area in the cartoon image, it is necessary to make the character's face area within the cartoon image. Image segmentation is a technique for separating an image area based on a threshold or an edge. In particular, separating a significant area such as a character's face is called semantic segmentation. However, there is a problem that an image editor needs a lot of time and effort to find a threshold or an edge every time for accurate and detailed image regionization without noise in different images.

In order to overcome this problem, image areaization technology through deep learning has recently been actively attempted. Representative image domainization technologies through deep learning include U-net, Segnet, Tiramisu, and Deep Lab. The representative image segmentation technology has a limitation in that it is a supervised learning method capable of training a neural network only when a vast amount of training data sets are prepared. The training data set includes a training image in the same domain as the image to be imaged and ground truth labels for the training image. However, in the cartoon image, training data sets for image regionization are very rare, and manual processing of the data set to create the cartoon image has a problem that it consumes a lot of time and cost.

In this supervised learning method, domain adaptation technologies such as XGAN and DTN were introduced to overcome the problem of requiring a training data set, but it was successful as a technology to convert an image of a specific domain into an image of another domain. It was impossible to use it for territorialization (A. Royer, K. Bousmalis, S. Gouws, F. Bertsch, I. Mosseri, F. Cole, and K. Murphy. XGAN: Unsupervised Image-to-Image Translation) for Many-to-Many Mappings.arXiv preprint arxiv:1711.05139, 2018, Y. Taigman, A. Polyak, and L. Wolf. Unsupervised Cross-Domain Image Generation.In ICLR, 2017.)

In addition, although a medical image domainization technique using domain application technology was introduced, it is composed of a very simple neural network, so if the difference in characteristics between domains exceeds a predetermined level, the domain application technology cannot be applied or even if it is applied There is a problem in that there is a performance limitation that cannot perform image segmentation (M. Javanmardi, and T. Tasdizen. DOMAIN ADAPTATION FOR BIOMEDICAL IMAGE SEGMENTATION USING ADVERSARIAL TRAINING. In ISBI, 2018.)

Non-Patent Document 1: O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI, 2015. Non-Patent Document 2: V. Badrinarayanan, A. Kendall, and R. Cipolla. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:2481-2495, 2017. Non-Patent Document 3: S. Jegou, M. Drozdzal, D. Vazquez, A. Romero, and Y. Bengio. The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. In CVPR, 2017. Non-Patent Document 4: L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv preprint arXiv:1706.05587, 2017.

In order to solve the above-described problems, the present invention relates to a deep learning-based image segmentation method and an apparatus therefor. In more detail, we propose a method and apparatus for regionizing a face in a cartoon image using a deep learning neural network.

In detail, an object of the present invention is to provide a deep learning-based image domainization method and apparatus capable of domainizing a character face in order to change a character face in a cartoon image.

In addition, an object of the present invention is to provide a deep learning-based image domainization method and apparatus capable of quickly, accurately and precisely domainizing a character face in a cartoon image.

In addition, an object of the present invention is to provide a deep learning-based image segmentation method and apparatus capable of segmenting a cartoon image by training a neural network without a training data set for a cartoon image.

In addition, an object of the present invention is to provide a deep learning-based image segmentation method and an apparatus for constructing a deep learning model capable of segmenting a cartoon image using a training data set for a real image.

The deep learning-based image segmentation apparatus according to the embodiment includes: a communication unit receiving a training data set including a target domain image as a cartoon image, a source domain image as a real image, and a correct answer label in which a face in the source domain image is region; A storage unit for storing a deep learning model; And a processor that trains the deep learning model to transform a face into an image region using the training data set, and performs face regionization of the target domain image using the trained deep learning model. The processor extracts a common content space from the source domain image using the deep learning model and extracts a common content space from the target domain image, and the common content space extracted from the source domain image is a face region of the source domain image. And the common content space extracted from the target domain image is a feature expression representing the face area of the target domain image, and the processor uses the deep learning model to extract the common content space from the source domain image. The training may be performed to make the image region, and the face region of the target domain image may be performed using a common content space extracted from the target domain image.

In addition, the deep learning model may include a source content encoder for extracting a common content space from the source domain image; A target content encoder for extracting a common content space from the target domain image; A domainization unit for performing image domainization using a common content space extracted from the source domain image or a common content space extracted from the target domain image; And a domain classifying unit for distinguishing whether the domain classifying unit is a common content space extracted from the source domain image or the common content extracted from the target domain image, wherein the processor comprises: the target Training to determine whether it is a common content space extracted from a content encoder, and the source content encoder and the target content encoder determine whether the domain classification unit is a common content space extracted from the source content encoder or a common content space extracted from the target content encoder. You can train to be indistinguishable.

In addition, the deep learning model may include: a source attribute encoder for extracting a specific attribute space from the source domain image; A source generator for generating a source virtual image using a specific attribute space and a common content space extracted from the source domain image; And a source determination unit configured to determine whether the source virtual image generated by the source generation unit is real or fake by comparing it with the source domain image, wherein the processor further comprises: the source determination unit to determine whether the source virtual image is fake or real. After training, the source generation unit may train so that the source virtual image cannot be determined by the source determining unit whether the source virtual image is real or fake.

In addition, the deep learning model may include a target attribute encoder for extracting a specific attribute space from the target domain image; A target generation unit generating a target virtual image using a specific attribute space and a common content space extracted from the target main image; And a target determination unit for determining whether the target virtual image generated by the target generation unit is real or fake by comparing with the target domain image, wherein the processor further comprises: the target determination unit to determine whether the target virtual image is fake or real. After training, the target generation unit may train so that the target virtual image cannot be determined whether the target virtual image is real or fake.

In addition, the source attribute encoder is a convolutional neural network (CNN), the target attribute encoder is a convolutional neural network (CNN), the original source content encoder is a fully convolutional network (FCN), and the target content encoder is a fully convolutional network. It is a network (FCN), the source generation unit is composed of a plurality of deconvolutional layers, the target generation unit is composed of a plurality of deconvolutional layers, the domainization unit is a fully convolutional network (FCN), and the pre-domain classification unit is convolutional It may be a neural network (CNN).

Further, the processor may train the deep learning model using domain hostile loss, self-recovery loss, image hostile loss, or segmentation loss.

In addition, the processor may train the deep learning model using a cross cycle loss or a semantic consistency loss.

The deep learning-based image segmentation method according to the embodiment is a deep learning method that performs face segmentation of a cartoon image by an image segmentation device including a communication unit, a storage unit storing the deep learning model, and a processor driving the deep learning model. A learning-based image segmentation method, comprising: receiving, by the communication unit, a target domain image that is the cartoon image; Receiving, by the communication unit, a training data set including a source domain image that is a photorealistic image and a correct answer label in which a face in the source domain image is regioned; Training, by the processor, the deep learning model to regionize the face of the target domain image using the training data set; Completing the training, by the Singgi processor, constructing an image domain-ized deep learning model: and the processor inputting the target domain image into the image-zoned deep learning model to localize a face of the target domain image; Including, wherein the processor extracts a common content space from the source domain image using the deep learning model, extracts a common content space from the target domain image, and the common content space extracted from the source domain image is the source domain A feature expression representing a face area of an image, and the common content space extracted from the target domain image is a feature expression representing a face area of the target domain image, and the processor includes the deep learning model extracted from the source domain image. It is possible to train the image region using space, and perform face regionization of the target domain image using a common content space extracted from the target domain image.

In addition, the deep learning model may include a source content encoder for extracting a common content space from the source domain image; A target content encoder for extracting a common content space from the target domain image; A domainization unit for performing image domainization using a common content space extracted from the source domain image or a common content space extracted from the target domain image; And a domain classifying unit for distinguishing whether the domain classifying unit is a common content space extracted from the source domain image or the common content extracted from the target domain image, wherein the processor comprises: the target Training to discriminate whether it is a common content space extracted from a content encoder, and the source content encoder and the target content encoder determine whether the domain classification unit is a common content space extracted from the source content encoder or a common content space extracted from the target content encoder. You can train to be indistinguishable.

In addition, the deep learning model may include: a source attribute encoder for extracting a specific attribute space from the source domain image; A source generator for generating a source virtual image using a specific attribute space and a common content space extracted from the source domain image; And a source determination unit configured to determine whether the source virtual image generated by the source generation unit is real or fake by comparing it with the source domain image, wherein the processor further comprises: the source determination unit to determine whether the source virtual image is fake or real. After training, the source generation unit may train so that the source virtual image cannot be determined by the source determining unit whether the source virtual image is real or fake.

In addition, the deep learning model may include a target attribute encoder for extracting a specific attribute space from the target domain image; A target generation unit generating a target virtual image using a specific attribute space and a common content space extracted from the target main image; And a target determination unit for determining whether the target virtual image generated by the target generation unit is real or fake by comparing with the target domain image, wherein the processor further comprises: the target determination unit to determine whether the target virtual image is fake or real. After training, the target generation unit may train so that the target virtual image cannot be determined whether the target virtual image is real or fake.

In addition, the source attribute encoder is a convolutional neural network (CNN), the target attribute encoder is a convolutional neural network (CNN), the source content encoder is a fully convolutional network (FCN), and the target content encoder is a fully convolutional neural network It is a network (FCN), the source generation unit is composed of a plurality of deconvolutional layers, the target generation unit is composed of a plurality of deconvolutional layers, the segmentation unit is a fully convolutional network (FCN), and the domain classification unit is convolutional It may be a neural network (CNN).

Further, the processor may train the deep learning model using domain hostile loss, self-recovery loss, image hostile loss, or segmentation loss.

In addition, the processor may train the deep learning model using a cross cycle loss or a semantic consistency loss.

The deep learning-based image segmentation method and the apparatus according to the embodiment may segment a character face to change a character face in a cartoon image.

In addition, the deep learning-based image segmentation method and the apparatus according to the embodiment can quickly, accurately and precisely segment a character face in a cartoon image.

In addition, the deep learning-based image segmentation method and the apparatus according to the embodiment may train a neural network without a training data set for the cartoon image to segment the cartoon image.

In addition, the deep learning-based image segmentation method and the apparatus according to the embodiment may construct a deep learning model capable of segmenting a cartoon image using a training data set for a real image.

1 is a conceptual diagram of a deep learning-based image segmentation system according to an embodiment of the present invention.
2 is an internal block diagram of a user terminal for executing an application requesting a deep learning-based image segmentation function according to an embodiment of the present invention.
3 is an internal block diagram of a machine learning server for executing an application having an image segmentation function based on deep learning according to an embodiment of the present invention.
4 is a diagram illustrating a face region in the source domain image and the target domain image, respectively.
5 is a diagram for explaining the structure of a deep learning model according to an embodiment of the present invention.
6 is a view for explaining a state of training a deep learning model according to an embodiment of the present invention.
7 is a diagram illustrating an image region of a target domain according to an embodiment of the present invention using a deep learning model.
8 is a diagram illustrating a loss for training a deep learning model according to an exemplary embodiment.
9 is a diagram for explaining a loss for training a deep learning model according to another embodiment.
10 is a diagram illustrating a comparison between an image segmentation result by a deep learning model according to an embodiment of the present invention and an image segmentation result by a deep learning model according to the prior art.
11 is a signal flow diagram of a deep learning-based image segmentation system according to an embodiment of the present invention.
12 is a deep learning-based image segmentation method according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will be apparent with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are not used in a limiting meaning, but for the purpose of distinguishing one component from another component. In addition, expressions in the singular include plural expressions unless the context clearly indicates otherwise. In addition, terms such as include or have means that the features or elements described in the specification are present, and do not preclude the possibility of adding one or more other features or elements in advance. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, and the present invention is not necessarily limited to what is shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted. .

1 is a conceptual diagram of a deep learning-based image segmentation system according to an embodiment of the present invention, and FIG. 2 is a user terminal for executing an application requesting a deep learning-based image segmentation function according to an exemplary embodiment of the present invention. 3 is an internal block diagram of a machine learning server for executing an application with a deep learning-based image segmentation function according to an embodiment of the present invention, and FIG. 4 is a source domain image and a target domain image. Each of the faces in is a regionalized appearance.

Referring to FIG. 1, a deep learning-based image segmentation system according to an embodiment may include a terminal 100, an image providing server 200, and a machine learning server 300.

Each component of FIG. 1 may be connected through a network. It refers to a connection structure in which information can be exchanged between nodes such as the terminal 100, the image providing server 200, and the machine learning server 300, and the like. An example of such a network is a 3GPP (3rd Generation Partnership Project) network. , Long Term Evolution (LTE) network, World Interoperability for Microwave Access (WIMAX) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and a Digital Multimedia Broadcasting (DMB) network, but are not limited thereto.

<Terminal (100)>

First, the terminal 100 is a terminal 100 of a user who wants to receive a face editing service for a cartoon image. In addition, the terminal 100 may access the machine learning server 300 through wired or wireless communication and may be provided with an image segmentation function for segmenting a face of a cartoon image. In addition, the terminal 100 may edit the character's face using the provided regionized image.

The terminal 100 is not limited to a specific terminal 100 if it is a terminal 100 for executing an application having a function of editing a face of a cartoon image. For example, in the terminal 100, a smart phone, a mobile phone, a laptop computer, a digital broadcasting terminal, a personal digital assistants (PDA), a portable multimedia player (PMP), a navigation system, and a tablet PC (tablet PC). PC), an ultrabook, a wearable device, a smart glass or a kiosk, etc. may be included.

Such a terminal 100 may include a data processing processor and a memory in which a cartoon image face editing program is installed, and the processor reads the cartoon image face editing program to provide a user with a face editing function of the cartoon image. can do.

In detail, the user executes a face editing program of the cartoon image through the terminal 100, selects a cartoon image to change the face, and transmits the selected cartoon image to the machine learning server 300, the machine learning server 300 ), and an editing function such as changing the skin color, hair style, and hair color of the regionized face, or changing to another face shape, may be provided.

-The internal unit of the terminal 100

Hereinafter, a detailed configuration of the terminal 100 will be described in detail with reference to FIG. 2.

The terminal 100 according to the embodiment may include an input unit 110, a display 140, a communication unit 120, a memory 130, and a processor 150.

First, the terminal 100 may include an input unit 110 that senses a user's input. For example, the input unit 110 may detect an execution input that turns on/off the power of the terminal 100, settings for various functions of the terminal 100, an execution input, etc. .

The input unit 110 is a touch input unit that senses a user's touch (eg, a touch sensor, a touch key, a push key, etc.) and a microphone that detects a voice input. By including at least one or more of, a user input may be detected.

In addition, the terminal 100 may include a display 140 that displays a progress screen of a face editing function of a cartoon image. The display 140 may be implemented as a touch screen by forming a layer structure or integrally with the touch sensor. Such a touch screen may function as a user input unit 110 that provides an input interface between the terminal 100 and a user, and may provide an output interface between the terminal 100 and a user.

In addition, the display 140 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display ( flexible display), a 3D display, and an e-ink display.

The user can select a hairstyle, hair color, skin color, face shape, etc. to be converted through such a touch screen, and can check the cartoon image in which the face has been converted as selected.

In addition, the terminal 100 may include a communication unit 120 for wireless communication with the machine learning server 300.

In detail, the terminal 100 may transmit a cartoon image (hereinafter, a target domain image (TD)) to the machine learning server 300 through the communication unit 120. Conversely, the terminal 100 may receive an image in which a face is region-coded from the machine learning server 300 through the communication unit 120.

In detail, the communication unit 120, the following communication methods (for example, GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access)) , LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network) Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), a wireless signal can be transmitted and received with at least one of a base station, an external terminal, and a server on a network.

In addition, the memory 130 may store a plurality of application programs or applications driven in the terminal 100, data for operation of the terminal 100, and commands.

In detail, the cartoon image face editing program may be stored in the memory 130, installed on the terminal 100, and driven by the processor 150 to perform an operation (or function) of the terminal 100. have.

In addition, the memory 130 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware, and the memory 130 performs a storage function of the memory 130 on the Internet. It may be a web storage that performs.

Finally, it may include a processor 150 that performs data processing for providing a face editing service of a cartoon image by controlling the overall operation of each unit.

These processors 150 include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, and microcontrollers. Controllers (micro-controllers), microprocessors (microprocessors), may be any type of processor for performing other functions.

<Image providing server (200)>

The image providing server 200 may provide a training data set to the machine learning server 300. More specifically, the training data set may include a source domain image and Ground Truth Labels for the source domain image. Referring to FIG. 4, the source domain image SD may be a real image including a human face. The correct answer label may be label data in which a face is correctly region-coded in the source domain image SD. For example, the image providing server 200 may store a training data set for a vast amount of source domain images SD previously disclosed in a database. As another example, the image providing server 200 may generate a new training data set by constructing a deep learning neural network for regionalizing a face of a new source domain image SD in the data processing unit.

The image providing server 200 may include a communication module for communicating with the terminal 100, a data processing unit for processing data, and a database storing various types of data.

<Machine Learning Server (300)>

The machine learning server 300 may include a separate cloud server or computing device. In addition, the machine learning server 300 may be a neural network system installed in the processor of the terminal 100 or the data processing unit of the image providing server 200, but hereinafter the machine learning server 300 provides the terminal 100 or an image It will be described as a separate device from the server 200.

Referring to FIG. 3, the machine learning server 300 may receive a target domain image TD from the terminal 100 through the communication unit 310 and receive a training data set from the image providing server 200. Referring to FIG. 3, the target domain image TD may be a cartoon image including a character face input by a user into the terminal 100. As described above, the training data set may include a source domain image SD including a face of a real person and a correct answer label in which the face of the source domain image SD is regioned. Since the target domain image TD is a cartoon image and the source domain image SD is a real image, domains are different from each other.

The machine learning server 300 may build an image segmentation deep learning model using a training data set including a source domain image SD, and perform image segmentation on a target domain image TD. As an example, referring to FIG. 4, the machine learning server 300 repeatedly compares a source domain image (SS) generated by inputting a source domain image (SD) to a deep learning neural network and a correct answer label to provide a deep learning neural network. You can learn enough to make an image area. The machine learning server 300 inputs a target domain image (TD) to a deep learning neural network (that is, an image localization deep learning model) that has been trained to localize an image to perform image domainization, thereby generating a target domainization image (TS). Can be generated. Accordingly, the present invention can implement zero-shot learning capable of regionalizing the target domain image TD by training a neural network without a training data set for the target domain image TD. A detailed description of training by the machine learning server 300 using the image segmentation deep learning model follows the description of the deep learning model of FIGS. 5 to 12.

In addition, the machine learning server 300 may include a processor 330 that processes data, and a storage unit 320 that stores an image segmentation deep learning model for performing image segmentation deep learning. 330 reads the image segmentation deep learning model, and performs image segmentation deep learning on the target domain image (TD) described below according to the constructed neural network system.

Depending on the embodiment, the processor 330 may include a main processor that controls all units, and a plurality of graphics processing units (GPUs) that process a large amount of computation required when driving a neural network according to image region deep learning. Can be configured.

5 is a diagram for explaining the structure of a deep learning model according to an embodiment of the present invention, FIG. 6 is a diagram for explaining a state of training a deep learning model according to an embodiment of the present invention, and FIG. 7 This is an image region of a target domain according to an embodiment of the present invention using a deep learning model.

Referring to FIG. 5, a deep learning model according to an embodiment of the present invention may include a plurality of encoders. The encoder may extract a shared content space or a specific attribute space from the input domain image. The common content space may be a feature vector or a feature representation for image segmentation. For example, a feature expression representing a face area of a source domain image (realistic image) and a feature expression representing a face area of a target domain image (cartoon image) may be a common content space. Also, the common content space may be a shared feature representation present in different domain images. The specific attribute space may be a characteristic expression specialized for a specific domain. For example, the background, facial skin tone, color, texture, etc. of the source domain image may be a specific attribute space. In addition, the face skin tone, color, texture, etc. of the target domain image may be a specific attribute space. That is, the specific attribute space may be a feature expression that is not used for image regionization.

The plurality of encoders may include a plurality of content encoders. The plurality of content encoders may include a source content encoder 11 and a target content encoder 13. The source content encoder 11 and the target content encoder 13 may have separate components. In addition, the content encoder may extract a common content space for image area of the face from the input domain image. For example, the source content encoder 11 may extract a common content space for image region of a face from the source domain image. The target content encoder 13 may extract a common content space for regionizing a face image from the target domain image. Also, the content encoder may be configured in a neural network structure. As an example, the content encoder may be a Fully Convolutional Network (FCN). For example, the content encoder may have a form in which several 3X3 convolutional layers are superimposed, a 64X64X3 size RGB image (domain image) may be input as an input, and a feature expression may be output as an output.

The plurality of encoders may include a plurality of attribute encoders. The plurality of attribute encoders may include a source attribute encoder 12 and a target attribute encoder 14. The source attribute encoder 12 and the target attribute encoder 14 may have separate configurations. In addition, the attribute encoder may extract a specific attribute space unnecessary for image regionization from the input domain image. For example, the source attribute encoder 12 may extract a specific attribute space such as background, facial skin tone, color, and texture from the source domain image. The target attribute encoder 14 may extract a specific attribute space such as background, facial skin tone, color, and texture from the target domain image. In addition, the attribute encoder may be configured in a neural network structure. As an example, the attribute encoder is a convolution neural network (CNN). For example, the attribute encoder may have a form in which several 3X3 convolution layers are superimposed and a fully connected layer is superimposed on the last layer, and a 64X64X3 size RGB image (domain image) is input as an input. , A feature expression may be output as an output.

The deep learning model according to an embodiment of the present invention may include a domain classifier 40. The domain classifying unit 40 may discriminate which domain the feature map input from the plurality of content encoders has come to. As an example, the domain classifier 40 may determine whether the input common content space is a common content space output from the source content encoder 11 or a common content space output from the target content encoder 13. As will be described later, the domain classifier 40 may be trained to discern which domain of the common content space, and the plurality of content encoders may be trained to prevent the domain classifier 40 from discriminating which domain of the common content space. That is, the domain classification unit 40 and the plurality of content encoders competently train each other based on a Generative Adversarial Network (GAN). For reference, GAN consists of a generator that generates a virtual data sample and a discriminator that determines whether the input data sample is real data, and performs adversarial training of the generator and the discrimination period. It refers to a machine learning model built through. Those skilled in the field of machine learning will already be familiar with the basic contents of GAN, so further description of GAN will be omitted. In addition, the domain classification unit 40 may be configured in a neural network structure. For example, the domain classifier 40 is a convolution neural network (CNN). For example, the domain classification unit 40 is a convolutional neural network having one output node, and may be a form in which a fully connected layer is superimposed on the last layer, and a common content space is input as an input, As an output, a sigmoid probability value having a range between 0 and 1 may be output. For example, if the sigmoid probability value is greater than 0.5, the domain classifier 40 determines it as a common content space output from the source content encoder, and if the sigmoid probability value is less than 0.5, the domain classifier 40 determines the common content space output from the target content encoder. I can judge. Through GAN-based training, the domain classifier 40 may output 0.5 even if a common content space is input from any content encoder. That is, the domain classification unit 40 cannot distinguish which domain image is the extracted feature map. In other words, the common content space output from the content encoders of different domains may be determined that the domain classifier 40 outputs the content encoders of the same domain.

The deep learning model according to an embodiment of the present invention may include a plurality of generation units. The plurality of generation units may include a source generation unit 21 and a target generation unit 22. The generator may generate a virtual image by receiving a specific property space of the property encoder and a common content space of the content encoder. For example, the source generator 21 may generate the source virtual image SF using a specific attribute space of the source attribute encoder 12 and a common content space of the source content encoder 11. The target generator 22 may generate the target virtual image TF using a specific attribute space of the target attribute encoder 14 and a common content space of the target content encoder 13. In addition, the generation unit may be configured in a neural network structure. As an example, the generation unit may be a network composed of a plurality of deconvolutional layers, and may output an RGB image with a feature expression concatenated with an output value of a content encoder and an attribute encoder as an input.

The deep learning model according to an embodiment of the present invention may include a plurality of discrimination units. The plurality of judgment units may include a source determination unit 31 and a target determination unit 32. The source determination unit 31 and the target determination unit 32 may be configured separately from each other. The determination unit may determine whether the virtual image generated by the generation unit is real or fake. The determination unit may help the virtual image generated by the generation unit to output a realistic and good quality image. As will be described later, the determination unit may be trained to determine whether the virtual image generated by the generation unit is fake, and the generation unit may be trained to determine whether the virtual image generated by the generation unit is genuine. In other words, the generation unit and the discrimination unit are training each other competitively based on GAN. As an example, the source determination unit 31 is trained to determine whether the source virtual image SF generated by the source generation unit 21 is fake or real, and the source generation unit 21 generates the generated source virtual image SF It may be trained so that the source determination unit 31 cannot determine whether it is real or fake. Similarly, the target determination unit 32 is trained to determine whether the target virtual image TF generated by the target generation unit 22 is fake or real, and the target generation unit 22 uses the generated target virtual image TF It may be trained so that the determination unit 32 cannot determine whether it is real or fake. In addition, the discrimination unit may be configured in a neural network structure. For example, the determination unit may be configured in a neural network structure. For example, the determination unit may be a convolution neural network (CNN). For example, the discrimination unit may be a convolutional neural network with one output node, and a fully connected layer may be superimposed on the last layer, and the source domain image (SD) as input and the output from the generation unit A virtual image is input, and a sigmoid probability value having a range between 0 and 1 may be output as an output.

The deep learning model according to an embodiment of the present invention may include a domainization unit 50. The domainization unit 50 may convert the target domain image into an image area when the target domain image is input after training to enable image domainization using the training data set for the source domain image. That is, the content encoder is trained in the direction of deceiving the domain classifier 40 so that the common content space output from the source content encoder 11 and the common content space output from the target content encoder 13 are Can be placed on the same line. The zoning unit 50 may receive a domain-invariant common content space as an input and output a zoning image obtained by performing image zoning. As an example, referring to FIG. 5, the source content encoder 11 may extract a common content space from the source domain image SD and input it to the region converter 50. The segmentation unit 50 performs image segmentation using the input common content space to generate a source segmentation image (SS), and compares the image segmentation result value and ground truth labels to correctly segment the image. Can be trained to perform. Referring to FIG. 6, the sufficiently trained regionization unit 50 receives the common content space extracted from the target domain image TD by the target content encoder 13 and performs image regionization to generate the target regionization image TS. Can be generated. In addition, the segmentation unit 50 may be configured in a neural network structure. As an example, the domainization unit 50 may be a Fully Convolutional Network (FCN).

The deep learning model according to an embodiment of the present invention may be trained to perform image segmentation using the loss described in FIGS. 8 and/or 9. As an example, the deep learning model may be trained by back-propagation of the loss described below to the corresponding configuration.

8 is a diagram illustrating a loss for training a deep learning model according to an exemplary embodiment.

)을 이용할 수 있다. 도메인 적대 손실(

)은 소스 컨텐츠 인코더(11)에서 출력한 공통 컨텐츠 공간과 타겟 컨텐츠 인코더(13)에서 출력한 공통 컨텐츠 공간이 서로 구분 불가능하도록 같은 잠재 공간(Latent Space) 선상에 놓이도록 하는 손실이다. 즉, 도메인 분류부(40)는 분류 정확도를 최대화하는 방향으로, 소스 컨텐츠 인코더(11)와 타겟 컨텐츠 인코더(13)는 분류 정확도를 최소화하는 방향으로 트레이닝 될 수 있다. 도메인 적대 손실(

)은 수학식 1과 같다.8, a deep learning model according to an embodiment includes Domain Adversarial Loss during training;

) Can be used. Domain hostility loss (

) Is a loss that causes the common content space output from the source content encoder 11 and the common content space output from the target content encoder 13 to be placed on the same latent space line so that they cannot be distinguished from each other. That is, the domain classifier 40 may be trained in a direction to maximize the classification accuracy, and the source content encoder 11 and the target content encoder 13 may be trained in a direction to minimize the classification accuracy. Domain hostility loss (

) Is the same as in Equation 1.

(Equation 1)

는 기대(expectation) 함수, xs는 소드 도메인 이미지(SD)이고, xt는 타겟 도메인 이미지(TD)이다.

Is the expectation function, xs is the sword domain image (SD), and xt is the target domain image (TD).

)을 이용할 수 있다. 자가 복구 손실(

)은 딥러닝 모델에 의해 생성된 가상 이미지가 입력된 도메인 이미지와 비교하여 원본 그대로 복원되도록 하는 손실이다. 일 예로, 소스 자가 복구 손실(

)은 소스 도메인 이미지(SD)와 소스 가상 이미지(SF)를 비교하여 소스 가상 이미지(SF)가 원본인 소스 도메인 이미지(SD)로 복원된 정도를 손실로 나타낸 것이다. 타겟 자가 복구 손실(

)은 타겟 도메인 이미지(TD)와 타겟 가상 이미지(TF)를 비교하여 타겟 가상 이미지(TF)가 원본인 타겟 도메인 이미지(TD)로 복원된 정도를 손실로 나타낸 것이다.A deep learning model according to an embodiment includes Self-Reconstruction Loss during training;

) Can be used. Self-healing loss (

) Is a loss that allows the virtual image generated by the deep learning model to be restored as it is by comparing it with the input domain image. For example, the source self-healing loss (

) Represents the degree to which the source virtual image SF is restored to the original source domain image SD by comparing the source domain image SD with the source virtual image SF as loss. Target self-healing loss (

) Denotes the degree to which the target virtual image (TF) is restored to the original target domain image (TD) by comparing the target domain image (TD) with the target virtual image (TF) as a loss.

)을 이용할 수 있다. 이미지 적대 손실(

)은 GAN 컨셉의 손실이며, 생성부에서 출력된 가상 이미지가 사실적이게 하는 손실이다. 예를 들어, 소스 이미지 적대 손실(

)은 소스 생성부(21)가 출력한 소스 가상 이미지(SF)를 소스 판별부(31)가 가짜라고 판별하지 못하는 정도를 손실로 나타낸 것이다. 타겟 이미지 적대 손실(

)은 타겟 생성부(22)가 출력한 타겟 가상 이미지(TF)를 타겟 판별부(32)가 가짜라고 판별하지 못하는 정도를 손실로 나타낸 것이다.The deep learning model according to an embodiment includes: Image Adversarial Loss during training;

) Can be used. Image hostility loss (

) Is the loss of the GAN concept, and is the loss that makes the virtual image output from the generation unit realistic. For example, source image hostile loss (

) Denotes the degree to which the source determining unit 31 cannot determine that the source virtual image SF output by the source generating unit 21 is a fake, as a loss. Target image hostile loss (

) Represents the degree to which the target virtual image TF output by the target generation unit 22 is not determined as a fake by the target determination unit 32 as a loss.

)을 이용할 수 있다. 시그멘테이션 손실(

)은 소스 도메인 이미지(SD)에 대한 이미지 영역화 결과가 정답 레이블(ground truth label)과 차이가 나는지에 대한 손실일 수 있다. 타겟 도메인 이미지(TD)에 대해서는 정답 레이블이 없기 때문에 소스 도메인 이미지(SD)에 대해서만 시그멘테이션 손실(

)을 적용할 수 있다. 시그멘테이션 손실(

)은 수학식 2와 같다.A deep learning model according to an embodiment includes: Segmentation Loss during training;

) Can be used. Loss of segmentation (

) May be a loss as to whether the result of image segmentation for the source domain image SD is different from the ground truth label. Since there is no correct answer label for the target domain image (TD), only the source domain image (SD) has a loss of signature (

) Can be applied. Loss of segmentation (

) Is the same as in Equation 2.

(Equation 2)

는 영역화부(50)의 출력 픽셀이고,

는 정답 레이블(ground truth label)의 픽셀을 나타낸다.In Equation 2,

Is an output pixel of the areaization unit 50,

Denotes the pixel of the ground truth label.

9 is a diagram for explaining a loss for training a deep learning model according to another embodiment.

)을 이용할 수 있다. 교차 사이클 손실(

)은 픽셀 수준의 교차 사이클 일관성 손실을 줄인 말이다. 교차 사이클 손실(

)은 딥러닝 모델의 전체 신경망의 도메인 적용(domain adaptation) 성능을 높이기 위한 손실일 수 있다. 일 도메인 이미지에 대한 콘텐츠 인코더의 출력을 타 도메인의 생성부로 스왑(swap)하면 타 도메인의 생성부는 일 도메인 이미지에서 얻은 특징 정보가 반영되어 제1 가상 이미지를 생성할 수 있다. 제1 가상 이미지를 다시 입력으로 사용하여 콘텐츠 인코더의 출력을 타 도메인의 생성부로 다시 스왑(swap)하여 생성된 제2 가상 이미지는 원본 이미지인 일 도메인 이미지로 복원되어야 한다. 일 예로, 딥러닝 모델은, 소스 속성 인코더(12)에 의해 소스 도메인 이미지(SD)에서 추출한 제1 특정 속성 공간과 타겟 컨텐츠 인코더(13)에 의해 타겟 도메인 이미지(TD)에서 추출한 제1 공통 컨텐츠 공간을 이용하여 제1 소스 가상 이미지(SF1)를 생성하고, 소스 컨텐츠 인코더(11)에 의해 소스 도메인 이미지(SD)에서 추출한 제1 공통 컨텐츠 공간과 타겟 속성 인코더(14)에 의해 타겟 도메인 이미지(TD)에서 추출한 제1 특정 속성 공간을 이용하여 제1 타겟 가상 이미지(TF1)를 생성하고, 소스 속성 인코더(12)에 의해 제1 소스 가상 이미지(SF1)에서 추출한 제2 특정 속성 공간과 타겟 컨텐츠 인코더(13)에 의해 제1 타겟 가상 이미지(TF1)에서 추출한 제2 공통 컨텐츠 공간을 이용하여 제2 소스 가상 이미지(SF2)를 생성하고, 소스 컨텐츠 인코더(11)에 의해 제1 소스 가상 이미지(SF1)에서 추출한 제2 공통 컨텐츠 공간과 타겟 속성 인코더(14)에 의해 제1 타겟 가상 이미지(TF1)에서 추출한 제2 특정 속성 공간을 이용하여 제2 타겟 가상 이미지(TF2)를 생성할 수 있다. 소스 도메인 이미지(SD)와 제2 소스 가상 이미지(SF2)와 비교한 손실을 소스 교차 사이클 손실(

)로 나타낼 수 있고, 타겟 도메인 이미지(TD)와 제2 타겟 가상 이미지(TF2)와 비교한 손실을 타겟 교차 사이클 손실(

)로 나타낼 수 있다. 전체 교차 사이클 손실(

)은 소스 교차 사이클 손실(

)과 타겟 교차 사이클 손실(

)의 합으로 나타낼 수 있으며 전체 교차 사이클 손실(

)은 수학식 3과 같다.Referring to FIG. 9, a deep learning model according to another embodiment includes: Pixel-Level Cross-Cycle Consistency Loss during training;

) Can be used. Cross cycle loss(

) Is an abbreviation for cross-cycle coherence loss at the pixel level. Cross cycle loss(

) May be a loss for improving the domain adaptation performance of the entire neural network of the deep learning model. When the output of the content encoder for one domain image is swapped with the generation unit of another domain, the generation unit of the other domain may generate a first virtual image by reflecting feature information obtained from the one domain image. The second virtual image generated by using the first virtual image as an input again and swapping the output of the content encoder back to the generation unit of another domain must be restored to a domain image that is the original image. As an example, the deep learning model includes a first specific attribute space extracted from the source domain image SD by the source attribute encoder 12 and the first common content extracted from the target domain image TD by the target content encoder 13 A first source virtual image SF1 is generated using the space, and the first common content space extracted from the source domain image SD by the source content encoder 11 and the target domain image ( A first target virtual image TF1 is generated using the first specific attribute space extracted from TD), and a second specific attribute space and target content extracted from the first source virtual image SF1 by the source attribute encoder 12 The second source virtual image SF2 is generated by using the second common content space extracted from the first target virtual image TF1 by the encoder 13, and the first source virtual image ( The second target virtual image TF2 may be generated using the second common content space extracted from SF1) and the second specific attribute space extracted from the first target virtual image TF1 by the target attribute encoder 14. The loss compared to the source domain image (SD) and the second source virtual image (SF2) is compared to the source cross cycle loss (

), and the loss compared with the target domain image TD and the second target virtual image TF2 is the target cross cycle loss (

Can be represented by ). Total cross cycle loss (

) Is the source cross-cycle loss (

) And target cross cycle loss (

It can be expressed as the sum of the total cross cycle loss (

) Is equal to Equation 3.

(Equation 3)

)을 이용할 수 있다. 시맨틱 일관성 손실(

)은 딥러닝 모델의 전체 신경망의 도메인 적용(domain adaptation) 성능을 높이기 위한 손실일 수 있다. 일 도메인 이미지에 대한 콘텐츠 인코더의 제1 공통 컨텐츠 공간 출력을 타 도메인의 생성부로 스왑(swap)하면 타 도메인의 생성부는 일 도메인 이미지에서 얻은 특징 정보가 반영되어 제1 가상 이미지를 생성할 수 있다. 제1 가상 이미지를 다시 입력으로 사용하여 타 도메인의 콘텐츠 인코더는 제2 공통 컨텐츠 공간을 출력할 수 있다. 이 때, 시맨틱 일관성 손실(

)은 타 도메인의 제2 공통 컨텐츠 공간이 일 도메인의 제1 공통 컨텐츠 공간을 그대로 유지하고 있는지에 대한 손실이다. 이러한 시맨틱 일관성 손실(

)은 신경망이 헤어 컬러, 헤어 스타일, 얼굴 윤곽과 같은 고레벨의 특징을 캡쳐할 수 있게 한다. 일 예로, 소스 컨텐츠 인코더(11)에서 출력하는 제1 공통 컨텐츠 공간과 타겟 컨텐츠 인코더(13)에서 출력하는 제2 공통 컨텐츠 공간을 비교하여 소스 도메인에서 타겟 도메인으로의 변환시의 시맨틱 일관성 손실(

)로 나타낼 수 있고, 타겟 컨텐츠 인코더(13)에서 출력하는 제1 공통 컨텐츠 공간과 소스 컨텐츠 인코더(11)에서 출력하는 제2 공통 컨텐츠 공간을 비교하여 타겟 도메인에서 소스 도메인으로의 변환시의 시맨틱 일관성 손실(

)로 나타낼 수 있다. 소스 도메인에서 타겟 도메인으로의 변환시의 시맨틱 일관성 손실(

)은 수학식 4와 같다.A deep learning model according to another embodiment may include Semantic Consistency Loss during training;

) Can be used. Loss of semantic consistency (

) May be a loss for improving the domain adaptation performance of the entire neural network of the deep learning model. When the output of the first common content space of the content encoder for one domain image is swapped to the generator of another domain, the generator of the other domain reflects feature information obtained from the one domain image to generate the first virtual image. Using the first virtual image again as an input, a content encoder of another domain may output a second common content space. At this time, the loss of semantic consistency (

) Is a loss of whether the second common content space of another domain maintains the first common content space of one domain as it is. This loss of semantic consistency (

) Allows neural networks to capture high-level features such as hair color, hairstyle, and facial contours. For example, by comparing a first common content space output from the source content encoder 11 and a second common content space output from the target content encoder 13, the loss of semantic consistency when converting from the source domain to the target domain (

), and semantic consistency loss when converting from the target domain to the source domain by comparing the first common content space output from the target content encoder 13 and the second common content space output from the source content encoder 11 (

Can be represented by ). Loss of semantic consistency when converting from source domain to target domain (

) Is the same as in Equation 4.

(Equation 4)

)은 마찬가지로 수학식 4에 적용할 수 있다.Loss of semantic consistency when converting from target domain to source domain (

) Can likewise be applied to Equation 4.

)을 이용하여 이미지 영역화를 수행하도록 트레이닝 될 수 있다. 전체 손실(

)은 수학식 5와 같다.The deep learning model according to an embodiment of the present invention includes a total loss (Full Loss) which linearly combines the losses described in FIGS. 7 and/or 8;

) Can be trained to perform image segmentation. Total loss(

) Is equal to Equation 5.

(Equation 5)

는 하이퍼 파라미터들이다. 사용자는 하이퍼 파라미터는 조절하여 각 손실의 상대적인 중요도를 조절할 수 있다. In Equation 5,

Are hyperparameters. The user can adjust the relative importance of each loss by adjusting the hyper parameter.

<Comparative Example>

10 is a diagram illustrating a comparison between an image segmentation result by a deep learning model according to an embodiment of the present invention and an image segmentation result by a deep learning model according to the prior art.

Referring to FIG. 10, a result of image domainization of a target domain image (Input), which is a cartoon image, in a deep learning model (Ours) according to an embodiment of the present invention, and a target to a network model that generally performs a semantic segmentation task. It shows the result of converting the domain image (Input) into an image area. Since there is no training data set for cartoon images for image domainization of cartoon images, the deep learning model must perform zero shot learning. Among the conventional deep learning models, U-Net could not properly learn to domainize cartoon images. . Mehran is an image domain deep learning model using domain application technology used in the medical field described in the background art. It can be seen that Mehran has some degree of image segmentation of cartoon images, but the performance is poor even with the naked eye compared to the image segmentation results of the deep learning model (Ours) of the present invention.

<Table 1>

Referring to Table 1, looking at the result of image segmentation for a cartoon image according to a deep learning model, the deep learning model (Ours) of the present invention is an average IoU (Mean Intersection over Union) than the conventional deep learning models U-Net and Mehran. ) And Pixel Accuracy are both dominant.

<Comparative example according to whether or not to separate the characteristic expression during training>

<Table 2>

Referring to Table 2, the deep learning model Ours according to an embodiment of the present invention performs training by inputting only the common content space of the real image into the domainization unit 50. Comparing the image domainization result of the cartoon image and the present invention in a deep learning model (Baseline) in which training was performed by inputting both the common content space and the specific property space of the real image into the domainization unit 50, the present invention (Ours ), it can be seen that both average IoU (Mean Intersection over Union) and pixel accuracy are superior to the comparative deep learning model (Bseline). This makes it possible to more efficiently perform image segmentation for a face because the content feature has more structural information than the attribute feature.

<Comparative example according to loss>

<Table 3>

Referring to Table 3, the degree to which the deep learning model for image segmentation is affected by the loss function during training was tested. That is, the deep learning model was trained by removing the effect of a specific loss from the total loss. As a result, it can be seen that the performance of the deep learning model is excellent during training considering all losses.

Accordingly, the deep learning-based image segmentation apparatus according to an embodiment can quickly, accurately, and precisely segment a character face in a cartoon image. In addition, the deep learning-based image segmentation apparatus according to the embodiment may train a neural network without a training data set for the cartoon image to segment the cartoon image. In addition, the deep learning-based image segmentation apparatus according to an embodiment may construct a deep learning model capable of segmenting a cartoon image using a training data set for a real image.

11 is a signal flow diagram of a deep learning-based image segmentation system according to an embodiment of the present invention.

Referring to FIG. 11, the terminal 100 may acquire a target domain image TD (S11). The target domain image TD may be a cartoon image including a character face. The terminal 100 may transmit the target domain image TD to the machine learning server 300 and request the image region for the target domain image TD (S12). The machine learning server 30 may request a training data set from the image providing server 200 upon receiving a request for image regionization of the target domain image TD (S13). The training data set may be data in which a real image including a person's face and a correct answer label of the corresponding real image are paired. The image providing server 200 may generate a training data set upon receiving a request for a training data set (S14). For example, the image providing server 200 may load a public training data set stored in a database. As another example, the image providing server 200 is a data set in which the real image and the correct answer label are paired by using the image segmentation image generated by performing image segmentation on the real image using the image segmentation deep learning model as the correct answer label. Can be created. The image providing server 200 may transmit the training data set to the machine learning server 300. The machine learning server 300 may train using the received training data set and a target domain image (TD) to generate an image regionized deep learning model (S16). Training of the deep learning model follows the description of training the deep learning model of FIGS. 5 to 10. The machine learning server 300 may perform image segmentation on the target domain image TD when the image segmentation deep learning model is built (S17). The machine learning server may transmit the target regionized image TS to the terminal 100 (S18). The terminal 100 may edit the face region of the cartoon image by using the target regionized image TS (S19).

12 is a deep learning-based image segmentation method according to an embodiment of the present invention.

Referring to FIG. 12, the deep learning-based image segmentation method according to an embodiment of the present invention includes the step of receiving a target domain image (TD) from the terminal 100 by the machine learning server 300 (S1102). I can.

The deep learning-based image segmentation method according to an embodiment of the present invention may include the step (S1101) of receiving a training data set from the image providing server 200 by the machine learning server 300.

The deep learning-based image segmentation method according to an embodiment of the present invention may include training, by the machine learning server 300, the deep learning model to image segmentation (S1103). Training of the deep learning model follows the description of training the deep learning model of FIGS. 5 to 10.

The deep learning-based image segmentation method according to an embodiment of the present invention may include the step (S1104) of constructing a deep learning model for regionalizing a target domain image when the machine learning server 300 sufficiently learns through training. have.

In the deep learning-based image segmentation method according to an embodiment of the present invention, the step of performing segmentation of the target domain image by inputting a target domain image (TD) to a deep learning model in which the machine learning server 300 is built (S1105). ) Can be included. In this case, the machine learning server 300 may generate the target regionized image TS.

The deep learning-based image segmentation method according to an embodiment of the present invention may include a step S1106 of transmitting, by the machine learning server 300, a target segmentation image TS to the terminal 100.

Accordingly, the deep learning-based image segmentation method according to an embodiment can quickly, accurately and precisely segment a character face in a cartoon image. In addition, the deep learning-based image segmentation method according to an embodiment may transform a cartoon image by training a neural network without a training data set for the cartoon image. In addition, the deep learning-based image segmentation method according to an embodiment may construct a deep learning model capable of segmenting a cartoon image using a training data set for a real image.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the computer-readable recording medium may be specially designed and configured for the present invention or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks. medium), and a hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be changed to one or more software modules to perform the processing according to the present invention, and vice versa.

The specific implementations described in the present invention are examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections. It may be referred to as a connection, or circuit connections. In addition, if there is no specific mention such as “essential” or “importantly”, it may not be an essential component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, the spirit of the present invention described in the claims to be described later if one skilled in the art or those of ordinary skill in the art And it will be appreciated that various modifications and changes can be made to the present invention within a range not departing from the technical field. Therefore, the technical scope of the present invention should not be limited to the content described in the detailed description of the specification, but should be determined by the claims.

100 terminals
200 image serving servers
300 machine learning server
11 source content encoder
12 source attribute encoder
13 target content encoder
14 target attribute encoder
21 source generator
22 Target generator
31 Source identification unit
32 Target determination unit
40 domain classification
50 area
SD source domain image
TD target main image
SS source zoning image
TS target zoning image
SF source virtual image
TF target virtual image

Claims (14)

A communication unit receiving a training data set including a target domain image that is a cartoon image, a source domain image that is a real-life image, and a correct answer label in which a face in the source domain image is region;
A storage unit for storing a deep learning model; And
A processor that trains the deep learning model to transform a face into an image region using the training data set, and performs a face regionization of the target domain image using the trained deep learning model; and
The processor extracts a common content space from the source domain image using the deep learning model and extracts a common content space from the target domain image,
The common content space extracted from the source domain image is a feature expression representing a face area of the source domain image,
The common content space extracted from the target domain image is a feature expression representing a face area of the target domain image,
The processor trains the deep learning model to image region using a common content space extracted from the source domain image, and performs face regionization of the target domain image using a common content space extracted from the target domain image. Deep learning-based image segmentation device, characterized in that. The method of claim 1,
The deep learning model,
A source content encoder for extracting a common content space from the source domain image;
A target content encoder for extracting a common content space from the target domain image;
A domainization unit for performing image domainization using a common content space extracted from the source domain image or a common content space extracted from the target domain image; And
A domain classifying unit for discriminating whether a common content space extracted from the source domain image or a common content extracted from the target domain image,
The processor trains the domain classification unit to discriminate whether it is a common content space extracted from the source content encoder or a common content space extracted from the target content encoder, and the source content encoder and the target content encoder include the domain classification unit. An image segmentation apparatus based on deep learning, characterized in that training not to discriminate whether it is a common content space extracted from a source content encoder or a common content space extracted from the target content encoder. The method of claim 2,
The deep learning model,
A source attribute encoder for extracting a specific attribute space from the source domain image;
A source generator for generating a source virtual image using a specific attribute space and a common content space extracted from the source domain image; And
The source virtual image generated by the source generation unit further comprises a source determining unit for determining whether the source virtual image is real or fake by comparing with the source domain image,
Wherein the processor trains the source determination unit to determine whether the source virtual image is fake or real, and trains the source generation unit to be unable to determine whether the source virtual image is real or fake by the source determination unit. Image areaization device based on deep learning. The method of claim 3,
The deep learning model,
A target attribute encoder for extracting a specific attribute space from the target domain image;
A target generation unit for generating a target virtual image using a specific attribute space and a common content space extracted from the target domain image; And
The target virtual image generated by the target generation unit further comprises a target determination unit for determining whether the target virtual image is real or fake by comparing it with the target domain image,
Wherein the processor trains the target determination unit to determine whether the target virtual image is fake or real, and trains the target generation unit so that the target virtual image cannot determine whether the target virtual image is real or fake Image areaization device based on deep learning. The method of claim 4,
The source attribute encoder is a convolutional neural network (CNN),
The target attribute encoder is a convolutional neural network (CNN),
The source content encoder is a Fully Convolutional Network (FCN),
The target content encoder is a fully convolutional network (FCN),
The source generator is composed of a plurality of deconvolution layers,
The target generator is composed of a plurality of deconvolution layers,
The segmentation unit is a fully convolutional network (FCN),
The domain classification unit is a deep learning-based image segmentation apparatus, characterized in that the convolutional neural network (CNN). The method of claim 4,
The processor trains the deep learning model using domain hostile loss, self-recovery loss, image hostile loss, or segmentation loss. The method of claim 6,
Wherein the processor trains the deep learning model using cross cycle loss or semantic coherence loss. In a deep learning-based image segmentation method for performing face segmentation of a cartoon image by an image segmentation apparatus including a communication unit, a storage unit storing a deep learning model, and a processor that drives the deep learning model,
Receiving, by the communication unit, a target domain image that is the cartoon image;
Receiving, by the communication unit, a training data set including a source domain image that is a photorealistic image and a correct answer label in which a face in the source domain image is regioned;
Training, by the processor, the deep learning model to regionize the face of the target domain image using the training data set;
Singgi processor completing the training to build an image domain deep learning model: And
Including, by the processor, inputting the target domain image to the image segmentation deep learning model to segment the face of the target domain image
The processor extracts a common content space from the source domain image using the deep learning model and extracts a common content space from the target domain image,
The common content space extracted from the source domain image is a feature expression representing a face area of the source domain image,
The common content space extracted from the target domain image is a feature expression representing a face area of the target domain image,
The processor trains the deep learning model to image region using a common content space extracted from the source domain image, and performs face regionization of the target domain image using a common content space extracted from the target domain image. Deep learning-based image segmentation method, characterized in that. The method of claim 8,
The deep learning model,
A source content encoder for extracting a common content space from the source domain image;
A target content encoder for extracting a common content space from the target domain image;
A domainization unit for performing image domainization using a common content space extracted from the source domain image or a common content space extracted from the target domain image; And
A domain classifying unit for discriminating whether a common content space extracted from the source domain image or a common content extracted from the target domain image,
The processor trains the domain classification unit to discriminate whether it is a common content space extracted from the source content encoder or a common content space extracted from the target content encoder, and the source content encoder and the target content encoder include the domain classification unit. A deep learning-based image segmentation method, characterized in that training is performed not to discriminate whether it is a common content space extracted from a source content encoder or a common content space extracted from the target content encoder. The method of claim 9,
The deep learning model,
A source attribute encoder for extracting a specific attribute space from the source domain image;
A source generator for generating a source virtual image using a specific attribute space and a common content space extracted from the source domain image; And
The source virtual image generated by the source generation unit further comprises a source determining unit for determining whether the source virtual image is real or fake by comparing with the source domain image,
Wherein the processor trains the source determination unit to determine whether the source virtual image is fake or real, and trains the source generation unit to be unable to determine whether the source virtual image is real or fake by the source determination unit. Image areaization device based on deep learning. The method of claim 10,
The deep learning model,
A target attribute encoder for extracting a specific attribute space from the target domain image;
A target generation unit for generating a target virtual image using a specific attribute space and a common content space extracted from the target domain image; And
The target virtual image generated by the target generation unit further comprises a target determination unit for determining whether the target virtual image is real or fake by comparing it with the target domain image,
Wherein the processor trains the target determination unit to determine whether the target virtual image is fake or real, and trains the target generation unit so that the target virtual image cannot determine whether the target virtual image is real or fake Image areaization method based on deep learning. The method of claim 11,
The source attribute encoder is a convolutional neural network (CNN),
The target attribute encoder is a convolutional neural network (CNN),
The source content encoder is a Fully Convolutional Network (FCN),
The target content encoder is a fully convolutional network (FCN),
The source generator is composed of a plurality of deconvolution layers,
The target generator is composed of a plurality of deconvolution layers,
The segmentation unit is a fully convolutional network (FCN),
The domain classification unit is a deep learning-based image segmentation method, characterized in that the convolutional neural network (CNN). The method of claim 11,
The processor trains the deep learning model using domain hostile loss, self-recovery loss, image hostile loss, or segmentation loss. The method of claim 13,
The processor trains the deep learning model using a cross cycle loss or a semantic coherence loss.

Images