KR102464704B1 - Method for posting data and apparatus for posting data - Google Patents

Abstract

The present invention relates to a data posting method to protect privacy of a user, and a data posting apparatus. The data posting method comprises the steps of allowing a server to: receive first data, which includes image data or video data including eye image data, from a user terminal; detect a first eye image from the first data; use a decision neural network analyzing the first eye image to determine whether a reflected scene exists in the first eye image; generate a second eye image in which the reflected scene is removed in response to determining that the reflected scene exists; synthesize the second eye image with the first data to generate second data from which the reflected scene is removed; and post an image or video corresponding to the second data.

Description

Data Posting Methods and Data Posting Devices

The present disclosure relates to a data posting method and a data posting device, and more particularly, to a data posting method and data posting device for protecting privacy by correcting and posting data.

Due to the recent development of camera resolution, the quality of photos and videos is improved, and it is possible to extract information by enlarging or restoring a small image with the development of artificial intelligence. Accordingly, it is possible to obtain information by restoring an image reflected by the eyes of a subject appearing in a photo or video. As a result, there is a possibility that users' personal information may be leaked from photos or videos that many people inadvertently upload to social media such as Facebook or Instagram.

Korean Patent Registration No. 10-2090891 Korean Patent Registration No. 10-2209595 US Patent Publication No. 2021/010373

Eye In-Painting with Exemplar Generative Adversarial Networks, Brian Dolhansky, Cristian Canton Ferrer Facebook Inc. 1 Hacker Way, Menlo Park (CA), USA Registration of eye reflection and scene images using an aspherical eye model, Atsushi Nakazawa et al. Globally and Locally Consistent Image Completion, Satoshi Iizuka et al. Single Image Reflection Removal Using Deep Encoder-Decoder Network. Zhixiang chi et al. Facial Attribute Editing via Style Skip Connections, Wenqing Chu et al.

The present disclosure provides a method and apparatus for protecting a user's privacy by removing an image reflected by the subject's eyes.

According to one aspect of the present disclosure, a data posting method includes: receiving, by a server, image data including eye image data or first data including moving picture data from a user terminal; detecting a first eye image from the first data; determining whether there is a scene reflected in the first eye image using a decision neural network that analyzes the first eye image; in response to determining that there is the reflected scene, generating a second eye image from which the reflected scene is removed;

generating second data from which the reflected scene is removed by synthesizing the second eye image with the first data; and posting an image or video corresponding to the second data.

In an embodiment, generating the second eye image from which the reflected scene is removed includes: receiving and using an output of the neural network to extract the reflected scene from the first eye image; and generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as inputs.

In one embodiment, the step of generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as an input includes: a reflected scene using the first eye image as an input generating an overall eye image without ophthalmic features, and generating a local image from which the reflected scene is removed by inputting the extracted reflected scene as an input; and generating the second eye image by synthesizing the overall eye image and the local image.

In one embodiment, the step of generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as an input includes: generating first information from the eye image; generating second information by using the extracted reflected scene as an input; and generating the second eye image based on the first information and the second information.

In one embodiment, the step of generating first information from the first eye image by inputting the first eye image as an input and the step of generating the second information by using the extracted reflected scene as an input are different generators. and generating the second eye image based on the first information and the second information is performed by a generator that receives the first eye image as an input and generates first information from the first eye image. can be performed.

In one embodiment, the step of generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as an input includes: generating a first feature map from the eye image; generating a second feature map by using the extracted reflected scene as an input; and generating the second eye image by concatenating the first feature map and the second feature map.

In an embodiment, the first feature map and the second feature map may be generated to have the same dimension.

In an embodiment, the step of extracting the reflected scene from the first eye image may be performed using an attention mechanism by receiving and using the output of the decision neural network.

In an embodiment, generating the second eye image from which the reflected scene has been removed includes: receiving and using an output of the neural network to extract the reflected scene from the first eye image; generating a masked image by masking the reflected scene from the first eye image; and generating a second eye image from which the reflected scene is removed by inputting the masked image and the extracted reflected scene as inputs.

In an embodiment, before detecting a first eye image from the first data, the method further comprises posting an image or video corresponding to the first data, and an image or video corresponding to the second data. The posting may include posting the posted image or video corresponding to the first data by replacing the image or video corresponding to the second data.

In an embodiment, the step of determining whether there is a scene reflected in the first eye image using a decision neural network that analyzes the first eye image includes the scene reflected in the first eye image output by the decision neural network It may include the step of determining based on the existence probability and a preset threshold (threshold).

According to one aspect of the present disclosure, a data posting device includes: a communication module configured to receive image data including eye image data or first data including moving picture data from a user terminal; a decision neural network configured to detect a first eye image from the first data and analyze the first eye image to determine if there is a reflected scene in the first eye image; receiving the output of the neural network, generating a second eye image from which the reflected scene is removed based on the output, and synthesizing the second eye image with the first data to remove the reflected scene a reconstructed neural network configured to generate data; a controller configured to post an image or video corresponding to the second data; and a storage configured to store the first data and the second data, and to enable deletion of the first data and the second data.

In an embodiment, the reconstructed neural network is configured to extract the reflected scene from the first eye image based on an output of the decision neural network, and based on the first eye image and the extracted reflected scene and a generator configured to generate a second eye image with the reflected scene removed.

In one embodiment, the generator comprises: a first generator configured to receive the first eye image as an input and generate an overall eye image without a reflected scene; and a second generator configured to receive the extracted reflected scene as an input and generate a local image from which the reflected scene is removed, wherein the second eye image is generated by synthesizing the global eye image and the local image. may be configured to generate.

In an embodiment, the reconstructed neural network may include: a first generator configured to receive the first eye image as an input and generate first information from the first eye image; and a second generator configured to generate second information using the extracted reflected scene as an input, wherein the first generator is configured to generate the second eye image based on the first information and the second information can be configured.

In one embodiment, the first generator has an encoder-decoder design structure comprising an encoder and a decoder, wherein the encoder and the decoder are of a skip connection for spatial information transfer and residual learning. Having a symmetric structure, the decoder may be configured to maintain the style of the first eye image using a normalization technique.

In an embodiment, the reconstructed neural network may include: a first generator configured to generate a first feature map from the first eye image by receiving the first eye image as an input; and a second generator configured to generate a second feature map using the extracted reflected scene as an input, wherein the first generator concatenates the first feature map and the second feature map to generate the second feature map. and may be configured to generate an eye image.

In an embodiment, the first generator and the second generator may each generate the first feature map and the second feature map to have the same dimension.

In an embodiment, the reconstructed neural network may be configured to receive an output of the decision neural network and extract the reflected scene from the first eye image using an attention mechanism based on the output.

In one embodiment, the reconstructed neural network extracts the reflected scene from the first eye image based on the output of the decision neural network and masks the reflected scene from the first eye image to generate a masked image, and take the masked image and the extracted reflected scene as inputs to generate a second eye image from which the reflected scene is removed.

In one embodiment, before detecting the first eye image from the first data, the controller controls the data posting device to publish an image or video corresponding to the first data, and After the generation, it may be configured to control the data posting device to replace the posted image or video corresponding to the first data with an image or video corresponding to the second data to be posted.

In an embodiment, the decision neural network may be trained to output a probability that the reflected scene exists in the first eye image.

According to one aspect of the present disclosure, a data posting method comprises: a first eye image including a reflective layer including a reflected scene on the cornea and a transport layer not including the reflected scene, wherein the first eye image is the reflection a layer and the transport layer are one image synthesized; receiving -; extracting information related to the reflection layer from the first eye image; extracting a reflection image including the reflected scene from the first eye image based on information related to the reflection layer; generating a second eye image including a cornea from which the reflected scene is removed, based on the first eye image and the reflected image; and posting the second eye image.

According to one aspect of the present disclosure, there is provided a method for training generative adversarial networks including a generator and a discriminator for correcting data including an eye image, the method comprising: receiving an output from a decision neural network configured to analyze a first eye image comprising a scene and separate a reflective layer comprising a reflected scene from the first eye image and a transport layer without the reflected scene; determining whether the reflected scene is reconstructable based on the output; in response to determining that the reflected scene is reconstructable, generating, by the generator, a second eye image from which the reflected scene is removed; inputting the first eye image and the second eye image into the discriminator to determine whether the image is real or fake; and providing feedback from the discriminator to the generator. and learning by the generator from the feedback.

It can remove images reflected in the subject's eyes and protect privacy, while synthesizing new eye images to provide a natural image or video.

1 is a conceptual diagram of a system according to an embodiment of the present disclosure.
2 is a block diagram of a server according to an embodiment of the present disclosure.
3 is a flowchart of a method of posting data according to an embodiment of the present disclosure;
4 is a flowchart of a method of posting data according to an embodiment of the present disclosure.
5 is a conceptual diagram of a decision neural network for determining whether a reflection region exists according to an embodiment of the present disclosure.
6A is a conceptual diagram of a network for correcting data according to an embodiment of the present disclosure.
6C is a conceptual diagram of correcting data using a network for correcting data according to an embodiment of the present disclosure.
6B is a conceptual diagram of correcting data using a network for correcting data according to an embodiment of the present disclosure.
7A is a conceptual diagram of a first generator according to an embodiment of the present disclosure.
7B is a conceptual diagram of a second generator according to an embodiment of the present disclosure.
8 is a conceptual diagram of a part of a network for correcting data according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the drawings. In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, the following examples may be modified in various other forms, and the scope of the technical spirit of the present disclosure is not limited to the following examples. Rather, these embodiments are provided to more fully and complete the present disclosure, and to fully convey the technical spirit of the present disclosure to those skilled in the art.

It is to be understood that the techniques described in the present disclosure are not intended to be limited to specific embodiments, and include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure.

In connection with the description of the drawings, like reference numerals may be used for like components.

In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features. In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used in the present disclosure, expressions such as “first,” “second,” “first,” or “second,” may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component) When referring to "connected to", it should be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression "configured to (or configured to)" as used in this disclosure depends on the context, for example, "suitable for," "having the capacity to" ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a processor configured (or configured to perform) A, B, and C," a module configured (or configured to perform) A, B, and C," refers to a dedicated processor (eg, : embedded processor) or a generic-purpose processor (eg, CPU or application processor) capable of performing corresponding operations by executing one or more software programs stored in a memory device.

Prior documents described in this disclosure are incorporated herein by reference in their entirety. Accordingly, the contents of which the description is simplified in the present disclosure may be supplemented through prior literature.

system

1 is a block diagram of a system 100 according to one embodiment. The system 100 may include a terminal 110 and a server 120 . The server 120 may be a server that receives and posts data from the terminal 110 . In one embodiment, the server 120 may be an entity that provides a social network service. For example, the server 120 may be various service providers such as Instagram, Facebook, and Kakao Story. In an embodiment, a “server” includes a computing system, hardware on which a program is executed, software running on the hardware, and cloud services, and may be connected to other servers through a network. In addition, “server” may refer to a distributed processing type that includes a system configured to provide a service at the request of a user or administrator, and operates one or more application programs in a mutually cooperative environment. “Server” may include hardware such as a processor, storage or database, and communication module. The processor manages and controls the components of the server. In one embodiment, the processor includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, It may be micro-controllers, microprocessors, or any other type of processor or controller for performing other functions. The storage is configured to store information for the operation of the server. The storage may store a plurality of application programs or applications running in the server, data readable by the processor, and instructions. For example, storage includes various storage spaces such as hard disk drive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM, EPROM, flash drive, hard drive, cloud using network, etc. can do. The server may include an artificial intelligence model. In one embodiment, an artificial intelligence model may be implemented by a combination of a processor, storage, and code stored in the storage. An artificial intelligence model is, in the present disclosure, by learning a learning model including an artificial neural network (ANN) through a large amount of learning data to optimize the parameters inside the artificial neural network, and a conversation system using the learned learning model It may mean a model involved in the operation of (not shown). In an embodiment, the artificial intelligence module (not shown) may be learned through Machine Reading Comprehension (MRC). In one embodiment, the artificial neural network model used in the artificial intelligence module is a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a constrained Boltzmann machine ( Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), Autoencoder, Variational Auto Encoder (VAE) or Deep Q-networks (Deep Q-Networks), generative adversarial networks (GAN), and at least any one or a combination thereof and a modified model, but is not limited to the above-described example. The artificial intelligence module may be implemented by any artificial intelligence neural network that may be implemented by prior art such as web search at the time of filing of the present disclosure.

The terminal 110 includes a display 115 . The terminal 110 may show an image or video to the user through the display 115 . The terminal 110 may share an image or video with other terminals 130 , 140 , and 150 through a service provided by the server 120 . That is, the user uploads or posts an image or video from the terminal 110 to the service provided by the server 120 , and the server 120 transmits the uploaded or posted image or video to the server 120 . It can be provided to users of other terminals 130 , 140 , and 150 through the service provided by . An image or video is exchanged between the terminals 110 , 130 , 140 , 150 and the server 120 in the form of data.

In an embodiment, the user of the terminal 110 posts the content (image or video) directly produced by the server 120 to the service provided. For example, a user may take a picture or a video with friends, or capture a place, moment, etc. that they want to remember and share with the server 120 . In this case, a person's face may be photographed. When a person's face is photographed, the human eye may reflect the surrounding environment or surrounding people. Therefore, if the photographed person's face is shared, the image reflected by the photographed person's eyes may also be shared, and when the human eye is enlarged or restored by a specific technique, information about the user and the user may be leaked. have. Such a reflection may occur in the cornea of the user's eye, and in particular, may mainly occur in the iris, pupil, etc. with a dark background. Of course, reflexes can also occur in the sclera of the eye. In addition, when an object reflecting an object such as a mirror or a glass object (eg, a glass bottle) as well as human eyes is photographed, information about the user and the user's surroundings may be leaked.

The communication method of the network between the server 120 and the terminal 110 is GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access) , LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network) Alliance), WiBro (Wireless Broadband), and WiMAX (World Interoperability for Microwave Access) may use a network established, but is not limited thereto, and may include all transmission method standards to be developed in the future. In addition, the communication method of the network may include both data transmission and reception via wired/wireless.

server

2 is a block diagram of a server 120 according to an embodiment of the present disclosure. The server 120 may include a service provider that provides a social network service. Referring to FIG. 2 , the server 120 includes a communication module 121 , a controller 122 , a storage 123 , a decision neural network 124 , and a reconstructed neural network 125 . The server 120 may receive an image or video from the terminal 110 through the communication module 121 , store it in the storage 123 , and post it. By posting of the server 120 , only the user who sent the image or video or all users of the server 120 can share the same image or video. The communication module 121 may exchange information with the terminal through the above-described communication method. The controller 122 controls the overall operation of the server 120, and the server 120 or the server 120 by a set of previously input commands (eg, software) or an input of an administrator who controls the server 120 ) can perform several functions. The storage 123 may store commands, software, data, and the like. The storage 123 may include a physical storage space or a network storage space such as a cloud.

The decision neural network 124 may determine whether it is necessary to correct the received data by analyzing or interpreting the received data (eg, image data, video data). In one embodiment, the decision neural network 124 may detect a human eye from image data or video data, and determine whether there is a reflected region of the human eye, and whether the reflected region is reconstructable. The reflected region may refer to a region in which another scene is reflected by the iris, the pupil, and the cornea. Even if the reflection actually occurs in the cornea, most of the cases in which the image is reflected by the cornea above the iris or pupil will be present. This is because the iris and pupil are dark, so reflection occurs well from the cornea above the iris and pupil. Therefore, in the present disclosure, when the reflection region of the eye image is referred to, a scene visible above the iris or pupil (a scene other than the iris and pupil) may be included. A more detailed description will be given later.

The reconstruction neural network 125 may generate an eye image or eye moving image data without a reflection area by removing and reconstructing the area reflected by the human eye from the image data or video data. In the present disclosure, the eye image or video without a reflection region includes a case in which only an image of the iris or pupil is provided without a reflection scene in the iris or pupil. A more detailed description will be given later.

Data Calibration Method

3 is a flowchart of a method of posting data according to an embodiment of the present disclosure; In an embodiment, the flowchart shown in FIG. 3 may be performed in the terminal 110 and/or the server 120 . Hereinafter, for convenience of description, it will be described that the server 120 performs the flowchart shown in FIG. 3 .

Referring to FIG. 3 , in block S305 the server 120 receives data. The data includes image data and/or moving picture data. In block S310, the server 120 detects the human eye from the data. As an example, the decision neural network 124 may detect a human eye from the data.

In one embodiment, a human head may be detected from image data or video data, and an eye may be detected from the detected head. Alternatively, the region proposal method can detect eyes at once. Eye detection may be performed using techniques that can be easily performed from information known to those skilled in the art of machine learning or deep learning. For example, as in Korean Patent Publication No. 2021-0060554, methods for detecting eyes in various ways have already been disclosed.

In the present disclosure, the following various embodiments may be used. In one embodiment, eye detection may be performed using landmark detection of object detection. In an embodiment, a human head may be detected from image data or video data using a network trained using data labeled as head and eyes, and eyes may be detected from the detected head. In an embodiment, the eye may be detected using a Haar Cascade Classifier. In another embodiment, a method using a region of interest (ROI) and a convolutional neural network (CNN) may be used. In another embodiment, an attention mechanism may be used to detect the eyes at once. The attention method is a method to look more carefully by judging a part related to a result to be predicted (eg, eye position or detection). For example, the detection ability may be improved by giving an attention score to a part or feature estimated to have an eye through learning. For example, by inputting a feature map of the reflective region input to the decoder, it is possible to pay more attention to the reflective region. This may be performed using a previously known attention model.

The eye detection method is not limited to this embodiment, and various eye detection techniques searched on the web or used by those skilled in the art at the time the present disclosure is filed may be used.

In block S315, the server 120 or the decision neural network 124 determines whether to restore the reflection region by obtaining a probability of the reflection region restoration. In one embodiment, the reflective region refers to a region or scene in which an object reflected by a human eye (eg, a cornea) is present. For example, there may be reflective areas in the cornea over the pupil and iris. Because the cornea is very thin and transparent, it may appear that there are reflective areas in the pupil and iris when viewed in a photograph or video. In the present disclosure, the region in which the reflective region is or may be present may be the cornea or the cornea over the pupil and iris.

In one embodiment, after detecting the eye, the reflective region may be detected. If the reflective region is detected, it may be determined to restore the reflective region.

In one embodiment, the reflective region may first detect the cornea or the cornea above the pupil and iris in the eye, and then detect the reflective region where the reflection is actually made. In this case, it is possible to detect only whether the reflection area exists without extracting or expanding the reflection area. The detection of the cornea may use neural networks and/or algorithms for object detection. For example, a model including a Region Proposal Network (RPN), a YOLO model, or the like may be used. In RPN, it is possible to detect a cornea and/or a reflective area by grabbing a plurality of object likelihood boxes based on the point at which the pixel value changes rapidly. In another embodiment, the cornea and/or the reflective region may be detected using a pre-trained model and database. For example, the pre-trained model may be trained from a data set of eyes, corneas, pupils, and iris, with and without reflection using eye image data. In the database, the eye, cornea, or pupil and iris, and feature vectors in the case of reflection and no reflection are learned and stored using the eye image data. For example, a reflective region may be detected by obtaining a feature vector such as an eye, cornea, iris, or pupil of the received data and comparing it with a previously stored feature vector.

Also, according to an embodiment, the restoration probability of the reflection region may be determined using a neural network modified from a network architecture designed to recognize and remove reflections from an input image. At least this neural network can be trained using paired image data sets, such as an identical eye image, but an eye image with and without reflection.

Similar to the image restoration method, the decision neural network 124 for determining the reflection region restoration probability may be implemented by applying the basic design of the convolutional encoder-decoder neural network. Also, the decision neural network 124 may be implemented by applying residual learning. Residual learning uses a method in which the gradient is transmitted well by attaching the previous result to the back for each predetermined node in a neural network with many layers. This is called residual mapping and is sometimes said to use a shortcut connection or skip connection.

More specifically, a number of prior art techniques for removing reflections from a single image have been proposed. In the prior art, except that the structure of the neural network used is different, a reflection layer and a transmission layer are separated by extracting a feature from a single image, and the reflection is removed by restoring the image. For example, the transport layer may be a layer including a desired scene from which unintentional reflection is removed, and the reflective layer may mean a layer in which a certain scene is reflected on eyes, glass, or the like.

The prior art Single Image Reflection Removal Using Deep Encoder-Decoder Network (Zhixiang chi et al.) discloses a feature extraction unit (encoder), a transport layer restoration unit (decoder), and a reflection restoration/removal neural network disposed between the encoder and the decoder. . The encoder separates the reflective layer and the transport layer by extracting features from the input image. The reflex/recovery cancellation neural network is a neural network that uses skip conncetion and is designed to learn and restore reflexes. Reflective/reconstructive neural networks can only preserve the characteristics of reflections by progressively removing the transport layer. The decoder reconstructs the transport layer by receiving the feature received from the encoder and the feature of the reflection from the reflection reconstruction/removal neural network. It will be understood that this prior document is only cited to show that the reflective layer and the transport layer can be separated from the input image, and is not intended to limit the present disclosure to the contents disclosed in the prior document. Many other neural network models can be designed to separate the reflection layer and the transport layer from the image. Complete restoration of the cornea, pupil, and iris cannot be performed with the contents disclosed in the prior literature.

In one embodiment, the restoration probability of the reflection region may be obtained by using a neural network (eg, the decision neural network 124 ) that separates the reflection layer and the transport layer and preserves and extracts only the reflection characteristics. For example, the reflection/reconstruction neural network may be designed to connect a plurality of convolutional layers and a plurality of deconvolutional layers and perform residual learning. For example, reflections appear in the form of feature vectors, each of which may mean a probability.

When there is a reflection in the input image, it is possible to determine whether the reflection can be restored by checking the output of the decision neural network 124 . For example, if a reflection is present, the decision neural network 124 may detect the reflection. The decision neural network 124 may determine that there are sufficient characteristics of the reflex that can be reconstructed from the detected reflex. Based on the probability of the restoration possibility, the server 120 may determine whether to restore the eye image or the reflection region of the received image. The decision neural network 124 transmits the output of the final layer to the next neural network (eg, the image reconstruction neural network 125).

In block 325, in response to the decision neural network 124 determining that the reflective region is not recoverable, the server 120 posts the received data. Accordingly, the terminal 110 and other terminals 130 , 140 , and 150 that have transmitted the data may share an image or a video corresponding to the posted data.

In block S320 , in response to determining that the reflective region is reconstructable, the server 120 or the reconstruction neural network 125 restores the reflective region. The decision neural network 124 transmits an output (eg, a feature map) of the final layer to the reconstruction neural network 125 in response to determining that the reflection region is reconstructable. In an embodiment, restoration of the reflective region may mean restoring an eye without corneal reflection by permanently removing the image reflected by the eye. In an embodiment, data corresponding to the cornea, iris, and pupil may be newly created by overwriting existing data by newly generating the cornea, iris, and pupil. Accordingly, data including an eye without corneal reflection may be obtained. The restoration of the reflection area may be performed by a network, module, etc. using artificial intelligence, and will be described in detail later.

In block S325, the server 120 publishes an image and/or a moving picture in which the reflection region is restored from the received data. Accordingly, the terminal 110 and other terminals 130 , 140 , and 150 that have transmitted the data may share an image or a video corresponding to the posted data.

4 is a flowchart of a method of posting data according to an embodiment of the present disclosure. In an embodiment, the flowchart shown in FIG. 4 may be performed in the terminal 110 and/or the server 120 . Hereinafter, for convenience of description, it will be described that the server 120 performs the flowchart shown in FIG. 4 .

Referring to FIG. 4 , in block S405 the server 120 receives data. The data includes image data and/or moving picture data. In block S410, the server 120 posts an image and/or a video corresponding to the received data. Accordingly, the user may check the image and/or video uploaded by the user. In an embodiment, the server 120 may control to check only the user's account or terminal who uploaded the image and/or video posted in block S410 . That is, the image and/or video posted in block S410 cannot be viewed by other users of the server 120 . Alternatively, the server 120 may control the terminal that transmits the posted image and/or video data in block S410 and the other terminal to share the image or video corresponding to the posted data.

In block S415 the server 120 detects the human eye from the data. Block S415 may detect a human eye in the same or similar manner as block S310.

In block S420 , the server 120 determines whether to restore the reflective region by obtaining a probability of restoring the reflective region. Block S420 may be implemented in the same or similar manner as block S315. In an embodiment, the reflective region refers to a region in which an object reflected by a human eye (eg, a cornea) exists. For example, there may be reflective areas in the cornea over the pupil and iris. Because the cornea is very thin, it may appear that there are reflective areas in the pupil and iris when viewed as a photograph or video. In the present disclosure, the region in which the reflective region is or may be present may be the cornea or the cornea over the pupil and iris.

In one embodiment, after detecting the eye, the reflective region may be detected.

In block S425 , in response to determining that the reflective region is not recoverable, the server 120 maintains the image and/or video posted in block S410 . In block S425, the server 120 may control the posted and maintained image and/or video to be shared by the terminal that has transmitted the data and other terminals. If the server 120 controls to check only the account or terminal of the user who has uploaded the image and/or video posted in block S410, in block S420, the server 120 determines the restoration possibility of the reflective area and says that restoration is not possible. Other users can only see your post if you decide.

In block S430 , in response to determining that the reflective region is reconstructable, the server 120 restores the reflective region. In an embodiment, restoration of the reflective region may mean restoring an eye without corneal reflection by permanently removing the image reflected by the eye.

In block S435, the cornea or iris and pupil may be newly created and overwrite existing data to generate an eye image including the new cornea or iris and pupil data. Accordingly, data including an eye without corneal reflection may be acquired. By using this, it is possible to obtain the entire image/video data reconstructed in which the eye reflection is removed from the data received in block S405. The restoration of the reflection area may be performed by a network, module, etc. using artificial intelligence, and will be described in detail later.

In block S440, the image and/or video posted in block S410 is replaced with an image and/or video corresponding to the data obtained in S435. In block S440, the server 120 may control the replaced image and/or video to be shared by the terminal that has transmitted the data and the other terminal. If the server 120 controls to check only the account or terminal of the user who uploaded the posted image and/or video in block S410, the post is replaced in block S440, and then other users can check the post.

decision neural network

5 is a conceptual diagram of a decision neural network 124 that determines whether a reflection region exists according to an embodiment of the present disclosure. In one embodiment, the decision neural network 124 may be constructed by applying a portion of the network 500 that cancels reflections. The network 500 may include a feature extraction network 510 , a reflection cancellation/recovery network 520 , and an image restoration network 530 . The network 500 may be an application of the network disclosed in Single Image Reflection Removal Using Deep Encoder-Decoder Network (Zhixiang chi et al.). For example, the decision neural network 124 may be configured to include a feature extraction network 510 and a reflection cancellation/restore network 520 .

The feature extraction network 510 , the reflection cancellation/reconstruction network 520 , and the image restoration network 530 each include a plurality of layers, and each layer may include a convolutional layer.

In one embodiment, the feature extraction network 510 may include a plurality of layers. For example, the layer of the feature extraction network 510 may be a convolutional layer. The feature extraction network 510 may receive the input image and generate a plurality of feature maps. The input image may include an image received by the server 120 or an extracted eye image. The feature extraction network 510 may generate a feature map of the input image using various filters. The reflection removal/recovery network 520 may generate a reflection layer (or a feature map of the reflection layer) and a transport layer (eg, a feature map of the layer or transport layer for generating an image from which the reflection region is removed). . The image restoration network 530 may receive the reflective layer and the transport layer to generate an image from which the reflective region is removed (or a feature map from which the reflective region is removed). The decision neural network 124 of the present disclosure may not include the image reconstruction network 530 . Therefore, it is possible to generate or not use the feature map of the image from which the reflection has been removed. For example, the decision neural network 124 may reconstruct only reflections.

In an embodiment, in the process of training the network 500 , a feature map including only a reflection region (or data corresponding thereto, a parameter, hereinafter referred to as a first output) and a possibility that reflection exists (or data corresponding thereto, parameter, hereinafter referred to as a second output) and output. The reflection cancellation/restore network 520 is trained to preserve only the reflection characteristics by gradually removing the transport layer from the input, so that the first output of the reflection removal/recovery network 520 is data corresponding to the characteristics of the reflection region in the input image. , for example, a feature map of the reflection region. Reflection can be reconstructed through the feature map of the reflective region and features such as position, size, and shape can be obtained. The feature map of this reflective region can be used to separate the reflective region from the input image. The second output may be an output indicating the likelihood that a reflection is present. In the process of learning the reflection cancellation/restore network 520 , the reflection cancellation/restore network 520 may learn that the reflection exists. The reflection cancellation/recovery network 520 may output the probability of reflection if reflection is present in the input image. For example, by comparing the probability of the reflection by a preset threshold, the reflection cancellation/recovery network 520 may determine that there is a reflection in the input image. For example, the reference value may be 50%. That is, the decision neural network 124 may determine whether a reflection is present in the input image based on the probability of the reflection. In response to the decision neural network 124 determining that a reflection region exists, an input image and a second output of the reflection cancellation/recovery network 520 may be provided to the reconstruction neural network 125 .

eye image restoration network

6A is a conceptual diagram of a network 600 for correcting data according to an embodiment of the present disclosure. In an embodiment, the network 600 may correspond to the reconstructed neural network 125 of FIG. 2 . The network 600 is a generative adversarial network (GAN), a neural network modified / applied to the GAN (eg, style GAN, Conditional GAN, Cycle GAN, Deep Convolution GAN, ExGAN, etc.) or at least a part thereof However, the example is not limited thereto, and a neural network capable of generating an image from an input may be included in an embodiment of the present disclosure. Server 120 may be configured to run network 600 .

The network 600 may be a neural network in which a generator and a discriminator compete to automatically generate an image, video, voice, etc. close to reality. The GAN includes a generator that receives an input (eg, image, noise, vector, etc.) and generates data therefrom, and a discriminator that distinguishes the data input from the generator. The generator is trained to deceive the discriminant model as much as possible by creating fake examples. The discriminator is trained to distinguish between fake and real examples as accurately as possible. Since a moving picture is a continuous playback of a plurality of image frames, the network 600 will be described below as generating and determining an image, but it is obvious to a person skilled in the art that the network 600 may be configured to generate and determine a moving picture. will be understood In addition, a generator and a discriminator are configured to generate a new image using various neural networks based on the input.

Referring to FIG. 6A , the network 600 may include two generators 620 and 630 and two discriminators 625 and 635 . For convenience of the invention, the first generator 620 , the second generator 630 , the first discriminator 625 , and the second discriminator 635 will be described. In FIG. 6A , a first discriminator 625 is shown as a global discriminator, and a second discriminator 635 is shown as a local discriminator.

Referring to FIG. 6A , the server 120 determines a reflection area from the eye image 610 . In one embodiment, network 600 may receive the output of the final layer of decision neural network 124 . The output of the final layer of decision neural network 124 may include reflective features (eg, feature maps) of the eye image. In the network 600 , the reflective region may be extracted from the eye image through an attention mechanism using a reflective feature. For example, a reflective region may be extracted from the eye image 610 to obtain a masked image (not shown) and a masking region 614 including the reflective region from the entire eye image. Also, the reflective region may include a plurality of patches 616 . The patch 616 may include an image that makes up all or part of the reflective area. In one embodiment, at least one of the patches 616 may include only reflective regions. At least one of the patches 616 may include a reflective area as well as some image of the pupil without reflection. Although the masking area 614 or patch 616 in FIG. 6A is shown as a rectangle, its shape may vary. For example, it may be extracted along the outline of the reflection area, or may be determined in various ways, such as a circle or a square. FIG. 6A is an example for explanation through the drawings, and extraction of the reflective region or patch 616 is performed within the network 600 and may include a layer including a reflective feature or a reflective feature vector. That is, the network 600 may extract the reflection region in the most suitable form through learning.

In one embodiment, masking region 614 consists of only reflective regions, includes regions that include regions other than reflective regions (eg, eyelids, eyebrows, iris, pupil, sclera, etc.), or includes reflective regions. The cornea or the iris, the pupil, and at least one of the sclera may be configured as a whole. The masking area 614 may be set by a user or may be set by the server 120 or the network 600 .

An eye image 610 is input to the first generator 620 . The first generator 620 may be concatenated with the second generator 630 to receive information from the second generator 630 and reconstruct the entire eye image using the information and the eye image 610 . . The first generator 620 may be configured to include a neural network using skip connection and spatial transfer. The information includes information related to the reflective region. The information may include at least one of a reflective feature generated by the second generator 630 , an image from which a reflective layer is removed, and a cornea, iris, and pupil image from which the reflective region is removed and reconstructed.

That is, for example, the first generator 620 may reconstruct the eye image from which the reflection region is removed by synthesizing the eye image 610 and the image reconstructed from the second generator 630 . Alternatively, the first generator 620 may extract a feature from the eye image 610 and reconstruct the eye image from which the reflection region is removed by using the feature received from the second generator 630 .

In another embodiment, the first generator 620 may be configured to reconstruct the image of the entire eye by filling in the masked region through learning independently without receiving information from the second generator 630 . For example, the first generator 620 may be trained or configured to reconstruct the masked reflective region based on the remaining corneal, pupil, and iris regions in which the reflective region does not exist.

In an embodiment, a masked image (not shown) may be input to the first generator 620 instead of the eye image 610 . The first generator 620 may be concatenated with the second generator 630 to receive information from the second generator 630 and reconstruct the entire eye image using the information and the eye image 610 . . In the masked image (not shown), the entire cornea may be masked, only the reflective region may be masked, or the reflective region and a portion of the cornea may be masked. The first generator 620 is configured to reconstruct an image of the entire eye by filling the masked region through learning. In an embodiment, since the first generator 620 is to reconstruct a masked portion, it may be configured to include a neural network using skip connection and spatial transfer to use the features of an unmasked portion.

The first generator 620 may be configured to reconstruct the image of the entire eye by filling the masked region through independent learning without receiving information from the second generator 630 . For example, the first generator 620 may be trained or configured to reconstruct the masked reflective region based on the remaining corneal, pupil, and iris regions in which the reflective region does not exist.

In an embodiment, the first generator 620 may reconstruct (generate) an eye image by inputting both the masked image 612 and the eye image 610 as inputs.

The first discriminator 625 receives the eye image generated by the first generator 620 . The first discriminator 625 may further receive the eye image 610 . The first discriminator 625 may determine whether the eye image that has been pre-learned as the learning data and received is a forgery (eg, whether it is a fake or an awkward image different from the real one). The training data includes various eye images, and may include real eye images and generated fake eye images. The first generator 620 and the first discriminator 625 are configured to perform learning while exchanging information with each other. That is, the first generator 620 learns to generate a sophisticated fake image to deceive the first discriminator 625, and the first discriminator 625 determines whether the generated image is a real image or not. It learns to determine whether it is a created image. The determination result of the first discriminator 625 is transmitted to the first generator 620 , so that the first generator 620 and the first discriminator 625 compete with each other to improve performance.

The masking region 614 and/or the plurality of patches 616 are input to the second generator 630 . The second generator 630 is configured to reconstruct/create the masked region through learning. The second generator 630 may be concatenated with the first generator 620 to transmit information to the first generator 620 . The second generator 630 may generate various information (eg, features, samples, filtered layers, channels, parameters, etc.) from the inputs through various neural networks based on the inputs. That is, at least a part of the various information obtained in the process of the second generator 630 reconstructing/generating the masked region based on or through the input may be included in the information transmitted to the first generator 620 . Since the second generator 630 receives the masking area 614 and/or the plurality of patches 616 as inputs, the information transmitted to the first generator 620 includes the masking area 614 and/or the plurality of patches ( 616) and related information.

The second discriminator 635 receives the image generated by the second generator 630 . The second discriminator 635 may further receive the masking region 614 and/or the plurality of patches 616 . The second discriminator 635 may determine whether an image that has been previously learned as the learning data and received is a forgery (eg, whether it is a fake or an awkward image different from the real one). The training data includes images in which reflective regions such as various corneas, iris, and pupils may exist, and may include real images and generated fake images. The second generator 630 and the second discriminator 635 are configured to perform learning while exchanging information with each other. That is, the second generator 630 learns to generate a sophisticated fake image to deceive the second discriminator 635, and the second discriminator 635 determines whether the generated image is a real image. It learns to determine whether it is a created image. In an embodiment, the second discriminator 635 may be trained to determine whether the image generated by the second generator 630 includes a reflective region. For example, the second determiner 635 may determine whether the image generated by the second generator 630 is the same image as a real cornea, iris, or pupil without a reflective region. The determination result of the second discriminator 635 is transmitted to the second generator 630 so that the second generator 630 and the second discriminator 635 compete with each other to improve performance.

The first generator 620 concentrates on generating the entire eye image or the entire iris and pupil using the style of the input eye image (eg, using AdaIN), and the second generator 630 focuses on generating the reflection area. can It is possible to use two generators to generate more sophisticated eye images compared to using one generator.

A network 600 in which the first generator 620, the second generator 630, the first discriminator 625, and the second discriminator 635 precisely generate an eye image without a reflective region through learning as described above. ) can be configured.

In an embodiment, the network 600 may be implemented by applying StyleGAN or a network modified therefrom. That is, the network 600 is configured to generate an eye image having characteristics similar to the input eye image by preferentially mapping the latent vector non-linearly so that the latent vector has a probability distribution similar to that of the training dataset, and using the mapped latent vector. can be implemented. In addition, the first generator 620 or the second generator 630 may use a normalization technique, for example, Adaptive Instance Normalization (AdaIN). In an embodiment, AdaIN may be located in front of the deconvolution layer of the first generator 620 to receive information through skip connection with the convolution layer.

The first generator 620 may adjust the regular information of the feature map using AdaIN. The second generator 630 may use AdaIN to normalize the cornea, pupil, and iris to a target style. As the input eye image passes through each layer, the scale and variance may change. To prevent this, a regularization technique is used for each layer. AdaIN can transmit the style of the input image in the future space by transmitting feature statistics called channel-wise mean and variance.

6B is a conceptual diagram of correcting data using a network for correcting data according to an embodiment of the present disclosure. Referring to FIG. 6B , the first and second discriminators 625 and 635 are components used to learn the first and second generators 620 and 630 , respectively, so the actual input image is corrected and provided to the user. It can be seen that it may not be used when The decision neural network 124 separates the reflection layer and the transport layer from the eye image data. The output of the final layer of the decision neural network 124 is transmitted to the reconstruction neural network 125 in response to the decision neural network 124 determining that the reflection region is reconstructable. The reconstructed neural network 125 extracts a reflection region from the eye image 610 using the output of the final layer. The eye image 610 is input to the first generator 620 , and the masking region 614 or patch 616 obtained from the reflective region is input to the second generator 630 . The second generator 630 generates a cornea, a pupil, an iris, etc. from the input, and transmits information acquired during generation to the first generator 620 . The first generator 610 reconstructs and outputs the eye image using the received eye image 610 and information.

6C is a conceptual diagram of a network for correcting data according to an embodiment of the present disclosure.

Referring to FIG. 6C , the first generator 620 may be configured to generate an entire eye image, and the second generator 630 may be configured to generate a corneal image. In one embodiment, the first generator 620 may be configured to generate an overall eye image without a reflected scene. In another embodiment, the first generator 620 may be configured to generate an entire eye image excluding the cornea. The restoration of the reflection region may be performed by synthesizing the image generated by the first generator 620 and the image generated by the second generator 630 to generate a complete eye image. By transmitting the final image and the eye image 610 to the discriminator, the entire neural network can be trained.

7A is a conceptual diagram of a first generator 620 according to an embodiment of the present disclosure. 7A is an exemplary diagram, and the first generator 620 of the present disclosure is not limited to this drawing and description.

Referring to FIG. 7A , the first generator 620 may be based on an encoder-decoder design structure. The first generator 620 may include the same number of down-sampling layers and up-sampling layers. In an embodiment, the down-sampling layer may correspond to the encoder structure and the up-sampling layer may correspond to the decoder structure. In one embodiment, the down-sampling layer may include four layers 622a, 622b, 622c, and 622d, and the up-sampling layer may include four corresponding layers 626a, 626b, 626c, 626d. In an embodiment, a down-sampling layer may correspond to an encoder structure, and an up-sampling layer may correspond to a decoder structure. The first generator 620 may further include a plurality of, for example, three AdaINs 625a, 625b, and 625c. The number of layers 622a, 622b, 622c, 622d, corresponding layers 626a, 626b, 626c, 626d, and AdaINs 625a, 625b, 625c are exemplary and can be changed when implementing the first generator 620 . Layers 622a, 622b, 622c, 622d and corresponding layers 626a, 626b, 626c, 626d may include latent feature maps at encoders and decoders, respectively. The layers 622a, 622b, 622c, and 622d and the corresponding layers 626a, 626b, 626c, and 626d have a spatial information transfer relationship. AdaINs 625a, 625b, and 625c may be used as information for instance normalization. The first generator 620 may further include a bottleneck layer 628 between the layer and the corresponding layer.

7B is a conceptual diagram of a second generator 630 according to an embodiment of the present disclosure. 7B is an exemplary diagram, and the second generator 630 of the present disclosure is not limited to this drawing and description.

Referring to FIG. 7B , the second generator 630 may be based on an encoder-decoder design structure. The second generator 630 may include the same number of down-sampling layers and up-sampling layers. In one embodiment, the down-sampling layer may correspond to the encoder structure 632 and the up-sampling layer may correspond to the decoder structure 636 . In one embodiment, the down-sampling layer may include three layers 632a, 632b, 632c. The decoder 636 of the second generator 630 may be a mirror structure of the encoder 632 . The second generator 630 may further include a bottleneck layer 638 between the encoder 632 and the decoder 636 . The bottleneck layer 628 of the first generator 620 may be concatenated with the bottleneck layer 638 of the second generator to receive a local feature of the eye. The bottleneck layers 628 and 638 may be a concept including a latent space of an encoder stage, a latent feature map, a latent vector, and the like. In one embodiment, the bottleneck layer 628 of the first generator 620 may have the same dimension as the bottleneck layer 638 of the second generator. For example, the bottleneck layer 628 of the first generator 620 may be understood as an n-D matrix having the same shape as the bottleneck layer 638 of the second generator.

In one embodiment, the first generator 620 may be configured to reconstruct the entire eye image and generate a style. The second generator 630 may be configured to learn the patch 616 to extract reflective features. Since the decoder 636 of the second generator 630 is configured to reconstruct an image that has removed reflections from the patches 616 , the second generator 630 can be trained to extract reflection-related features from the patches 616 . . That is, the bottleneck layer 638 of the second generator 630 may include a reflection-related feature map, a feature map from which reflection is removed, and the like.

The bottleneck layer 628 of the first generator 620 is concatenated with the bottleneck layer 638 of the second generator 620 so that the first generator 620 can reconstruct the entire eye image. In order to maintain the style of the input image, the first generator 620 may use skip connection and AdaIN.

In the first generator, the style information of the encoder may be used to adjust the latent feature map at the decoder. Information of the latent feature map in the encoder is transferred to the corresponding layer of the decoder by skip connection, and residual learning can be performed. Then, it can be used as information for instance normalization in AdaIN of the decoder. Spatial information transmission may be used to convey spatial information to a decoder. For example, the style information may be used to convey the style of the overall eye image, and the spatial information may be used to convey local information. Accordingly, all features of the input eye image 610 can be well conveyed and preserved.

The decoder 636 of the second generator 630 may not be used to reconstruct the eye image. That is, the second generator 630 shown in FIG. 7B may be used for learning together with the discriminator 635 , and the decoder 636 may not be used to restore the eye image.

8 is a conceptual diagram of a portion of a network 700 for correcting data according to an embodiment of the present disclosure. Referring to FIG. 8 , an omitted portion of the network 700 may be the same as the network 600 illustrated in FIG. 6A . The network 700 includes a first feature extractor 725 and a second feature extractor 735 instead of a first discriminator 625 and a second discriminator 635 . The first feature extractor 725 and the output of the second feature extractor 735 are connected to provide one output (the output that determines whether it is real or fake). From the connected output, it is trained to determine whether the input full eye image is real or fake. The first feature extractor 725 and the second feature extractor 735 transmit the determination result to the first generator 620 and the second generator 630 to train the network 600 .

Although a method for correcting an eye image using an image has been described in the present disclosure, it will be understood by those skilled in the art that such a method and apparatus can be applied to a method of removing and reconstructing a reflective region occurring in an eye in a moving picture.

This disclosure focuses on the creation of the cornea, iris, and pupil. In other words, instead of removing the reflective area from all input images, we are focusing on removing the reflection from the eye. Therefore, it is possible to generate a sophisticated eye image with the reflective region removed using the trained eye image generation network.

In the present disclosure, an end goal, a sub-goal, a goal, and the like are exemplary, and the present disclosure may be interpreted as a conversational method for inducing a specific goal.

The apparatus and method described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to a person skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

A computing device may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s), via wired and/or wireless communications. The remote computer(s) may be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, generally comprising the components described for the computing device. includes many or all of Logical connections include wired/wireless connections to local area networks (LANs) and/or larger networks, eg, wide area networks (WANs). Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, for example, the Internet.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

100: system 110: terminal
115: display 120: server
121: communication module 122: controller
123: storage 124: decision neural network
125: reconstructed neural network 600: network
610: eye image 614: masking area
616: patch 620: first generator
625: first discriminator 630: second generator
635: second discriminator

Claims (22)

A method in which a server providing a social network service posts data,
receiving, by the server, image data including eye image data or first data including moving picture data from the user terminal;
detecting a first eye image from the first data;
determining whether there is a reflected scene in the first eye image using a decision neural network analyzing the first eye image, wherein the reflected scene includes a human image or a background image;
in response to determining that there is the reflected scene, generating a second eye image from which the reflected scene is removed;
generating second data from which the reflected scene is removed by synthesizing the second eye image with the first data; and
Posting an image or video corresponding to the second data
How to post data. According to claim 1,
The step of generating the second eye image from which the reflected scene is removed comprises:
receiving and using the output of the decision neural network to extract the reflected scene from the first eye image; and
and generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as inputs.
How to post data. 3. The method of claim 2,
generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as inputs;
generating an overall eye image without a reflected scene by receiving the first eye image as an input, and generating a local image in which the reflected scene is removed by receiving the extracted reflected scene as an input; and
generating the second eye image by synthesizing the global eye image and the local image;
How to post data. 3. The method of claim 2,
generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as inputs;
generating first information from the first eye image by receiving the first eye image as an input;
generating second information by using the extracted reflected scene as an input; and
generating the second eye image based on the first information and the second information;
How to post data. 5. The method of claim 4,
Generating the first information from the first eye image by taking the first eye image as an input and generating the second information by using the extracted reflected scene as an input are performed by different generators,
generating the second eye image based on the first information and the second information is performed by a generator that generates first information from the first eye image by receiving the first eye image as an input;
How to post data. 3. The method of claim 2,
generating a second eye image from which the reflected scene is removed by inputting the first eye image and the extracted reflected scene as inputs;
generating a first feature map from the first eye image by receiving the first eye image as an input;
generating a second feature map by using the extracted reflected scene as an input; and
generating the second eye image by concatenating the first feature map and the second feature map;
How to post data. 7. The method of claim 6,
The first feature map and the second feature map are generated to have the same dimension,
How to post data. 3. The method of claim 2,
The step of extracting the reflected scene from the first eye image is performed using an attention mechanism by receiving and using the output of the decision neural network,
How to post data. According to claim 1,
The step of generating a second eye image from which the reflected scene is removed includes:
receiving and using the output of the decision neural network to extract the reflected scene from the first eye image;
generating a masked image by masking the reflected scene from the first eye image; and
generating a second eye image from which the reflected scene is removed by inputting the masked image and the extracted reflected scene as inputs
How to post data. According to claim 1,
Prior to detecting a first eye image from the first data,
Further comprising the step of posting an image or video corresponding to the first data,
Posting the image or video corresponding to the second data comprises the step of replacing the posted image or video corresponding to the first data with an image or video corresponding to the second data and posting it
How to post data. According to claim 1,
The step of determining whether there is a scene reflected in the first eye image using a decision neural network that analyzes the first eye image includes:
Comprising the step of determining based on the probability that the reflected scene exists in the first eye image output by the neural network and a preset threshold,
How to post data. A data posting device configured to provide a social network service, comprising:
a communication module configured to receive image data including eye image data or first data including moving picture data from the user terminal;
a decision neural network configured to detect a first eye image from the first data and analyze the first eye image to determine if there is a reflected scene in the first eye image, wherein the reflected scene comprises a human image or a background image -;
receiving the output of the neural network, generating a second eye image from which the reflected scene is removed based on the output, and synthesizing the second eye image with the first data to remove the reflected scene a reconstructed neural network configured to generate data;
a controller configured to post an image or video corresponding to the second data; and
a storage configured to store the first data and the second data, and to enable deletion of the first data and the second data;
data posting device. 13. The method of claim 12,
The reconstructed neural network is
and extract the reflected scene from the first eye image based on an output of the decision neural network;
a generator configured to generate a second eye image with the reflected scene removed based on the first eye image and the extracted reflected scene;
data posting device. 14. The method of claim 13,
The generator is
a first generator configured to receive the first eye image as an input and generate an overall eye image without a reflected scene; and
a second generator configured to take the extracted reflected scene as input and generate a local image from which the reflected scene has been removed;
configured to synthesize the global eye image and the local image to produce the second eye image;
data posting device. 14. The method of claim 13,
The reconstructed neural network is
a first generator configured to take the first eye image as an input and generate first information from the first eye image; and
a second generator configured to generate second information using the extracted reflected scene as input;
the first generator is configured to generate the second eye image based on the first information and the second information;
data posting device. 16. The method of claim 15,
The first generator has an encoder-decoder design structure including an encoder and a decoder, wherein the encoder and the decoder have a symmetric structure of spatial information transfer and skip connection for residual learning, wherein the decoder is configured to maintain a style of the first eye image using a normalization technique;
data posting device. 14. The method of claim 13,
The reconstructed neural network is
a first generator configured to take the first eye image as an input and generate a first feature map from the first eye image; and
a second generator configured to generate a second feature map using the extracted reflected scene as input;
The first generator generates the second eye image by concatenating the first feature map and the second feature map,
data posting device. 18. The method of claim 17,
wherein the first generator and the second generator respectively generate the first feature map and the second feature map to have the same dimension;
data posting device. 14. The method of claim 13,
the reconstructed neural network is configured to receive an output of the decision neural network and extract the reflected scene from the first eye image using an attention mechanism based on the output;
data posting device. 14. The method of claim 13,
The reconstructed neural network is
extracting the reflected scene from the first eye image based on the output of the decision neural network;
masking the reflected scene from the first eye image to generate a masked image;
configured to take the masked image and the extracted reflected scene as inputs to generate a second eye image with the reflected scene removed;
data posting device. 13. The method of claim 12,
the controller is
Prior to detecting the first eye image from the first data,
controlling the data posting device to publish an image or video corresponding to the first data,
after generating the second data, control the data posting device to replace the posted image or video corresponding to the first data with an image or video corresponding to the second data to be posted
data posting device. 13. The method of claim 12,
The decision neural network is trained to output the probability that the reflected scene exists in the first eye image.
data posting device.

Images