KR20210030573A - Apparatus and method for controlling weather effects in marine video based on neural network - Google Patents

Abstract

The present invention provides a method of controlling a weather effect in a marine image by using a generative adversarial network (GAN) including a generator for generating an image and a discriminator for discriminating the generated image from an actual image. The method includes a step of training a first generative adversarial neural network (GAN) by extracting style features and context features of an image including a first weather effect from a generator of the first generative adversarial neural network (GAN); a step of regenerating an image including a second weather effect by inputting the style feature or context feature of the image including the extracted first weather effect into a generator of the second generative adversarial neural network; a step of training the second generative adversarial neural network to allow the discriminator of the second generative adversarial neural network to identify the image including actual second weather effect by comparing the regenerated image including the second weather effect with the image including the actual second weather effect; and a step of controlling a weather effect in the input marine image by using the learned first generative adversarial neural network or second generative adversarial neural network.

Description

Method and apparatus for controlling weather effects in maritime images based on neural networks {APPARATUS AND METHOD FOR CONTROLLING WEATHER EFFECTS IN MARINE VIDEO BASED ON NEURAL NETWORK}

The present invention relates to a method and apparatus for controlling meteorological effects in a maritime image, and more particularly, to identify maritime objects such as ships and artificial structures from optical maritime images in which various meteorological effects such as sea clouds, raindrops, and reflections of water exist. It relates to a method and apparatus for controlling a weather effect in a marine image.

Ships in operation, coastal guard posts, and control centers must identify surrounding ships and structures from optical images for various purposes, such as controlling traffic or supporting maritime accidents. Therefore, in order to identify marine objects in a marine environment different from the land environment, there must be an identification model capable of identifying marine objects even with changes in various weather effects such as sea clouds, raindrops, and reflections on the surface of the marine environment.

Prior art techniques have been proposed to remove the meteorological effect through pre-processing of an optical image, and in particular, techniques for pre-processing the fog existing in the optical image have been proposed. Here, the prior art proposed an algorithm to remove the fog through the channel image of the darkest color in the image, focusing on the fact that the pixels in which the fog exists is a bright color. In addition, the prior art has proposed a technique for identifying marine objects in an image and removing fog through a neural network based on the result.

On the other hand, the prior art has a problem that it does not work properly for the image dominated by a very bright area such as the sky, so it is unsuitable for identifying a marine object in a marine environment.

In addition, in order to identify marine objects in an image and create a neural network based on the result, the identification of marine objects must be preceded, so it is not suitable to be used for object identification purposes.

In addition, most of the prior art assumes an identification model that removes fog in a terrestrial environment, and thus, there is a problem that it is unsuitable to be used as a model for identifying a marine object in a marine environment in which there are many water particles in the atmosphere and diffuse reflection occurs due to the water surface.

An object of the present invention for solving the above problems is to provide a method for generating sea data according to various weather conditions by generating a meteorological model suitable for a marine environment.

Another object of the present invention for solving the above problems is to provide a method of preprocessing an optical maritime image in a form suitable for identifying maritime objects such as marine vessels and various artificial structures by removing weather effects that hinder object identification. To provide.

In order to achieve the above object, a method for controlling weather effects in a marine image according to an embodiment of the present invention includes a generator for generating an image and a discriminator for distinguishing the generated image from the actual image. As a method of controlling meteorological effects in maritime images using a Generative Adversarial Network (GAN), the style features and context features of the image including the first meteorological effect are extracted from the generator of the first generative adversarial network (GAN). Thus learning the first generative adversarial neural network; Regenerating an image including a second meteorological effect by inputting a style feature or a context feature of the image including the extracted first meteorological effect as a generator of a second generative hostile neural network; The second generative hostility is such that the identifier of the second generative hostile neural network identifies the image including the actual second weather effect by comparing the regenerated image including the second weather effect with the image including the actual second weather effect. Training a neural network; And adjusting the weather effect in the input maritime image using the learned first generative hostile neural network or the second generative hostile neural network.

According to an embodiment of the present invention, it is possible to generate data for learning a marine object identification device suitable for a marine environment by generating images to which various weather effects are given through neural network learning.

According to an embodiment of the present invention, various weather effects can be generated and removed by using an image encoder and a decoder capable of domain transformation during neural network learning.

According to an embodiment of the present invention, a problem of securing training data required for neural network training may be solved by learning a generative hostile neural network using an image encoder and an image decoder capable of domain transformation.

1 is a block diagram of a weather effect control system that applies a fog effect to a clear image.
2 is a conceptual diagram illustrating a generative hostile neural network (GAN).
3 is a block diagram of an apparatus for controlling a weather effect according to an embodiment of the present invention.
4 is a block diagram of an apparatus for controlling a weather effect according to an embodiment of the present invention.
5 is a diagram for explaining a method of learning a generative adversarial neural network using an encoder and a decoder in the same domain in the present invention.
6 is a diagram for explaining a method of learning a generative adversarial neural network by combining encoders and decoders between different domains in the present invention.
7 is a diagram illustrating a method of learning a generative adversarial neural network using an identifier in the present invention.
8A is a table showing an algorithm of a method for controlling a weather effect according to an embodiment of the present invention.
8B is a table showing an algorithm of a method for controlling a weather effect according to an embodiment of the present invention.
9 is a flowchart illustrating an operation of a method for controlling a weather effect according to an embodiment of the present invention.
10 is a block diagram of an apparatus for controlling a weather effect according to another embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term “and/or” includes a combination of a plurality of related described items or any of a plurality of related described items.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

Unless otherwise defined, all terms including technical or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present application. Does not.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a weather effect control system that applies a fog effect to a clear image.

Referring to FIG. 1, an algorithm for applying a fog effect to a clear image can be seen. According to Fig. 1 and Equations 1 and 2, when a clear image is input to an image encoder, and a randomly sampled style feature and a context feature of the input clear image are input to the image decoder, the image decoder An image with an effect can be output.

는 는 투과 계수를 의미할 수 있고, d는 영상의 깊이 맵을 의미할 수 있다.Here, I may refer to an image to which a fog effect is applied, J may refer to a clear image, A may refer to the global atmospheric brightness, and t may refer to a transmission map of the image,

May denote a transmission coefficient, and d denote a depth map of an image.

However, the algorithm is a problem because the depth map (d) of the image must be essential. Therefore, by using a Generative Adversarial Network (GAN), which will be described later, a fog effect can be applied to a clear image without a depth map of the image.

2 is a conceptual diagram illustrating a generative hostile neural network (GAN).

The Generative Adversarial Network (GAN) can refer to a deep neural network structure composed of two networks. In other words, it is possible to create a fake image that looks like the real through a competition between a model that creates a fake like real and a model that determines its authenticity.

Referring to FIG. 2, a generative adversarial neural network may include two models called a generator and a discriminator. In other words, the generator learns real data and generates false data based on it, thereby generating false data close to real. In addition, the identifier can identify whether the data provided by the creator is real or false.

3 is a block diagram of an apparatus for controlling a weather effect in a marine image according to an embodiment of the present invention.

Referring to FIG. 3, an apparatus 1000 for adjusting a weather effect within a maritime image includes a first generative hostile neural network learning unit 1100, a second generative hostile neural network learning unit 1200, and a weather effect adjusting unit 1300. ) Can be included.

Here, the first generative hostile neural network learning unit 1100 generates an image to which a fog effect is applied from a clear image by the creator, and compares the image to which the identifier is generated with the fog effect to the image including the actual fog effect. It can be learned to identify an image including an actual fog effect.

In addition, the second generative hostile neural network learning unit 1200 allows the generator to generate a clear image from the image to which the fog effect is applied, and to identify the actual clear image by comparing the clear image with the identifier generated with the actual clear image. Can be learned.

Here, the first generative hostile neural network learning unit 1100 may learn to identify an actual clear image, and the second generative hostile neural network learning unit 1200 may identify an image including the actual fog effect. Can be learned to be.

In addition, the meteorological effect controller 1300 may generate and remove the meteorological effect in the maritime image by using the learning results of the first generative hostile neural network learning unit and the second generative hostile neural network learning unit.

In addition, errors in the identification result of the identifier and the image generated by the creator can be back propagated, and the hostile generated neural network can be learned by gradient descent by alternately between the generator and the identifier.

4 is a block diagram of an apparatus for controlling a weather effect in a marine image according to an embodiment of the present invention.

4, the first generative adversarial neural network learning unit 1100 may include an image generator 1 1110 and an image identifier 1 1120, and the second generative adversarial neural network training unit 1200 It may include a creator 2 1210 and an image identifier 2 1220.

In addition, the image generator 1 1110 may include an image encoder 1 1111 and an image decoder 1 1112, and the image generator 2 1210 includes an image encoder 2 1211 and an image decoder 2 1212. can do.

Here, the image generator 1 1110 and the image generator 2 1210 may exchange information with each other to learn a generative adversarial neural network. Mutual information exchanged between the video creators will be described later.

5 is a diagram for explaining a method of learning a generative adversarial neural network using an encoder and a decoder in the same domain in the present invention.

The video encoders 1111 and 1211 may be composed of a style encoder and a context encoder. Therefore, referring to FIG. 5, the style encoder of the image encoder 1 1111 can receive a real clear image _{I 1 to generate the style feature S e1} of the real clear image, and the context encoder can generate a real clear image Context feature (C ₁ ) can be extracted by receiving (I _{1 ).} Also, the image decoder 1 1112 may generate _{an image I 1} ′ including feature information of the clear image by receiving style features and context features of the actual clear image.

Similarly, the style encoder of the image encoder 2 1211 _{can receive the image (I 2} ) including the actual fog effect and generate _{the style feature (S e2} ) of the image including the actual fog effect. The context feature C ₂ may be extracted by _{receiving the image I 2} including the fog effect. In addition, the image decoder 2 1212 may generate _{an image I 2} ′ including feature information of the image to which the fog effect is applied by receiving style features and context features of the image including the actual fog effect.

_{Referring back to FIG. 4, the image ID 1 is actually clear by comparing the image (I 1} ′) including the characteristic information of the clear image generated by the image decoder 1 and the actual clear image input to the image identifier 1 (1120). A first generative adversarial neural network may be trained to identify an image.

_{Similarly, by comparing the image (I 2} ′) including feature information of the image to which the fog effect is applied generated by the image decoder 2 and the image including the actual fog effect input to the image identifier 2 1220, the image identifier 2 A second generative adversarial neural network may be trained to identify an image including an actual fog effect.

Accordingly, after the learning is completed, the style feature may include information related to the fog effect, and the context feature may include information about the appearance of the object in the marine image. In addition, since the style feature can follow a probability distribution arbitrarily designated by the user, the user can arbitrarily change the style feature, and thus the fog effect to be applied to a clear image can be adjusted as the style feature changes.

6 is a diagram for explaining a method of learning a generative adversarial neural network by combining encoders and decoders between different domains in the present invention.

Referring to FIG. 6, the image to which the fog effect is applied by inputting the _{context feature (C 1} ) of the clear image extracted from the image encoder 1 (1111) and the style feature (S _{2) randomly sampled to the image decoder 2 (1212). An image I 12} including feature information of may be generated.

_{Similarly, by inputting the context feature (C 2-} ) and the randomly sampled style feature (S _1- ) of the image to which the fog effect extracted from the image encoder 2 (1211) is applied to the image decoder 1 (1112), the characteristics of a clear image An image I ₂₁ including information may be generated.

_{Referring back to FIG. 4, the image I 12} including feature information of the image to which the fog effect generated by the image decoder 2 1212 is applied and the actual fog effect input to the image identifier 2 1220 are included. The second generative adversarial neural network may be trained so that the image identifier 2 can identify the image including the actual fog effect by comparing the images.

_{Similarly, by comparing the image I 21} including the characteristic information of the clear image generated by the image decoder 1 1112 with the actual clear image input to the image identifier 1 1120, the image identifier 1 identifies the actual clear image. The first generative adversarial neural network may be trained to do so.

7 is a diagram illustrating a method of learning a generative adversarial neural network using an identifier in the present invention.

Referring to FIG. 7, the image identifier 1 1120 is a clear image (I _{1 ′) generated using the image encoder 1 and the image decoder 1 or the clear image (I 21} ′) generated using the image encoder 2 and the image decoder 1. ) Is received as an input and compared with an actual clear image (I ₁ ), the first generative adversarial neural network may be trained so that the image identifier 1 identifies an actual clear image.

_{Similarly, the image identifier 2 1220 is an image (I 2} ′) with a fog effect generated using the image encoder 2 and the image decoder 2, or a fog effect generated using the image encoder 1 and the image decoder 2. The second generative hostile neural network may be trained so that the image identifier 2 identifies an image including the actual fog effect by receiving the image I _{12 as an input and comparing it with the image I 2 including the actual fog effect.}

Here, the image identifier may be returned as a probability value between 0 and 1 by identifying an actual image and a generated image. Here, 1 may be a value returned when the video identifier identifies the image generated by the image creator as an actual image, and 0 is the image identifier identifying the image generated by the image creator as an image for which the image identifier is generated. May be a return value.

Also, the image identifier may receive feedback from the correct answer value of the actual image, and the image creator may receive feedback from the image identifier.

8A and 8B are tables showing an algorithm of a method for controlling a weather effect according to an embodiment of the present invention.

Referring to FIGS. 8A and 8B, first, the style encoders of the video encoders 1111 and 1211 may be composed of a convolutional layer, a pooling layer, and an output layer. Here, the output layer may be configured as a convolutional layer. Here, the pooling layer may mean an adaptive average pooling layer, and the pooling layer has a kernel size of H x W, so that no matter what input is received, the output is always 1 x The style encoder can be average pooled so that it can be 1.

In addition, the context encoder of the video encoder may be composed of a convolution layer, a residual block, and an output layer. Here, the output layer may be composed of a recycler block. In addition, the resider block may be composed of a 3 x 3 convolutional layer, and two convolutional layers per residual connection may be used, and a total of four residual connections may exist.

In addition, the image decoders 1112 and 1212 may be composed of a fully-connected layer, a parameter output layer, a resider block, an upsampling layer, a convolution layer, and an output layer. Here, the output layer may be configured as a convolutional layer. In addition, the fully connected layer may mean a fully connected layer for adaptive instance normalization (AdaIN). In addition, the parameter output layer can mean a layer that outputs parameters for adaptive instance normalization, and because 256 Mean values and 256 Standard Deviations can be allocated per resider block, the parameter is output. The layer can output a total of 2048 dimensional vectors. In addition, the upsampling layer may refer to a layer that upsampling (Nearest Neighbor Upsampling) pixels that are adjacent to each other by the size of the kernel.

Also, the image identifiers 1120 and 1220 may be composed of a convolution layer and an output layer.

An algorithm of a method of adjusting a weather effect in a marine image according to an embodiment of the present invention according to the above FIGS. 8A and 8B and Equations 3 to 25 is as follows.

은 실제 청명한 영상을 의미할 수 있고,

는 실제 안개 낀 영상을 의미할 수 있고,

는 청명한 영상의 인코더를 의미할 수 있고,

는 안개 효과를 포함하는 영상의 인코더를 의미할 수 있고,

는 영상 인코더

_-를 이용하여 추출한 영상

의 맥락 특징을 의미할 수 있고,

는 영상 인코더

를 이용해 추출한

_-의 스타일 특징을 의미할 수 있다. 또한,

는 영상 인코더의 출력을 의미할 수 있다.here,

Can mean a real clear image,

May mean an actual foggy image,

May mean an encoder of a clear image,

May mean an encoder of an image including a fog effect,

Video encoder

_-Extracted using the image

Can mean contextual features of,

Video encoder

Extracted using

_- it can mean the style Features. Also,

May mean the output of the video encoder.

는 신경망을 통해 재구성된 영상을 의미할 수 있고,

은 청명한 영상의 디코더를 의미할 수 있고,

은 안개 효과를 포함하는 영상의 디코더를 의미할 수 있고,

는 맥락 특징

와 스타일 특징

에 대한 영상 디코더의 출력을 의미할 수 있다.here,

Can mean an image reconstructed through a neural network,

May mean a clear video decoder,

May mean a decoder of an image including a fog effect,

Is a context feature

And style features

It may mean an output of an image decoder for.

는 미리 정해 놓은 임의의 분포도에 따라 임의로 샘플링된 스타일 특징을 의미할 수 있고,

는 생성된 안개 효과가 부여된 영상을 의미할 수 있고,

은 생성된 청명한 영상을 의미할 수 있다. here,

May mean a style feature randomly sampled according to a predetermined distribution map,

May mean an image to which the generated fog effect is applied,

May mean the generated clear image.

은 안개 효과가 부여된 영상에서 안개 효과를 제거한 영상을 의미할 수 있고,

는 안개 효과가 제거된 영상에서 안개 효과를 부여한 영상을 의미할 수 있다.here,

May mean an image in which the fog effect is removed from the image to which the fog effect is applied,

May mean an image to which a fog effect is applied from an image from which the fog effect is removed.

은 청명한 영상의 재구성 손실 함수(Reconstruction Loss)를 의미할 수 있고,

는 안개 효과가 포함된 영상의 재구성 손실 함수를 의미할 수 있다.here,

May mean Reconstruction Loss of a clear image,

May mean a reconstruction loss function of an image including a fog effect.

는 안개 효과가 포함된 영상의 스타일 특징 재구성 손실 함수를 의미할 수 있다.here,

May mean a loss function of reconstruction of style features of an image including a fog effect.

은 청명한 영상의 맥락 특징 재구성 손실 함수를 의미할 수 있고,

는 안개 효과가 포함된 영상의 맥락 특징 재구성 손실 함수를 의미할 수 있다.here,

May mean the loss function of reconstructing the context feature of a clear image,

May mean a loss function for reconstructing the context feature of the image including the fog effect.

은 청명한 영상의 순환 일관성 손실 함수(Cycle Consistency Loss)를 의미할 수 있고,

는 안개 효과가 포함된 영상의 순환 일관성 손실 함수를 의미할 수 있다.here,

May mean the cyclic consistency loss function (Cycle Consistency Loss) of a clear image,

May mean a cyclic coherence loss function of an image including a fog effect.

은 청명한 영상의 식별자를 의미할 수 있고,

는 안개 효과가 포함된 영상의 식별자를 의미할 수 있고,

는 영상

에 대한 식별자

의 출력을 의미할 수 있고,

은 영상 식별자 1을 사용한 적대적 손실 함수를 의미할 수 있고,

는 영상 식별자 2를 사용한 적대적 손실 함수를 의미할 수 있다.here,

May mean an identifier of a clear image,

May mean the identifier of the image including the fog effect,

The video

Identifier for

Can mean the output of,

May mean a hostile loss function using image identifier 1,

May mean a hostile loss function using image identifier 2.

은 영상 식별자 1을 학습하기 위한 손실 함수를 의미할 수 있고,

는 영상 식별자 2를 학습하기 위한 손실함수를 의미할 수 있다.here,

May mean a loss function for learning the image identifier 1,

May mean a loss function for learning the image identifier 2.

는 영상 생성자 1 및 영상 생성자 2를 학습하기 위해 사용되는 손실 함수를 의미할 수 있고,

는 영상 식별자 1 및 영상 식별자2를 학습하기 위해 사용되는 손실함수를 의미할 수 있다.here,

May mean a loss function used to learn image creator 1 and image creator 2,

May mean a loss function used to learn the image identifier 1 and the image identifier 2.

The above-described algorithm has been described by taking only the fog effect among various weather effects that exist in the maritime image, but the style characteristic can follow the probability distribution arbitrarily specified by the user, so the user can arbitrarily change the style characteristic, as the style characteristic changes. Various weather effects can be adjusted to give a clear image.

9 is a flowchart illustrating a method of controlling a weather effect in a marine image according to an embodiment of the present invention.

A method for adjusting weather effects in a marine image according to an embodiment of the present invention is a method for adjusting weather effects in a marine image using a Generative Adversarial Network (GAN) consisting of a generator and an identifier (Discriminator), The first generative hostile neural network may be trained (S910) by extracting style features and context features of the image including the first weather effect from the generator of the first generative hostile neural network (GAN). Here, the plurality of generative adversarial neural networks (GANs) may include a first generative adversarial neural network and a second generative adversarial neural network. Also, an image including different weather effects may be an image including a clear image, an image to which a fog effect is applied, and an image to which various weather effects are applied.

Then, it may include regenerating the image including the second weather effect by inputting the style feature or the context feature of the image including the extracted first meteorological effect as a generator of the second generative hostile neural network (S920). have.

Subsequently, by comparing the image including the regenerated second weather effect with the image including the actual second weather effect, the identifier of the second generative hostile neural network is generated so that the image including the actual second weather effect is identified. It may include the step of training the hostile neural network (S930). Here, the identifier of the generative adversarial neural network may be returned as a probability value between 0 and 1 by comparing the generated image and the real image, and the generative adversarial neural network may be trained based on this.

Subsequently, it may include a step (S940) of adjusting the weather effect in the input maritime image using the learned first generative hostile neural network or the second generative hostile neural network. Here, by using the learned generative hostile neural network to generate and remove weather effects from the input marine image, it is possible to more specifically identify the marine object existing in the marine image.

10 is a block diagram of an apparatus for controlling a weather effect in a marine image according to another embodiment of the present invention.

According to an embodiment of the present invention, the apparatus 1000 for controlling the weather effect in a marine image includes a processor 1010 and a memory 1020 for storing at least one command executed through the processor and a result of execution of the command, and a network. It may include a transmission/reception device 1030 connected to perform communication.

The apparatus 1000 for adjusting the weather effect in the marine image may further include an input interface device 1040, an output interface device 1050, a storage device 1060, and the like. Each of the constituent elements included in the apparatus 1000 for controlling the meteorological effect in the marine image may be connected by a bus 1070 to communicate with each other.

The processor 1010 may execute a program command stored in at least one of the memory 1020 and the storage device 1060. The processor 1010 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which methods according to an embodiment of the present invention are performed. Each of the memory 1020 and the storage device 1060 may be configured with at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory 1020 may be composed of at least one of read only memory (ROM) and random access memory (RAM).

The storage device 1060 also stores the results of learning the first generative hostile neural network using the image encoder 1 and the image decoder 1 and the results of learning the second generative hostile neural network using the image encoder 1 and the image decoder 2. Can be saved. In addition, the result of learning the second generative hostile neural network using the image encoder 2 and the image decoder 2 and the result of learning the first generative hostile neural network using the image encoder 2 and the image decoder 1 may be stored.

Here, the at least one command includes an instruction for learning the first generative hostile neural network by extracting the style feature and the context feature of the image including the first meteorological effect from the generator of the first generative hostile neural network (GAN); A command for regenerating an image including a second weather effect by inputting a style feature or a context feature of the image including the extracted first weather effect as a generator of a second generative hostile neural network; The second generative hostility is such that the identifier of the second generative hostile neural network identifies the image including the actual second weather effect by comparing the regenerated image including the second weather effect with the image including the actual second weather effect. Instruction to train a neural network; And a command to adjust the weather effect in the marine image input by using the learned first generative hostile neural network or the second generative hostile neural network.

The operation of the method according to the embodiment of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. In addition, a computer-readable recording medium may be distributed over a network-connected computer system to store and execute a computer-readable program or code in a distributed manner.

Further, the computer-readable recording medium may include a hardware device specially configured to store and execute program commands, such as ROM, RAM, and flash memory. The program instructions may include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

While some aspects of the invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or characteristic of a method step. Similarly, aspects described in the context of a method can also be represented by a corresponding block or item or a feature of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

Although the above has been described with reference to the preferred embodiments of the present invention, those skilled in the art will be able to variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. You will understand that you can.

Claims (1)

As a method of controlling weather effects in a marine image using a generative adversarial network (GAN) including a generator that generates an image and a discriminator that distinguishes the generated image from the actual image,
Learning a first generative adversarial neural network by extracting a style feature and a context feature of an image including a first meteorological effect from a generator of a first generative adversarial neural network (GAN);
Regenerating an image including a second meteorological effect by inputting a style feature or a context feature of the image including the extracted first meteorological effect as a generator of a second generative hostile neural network;
The second generative hostility is such that the identifier of the second generative hostile neural network identifies the image including the actual second weather effect by comparing the regenerated image including the second weather effect with the image including the actual second weather effect. Training a neural network; And
And adjusting a weather effect in an input marine image using the learned first generative hostile neural network or a second generative hostile neural network.

Images