KR102427861B1 - Apparatus and method for generating underwater image data - Google Patents

Abstract

The present disclosure relates to an apparatus for generating underwater image data using a GAN, a pixel distribution predictor for extracting pixel distribution characteristics of an input image, a generator for generating underwater image data using the extracted pixel distribution characteristics, and an input By including a determining unit for determining whether the image and the underwater image data is a forged image, it is possible to effectively generate a plurality of underwater image data.

Description

Apparatus and method for generating underwater image data {APPARATUS AND METHOD FOR GENERATING UNDERWATER IMAGE DATA}

The present disclosure provides an apparatus and method for generating underwater image data.

Underwater imaging is used for ocean exploration. For example, side-scan sonars are an important element in various marine industries, such as subsea topography surveying and debris detection. The side scan sonar data composes an underwater image in line units using the received signal reflected from the target or terrain. According to the operating frequency, the side scan sonar image can be classified into high frequency and low frequency types, and can be simply divided into a seabed image and an object image according to the presence or absence of an object. A number of side-scan sonar images are needed for related research such as underwater image recognition using side-scan sonar images. Acquisition of multiple side-scan sonar images requires extensive field work and actual experiments at sea. Since actual experiments have to be performed at sea, high costs are incurred and a lot of time is consumed. Collecting and processing various data is not an easy task. It is difficult to acquire a large number of side scan sonar images due to high cost and time consumption.

Generative Adversarial Networks (GANs) are useful in various fields such as image synthesis, content generation, and data augmentation. In addition, GAN is widely applied to sonar image simulation tasks. GANs leverage the high-function extraction capabilities of convolutional neural networks (CNNs), which means that realistic synthetic data can be generated from real data.

Accordingly, in order to acquire a plurality of side-scan sonar images, a method of generating sonar image data using a GAN is required.

10-2020-0009852 A 10-2018-0065417 A

An apparatus and method for generating underwater image data are provided. Another object of the present invention is to provide a recording medium in which a program for executing the method in a computer is recorded. The technical problems to be achieved by the present embodiment are not limited to the technical problems as described above, and other technical problems may be inferred from the following embodiments.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure includes: a pixel distribution prediction unit for extracting pixel distribution characteristics of an input image; a generator for generating underwater image data by using the extracted pixel distribution characteristics; and a determining unit that determines whether the input image and the underwater image data are forged images.

In addition, the pixel distribution prediction unit learns the pixel distribution of the input image through unsupervised learning, extracts the pixel distribution characteristic from the learned pixel distribution, and uses the extracted pixel distribution characteristic. An apparatus for generating underwater image data for generating a random vector may be provided.

Also, the pixel distribution prediction unit may provide an apparatus for generating underwater image data including at least one of a convolution layer and a fully-connected layer (FC).

Also, the generator may provide an apparatus for generating underwater image data that generates the underwater image data using a Pix2pixHD model.

In addition, the generator, an upsampling convolution layer (upsampling convolution layer) and an underwater image data comprising at least one of a spade block for adding a segmentation map to the output of the upsampling convolution layer A generating device may be provided.

Also, the spade block may provide an apparatus for generating underwater image data including at least one of a convolution layer and a batch normalization layer.

In addition, the determining unit may provide an apparatus for generating underwater image data using a Patch Generative Adversarial Network (GAN) structure.

In addition, the determining unit, an underwater image data generating device for determining whether the input image concatenated in the divided map and the channel and the underwater image data connected in the divided map and the channel are the forged images can provide

In addition, the pixel distribution predictor is learned through a loss function, the loss function may provide an apparatus for generating underwater image data in which the following Equation (1).

[Equation 1]

는 상기 손실 함수, 상기

는 KL 다이버전스(KL divergence) 함수, 상기

는 상기 입력 영상, 상기

는 잠재 변수(latent variable), 상기

은 상기 입력 영상에 따른 상기 학습된 픽셀 분포, 상기

는 표준 가우스 분포임)(here, the

is the loss function, where

is the KL divergence function, where

is the input image, the

is a latent variable,

is the learned pixel distribution according to the input image, the

is the standard Gaussian distribution)

In addition, the objective function of the generating unit and the determining unit may provide an apparatus for generating underwater image data, which is the following Equation (2).

[Equation 2]

은 GAN 손실, 상기

은 특징 매칭 손실(feature matching loss), 상기

는 상기 생성부, 상기

는 상기 판별부, 상기

는 상기 입력 영상, 상기

는 상기 입력 영상의 크기에 해당되는 시맨틱 라벨 지도(semantic label map) 값임)(here, the

is the GAN loss, above

is the feature matching loss,

is the generator, the

is the determination unit, the

is the input image, the

is a semantic label map value corresponding to the size of the input image)

A second aspect of the present disclosure provides a method for generating underwater image data, the method comprising: extracting pixel distribution characteristics of an input image; generating underwater image data by using the extracted pixel distribution characteristics; and determining whether the input image and the underwater image data are forged images.

A third aspect of the present disclosure may provide a recording medium in which a program for executing the method according to claim 11 in a computer is recorded.

A fourth aspect of the present disclosure is an underwater image data generating apparatus, comprising: a memory; and a processor, wherein the processor extracts pixel distribution characteristics of the input image, generates underwater image data using the extracted pixel distribution characteristics, and determines whether the input image and the underwater image data are forged images. It is possible to provide an underwater image data generating device for determining whether or not.

The present disclosure may generate underwater image data using a GAN. Specifically, by using the GAN, the generating unit and the discriminating unit can perform competitive learning with each other, and as the learning progresses, it is possible to generate underwater image data similar to the input image. In addition, it is possible to effectively generate underwater image data even at different frequencies.

Since actual experiments do not have to be conducted at sea, the cost may be low and the efficiency may be high, and excellent quality and detailed image expression may be possible.

1 is a block diagram of an apparatus for generating underwater image data according to an exemplary embodiment.
2 is a diagram for describing a pixel distribution predictor according to an exemplary embodiment.
3 is a view for explaining a generator and a determiner according to an embodiment.
FIG. 4 is a view for explaining the structure of the spade block of FIG. 3 .
5 is a diagram for comparing an input image and underwater image data.
6 is a table for explaining the performance of the apparatus for generating underwater image data according to an embodiment.
7 is a block diagram of an apparatus for generating underwater image data according to another embodiment.

The terms used in the present embodiments have been selected as currently widely used general terms as possible while considering the functions in the present embodiments, but this may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. have. In addition, in certain cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant part. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents throughout the present embodiments, rather than the simple name of the term.

Since the present embodiments may have various changes and may have various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosure form, and it should be understood that all modifications, equivalents and substitutes included in the spirit and scope of the present embodiments are included. The terms used herein are used only for description of the embodiments, and are not intended to limit the present embodiments.

Unless otherwise defined, terms used in the present embodiments have the same meanings as commonly understood by those of ordinary skill in the art to which the present embodiments belong. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present embodiments, have an ideal or excessively formal meaning. should not be interpreted.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or by circuit configurations for a given function. Also, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as an algorithm running on one or more processors. Also, the present disclosure may employ prior art for electronic configuration, signal processing, and/or data processing, and the like.

Also, terms including ordinal numbers such as 'first' or 'second' used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms may be used for the purpose of distinguishing one component from another.

In addition, the connecting lines or connecting members between the components shown in the drawings only exemplify functional connections and/or physical or circuit connections. In an actual device, a connection between components may be represented by various functional connections, physical connections, or circuit connections that are replaceable or added.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating underwater image data according to an exemplary embodiment.

Referring to FIG. 1 , the apparatus 100 for generating underwater image data may include a pixel distribution predictor 110 , a generator 120 , and a determiner 130 .

) 벡터, 분산(

²) 벡터를 의미할 수 있다.The pixel distribution predictor 110 may extract a pixel distribution characteristic of the input image. The pixel distribution predictor 110 may mean a pixel-level feature extractor, and may effectively extract the style and content of the input image by extracting the pixel-level feature. The input image is an image input to the underwater image data generating apparatus 100 and may mean a side scan sonar image. The pixel distribution characteristic is the pixel distribution parameter vector, the average (

) vector, variance(

² ) can mean a vector.

In an embodiment, the pixel distribution prediction unit 110 learns the pixel distribution of the input image through unsupervised learning, extracts a pixel distribution characteristic from the learned pixel distribution, and uses the extracted pixel distribution characteristic. Thus, a random vector can be generated. Unsupervised learning may refer to a learning method in which a computer finds regularity of inputs using training data having only input values in machine learning.

The pixel distribution prediction unit 110 may be learned through a loss function. The loss function can be expressed by Equation 1 below.

[Equation 1]

는 손실 함수를 의미하고,

는 KL 다이버전스(KL divergence) 함수를 의미하고,

는 상기 입력 영상을 의미하고,

은 입력 영상에 따른 학습된 픽셀 분포를 의미하고,

는 표준 가우스 분포를 의미할 수 있다.

는 잠재 변수(latent variable)로, 픽셀 분포 예측부(110)의 출력을 의미할 수 있다. here,

is the loss function,

is KL divergence (KL divergence) function,

means the input image,

is the learned pixel distribution according to the input image,

may mean a standard Gaussian distribution.

is a latent variable and may mean an output of the pixel distribution predictor 110 .

The difference between the pixel distribution generated from the input image and the pixel distribution generated from the random variable can be calculated using the KL divergence function as the loss function. By using the KL divergence function, we can learn the pixel distribution and force it to follow a normal distribution.

The generator 120 may generate underwater image data by using the extracted pixel distribution characteristic. The generator 120 may generate underwater image data having a shape similar to that of the input image by using the random vector generated by the pixel distribution predictor 110 . The underwater image data may refer to a side-scan sonar image generated by the underwater image data generating apparatus. The generator 120 will be described in detail later with reference to FIG. 3 .

The determining unit 130 may determine whether the input image and the underwater image data are forged images. The generating unit 120 and the determining unit 130 may compete with each other for learning. The determining unit 130 may evaluate the underwater image data generated by the generating unit 120 , and the generating unit 120 may generate underwater image data more similar to the input image by reflecting the evaluation result of the determining unit 130 . can As learning progresses, the underwater image data generated by the generator 120 may be similar to the input image.

2 is a diagram for describing a pixel distribution predictor according to an exemplary embodiment.

Referring to FIG. 2 , an input image may pass through a convolution (conv) layer, a reshape layer, and a linear layer.

는 입력 영상의 너비 및 높이 각각을 반으로 줄이는 것을 의미할 수 있다. 예를 들어, 크기가 100 × 100인 입력 영상이 컨볼루션 레이어를 통과하면 입력 영상의 크기가 50 × 50으로 줄어들 수 있다. 도 2의

은 입력 영상의 크기를 줄였으므로 채널을 늘리는 것을 의미할 수 있다. In an embodiment, the pixel distribution predictor (eg, the pixel distribution predictor 110 of FIG. 1 ) may include at least one of a convolution layer and a fully-connected layer (FC). . The convolution layer may refer to the conv layer of FIG. 2 , and a convolution operation may be performed. A plurality of convolutional layers may exist, and preferably, there may be six convolutional layers. In the convolutional layer, 3 × 3 means the size of the convolutional layer,

may mean reducing each of the width and height of the input image by half. For example, when an input image having a size of 100 × 100 passes through a convolutional layer, the size of the input image may be reduced to 50 × 50. 2 of

Since the size of the input image is reduced, it may mean increasing the channel.

The Reshape layer may transform the shape of the output of the convolution layer so that the output of the convolution layer becomes the input of the Linear layer. For example, when the output of the convolution layer is two-dimensional, the reshape layer may transform the output of the two-dimensional convolution layer into one dimension so that the output of the linear layer becomes the input of the linear layer.

The pixel distribution predictor may include a plurality of FC layers, and preferably may include two FC layers. The Linear layer of FIG. 2 may correspond to the FC layer.

) 벡터, 분산(

²) 벡터)이 추출될 수 있고, 추출된 픽셀 분포 특성을 이용하여 랜덤 벡터가 생성될 수 있다.By passing the input image through the six convolutional layers, the Reshape layer, and the Linear layer of the pixel distribution prediction unit, the pixel distribution characteristics (average (

) vector, variance(

² ) vector) may be extracted, and a random vector may be generated using the extracted pixel distribution characteristics.

3 is a view for explaining a generator and a determiner according to an embodiment.

Referring to FIG. 3 , the apparatus 300 for generating underwater image data may include a pixel distribution predictor 310 , a generator 320 , and a determiner 330 . The generator 320 may generate underwater image data by using a segmentation map from the random vector output from the pixel distribution predictor 310 . The underwater image data is an image similar to the input image, and may mean a side scan sonar image generated by the underwater image data generating apparatus 300 . The apparatus 300 for generating underwater image data of FIG. 3 , the pixel distribution prediction unit 310 , the generation unit 320 , and the determining unit 330 include the underwater image data generation apparatus 100 of FIG. 1 , the pixel distribution prediction unit ( 110), the generating unit 120, and the determining unit 130, so overlapping content is omitted.

In an embodiment, the generator 320 may generate the underwater image data by using the Pix2pixHD model. Pix2pixHD is a method of synthesizing high-resolution realistic images from a semantic label map using a Generative Adversarial Network (GAN), and may refer to a GAN model of a special structure designed to generate an HD image. Since the generation unit 320 starts the generation process with the random vector generated by the pixel distribution prediction unit 310, only the upsampling convolution layers 321, 322, and 323 of Pix2pixHD may be included.

The generator 320 may include at least one of upsampling convolutional layers 321 , 322 , and 323 and spade blocks 324 , 325 , and 326 . The generator 320 may include a plurality of upsampling convolution layers 321 , 322 , 323 and a plurality of spade blocks 324 , 325 , 326 , and preferably, the upsampling convolution layer 321 , The number of 322 and 323 and the spade blocks 324 , 325 and 326 may be seven, respectively. In addition, the upsampling convolution layers 321 , 322 , 323 and the spade blocks 324 , 325 , 326 of the generator 320 may be alternately disposed one by one, and the upsampling convolution layers 321 , 322 , 323 . The structure in which the spade blocks 324 , 325 , and 326 follow may be repeated seven times. For example, in the generator 320 , the upsampling convolution layer 321 , the spade block 324 , the upsampling convolution layer 322 , the spade block 325 , the upsampling convolution layer 323 , and Spade blocks 326 may be placed or performed in the order listed.

The spade blocks 324 , 325 , and 326 may add a segmentation map to the output of the upsampling convolutional layers 321 , 322 , and 323 . The output of the split map and the immediately preceding upsampling convolution layer 321 , 322 , 323 may be an input of the spade blocks 324 , 325 , 326 . For example, the input of the spade block 324 may be the output of the segmentation map and upsampling convolution layer 321 . The input of the spade block 325 may be the output of the segmentation map and upsampling convolution layer 322 . The spade blocks 324 , 325 , and 326 will be described in detail later with reference to FIG. 4 .

The determining unit 330 may determine whether the underwater image data generated by the input image and the generator 320 is a forged image.

In an embodiment, the determining unit 330 may determine whether the input image concatenated in the segmented map and the channel and the underwater image data concatenated in the segmented map and the channel are forged images. Concatenate in FIG. 3 means a concatenate operation, and may mean that two images are connected to each other to make one image. For example, the divided map and the input image may be connected to each other in a channel to make one image, and the underwater image data generated by the divided map and generator 320 may be connected to each other in the channel to make one image. can

The determining unit 330 may determine whether the input image connected to the divided map and the channel and the underwater image data connected to the divided map and the channel are forged images to calculate a result value Y.

In an embodiment, the determining unit 330 may use a Patch Generative Adversarial Network (GAN) structure. The patch GAN may refer to a method of determining whether a forged image is a forged image in a patch unit of a specific size rather than the entire area, and taking an average of the result. The number of parameters may be reduced because the operation is performed while a sliding window passes through the patch unit rather than the entire area. Accordingly, the calculation speed may be increased. In addition, since it is not affected by the size of the image, it may be structurally flexible.

The objective function of the generating unit 320 and the determining unit 330 may be composed of two loss functions: a GAN loss and a feature matching loss. The objective function can be expressed by the following Equation (2).

[Equation 2]

은 GAN 손실을 의미하고,

은 특징 매칭 손실(feature matching loss)을 의미하고,

는 생성부(320),

는 판별부(330),

는 입력 영상,

는 입력 영상의 크기에 해당되는 시맨틱 라벨 지도 값을 의미할 수 있다. 특징 매칭 손실은 생성부(320)가 여러 해상도에서 사실적인 영상을 생성하도록 하여 훈련 과정을 안정화시키는 데 도움을 줄 수 있고, GAN 손실은 분할 지도와 함께 입력 영상의 조건부 분포를 학습하는 데 중점을 둘 수 있다. here,

means GAN loss,

is a feature matching loss,

is the generator 320,

is the determination unit 330,

is the input image,

may mean a semantic label map value corresponding to the size of the input image. The feature matching loss can help stabilize the training process by causing the generator 320 to generate realistic images at different resolutions, and the GAN loss focuses on learning the conditional distribution of the input image with the segmentation map. can be put

FIG. 4 is a view for explaining the structure of the spade block of FIG. 3 .

Referring to FIG. 4 , the spade block 400 of FIG. 4 may include a convolution layer and a batch normalization layer. The spade block 400 and the upsampling convolutional layer 410 of FIG. 4 correspond to the spade blocks 324, 325, 326 and the upsampling convolutional layers 321, 322, and 323 of FIG. content is omitted.

The input of the spade block 400 may be the output of the split map and the previous upsampling convolutional layer. The previous upsampling convolutional layer may mean an upsampling convolutional layer immediately before the spade block 400 to which the output of the previous upsampling convolutional layer is input. The output of the spade block 400 may be an input of the next upsampling convolution layer 410 . Since the spade block 400 is included in the generator, it is possible to additionally obtain information on the divided map.

In an embodiment, the spade block 400 may include at least one of a convolutional layer and a placement normalization layer. The batch normalization layer is a layer that performs batch normalization, and may mean rearranging the input so that the mean is 0 and the variance is 1.

) 및 시프팅 요소(shifting factor:

)로 변환될 수 있다. 스케일링 요소는 정규화된 데이터에 대한 스케일 조정 파라미터를 의미할 수 있고, 시프팅 요소는 정규화된 데이터에 대한 이동 조정 파라미터를 의미할 수 있다. The segmentation map may pass through the convolutional layer of the spade block 400, and a scaling factor for the batch normalization layer:

) and a shifting factor:

) can be converted to The scaling factor may mean a scale adjustment parameter for normalized data, and the shifting element may mean a movement adjustment parameter for normalized data.

5 is a diagram for comparing an input image and underwater image data.

OURS HO (High frequency Object) of FIG. 5 may mean an underwater image including an object measured at a high frequency, and OURS LO (Low Frequency Object) may mean an underwater image including an object measured at a low frequency. . HO and LO may contain more complex subsea environments than pure subsea environments such as canyons or ridges. A first row may represent a segmentation map, a second row may represent an input image, and a third row may represent underwater image data. High frequency and low frequency may include ranges easily understood by those of ordinary skill in the art to which the present invention pertains.

The input image of the HO of FIG. 5 includes the rocks of the seabed displayed on the divided map. The input image may consist of three areas: a highlighted area (facing the side scan sonar signal), a shaded area (the side scan sonar signal is blocked by the highlighted area), and a background (undersea area). When comparing the input image and the underwater image data, it can be seen that the underwater image data can reflect the three areas well while maintaining high image quality compared to the input image. The underwater image data may refer to a side-scan sonar image generated by the apparatus for generating underwater image data according to the present disclosure.

LO of FIG. 5 represents an underwater image generated at a low frequency. Since the low frequency side scan sonar operates in a deep sea environment, the LO may have relatively lower image quality and resolution than the HO. Since a segmentation map can be added to the underwater image data, it can be realistically expressed in a detailed structure and low-level noise can be included.

6 is a table for explaining the performance of the apparatus for generating underwater image data according to an embodiment.

6, Acc (Acuracy) may mean pixel accuracy, and mIoU (Mean Intersection over Union) may mean an average combination intersection point. Pixel accuracy refers to the proportion of correctly classified pixels. The IoU may refer to an overlapping area in which the segmented result and the label map are divided by the combined area between the segmented result and the label map.

Acc and mIoU can be used as indicators of semantic segmentation because accurate prediction results can be calculated using the underwater image data when the underwater image data is sufficiently realistic. The semantic segmentation method of PSPNet and FCN8s can be used to evaluate the segmentation results on the underwater image dataset and the input image set. The generated data (%) of FIG. 6 may mean a ratio of the input image and the underwater image data. For example, when the generated data is 0%, it may mean that the ratio of the underwater image data is 0%. When the generated data is 50%, it may mean that the ratio of the underwater image data and the input image is 50%, respectively.

Acc and mIoU may mean indicators indicating reliability of underwater image data. For example, if the values of 0% and 50% of the generated data are similar, or the value of Acc is higher at 50% than 0%, it may mean that the underwater image data is reliable.

In the case of using PSPNet, as the ratio of the underwater image data to the input image increases, the metric may be improved, and the reliability of the underwater image data may be high when the generated data is 50%. For example, when the generated data is 0% at high frequency, Acc may be 87.62%, mIoU may be 65.95%, and when generated data is 50%, Acc may be 90.95%, and mIoU may be 73.84%. As another example, when the generated data is 0% at a low frequency, Acc is 87.71%, mIoU is 68.08%, and when generated data is 50%, Acc is 90.95%, and mIoU is 73.84%. Since Acc and mIoU when generated data is 50% are higher than when generated data is 0%, it can be seen that the underwater image data is reliable.

When FCN8s is used, when the ratio of the underwater image data to the input image is less than 50% of the generated data, the reliability of the underwater image data may be high. For example, at high frequency, when the generated data is 0%, the Acc is 88.72%, the mIoU is 69.19%, and when the generated data is 10%, the Acc is 89.02%, the mIoU is 69.8%, and the generated data is 30 In the case of %, Acc may be 89.01%, and mIoU may be 69.89%. Acc and mIoU when generated data is 10% and 30% can be higher than when generated data is 0%, Acc is highest when generated data is 10%, and mIoU is highest when generated data is 30% can be the highest.

As another example, when the generated data is 0% at a low frequency, Acc may be 85.60%, mIoU may be 73.75%, Acc may be 86.84% when generated data is 20%, and mIoU may be 75.43%. Acc and mIoU when generated data is 20% may be higher than when generated data is 0%, and Acc and mIoU when generated data is 20% may be the highest. When the generated data is 10%, 20%, and 30%, Acc and mIoU are higher than when it is 0%, so it can be seen that the underwater image data is reliable. Accordingly, the device according to the present disclosure can improve the quality of the side-scan sonar image by generating reliable underwater image data.

7 is a block diagram of an apparatus for generating underwater image data according to another embodiment.

Referring to FIG. 7 , the apparatus 700 for generating underwater image data may include a memory 710 and a processor 720 . Only the components related to the embodiment are shown in the underwater image data generating apparatus 700 of FIG. 7 . Accordingly, it can be understood by those skilled in the art that other general-purpose components may be further included in addition to the components shown in FIG. 7 .

The memory 710 is hardware for storing various data processed in the underwater image data generating apparatus 700 . For example, the memory 710 may store an input image, a pixel distribution characteristic, a segmentation map, a random vector extracted from the pixel distribution characteristic, and the like. The memory 710 includes random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD- It may include ROM, Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or flash memory.

The processor 720 performs an overall function for generating the underwater image data described above with reference to FIGS. 1 to 6 .

In an embodiment, the processor 720 may extract a pixel distribution characteristic of the input image, generate underwater image data using the extracted pixel distribution characteristic, and determine whether the input image and the underwater image data are forged images. have.

The present embodiments may also be implemented in the form of a recording medium in which a program such as a program module executed by a computer is recorded. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, other data in modulated data signals, such as program modules, or other transport mechanisms, and includes any information delivery media.

Also, in this specification, "unit" may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The description of the present specification described above is for illustration, and those of ordinary skill in the art to which the content of the present specification pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be able Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The description of the above-described embodiments is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true scope of protection of the invention should be defined by the appended claims, and all differences within the scope of equivalents to those described in the claims should be construed as being included in the protection scope defined by the claims.

100: underwater image data generating device 110: pixel distribution prediction unit
120: generating unit 130: determining unit
300: Underwater image data generating device 310: Pixel distribution prediction unit
320: generating unit 330: determining unit
321: upsampling convolutional layer 322: upsampling convolutional layer
323: upsampling convolution layer 324: spade block
325: spade block 326: spade block
400: spade block 410: upsampling convolutional layer
700: underwater image data generating device 710: memory
720: processor

Claims (13)

a pixel distribution prediction unit for extracting pixel distribution characteristics of the input image;
a generator for generating underwater image data by using the extracted pixel distribution characteristics; and
A determining unit for determining whether the input image and the underwater image data are forged images;
The input image is a side scan sonar image,
The generator includes at least one of an upsampling convolution layer and a spade block that adds a segmentation map to the output of the upsampling convolution layer,
The upsampling convolution layer or the spade block is provided in plurality,
The spade block is
at least one of a convolutional layer and a batch normalization layer,
The determining unit,
A device for generating underwater image data using a Patch GAN (Generative Adversarial Network) structure. The method of claim 1,
The pixel distribution prediction unit,
Learning the pixel distribution of the input image through unsupervised learning,
extracting the pixel distribution characteristic from the learned pixel distribution,
An apparatus for generating an underwater image data for generating a random vector by using the extracted pixel distribution characteristic. The method of claim 1,
The pixel distribution prediction unit,
Convolutional layer (convolution layer) and FC layer (Fully-Connected layer) comprising at least one of the, underwater image data generating apparatus. The method of claim 1,
The generating unit,
An underwater image data generating apparatus for generating the underwater image data by using a Pix2pixHD model. delete delete delete The method of claim 1,
The determining unit,
An apparatus for generating underwater image data, which determines whether the input image concatenated in a segmented map and a channel and the underwater image data concatenated in the segmented map and the channel are the forged images. The method of claim 1,
The pixel distribution prediction unit is learned through a loss function,
The loss function is the following equation (1), underwater image data generating device.
[Equation 1]

(here, the

is the loss function, where

is the KL divergence function, where

is the input image, the

is a latent variable,

is the learned pixel distribution according to the input image, the

is the standard Gaussian distribution) The method of claim 1,
The objective function of the generating unit and the determining unit is the following equation (2), an underwater image data generating device.
[Equation 2]

(here, the

is the GAN loss, above

is the feature matching loss,

is the generator, the

is the determination unit, the

is the input image, the

is a semantic label map value corresponding to the size of the input image) A method for generating underwater image data, the method comprising:
extracting a pixel distribution characteristic of an input image;
generating underwater image data by using the extracted pixel distribution characteristics; and
determining whether the input image and the underwater image data are forged images;
The input image is a side scan sonar image,
The generating of the underwater image data using the extracted pixel distribution characteristic includes at least one of an upsampling convolution layer and a spade block for adding a split map to the output of the upsampling convolutional layer,
The upsampling convolution layer or the spade block is provided in plurality,
The spade block is
at least one of a convolutional layer and a batch normalization layer;
The step of determining whether the input image and the underwater image data are forged images,
A method of generating underwater image data using the patch GAN structure. A recording medium recording a program for executing the method according to claim 11 in a computer. An apparatus for generating underwater image data, comprising:
Memory; and
processor; including;
The processor is
Extracting the pixel distribution characteristics of the input image,
Generate underwater image data using the extracted pixel distribution characteristics,
determining whether the input image and the underwater image data are forged images,
The input image is a side scan sonar image,
Generating the underwater image data using the extracted pixel distribution characteristic includes at least one of an upsampling convolution layer and a spade block for adding a split map to the output of the upsampling convolution layer,
The upsampling convolution layer or the spade block is provided in plurality,
The spade block is
at least one of a convolutional layer and a batch normalization layer;
Determining whether the input image and the underwater image data are forged images comprises:
A device for generating underwater image data using the patch GAN structure.

Images