KR20230164517A - Method and apparatus of generating Super-Resolution image for the target object in Synthetic Aperture Radar image using deep learning - Google Patents

Abstract

A method and device for generating a super-resolution image using deep learning for a target image of interest acquired through synthetic aperture radar (SAR) are disclosed. Generate training data consisting of high-resolution (HR, High Resolution) images and low-resolution (LR) images for the target of interest, create a database with the generated training data as a component, and create super-resolution images for the target of interest. (SR, Super-Resolution) After creating a deep learning model for image generation, a process of training the deep learning model is performed using training data in the database, and the trained deep learning model is used to determine actual SAR interest. By generating ultra-high resolution images of target images, it has the effect of providing high-quality images with improved resolution compared to target images that synthetic aperture radar can provide.

Description

{Method and apparatus of generating Super-Resolution image for the target object in Synthetic Aperture Radar image using deep learning}

The present invention relates to a method and device for generating super-resolution images using deep learning for target images of interest acquired through synthetic aperture radar (SAR), and more specifically, to generate self-training data. This relates to a method and device for training a deep learning model for generating super-resolution images of a target of interest and using it to generate a super-resolution image of the target of interest from an actual synthetic aperture radar image.

Super-resolution (SR) image refers to a high-resolution (HR) image converted from a low-resolution (LR) image. Super-resolution image creation technology refers to converting a low-resolution image into a high-resolution image. Or it means a restoration technique or method. Application fields such as converting input low-resolution images into high-resolution images to obtain more information from surveillance images or medical images obtained from sensors, or converting currently owned low-resolution broadcast content into high-resolution content and transmitting it through broadcasting. And its importance within the industry is gradually expanding. Super-resolution image generation methods are broadly classified into three types: interpolation-based methods, reconstruction-based methods, and learning-based methods.

The interpolation-based method is a method of converting a low-resolution image into a high-resolution image by adding new pixel values using adjacent pixel values using interpolation. The processing time for applying the interpolation method is relatively short compared to other methods, but , there is a disadvantage in that the image quality deteriorates significantly when adding a large number of pixel values.

Ultra-high-resolution image generation using a reconstruction-based method analyzes the process by which a low-resolution image is created from a high-resolution original image, and devises a method to reverse the process to create a high-resolution image that becomes the original image. How to create it.

The learning-based method utilizes deep learning technology, a type of artificial intelligence, and constructs a deep learning model for generating ultra-high resolution images with DNN (Deep Neural Net), which refers to a multi-layer artificial neural network. When training data is composed of multiple low-resolution images and high-resolution images, and a deep learning model is trained and the low-resolution images are input to the deep learning model in actual operation mode, the trained deep learning model provides detailed image information about the input low-resolution images. Through inference, a super-resolution image is generated. Although it is the latest technology and has excellent performance, it has the disadvantage of requiring a large amount of training data to train a deep learning model.

The resolution of the high-resolution image used as training data for the deep learning model and the ultra-high-resolution image generated in actual operation mode are the same. However, in the case of training data, it is generally named a high-resolution image, and the high-resolution image generated by a deep learning model for the actual input low-resolution image is named a super-resolution image.

Meanwhile, Synthetic Aperture Radar (SAR) is a device that transmits electrical signals into the air, touches distant objects and terrain, and then collects the reflected signals to obtain image signals for the objects and terrain. Or, it is mounted and operated on an aircraft.

Image signals collected by synthetic aperture radar are converted into two-dimensional images that can be recognized by humans (hereinafter referred to as 'SAR images') through a series of digital signal processing called SAR Processing. The synthetic aperture radar mounted on an aircraft can perform SAR processing on its own, and the synthetic aperture radar mounted on a satellite usually has its own SAR processing function, but has the function of transmitting video signals to the ground. Video signals transmitted to the ground are processed by a SAR processing device installed on the ground.

Synthetic aperture radar can display objects, features, or terrain information on the ground (hereinafter referred to as 'targets of interest') as images, so this function can be used to detect and identify targets of interest in a specific area or to move terrain. It is widely used to observe topographical changes such as.

Synthetic aperture radar has the advantage of being able to image an area ranging from several km ² to hundreds of km ² at once, but the resolution of SAR images is generally only 0.5 m to 1 m, which is significantly lower than the resolution of images obtained on a daily basis. has Therefore, there are many attempts to make low-resolution SAR images closer to high-resolution actual images. In particular, many studies are being conducted to apply learning-based super-resolution image generation methods to SAR images, but the acquisition of SAR images themselves is small and it is difficult to secure training image data with higher resolution compared to SAR images. There is a problem that makes it difficult to train deep learning models for high-resolution image generation. Therefore, when applying learning-based super-resolution image generation technology to SAR images, the existing technology has a minimal increase in actual resolution for SAR images with a resolution of 0.5m to 1m.

Domestic Registered Patent No. 10-2119132 (registered on May 29, 2020)

In order to solve the above-mentioned problems and technical limitations, the technical task to be achieved by the present invention is to generate a deep learning model to generate a super-resolution image for the target of interest in the SAR image, and to train the generated deep learning model by Generate a large amount of training data for the target, that is, a large amount of low-resolution images for training and high-resolution images for training corresponding to them, train the deep learning model using the generated training data, and use the trained deep learning model to The purpose of the present invention is to present a method and device for generating a super-resolution image of a target of interest in a SAR image, which can generate a super-resolution image of a target of interest in a SAR image.

The problems of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

As a means to solve the above-described technical problem, the method for generating a super-resolution image for a target of interest in a SAR image according to an embodiment of the present invention includes a high-resolution (HR, High Resolution) image and a low-resolution (LR, Low) image for the target of interest. Resolution) training data generation step consisting of images; Creating a database containing the generated training data as a component; It includes generating a deep learning model for generating a super-resolution (SR, Super-Resolution) image for a target of interest in a SAR image, and training the deep learning model using training data in the database.

The step of generating training data consisting of a high-resolution image and a low-resolution image for the target of interest includes (Generation A) generating a 3D target model for the target of interest; (Generation B) setting the posture of the 3D target model based on the direction in which the SAR is looking at the target of interest; (Generation C) converting the 3D target model whose posture is set in the (Generation B) step into a high-resolution 2D image to generate a high-resolution image for training for the target model; (Generation D) generating a low-resolution image for training on a target of interest from the 3D target model through a series of signal processing processes; (Generation E) generating a training data pair consisting of the high-resolution image for training generated in the (Generate C) step and the low-resolution image for training generated in the (Generate D) step.

The (Generation D) step is, (Generation D1) a step of generating a SAR simulated received signal for the 3D target model by simulating a received signal input to the SAR by reflecting the transmitted signal of the SAR from the 3D target model; (Generation D2) performing SAR Processing from the SAR simulated reception signal to generate a low-resolution image for training on the target of interest.

In addition, the (Generate D1) step includes: (Generate D1-1) generating a transmission waveform of the SAR transmission signal; (Generation D1-2) Projecting the 3D target model generated in the (Generation A) step at the location of the SAR and dividing it into a plurality of cells; (Generation D1-3) calculating the electromagnetic wave reflectivity for each of the divided cells and the cell distance, which is the distance between the SAR and the cell, at each transmission time; (Generation D1-4) generating a simulated received signal for the 3D target model using the electromagnetic wave reflectivity, cell distance, and transmission waveform calculated for each of the plurality of cells, and the (Generation D1-4) step. Generates a simulated reception signal for the entire simulated movement path of the SAR.

In the step (Generation D1-3), the reflectivity of the cell is calculated using the change in distance between a number of specific points located in the cell and the center of the SAR antenna.

Additionally, the (Generate D1-3) step calculates the reflectivity of the cell using the following equation.

Here, Reflectivity is the reflectivity at one of a number of specific points (point_K) located in the cell corresponding to (i, j) at a random transmission point, is a positive arbitrary constant, point_K0 means the same point as point_K selected from among the plurality of specific points, point_K1 and point_K2 are the surrounding points of the specific point (point_K), d_cell is the distance between SAR and the specific point or surrounding points am.

In the (Generate D1-4) step, a simulated reception signal (hereinafter referred to as 'specific point simulated reception signal') is generated at a random transmission time for each of a plurality of specific points located in the (Generate D1-4-1) cell. ) generating; (Generation D1-4-2) A step of generating a simulated received signal of a cell (hereinafter referred to as 'cell simulated received signal') by adding the specific point simulated received signals generated in the (Generated D1-4-1) step; (Generation D1-4-3) Generating a simulated received signal for the 3D target model at a random transmission point by summing the cell simulated received signals generated for each plurality of cells in the (Generate D1-4-2) step; Includes.

The (Generation D1-4-1) step generates a simulated reception signal at one of a number of specific points located in the cell using the following equation.

Here, rx_timed_cell is a simulated reception signal at one of a number of specific points (A0 ~ D0) (point_K, K = A0, B0, C0, D0) located in the cell corresponding to (i, j) at a random transmission point. , Reflectivity is the electromagnetic wave reflectivity at one (point_K, K=A0, B0, C0, D0) of multiple specific points (A0~D0), d_cell is a specific point of the i and jth cells at the antenna center of the SAR (point_K ) or means the distance between surrounding points, t is time, pi is the pi, f0 is the center frequency of the SAR transmission signal, c is the speed of light, and Kr is the chirp rate, which represents the change in frequency per unit time within the chirp waveform. means Hz/sec, exp means Euler's formula expressed as exp(jХArg) = cos(Arg) + jХsin(Arg) .

In addition, the transmission waveform of the transmission signal is assumed to be pulse_shape(t) × exp(j ^* × pi ^* × Kr ^* × t^2), and pulse_shape(t) is the envelope function of the transmission waveform and has the following value. has

The method for generating a super-resolution image for a target of interest in a SAR image according to an embodiment of the present invention is to change the posture of the 3D target model in the (Generate F) step and the (Generate B) step after the (Generate E) step. Generating a plurality of low-resolution images for training and high-resolution images for training for a specific target of interest by repeatedly executing the (Generate E) step in the Generation B) step; (Generation G) Repeating the steps from the (Generation A) step to the (Generation F) step for a number of new targets of interest to generate a number of low-resolution images for training and high-resolution images for training for a number of new targets of interest. Step; includes.

The step of creating a database with the generated training data as a component involves setting the low-resolution image for training as a key for the low-resolution image for training and the high-resolution image for training generated for a specific target of interest, and training corresponding thereto. It includes generating a dictionary that sets the high-resolution image for value as a value; and generating the dictionary for each target of interest and assigning a target identifier to each generated dictionary to construct a database with the following structure.

Start database [

Target_A: Dictionary

Target_B: Dictionary

Target_C: Dictionary

] Database shutdown

Here, the structure of the dictionary is as follows:

{key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training,

The step of creating a deep learning model for generating a super-resolution (SR, Super-Resolution) image for a target of interest in a SAR image, and training the deep learning model using training data in the database, (training A) input Creating a generator with a multi-layer artificial neural network structure including an upsampling layer and a convolutional layer capable of increasing the resolution of the image and expressing image information; (Training B) generating a reference model with a multi-layer artificial neural network structure including at least one feature extractor including a convolutional layer to extract image features for the input image data; (Training C) generating a discriminator with a multi-layer artificial neural network structure, including a feature extractor with the same structure as the feature extractor constituting the Reference Model, in order to extract image features for the input image data; (Training D) Setting an optimization objective function to optimize the artificial neural network parameters (weight, bias) of the generator and discriminator; (Training E) Setting a Reference Model optimization objective function to optimize the artificial neural network parameters of the Reference Model created in the (Training B) step; (Training F) A step of optimizing the artificial neural network parameters of the Reference Model using the high-resolution image and noise image for training output from the database unit as input data of the Reference Model; (Training G) outputting high-resolution images for training from the database unit to update the artificial neural network parameters of the discriminator; (Training H) outputting low-resolution images for training from the database unit to update the artificial neural network parameters of the generator and discriminator; (Training I) determining whether the training of the generator and discriminator has been completed, and repeating the above (training G) to (training H) steps until the training completion state is reached.

The (training D) step sets an optimization objective function to optimize the artificial neural network parameters (weight, bias) of the generator and discriminator using the following equation.

Optimization objective function for discriminator

Here, I _{HR_tr} is a high-resolution image for training, I _{LR_tr} is a low-resolution image for training, paramsD is an artificial neural network parameter constituting the discriminator, paramsG is an artificial neural network parameter constituting the generator, and D _paramsD (I) is a parameter value named paramsD. is the output value for the input I of the Discriminator, G _paramsG (I) is the output value for the input I of the Generator with the parameter value named paramsG, E[ ] is the expected value, and N_featuremap is the feature map included in all convolutional layers in the Discriminator. The total number of, featuremap_Discriminator(n) is the nth featuremap among each featuremap existing in the Discriminator, and featuremap_ReferenceModel(n) is the nth featuremap among each featuremap existing in the Reference Model.

Optimization objective function for generator

In the (training E) step, the Reference Model is trained using the following optimization objective function.

Optimization objective function for reference model

here, is a super-resolution video for training; is the noise image, paramsR is the artificial neural network parameter constituting the Reference Model, and R _paramsR (I) is the output value for the input I of the Reference Model with the parameter value named paramsR.

Meanwhile, in the method of generating a super-resolution image for a target of interest in a SAR image according to another embodiment of the present invention, after the (training I) step (training J), the target of interest in the actual SAR image is imaged as an input from the trained generator. It further includes the step of inputting and outputting the generator output as a super-resolution image for the input target image.

In addition, the method of generating a super-resolution image for a target of interest in a SAR image according to another embodiment of the present invention includes receiving an image of a target of interest and target identification information in an actual SAR image after the (training I) step (training J). step; (Training K) selecting a dictionary corresponding to the target identification information from a database using the target identification information; (Training L) Check the similarity between the low-resolution image for training stored in the Key value of the selected dictionary and the target image of interest received in the (Training J) step, and generate Value data corresponding to the Key value with the highest similarity. It further includes outputting a super-resolution image for the target image of interest received in the (training J) step.

Meanwhile, the apparatus for generating a super-resolution image for a target of interest in a SAR image according to an embodiment of the present invention generates training data consisting of a high-resolution (HR, High Resolution) image and a low-resolution (LR, Low Resolution) image for the target of interest. a self-training data generation unit; a database unit that creates and manages a database containing the generated training data as a component; A super-resolution image generation deep learning model training unit that generates a deep learning model for generating super-resolution (Super-Resolution) images for targets of interest in the SAR image, and trains the deep learning model using training data in the database; Includes.

The self-training data generation unit includes a 3D modeling unit that generates a 3D target model for a target of interest; a model pose setting unit that sets the pose of the 3D target model in various directions based on the direction in which the SAR looks at the target of interest; a high-resolution image generator for training that converts the 3D target model with the set posture into a high-resolution 2D image and generates a high-resolution image for training for the target model; A SAR simulated received signal generator that generates a simulated received signal for the 3D target model by simulating a received signal input to the SAR by reflecting the transmitted signal of the SAR from the 3D target model; It includes a low-resolution image generator for training that performs SAR processing from the generated SAR simulated reception signal to generate a low-resolution image for training on the target of interest.

The SAR simulated received signal generator includes a waveform generator that generates a transmission waveform of the SAR transmission signal; a cell division unit that projects the 3D target model generated by the 3D modeling unit at the location of the SAR and divides it into a plurality of cells; a calculation unit that calculates electromagnetic wave reflectivity for each of the plurality of divided cells and a cell distance, which is the distance between the SAR and the cell, at each transmission time; a simulation signal unit that generates a SAR simulated reception signal for the 3D target model using the electromagnetic wave reflectivity, cell distance, and transmission waveform calculated for each of the plurality of cells; It includes, and the simulated signal unit generates a SAR simulated reception signal for the entire simulated movement path of the SAR.

The calculation unit calculates the reflectivity of the cell using the change in distance between a plurality of specific points located in the cell and the center of the SAR antenna.

Additionally, the calculation unit calculates the reflectivity of the cell using the following equation.

Here, Reflectivity is the reflectivity at one of a number of specific points (point_K) located in the cell corresponding to (i, j) at a random transmission point, is a positive arbitrary constant, point_K0 means the same point as point_K selected from among the plurality of specific points, point_K1 and point_K2 are the surrounding points of the specific point (point_K), d_cell is the distance between SAR and the specific point or surrounding points am.

The simulated signal unit includes a specific point simulated signal generator that generates a simulated received signal (hereinafter referred to as a 'specific point simulated received signal') at a random transmission point for each of a plurality of specific points located in the cell; a cell simulated signal generator that generates a cell simulated received signal (hereinafter referred to as a 'cell simulated received signal') by combining the generated specific point simulated received signals; It includes a 3D target model simulated signal generator that generates a simulated received signal for the 3D target model at a random transmission point by combining the cell simulated received signals generated for each of the plurality of cells.

The specific point simulated signal generator generates a simulated received signal at one of a plurality of specific points located in the cell using the following equation.

Here, rx_timed_cell is a simulated reception signal at one of a number of specific points (A0 ~ D0) (point_K, K = A0, B0, C0, D0) located in the cell corresponding to (i, j) at a random transmission point. , Reflectivity is the electromagnetic wave reflectivity at one (point_K, K=A0, B0, C0, D0) of multiple specific points (A0~D0), d_cell is a specific point of the i and jth cells at the antenna center of the SAR (point_K ) or means the distance between surrounding points, t is time, pi is the pi, f0 is the center frequency of the SAR transmission signal, c is the speed of light, and Kr is the chirp rate, which represents the change in frequency per unit time within the chirp waveform. means Hz/sec, exp means Euler's formula expressed as exp(jХArg) = cos(Arg) + jХsin(Arg) .

In addition, the transmission waveform of the transmission signal is assumed to be pulse_shape(t) ^* × exp(j ^* × pi ^* × Kr ^* × t^2), and pulse_shape(t) is the envelope function of the transmission waveform as follows. It has value.

One training data is generated each time the posture is adjusted by the model posture setting unit, and a plurality of training data for the target of interest is generated while the posture is adjusted.

The 3D modeling unit generates models for multiple targets of interest and generates multiple training data for each target of interest.

The database unit creates a dictionary that sets the low-resolution image for training as a key and sets the corresponding high-resolution image for training as a value for the low-resolution image for training and the high-resolution image for training generated for a specific target of interest, After creating the dictionary for each target of interest, a target identifier is assigned to each generated dictionary to form a database with the following structure.

Start database [

Target_A: Dictionary

Target_B: Dictionary

Target_C: Dictionary

] Database shutdown

Here, the structure of the dictionary is as follows.

{key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training}

The ultra-high-resolution image generation deep learning model training unit includes a generator having a multi-layer artificial neural network structure including an upsampling layer and a convolutional layer capable of increasing the resolution of the input image and expressing image information; In order to extract image features for input image data, a discriminator with a multi-layer artificial neural network structure including at least one feature extractor including a convolutional layer; An optimization unit that optimizes the artificial neural network parameters (weight, bias) of the generator and discriminator through training, wherein the optimization unit includes an optimization calculation unit, a Reference Model, and a Reference Model optimization unit, and the Reference Model is input image data. It is created with a multi-layer artificial neural network structure that includes at least one feature extractor including a convolutional layer to extract image features, and each feature extractor included in the discriminator is included in each reference model. It has the same structure as the feature extractor, and the Reference Model is optimized using high-resolution images for training and noise images as training data, and the optimization unit trains the Generator and Discriminator using the optimization objective function.

The optimization unit uses the following optimization objective function to optimize the artificial neural network parameters (weight, bias) of the generator and discriminator.

Optimization objective function for discriminator

Here, I _{HR_tr} is a high-resolution image for training, I _{LR_tr} is a low-resolution image for training, paramsD is an artificial neural network parameter constituting the discriminator, paramsG is an artificial neural network parameter constituting the generator, and D _paramsD (I) is a parameter value named paramsD. is the output value for the input I of the Discriminator, G _paramsG (I) is the output value for the input I of the Generator with the parameter value named paramsG, E[ ] is the expected value, and N_featuremap is the feature map included in all convolutional layers in the Discriminator. The total number of, featuremap_Discriminator(n) is the nth featuremap among each featuremap existing in the Discriminator, and featuremap_ReferenceModel(n) is the nth featuremap among each featuremap existing in the Reference Model.

Optimization objective function for generator

Additionally, the optimization unit trains the Reference Model using the following optimization objective function.

Optimization objective function for reference model

here, is a super-resolution video for training; is the noise image, paramsR is the artificial neural network parameter constituting the Reference Model, and R _paramsR (I) is the output value for the input I of the Reference Model with the parameter value named paramsR.

The optimizer inputs the high-resolution training image output from the database into the discriminator, observes the output value of the discriminator, and Update the artificial neural network parameters of the discriminator so that is the maximum; After inputting the output high-resolution training image into the Reference Model, check each feature map value calculated by the Reference Model; After inputting the output high-resolution image for training into the discriminator, updating the artificial neural network parameters of the discriminator in such a way that each feature map value existing in the discriminator follows the feature map value of the corresponding reference model; Input the low-resolution image for training output from the database into the generator, input the output of the generator into the discriminator, and observe the output of the discriminator, Update the artificial neural network parameters of the generator to minimize Update the artificial neural network parameters of the discriminator so that is maximized; By repeating the output of the high-resolution image for training and the output of the low-resolution image for training, the artificial neural network parameters of the generator and discriminator are optimized.

Meanwhile, a super-resolution image generating device for a target of interest in a SAR image according to another embodiment of the present invention applies a trained generator to receive a target of interest image in an actual SAR image, and provides information about the received target of interest image. It further includes a super-resolution image generator that outputs a super-resolution image.

Meanwhile, a super-resolution image generating device for a target of interest in a SAR image according to another embodiment of the present invention receives an image of a target of interest and target identification information in an actual SAR image, and uses the received target identification information to store it in a database. After selecting the dictionary corresponding to the target identification information, check the similarity between the low-resolution image for training stored in the key value of the selected dictionary and the received target image of interest, and select the value corresponding to the key value with the highest similarity. It further includes a super-resolution image generator that outputs data as a super-resolution image for the received target image of interest.

According to one embodiment of the present invention, in generating a super-resolution image for a target image of interest acquired through SAR, a target image with higher resolution than the target image of interest acquired by applying deep learning technology is obtained through SAR. It has the effect of providing a method and device for obtaining.

According to one embodiment of the present invention, the problem of lack of training data for SAR images can be solved by providing a large number of low-resolution images for training and high-resolution images for training to train a deep learning model for generating super-resolution images.

According to an embodiment of the present invention, in generating a high-resolution image for training to train a deep learning model for generating a super-high-resolution image, the process of creating a 3D target model, setting the model pose, and converting the 2D image to the 3D target model is performed. By generating high-resolution images for training, it can be effective in providing a method and device for generating ultra-high resolution images that acquire ultra-high resolution target images that are 10 times higher than the 0.5m resolution target images typically provided by SAR. there is.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram showing a super-resolution image generating device 10 for a target of interest in a SAR image according to an embodiment of the present invention;
Figure 2 is a block diagram showing the self-training data generator 100 shown in Figure 1;
Figure 3 is an example of 3D target modeling,
Figure 4 is an example diagram defining the geometric structure between the SAR and the target of interest (3D target model);
Figure 5 is a schematic diagram modeling the geometry of Figure 4 for generating a simulated received signal of SAR;
Figure 6 is an example diagram to explain the Depression Angle and Aspect Angle that set the posture of the 3D target model;
Figure 7 is a block diagram showing the SAR simulated received signal generator 140 shown in Figure 2;
Figure 8 shows an example of generating a high-resolution image for training by two-dimensionally projecting a 3D target model based on the direction in which the SAR is facing, and an example of cell division.
FIG. 9 is an example diagram illustrating a number of points assigned to cells at (i, j) positions among a number of cells divided by the cell division unit 145;
Figure 10 is a block diagram showing the simulated signal unit 149 shown in Figure 7;
Figure 11 is a diagram schematically illustrating the dimensions of the SAR simulated received signal;
Figure 12 is a block diagram showing the ultra-high resolution image generation deep learning model training unit 300 shown in Figure 1;
Figure 13 is an example diagram illustrating the artificial neural network structure of the Reference Model (331) shown in Figure 12;
Figure 14 is an example diagram illustrating the artificial neural network structure of the discriminator 320 shown in Figure 12;
Figure 15 is an example diagram for explaining the structure of the internal filter of the convolutional layer shown in Figures 13 and 14;
Figure 16 is an example diagram for explaining the convolution operation of the convolutional layer shown in Figures 13 and 14;
FIG. 17 is a diagram showing the operation of the Reference Model 331 and the Reference Model optimization unit 331-1 shown in FIG. 12 to optimize the artificial neural network parameters of the Reference Model 331;
Figure 18 is a block diagram showing a super-resolution image generating device 20 for a target of interest in a SAR image according to another embodiment of the present invention.
Figure 19 is a block diagram showing the ultra-high resolution image generator 400 shown in Figure 18, according to one embodiment.
FIG. 20 is a block diagram showing the ultra-high resolution image generator 400 shown in FIG. 18 according to another embodiment;
21 is a flowchart showing a method for generating a super-resolution image of a target of interest in a SAR image according to an embodiment of the present invention;
FIG. 22 is a flowchart illustrating in detail the self-training data generation (S2100) step and the database creation (S2102) step including the generated training data shown in FIG. 21;
FIG. 23 is a flowchart detailing the step of generating a SAR simulated received signal for a 3D target model (S2206) by simulating a received signal that is input to the SAR by reflecting the transmitted signal of the SAR shown in FIG. 22 from the 3D target model;
Figure 24 is an example diagram showing a series of operations for generating training data according to the self-training data generation (S2100) step shown in Figure 21;
Figure 25 is a flowchart detailing the deep learning model creation and deep learning model training (S2104) steps for ultra-high resolution image generation shown in Figure 21;
Figure 26 is a flowchart showing a method for generating a super-resolution image for a target of interest in a SAR image using a trained generator according to an embodiment of the present invention;
Figure 27 is a flowchart showing a method for generating a super-resolution image of a target of interest in a SAR image using an actual target of interest image and target identification information according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, a method and device for generating a super-resolution image of a target of interest in a SAR image according to the present invention will be described with reference to the attached drawings.

Figure 1 is a block diagram showing an apparatus 10 for generating a super-resolution image of a target of interest in a SAR image according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus 10 for generating a super-resolution image for a target of interest in a SAR image according to an embodiment of the present invention includes a self-training data generation unit 100, a database unit 200, and deep learning for generating super-resolution images. It may include a model training unit 300.

The self-training data generator 100 can generate a plurality of training data required to train a deep learning model for generating a super-resolution image of a target of interest in a SAR image.

The database unit 200 can create, store, and manage a large number of training data generated by the self-training data generation unit 100 as a database so that it can be used for training a deep learning model for generating ultra-high resolution images in the future.

The ultra-high-resolution image generation deep learning model training unit 300 may generate a deep learning model for ultra-high-resolution image generation, and train the deep learning model using a plurality of training data stored in the database unit 200.

In the above embodiment, the self-training data is training data generated entirely internally by the super-resolution image generating device 10 for the target of interest in the SAR image, and the training data required for training the deep learning model for generating super-resolution images is as desired. can be created.

This is a situation in which, in existing technology, a typical ultra-high-resolution image generation device trains a deep learning model using training data received from an external source, or the amount of training data received from an external source is insufficient to train the deep learning model. This is in contrast to the data augmentation method used in . The data augmentation method is a method of artificially increasing the amount of training data by creating a copy of the original training data by performing artificial processing on external training data.

FIG. 2 is a block diagram showing the self-training data generator 100 shown in FIG. 1.

Referring to FIG. 2, the self-training data generator 100 includes a 3D modeling unit 110, a model posture setting unit 120, a high-resolution image generator for training 130, a SAR simulated received signal generator 140, It may include a low-resolution image generator 150 for training.

The 3D modeling unit 110 creates a 3D target model by modeling the physical shape of the target of interest in three dimensions. Figure 3 is an example diagram of 3D target modeling, and the 3D modeling unit 110 can generate 3D target models for different targets of interest.

The model pose setting unit 120 may define the geometric structure between the SAR and the 3D target model and adjust the pose of the 3D target model based on the direction in which the SAR faces the target of interest.

FIG. 4 is an example diagram defining the geometric structure between the SAR and the target of interest (3D target model), and FIG. 5 is a schematic diagram modeling the geometry of FIG. 4 for generating a simulated received signal of the SAR.

Referring to FIG. 4, the model pose setting unit 120 defines the geometric structure between the SAR and the 3D target model in order to generate a SAR simulation image for the target of interest constituting the training data. SAR transmits the transmitted waveform toward the target of interest from t(0), the time when the first transmitted waveform is transmitted, to t(N-1), the time when the transmitted waveform is finally transmitted (where N is the number of transmitted waveforms). It transmits and receives the signal reflected from the target of interest. At this time, the transmission cycle is PRI (Pulse Repetition Interval).

Referring to FIG. 5, the SAR moves the orbit in the azimuth direction at a moving speed (v) and transmits a transmission waveform at a period of PRI. Therefore, t(azi_idx=0) shown in FIG. 5 is the time when the first transmission waveform is transmitted, and the SAR is located in different orbits from time t(azi_idx=0) to time t(azi_idx=N-1). A transmission waveform is transmitted from that point. At this time, the SAR simulated reception signal depends on the cell distance (d_cell), which is the distance between the SAR location and the target. d_cell(i, j, point_K) is the distance between each cell created by dividing the 3D target model into two dimensions, a specific point (point_K) within the corresponding cell (i, j), and the center of the SAR antenna.

In addition, the model posture setting unit 120 can set the Depression Angle and Aspect Angle as geometric positions between the SAR and the 3D target model, and adjust the posture of the 3D target model according to the set angle and geometric structure.

Figure 6 is an example diagram to explain the Depression Angle and Aspect Angle that set the posture of the 3D target model.

Referring to Figure 6, the angle formed between the SAR compared to the ground surface and the center point of the 3D target model is called the Depression Angle, and the difference between the direction in which the front of the 3D target model faces and the direction in which the SAR faces the 3D target model is called the Aspect Angle. do. When generating SAR training data, the model attitude setting unit 120 randomly sets the Depression Angle and the Aspect Angle each time one piece of training data is generated, and the shape of the 3D target model as seen from the SAR according to the set Depression Angle and Aspect Angle. (i.e. posture) can be adjusted. This is to secure training data with various shapes from one target of interest, that is, one 3D model.

Referring again to FIG. 2, the high-resolution image generator 130 for training may generate a high-resolution image for training (I _{HR_tr} ) among the training data required to train a deep learning model for generating a super-high-resolution image.

The high-resolution image generator for training 130 projects the 3D target model two-dimensionally based on the direction in which the SAR faces the 3D target model in the geometric structure between the SAR and the 3D target model defined in the model attitude setting unit 120. You can create and save 2D images. The generated 2D image is a high-resolution image for training.

The resolution when generating high-resolution images for training can be set to have the target resolution value when generating ultra-high-resolution images. For example, if the resolution of the image of the target of interest to be acquired through actual SAR is 0.5m, that is, the horizontal or vertical size of the actual object represented by one pixel in the image is 0.5m, ultra-high resolution image is generated. The resolution of the input low-resolution image (I _LR ) is 0.5m, and a super-resolution image (I _SR ) with an improved resolution of 0.05m can be output for the input low-resolution image. At this time, the resolution of the high-resolution image for training is set to 0.05m, which is the same resolution as the target ultra-high-resolution image. This means that when generating a high-resolution image for training for a 3D target model, a 5m long physical shape is expressed with 100 pixels.

Figure 8 is an example of generating a high-resolution image for training by two-dimensionally projecting a 3D target model based on the direction in which the SAR is facing, and an example diagram of cell division.

Referring to FIG. 8, the 3D target model can be projected two-dimensionally based on the viewing direction in SAR. The 3D target model is projected as a 2D image with the target resolution, and one high-resolution image for training is created. Description of cell division will be described later.

Referring again to FIG. 2, the SAR simulated received signal generator 140 simulates a received signal input to the SAR when the SAR transmitted signal is reflected from the 3D target model to generate a simulated received signal in the time domain for the 3D target model (i.e. , SAR simulated reception signal) can be generated.

FIG. 7 is a block diagram showing the SAR simulated received signal generator 140 shown in FIG. 2.

Referring to FIG. 7, the SAR simulated received signal generation unit 140 includes a parameter setting unit 141, a waveform generating unit 143, a cell dividing unit 145, a calculating unit 147, and a simulated signal unit 149. It can be included.

The parameter setting unit 141 sets physical parameters used by the SAR that is the subject of simulation to generate a SAR simulated reception signal, and hereinafter, these are referred to as 'SAR system parameters'. The SAR system parameters include the center frequency of the SAR transmission signal (f0), the size of the envelope of the transmission waveform, the length of the transmission waveform (T_p), the chirp rate (Kr) indicating the amount of change in frequency per unit time within the transmission waveform, SAR and It can include the distance between 3D target models, the moving speed of the SAR, and the aperture of the SAR antenna.

SAR system parameters determine the resolution of the SAR image. The SAR system parameters applied to the parameter setting unit 141 apply the same values as the SAR system parameters applied to the actual SAR, so that the target image of interest obtained from the low-resolution image for training and the actual SAR image has the same resolution.

The waveform generator 143 may determine the point in time at which the SAR transmits the transmission waveform of the transmission signal (initial t=t(0)), and update the transmission point in each PRI cycle. In addition, the waveform generator 143 generates a transmission waveform of the SAR at a specific transmission point to simulate the behavior of the SAR transmitting the transmission waveform.

The cell division unit 145 may divide the 3D model generated by the 3D modeling unit 110 into a plurality of cells by projecting it in two dimensions at the location of the SAR.

Referring again to FIG. 8, when the posture of the 3D target model is set in the model posture setting unit 120, the two-dimensional surface, that is, the 2D shape, can be confirmed when the 3D target model is viewed from any direction. Accordingly, the cell division unit 145 may generate a 2D image by projecting the 3D target model after the model posture is set, and divide the generated 2D image into a plurality of cells as shown in FIG. 8.

The 2D image generation performed by the cell division unit 145 does not aim at generating the 2D image itself, but rather splits the 3D target model into small cells and applies a mathematical model to be described later to each cell, and provides high-resolution training for training. The difference is that the image generator 130 generates a 2D image with a specific resolution and uses it as training data.

The calculation unit 147 calculates the electromagnetic wave reflectivity for each of the plurality of cells divided by the cell division unit 145 and the cell distance, which is the distance between the SAR and the cell, at each transmission time (i.e., at each transmission cycle) for the entire simulated movement path of the SAR. It can be calculated.

FIG. 9 is an example diagram illustrating a plurality of points assigned to a cell at the (i, j) position among the plurality of cells divided by the cell division unit 145.

Referring to FIG. 9, the calculation unit 147 may designate two or more specific points for each cell and one or more surrounding points for each specific point. In Figure 9, four specific points (A0, B0, C0, D0) and two surrounding points (A1, A2, B1, B2, C1, C2, D1, D2) are designated.

The calculation unit 147 calculates the distance between each of a plurality of points (A0, B0, C0, D0, A1, A2, B1, B2, C1, C2, D1, D2) located in the cell for each cell and the SAR antenna central body. Calculate cell distance (or point distance). The operation of generating a 3D target model and calculating the cell distance of the target of interest is based on the relative position (or distance) between the position of the SAR set in the model posture setting unit 120 or the parameter setting unit 141 and the target model. This can be performed through 3D target modeling software.

In addition, the calculation unit 147 can calculate the electromagnetic wave reflectivity of each cell by using the change in distance between a plurality of specific points (A0, B0, C0, D0) and the center of the SAR antenna.

The calculation unit 147 can calculate the electromagnetic wave reflectivity of the cell using [Equation 1].

[Equation 1]

In [Equation 1], Reflectivity is one of a number of specific points (A0 to D0) located in the (i, j)th cell at a random transmission point (t) (point_K, K=A0, B0, C0, D0) Reflectivity at, is a positive arbitrary constant, point_K0 means the same point as point_K selected from among the plurality of specific points, point_K1 and point_K2 are the surrounding points of the specific point (point_K), d_cell is the SAR and the specific point or the surrounding points and the SAR is the distance between

Referring to [Equation 1], i, j are identifiers that assign a position to each cell after dividing the 3D target model into multiple cells, and within the cell, four specific points named A0, B0, C0, and D0 The electromagnetic wave reflectivity can be calculated as the gradient with respect to the cell distance of each point.

Referring again to FIG. 7, the simulated signal unit 149 generates a simulated received signal for each cell using the transmission waveform and the electromagnetic wave reflectivity and cell distance calculated for each cell, and collects the simulated received signals for each cell to create a 3D target. A simulated received signal (hereinafter referred to as 'SAR simulated received signal') for the model can be generated.

The simulation signal unit 149 can generate a SAR simulated reception signal by simulating the operation of transmitting and reflecting the SAR transmission waveform with a 3D target model according to the transmission cycle.

Additionally, the simulated signal unit 149 may generate a SAR simulated reception signal for the entire simulated movement path of the preset SAR.

FIG. 10 is a block diagram showing the simulated signal unit 149 shown in FIG. 7.

Referring to FIG. 10, the simulation signal unit 149 may include a specific point simulation signal generation unit 149a, a cell simulation signal generation unit 149b, a 3D model simulation signal generation unit 149c, and a collection unit 149d. You can.

The specific point simulated signal generator 149a generates a simulated reception signal (hereinafter referred to as 'specific point simulated reception signal') (a ) can be created.

The specific point simulation signal generator 149a uses [Equation 2] to generate a plurality of specific points (point_K) located in the (i, j)-th cell (point_K=A0, A simulated reception signal can be generated from any one of B0, C0, and D0).

[Equation 2]

In [Equation 2], rx_timed_cell is one of a number of specific points (A0 ~ D0) located in the cell corresponding to (i, j) at a random transmission point (point_K, K = A0, B0, C0, D0) The simulated received signal, Reflectivity is the electromagnetic wave reflectivity at one of multiple specific points (A0~D0) (point_K, K=A0, B0, C0, D0), and d_cell is the i, j cell from the center of the SAR antenna. It is the distance between a specific point (point_K) or surrounding points.

In addition, t is time, pi is the circumference, f0 is the center frequency of the SAR transmission signal, c is the speed of light, and Kr is the chirp rate representing the amount of change in frequency per unit time within the chirp waveform. The units are Hz/sec and exp is exp( It means Euler's formula expressed as jХArg) = cos(Arg) + jХsin(Arg) .

In addition, the transmission waveform of the transmission signal is assumed to be pulse_shape(t) ^* × exp(j ^* × pi ^* × Kr ^* × t^2), and pulse_shape(t) is the envelope function of the transmission waveform as follows. It has value.

The cell simulated signal generator 149b generates a cell simulated received signal (hereinafter, (referred to as ‘cell simulated reception signal’) (b) can be generated.

When four specific points are designated in one cell as shown in FIG. 9, the cell simulated received signal is the result of adding the four specific point simulated received signals (a) calculated by [Equation 2] at a random transmission time. Accordingly, the cell simulation signal generation unit 149b can generate a cell simulation reception signal (b) equal to the number of cells divided by the cell division unit 145 at any transmission time.

The 3D model simulated signal generator 149c generates a simulated received signal (c) for the 3D target model by summing the cell simulated received signals (b) generated for each multiple cell in the cell simulated signal generator (139b) at a random transmission point. can be created. As a result, a simulated reception signal (c) for the 3D target model is generated at a random transmission point (for example, t(azi_idx=0)).

The collection unit 149d collects the simulated reception signals for the 3D target model generated at all transmission times (t(azi_idx=0), t(azi_idx=1), ??, t(azi_idx=N-1)) A SAR simulated reception signal can be generated. That is, the collection unit 149d generates a SAR simulation signal by collecting simulated signals generated by simulating a situation in which the SAR moves along a simulated movement path and transmits a transmission waveform to the 3D target model at each transmission cycle.

As described above, each time the posture is adjusted in the model posture setting unit 120 and a new 3D target model is set, one SAR simulated reception signal is generated, which means that one SAR simulated reception signal is generated in one posture. means that Therefore, when the attitude of the same 3D target model is readjusted, a new SAR simulated received signal is generated.

Figure 11 is a diagram schematically illustrating the dimensions of the SAR simulated received signal.

Referring to FIG. 11, the time series signal strength as much as N_azi is simulated for each i, j, and point_K. N_azi is the number of transmitted waveforms transmitted by the SAR along the movement path, N_range is the number of samples generated when the signal received for each transmitted waveform is sampled in the time domain, and Ts refers to the sampling period. azi_idx=0 is the first transmission time of the waveform (t(0)), and azi_idx=N_azi-1 is the last transmission time (t(N-1)).

At a specific transmission point, for example, when azi_idx=0, the sum of the specific point simulation signals (time domain signals within the red dotted line) generated creates a cell simulation signal, and since there are as many transmission points as N_azi, N_azi Х Ni Х Nj The number of cell simulated received signals (b) generates a SAR simulated received signal.

Referring again to FIG. 2, the low-resolution image generator for training 150 may generate a low-resolution image for training of a target of interest from the SAR simulated reception signal generated by the simulation signal unit 149.

The low-resolution image generator for training 150 performs a signal processing procedure on the simulated SAR simulated received signal using the SAR Processing algorithm to generate a low-resolution image for training for the 3D target model. Examples of SAR Processing algorithms include Backprojection algorithm and RDA (Range Doppler Algorithm), but are not limited thereto.

The resolution of the generated low-resolution image for training is the same as the SAR image targeted for actual application. A typical SAR image has a resolution of 0.5 m to 1 m, but the present invention is not limited to the typical resolution specified above.

Once the model pose is determined for a specific 3D target model, the high-resolution image for training generated through the high-resolution image generator 130 for training and the low-resolution image for training generated through the low-resolution image generator 150 for training are mutually They correspond to form one training data pair. The training data pair is input to the database unit 200 for deep learning training to generate ultra-high resolution images in the future.

Referring again to FIG. 1, the database unit 200 stores/manages training data pairs consisting of ultra-high resolution images for training and low-resolution images for training generated by the above-described operation in the self-training data generation unit 100 as a database. do. At this time, a different training data pair is generated each time the model posture is adjusted for a specific target of interest, and multiple training data pairs may be generated for different targets of interest.

The configuration of the database applies a dictionary method that sets low-resolution images for training as Key for one training data pair, and sets high-resolution images for training that exist within the same training data pair as Value. One dictionary can store multiple keys and values, but the keys and values contained in the same dictionary are composed of training data for the same target of interest.

A dictionary is created for each target for a plurality of targets of interest, and the database created by the database unit 200 has the following structure.

Start database [

Target_A: Dictionary

Target_B: Dictionary

Target_C: Dictionary

] Database shutdown

Here, the structure of the dictionary is as follows.

{key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training}

The ultra-high-resolution image generation deep learning model training unit 300 generates a deep learning model for ultra-high-resolution image generation with a multi-layer artificial neural network structure, and creates an actual deep learning model based on a plurality of training data constituting the database unit 200. The deep learning model can be trained to generate a super-resolution image for the SAR target image of interest.

FIG. 12 is a block diagram showing the ultra-high resolution image generation deep learning model training unit 300 shown in FIG. 1.

Referring to FIG. 12, the ultra-high resolution image generation deep learning model training unit 300 may include a generator 310, a discriminator 320, and an optimization unit 330.

The Generator (310) and Discriminator (320) configure a Generative Adversarial Network (GAN) with the purpose of converting a low-resolution image into a super-resolution image, and the Optimizer (330) uses an optimization objective function and multiple training methods. Using data, the artificial neural network parameters of the Generator (310) and Discriminator (320) can be optimized.

Generator (310) can be composed of DNN (Deep Neural Network), a multi-layer artificial neural network. More specifically, it includes an Upsampling Layer and a Convolutional Layer that can increase the resolution of the input image and express image information. May include hierarchical artificial neural networks.

The generator 310 generates a generator output image (G _paramsG (I _{LR_tr} )) having the same resolution as the resolution of the super-resolution image when inputting the low-resolution image for training (I _{LR_tr} ) output from the database unit. The untrained generator 310 generates a random image determined by the initial values of the artificial neural network parameters (weight, bias) that make up the generator 310 for the input low-resolution image for training, but the training described later is performed. The generator 310 that has completed training can generate a super-resolution image with more detailed image information for the input low-resolution image.

The discriminator 320 can be composed of a multi-layer artificial neural network, and more specifically, it can include a convolutional layer to enable extraction of local image features for the input image. The input layer of the multi-layer artificial neural network of the discriminator 320 has more than the number of input nodes capable of receiving each pixel data of a high-resolution image. Discriminator 320 outputs the probability that I _x is a super-resolution image for the input image I _x , and the output value has a value between 0 and 1.

The discriminator (320) receives the high-resolution image for training output from the database unit according to the path setting of the input unit and outputs D _paramsD (I _{HR_tr} ) or the generator output image (G _paramsG (I _{LR_tr} )) from the generator (310). You can receive and output D _paramsD (G _paramsG (I _{LR_tr} )).

According to the description of the output of the Discriminator (320), D _paramsD (I _{HR_tr} ) means the probability that the input high-resolution image for training is a super-resolution image, and D _paramsD (G _paramsG (I _{LR_tr} )) is the probability that the input high-resolution image for training is a super-resolution image. This means the probability that the output image (G _paramsG (I _{LR_tr} )) is a super-resolution image. The training process of the discriminator (320) involves optimizing the values of the artificial neural network parameters of the discriminator (320) so that D _paramsD (I _{HR_tr} ) is close to 1 and D _paramsD (G _paramsG (I _{LR_tr} )) is close to 0. Includes.

The optimization unit 330 may be composed of an optimization calculation unit 332, a Reference Model 331, and a Reference Model optimization unit 331-1.

The optimization unit 330 optimizes the weight and bias values, which are each artificial neural network parameters constituting the generator 310 and the discriminator 320.

The optimizer 330 uses an optimization objective function based on the output values of the generator 310 and discriminator 320 according to the training data input and the state information of each node of the artificial neural network of the reference model that received the high-resolution image for training. Optimize Generator (310) and Discriminator (320). The detailed operation of the optimization unit will be described later.

FIG. 13 is an example diagram showing the artificial neural network structure of the Reference Model 331 shown in FIG. 12.

The Reference Model 331 may be composed of a multi-layer artificial neural network, and more specifically, may include a convolutional layer to enable extraction of local image features for the input image. The Reference Model 331 receives a high-resolution image for training (I _{HR_tr} ) or a noise image (I _Noise ) in which each pixel is set to a random noise value, and outputs the probability that the input image is a super-resolution image.

The Reference Model 331 may include one or more feature extractors that learn image features of ultra-high-resolution images by learning from multiple high-resolution images for training and multiple noise images. The learned image features are saved as a filter value, which is a component of the feature extractor.

FIG. 14 is an exemplary diagram showing the artificial neural network structure of the discriminator 320 shown in FIG. 12.

The discriminator 320 includes one or more feature extractors capable of extracting image features for the input image, and each feature extractor has the same structure as each feature extractor possessed by the reference model 331 described above. You can set it. For example, the Feature extractor (1) owned by the Discriminator (320) and the Reference Model (331) are set to have the same number of input nodes and the same filter dimension. This is to ensure that the featuremap information of the Discriminator (320) follows the corresponding featuremap information of the Reference Model (331) during the training process of the Discriminator (320).

Figure 15 is an example diagram for explaining the structure of the internal filter of the convolutional layer shown in Figures 13 and 14.

The filter, a component of the convolutional layer, has a two-dimensional structure. This is to perform a convolution operation between the input image expressed in two dimensions and the filter. The two-dimensional matrix that makes up the filter forms a kind of image pattern according to the value of each element that makes up the matrix. For example, if an image pattern similar to filter (1) shown in FIG. 15 exists in the input image, the output value of the convolutional layer appears high, and information about the location of the image pattern in the input image can be obtained. These operating characteristics of the convolutional layer are useful in constructing a feature extractor.

Figure 16 is an example diagram for explaining the convolution operation of the convolutional layer shown in Figures 13 and 14.

The convolution operation is the same as a normal 2D convolution operation. This is the process of calculating between input 2D data and 2D filter and adding bias after the calculation. For the input data, the filter is placed at the upper leftmost corner of the input data, and the process of multiplying and adding the element values of the input data and the filter is repeated for each corresponding element value. When the filter reaches the lower rightmost corner of the input data, the above operation is performed. Once completed, the convolution operation ends.

The two-dimensional output obtained after completing the convolution operation is named featuremap. The featuremap expresses information about where the image pattern expressed by the filter exists within the input data and how high its similarity is.

FIG. 17 is a diagram showing the operation of the Reference Model 331 and the Reference Model optimization unit 331-1 shown in FIG. 12 to optimize artificial neural network parameters of the Reference Model 331.

Training data for optimization of the Reference Model (331) uses high-resolution images for training (I _{HR_tr} ) and noise images (I _Noise ).

Reference Model 331 outputs RparamsR(IHR_tr) or RparamsR(INoise) for the input training data. At this time, RparamsR(IHR_tr) means the probability that the input high-resolution image for training is a super-resolution image, and RparamsR(INoise) means the probability that the input noise image is a super-resolution image.

The Reference Model optimization unit (331-1) is an artificial neural network that observes the output of the input of the Reference Model (331) and constructs the Reference Model (331) using the optimization objective function expressed in the following [Equation 3] Parameters can be optimized.

[Equation 3]

Optimization objective function for reference model

here, is a super-resolution video for training; is the noise image, paramsR is the artificial neural network parameter constituting the Reference Model, and RparamsR(I) is the output value for the input I of the Reference Model with the parameter value named paramsR.

By training through the optimization objective function, the feature extractor of the reference model 331 can learn the image information contained in the super-resolution image.

Referring again to FIG. 12, training of the deep learning model for generating ultra-high resolution images is performed after optimization of the Reference Model according to [Equation 3] is completed, and then optimization of the Generator (310) and Discriminator (320) is performed.

The ultra-high-resolution image generation deep learning model training unit 300 uses low-resolution images for training or high-resolution images for training stored in the database unit 200 as training data, and uses the optimization objective function expressed in the following [Equation 4]. Thus, the artificial neural network parameters of the Generator (310) and Discriminator (320) can be optimized.

[Equation 4]

Optimization objective function for discriminator

Here, I _{HR_tr} is a high-resolution image for training, I _{LR_tr} is a low-resolution image for training, paramsD is an artificial neural network parameter constituting the discriminator, paramsG is an artificial neural network parameter constituting the generator, and D _paramsD (I) is a parameter value named paramsD. is the output value for the input I of the Discriminator, G _paramsG (I) is the output value for the input I of the Generator with the parameter value named paramsG, E[ ] is the expected value, and N_featuremap is the feature map included in all convolutional layers in the Discriminator. The total number of, featuremap_Discriminator(n) is the nth featuremap among each featuremap existing in the Discriminator, and featuremap_ReferenceModel(n) is the nth featuremap among each featuremap existing in the Reference Model.

Optimization objective function for generator

Through the above optimization, the discriminator (320) receives the super-resolution image for training output from the database unit and learns the characteristics of the super-resolution image, thereby determining whether the super-resolution image generated by the generator (310) possesses the characteristics of the super-resolution image. Determine whether or not

The generator 310 generates a super-resolution image from the input low-resolution image for training, but generates a random image that has almost no features of the super-resolution image in its initial untrained state. While learning is in progress, the Generator (310) learns the characteristics of the ultra-high resolution image, and upon completion of optimization, the ultra-high resolution image generated by the Generator (310) retains the characteristics of the ultra-high resolution image similar to the high-resolution image for training. As a result, the discriminator 320 cannot distinguish between the sources of the two images.

Figure 18 is a block diagram showing a super-resolution image generating device 20 for a target of interest in a SAR image according to another embodiment of the present invention.

Referring to FIG. 18, the device 20 for automatically identifying a target of interest in a SAR image according to another embodiment of the present invention includes a self-training data generation unit 100, a database unit 200, and a deep learning model for generating super-resolution images. It may include a training unit 300 and a super-resolution image generation unit 400.

The self-training data generation unit 100, the database unit 200, and the ultra-high-resolution image generation deep learning model training unit 300 have been described with reference to FIGS. 1 to 17, so detailed descriptions will be omitted.

The super-resolution image generator 400 may receive a target of interest obtained from an actual SAR and generate a super-resolution image for the input low-resolution image.

The device 20 shown in FIG. 18 generates a super-resolution image of the target of interest in real time by inputting the image of the target of interest obtained through actual SAR, and at the same time generates training data for the target of interest to create a deep learning model. can be updated.

FIG. 19 is a block diagram illustrating the ultra-high resolution image generator 400 shown in FIG. 18 according to an embodiment.

Referring to FIG. 19, the ultra-high resolution image generator may include a generator 410. The generator 410 can receive a SAR target of interest image obtained through actual SAR and generate an ultra-high resolution image (I _SR ) for the input image. The input SAR target of interest image (I _LR ) is a low-resolution image that is the target of super-resolution image generation.

Generator (410) is a deep learning model that has completed training of Generator (310) created by the ultra-high-resolution image generation deep learning model training department. The deep learning model creation and deep learning model training of the generator 310 were explained in detail with reference to FIGS. 12 to 17, [Equation 4], and FIG. 25.

FIG. 20 is a block diagram illustrating the ultra-high resolution image generator 400 shown in FIG. 18 according to another embodiment.

Referring to FIG. 20, the ultra-high resolution image generator may include a database-based ultra-high resolution image generator 420.

The database-based super-resolution image generator 420 may receive the SAR target-of-interest image and target identification information obtained through actual SAR and generate a super-resolution image (I _SR ) for the input image. The target identification information may be generated by a typical target identification device for SAR images.

The database-based super-resolution image generation unit 420 selects a dictionary corresponding to a target identifier matching the input target identification information from the database managed by the database unit 200. The similarity between the low-resolution training image stored in the key value of the selected dictionary and the input SAR target image of interest is checked, the key value with the highest similarity is selected, and the value data corresponding to the selected key value is stored in the input SAR. The image of the target of interest can be output as a super-resolution image.

Figure 21 is a flowchart showing a method for generating a super-resolution image of a target of interest in a SAR image according to an embodiment of the present invention.

Referring to FIG. 21, in step S2100, self-training data is generated.

In step S2102, a database containing the generated training data is created.

In step S2104, a deep learning model for generating ultra-high resolution images is created and the deep learning model is trained.

FIG. 22 is a flowchart illustrating in detail the self-training data generation (S2100) step and the database creation (S2102) step including the generated training data shown in FIG. 21;

Referring to FIG. 22, in step S2200, a 3D target model for the target of interest is created.

In step S2202, the pose of the 3D target model is set based on the direction in which the SAR is looking at the target of interest.

In step S2204, a high-resolution image for training is generated by projecting the 3D target model in two dimensions in the direction the SAR is looking.

In step S2206, the SAR transmitted signal is reflected from the 3D target model and simulates the received signal input to the SAR to generate a SAR simulated received signal for the 3D target model. The SAR simulated received signal is generated by applying SAR system parameters in a simulated environment that sets the geometric structure between the SAR and the 3D target model. The SAR simulated received signal is a signal received by the SAR when the transmitted waveform transmitted by the SAR in the PRI cycle is reflected and returned to the 3D target model.

In step S2208, a low-resolution image for training of the target of interest is generated from the SAR simulated received signal. SAR simulated reception signals are processed with the SAR Processing algorithm to generate SAR images, and the generated SAR images are low-resolution images for training.

In step S2210, a training data pair consisting of the generated low-resolution image for training and the high-resolution image for training is generated.

In step S2212, it is determined whether to complete the generation of training data for the target of interest. When steps S2202 to S2210 are performed, one training data pair for the target of interest is generated. It is desirable to train a deep learning model for ultra-high resolution image generation by generating a plurality of training data pairs for the target of interest, and repeat steps S2202 to S2210 until a sufficient amount of training data pairs for the target of interest are generated. It can be done by doing this.

In step S2214, a database consisting of a dictionary for each target of interest is constructed. Generating multiple training data for the specific target of interest can be completed by performing steps S2200 to S2212. The plurality of training data generated for the specific target of interest is configured as a dictionary, and different dictionaries can be created for each target of interest and converted into a database as follows.

Start database [

Target_A: Dictionary

Target_B: Dictionary

Target_C: Dictionary

] Database shutdown

Here, the structure of the dictionary is as follows.

{key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training,

key: low-resolution image for training, value: high-resolution image for training}

In step S2216, it is determined whether it is necessary to generate training data for other targets of interest. According to a preferred embodiment of the present invention, it is desirable to train a deep learning model for generating super-resolution images by generating a plurality of training data for a plurality of targets of interest. Steps S2200 to S2214 are performed for new targets of interest that require training data generation.

FIG. 23 is a flowchart illustrating in detail the step of generating a SAR simulated received signal for a 3D target model (S2206) by simulating a received signal that is input to the SAR by reflecting the transmitted signal of the SAR shown in FIG. 22 from the 3D target model.

Referring to FIG. 23, in step S2300, physical parameters used by SAR to generate a SAR simulated received signal, that is, SAR system parameters, are set.

In step S2302, the time point at which the SAR transmits the transmission waveform (initial t=t(0)) is determined, and the transmission time point can be updated with the PRI cycle.

In step S2304, the transmission waveform of the SAR at a random transmission point can be generated to simulate the behavior of the SAR transmitting the transmission waveform.

In step S2306, the 3D model can be divided into a number of cells by projecting it in two dimensions at the location of the SAR.

In step S2308, the electromagnetic wave reflectivity for each of the plurality of cells divided in step S2306 and the cell distance, which is the distance between the SAR and the cell, can be calculated at each transmission time (i.e., at each transmission cycle) for the entire simulated movement path of the SAR. In step S2308, the reflectance can be calculated using [Equation 1], and the cell distance can be calculated through 3D target modeling software.

In step S2310, a specific point simulated reception signal (a) may be generated at a random transmission time for each of a plurality of specific points (A0 to D0) located in the cell. Step S2310 uses [Equation 2] to simulate each of a number of specific points (point_K) (point_K=A0, B0, C0, D0) located in the (i, j)-th cell at a random transmission time. A received signal (a) can be generated.

In step S2312, a simulated received signal of a cell (i.e., a simulated cell received signal) (b) can be generated by summing the simulated received signals (a) of a specific point for a random transmission time point generated in step S2310. When four specific points are designated in one cell as shown in FIG. 9, the cell simulated received signal is the result of adding the four specific point simulated received signals (a) calculated by [Equation 2] at a random transmission time.

In step S2314, steps S2308 to S2312 are repeated until generation of cell simulated received signals for all cells is completed.

When all cell simulated received signals (b) for all divided cells are generated (S2314-yes), in step S2316, the cell simulated received signals (b) generated for each multiple cell are added to create a simulated received signal for the 3D target model. (c) can be generated. As a result, a simulated reception signal (c) for the 3D target model is generated at a random transmission point (for example, t(azi_idx=0)).

In step S2318, it is determined whether the generation of a simulated received signal for the entire SAR path has been completed.

While updating the SAR transmission waveform transmission point (t(azi_idx=0), t(azi_idx=1), ??, t(azi_idx=N-1)) until the generation of simulated received signals for the entire SAR path is completed. Generate a simulated received signal (c) for the 3D target model.

Upon completion of generating a simulated reception signal for the entire SAR path (S2318-yes), the generated signal is generated at every transmission point (t(azi_idx=0), t(azi_idx=1), ??, t(azi_idx=N-1)). A SAR simulated received signal can be generated by collecting the simulated received signal (c) for the 3D target model.

In steps S2300 to S2320, a situation in which the SAR transmits a transmission waveform to the 3D target model at each transmission cycle while moving along the simulated movement path is simulated, and the generated simulated signals are collected to generate a SAR simulated reception signal. .

FIG. 24 is an example diagram showing a series of operations for generating training data according to the self-training data generation (S2100) step shown in FIG. 21.

Referring to FIG. 24, the training data generation step (2100) generates a 3D target model (1) for the target of interest.

Based on the direction in which the SAR is looking at the target of interest, the posture of the generated 3D target model is set in various ways (2), and the 3D target model is projected in two dimensions in the direction in which the SAR is looking to generate a high-resolution image for training. (3).

The SAR's transmitted signal is reflected from the 3D target model and simulates the received signal input to the SAR to generate a SAR simulated received signal for the 3D target model (4).

Low-resolution images for training on targets of interest are generated from SAR simulated received signals (5).

One training data pair is created according to the above process, and the high-resolution image for instruction and the low-resolution image for training generated through the above process constitute the same training data pair (see blue dotted line).

Showing multiple example images in (2), (3), and (5) is to explain that the above process is repeated to generate high-resolution images for training and low-resolution images for training in various postures for the 3D target model. This does not mean that multiple training data pairs are created when the above process is executed once.

FIG. 25 is a flowchart detailing the deep learning model creation and deep learning model training (S2104) steps for ultra-high resolution image generation shown in FIG. 21.

Referring to FIG. 25, in step S2500, a generator, discriminator, and reference model are created. Generator, Discriminator, and Reference Model may be deep learning models with a multi-layer artificial neural network structure.

The generator may include an upsampling layer and a convolutional layer that can increase the resolution of the input image and express image information.

The discriminator and reference model may include a feature extractor with a convolutional layer to enable extraction of image features for the input image. Each feature extractor owned by the discriminator and the reference model can be set to have the same structure so that the discriminator's featuremap information follows the corresponding featuremap information of the reference model during the discriminator's training process.

In step S2502, an optimization objective function for training the artificial neural network parameters of the generator and discriminator is set. At this time, [Equation 4] can be used as the optimization objective function.

In step S2504, an optimization objective function for training the artificial neural network parameters of the reference model is set. At this time, [Equation 3] can be used as the optimization objective function.

In step S2506, the reference model is trained using the high-resolution training image (I _{HR_tr} ) and the noise image (I _Noise ) of the database unit as input.

The high-resolution image for training (I _{HR_tr} ) is training data output from the database managed by the database department. The noise image has the same resolution as the high-resolution image for training, and can be easily generated by randomly setting each pixel value within the range that the pixel value can have.

When training the Reference Model, multiple high-resolution images for training and noise images are alternately input to the Reference Model, and the artificial neural network parameters of the Reference Model are optimized using the optimization objective function for training the Reference Model set in step S2504 above. While performing optimization, the Reference Model learns the image characteristics of high-resolution images and stores the information as the filter value of the convolutional layer.

In step S2508, the database unit outputs a high-resolution image for training.

In step S2510, train the discriminator. Referring to [Equation 4], the training of the discriminator performed in step S2510 is The process of finding the artificial neural network parameters of the discriminator that maximizes , It may include a process of minimizing .

remind The process of minimizing may include updating the artificial neural network parameters of the discriminator in a direction in which each feature map value existing in the discriminator follows each feature map value calculated by the reference model for the input super-resolution image for training. there is.

In step S2512, the database unit outputs a low-resolution image for training.

In step S2514, the generator and discriminator are trained.

Referring again to [Equation 4], the training of the discriminator performed in step S2514 is This is the process of finding the artificial neural network parameters of the discriminator that maximizes .

Generator training performed in step S1514 is This is the process of finding the artificial neural network parameters of the generator that minimize .

In step S2516, it is determined whether training of the generator and discriminator will be completed. To end training, a method for determining whether training has ended for a typical deep learning model can be applied. For example, it is possible to decide whether to end the training when a predetermined number of training times is reached or by observing the output of the generator.

Figure 26 is a flowchart showing a method for generating a super-resolution image of a target of interest in a SAR image using a trained generator according to an embodiment of the present invention.

Referring to FIG. 26, steps S2600 to S2604 have been described with reference to FIGS. 21 to 25, so detailed description will be omitted.

In step S2606, a super-resolution image is generated for the actual target image of interest using the trained generator.

In step S2604, the generator was trained to generate super-resolution images for the input low-resolution images for training. The target-of-interest image input in step S2606 is a target-of-interest image obtained from an actual SAR, and has image features similar to the low-resolution image used as training data in step S2604. Therefore, the generator can generate a super-resolution image by applying artificial neural network parameters optimized through training to the actual target image of interest.

Figure 27 is a flowchart showing a method for generating a super-resolution image of a target of interest in a SAR image using an actual target of interest image and target identification information according to an embodiment of the present invention.

Referring to FIG. 27, steps S2700 to S2704 have been described with reference to FIGS. 21 to 25, so detailed description will be omitted.

In step S2706, an actual target image of interest and target identification information are received.

In step S2708, a dictionary with target information matching the received target identification information is selected from the database.

In step S2710, after confirming the similarity between the Key value of the selected dictionary and the actual image of the target of interest, a super-resolution image is output.

Since the key value of the dictionary is a low-resolution training image, the similarity between all low-resolution training images constituting the dictionary and the actual target image of interest is checked, and the value corresponding to the key value with the highest similarity is output. Since the Value value corresponding to the Key value is a high-resolution image for training that corresponds to a low-resolution image for training, the output high-resolution image for training can be selected as a super-resolution image for the actual target image of interest.

Meanwhile, although the preferred embodiments have been described and illustrated to illustrate the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described, and does not deviate from the scope of the technical idea. Without limitation, those skilled in the art will understand that many changes and modifications can be made to the present invention. Accordingly, all such appropriate changes, modifications and equivalents should be considered to fall within the scope of the present invention. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached registration claims.

10: Device for generating ultra-high resolution images for targets of interest in SAR images 100: Self-training data generation unit
110: 3D modeling unit 120: Model posture setting unit
130: High-resolution image generator for training 140: SAR simulated reception signal generator
150: Low-resolution image generation unit for training
141: Parameter setting unit 143: Waveform generation unit
145: cell division unit 147: calculation unit
149: Simulated signal unit
149a: Specific point simulation signal generation unit 149b: Cell simulation signal generation unit
149c: 3D model simulated signal generation unit 149d: Gathering unit
200: Database unit
300: Ultra-high-resolution image generation deep learning model training department
310: Generator 320: Discriminator
330: Optimization unit 331: Reference Model
332: Optimization calculation 331-1: Reference Model optimization unit
20: Device for generating super-resolution images of targets of interest in SAR images
400: Ultra-high resolution image generator 410: Generator
420: Database-based ultra-high-resolution image generation unit

Claims (31)

In generating super-resolution images of targets of interest from Synthetic Aperture Radar (SAR) images,
A training data generation step consisting of high-resolution (HR, High Resolution) images and low-resolution (LR, Low Resolution) images of the target of interest;
Creating a database containing the generated training data as a component;
Creating a deep learning model for generating a super-resolution (SR, Super-Resolution) image for a target of interest in a SAR image, and training the deep learning model using training data in the database;
A method of generating a super-resolution image of a target of interest in a SAR image including.
According to paragraph 1,
The training data generation step consisting of high-resolution and low-resolution images for the target of interest is,
(A) generating a 3D target model for the target of interest;
(B) setting the posture of the 3D target model based on the direction in which the SAR faces the target of interest;
(C) generating a high-resolution image for training for the target model by converting the 3D target model whose posture is set in step (B) into a high-resolution 2D image;
(D) generating a low-resolution image for training on a target of interest from the 3D target model through a series of signal processing processes;
(E) generating a training data pair consisting of the high-resolution image for training generated in step (C) and the low-resolution image for training generated in step (D);
A method of generating a super-resolution image for a target of interest in a SAR image, comprising:
According to paragraph 2,
In step (D),
(D1) generating a SAR simulated received signal for the 3D target model by simulating a received signal input to the SAR by reflecting the transmitted signal of the SAR from the 3D target model;
(D2) performing SAR processing from the SAR simulated reception signal to generate a low-resolution image for training on the target of interest;
A method of generating a super-resolution image for a target of interest in a SAR image, comprising:

According to paragraph 3,
The step (D1) is,
(D1-1) generating a transmission waveform of the SAR transmission signal;
(D1-2) projecting the 3D target model generated in step (A) at the location of the SAR and dividing it into a plurality of cells;
(D1-3) calculating the electromagnetic wave reflectivity for each of the divided cells and the cell distance, which is the distance between the SAR and the cell, at each transmission time;
(D1-4) generating a simulated received signal for the 3D target model using the electromagnetic wave reflectivity, cell distance, and transmission waveform calculated for each of the plurality of cells;
Including,
The step (D1-4) is,
A method of generating a super-resolution image for a target of interest in a SAR image, characterized in that generating a simulated received signal for the entire simulated movement path of the SAR.

According to paragraph 4,
The step (D1-3) is,
A method of generating a super-resolution image for a target of interest in a SAR image, characterized in that the reflectivity of the cell is calculated using the amount of change in the distance between a plurality of specific points located in the cell and the center of the SAR antenna.
According to clause 5,
In the step (D1-3), a super-resolution image generation method for a target of interest in a SAR image is characterized in that the reflectivity of the cell is calculated using the following equation:

Here, Reflectivity is the reflectivity at one of a number of specific points (point_K) located in the cell corresponding to (i, j) at a random transmission point, is a positive arbitrary constant, point_K0 means the same point as point_K selected from among the plurality of specific points, point_K1 and point_K2 are the surrounding points of the specific point (point_K), d_cell is the distance between SAR and the specific point or surrounding points lim.
According to paragraph 4,
The step (D1-4) is,
(D1-4-1) generating a simulated reception signal (hereinafter referred to as 'specific point simulated reception signal') at a random transmission point for each of a plurality of specific points located in the cell;
(D1-4-2) generating a simulated received signal of a cell (hereinafter referred to as 'cell simulated received signal') by adding the specific point simulated received signals generated in step (D1-4-1);
(D1-4-3) generating a simulated received signal for the 3D target model at a random transmission point by adding the cell simulated received signals generated for each plurality of cells in step (D1-4-2);
A method of generating a super-resolution image for a target of interest in a SAR image, comprising:
In clause 7,
The step (D1-4-1) is a super-resolution image of a target of interest in a SAR image, characterized in that it generates a simulated received signal at one of a plurality of specific points located in the cell using the following equation: How to create:

Here, rx_timed_cell is a simulated reception signal at one of a number of specific points (A0 ~ D0) (point_K, K = A0, B0, C0, D0) located in the cell corresponding to (i, j) at a random transmission point. , Reflectivity is the electromagnetic wave reflectivity at one (point_K, K=A0, B0, C0, D0) of multiple specific points (A0~D0), d_cell is a specific point of the i and jth cells at the antenna center of the SAR (point_K ) or means the distance between surrounding points, t is time, pi is the pi, f0 is the center frequency of the SAR transmission signal, c is the speed of light, and Kr is the chirp rate, which represents the change in frequency per unit time within the chirp waveform. means Hz/sec, exp means Euler's formula expressed as exp(jХArg) = cos(Arg) + jХsin(Arg) .
In addition, the transmission waveform of the transmission signal is assumed to be pulse_shape(t) ^* × exp(j ^* × pi ^* × Kr ^* × t^2), and pulse_shape(t) is the envelope function of the transmission waveform as follows. Has value.



According to paragraph 2,
After step (E) above,
(F) While changing the posture of the 3D target model in step (B), steps (B) to (E) are repeatedly executed to generate a number of low-resolution images for training and high-resolution images for training for a specific target of interest. generating step;
(G) generating a plurality of low-resolution images for training and high-resolution images for training for a plurality of new targets of interest by repeatedly performing steps (A) to (F) for a plurality of new targets of interest;
A method of generating a super-resolution image for a target of interest in a SAR image, further comprising:

According to paragraph 1,
The step of creating a database containing the generated training data as a component is,
For low-resolution images for training and high-resolution images for training generated for a specific target of interest, creating a dictionary that sets the low-resolution image for training as a Key and sets the corresponding high-resolution image for training as a Value;
Constructing a database with the following structure by generating the dictionary for each target of interest and assigning a target identifier to each generated dictionary;
A method for generating a super-resolution image for a target of interest in a SAR image, comprising:
Start database [
Target_A: Dictionary
Target_B: Dictionary
Target_C: Dictionary
] Database shutdown
Here, the structure of the dictionary is as follows.
{key: low-resolution image for training, value: high-resolution image for training,
key: low-resolution image for training, value: high-resolution image for training,
key: low-resolution image for training, value: high-resolution image for training}

According to paragraph 1,
The step of creating a deep learning model for generating a super-resolution (SR) image for a target of interest in a SAR image and training the deep learning model using training data in the database,
(A) Creating a generator with a multi-layer artificial neural network structure including an upsampling layer and a convolutional layer capable of increasing the resolution of the input image and expressing image information;
(B) generating a reference model with a multi-layer artificial neural network structure including at least one feature extractor including a convolutional layer to extract image features for the input image data;
(C) generating a discriminator with a multi-layer artificial neural network structure, including a feature extractor with the same structure as the feature extractor constituting the Reference Model, in order to extract image features for the input image data;
(D) setting an optimization objective function to optimize the artificial neural network parameters (weight, bias) of the generator and discriminator;
(E) setting a reference model optimization objective function to optimize the artificial neural network parameters of the reference model generated in step (B);
(F) optimizing the artificial neural network parameters of the Reference Model using the high-resolution training image and noise image output from the database unit as input data of the Reference Model;
(G) outputting high-resolution images for training from the database unit to update the artificial neural network parameters of the discriminator;
(H) outputting low-resolution images for training from the database unit to update the artificial neural network parameters of the generator and discriminator;
(I) determining whether the training of the generator and discriminator has been completed and repeating steps (G) to (H) until the training is completed;
A method of generating a super-resolution image for a target of interest in a SAR image, comprising:

According to clause 11,
In step (D),
A method of generating a super-resolution image for a target of interest in a SAR image, characterized by setting an optimization objective function to optimize the artificial neural network parameters (weight, bias) of the generator and discriminator using the following equation:
Optimization objective function for discriminator

Here, I _{HR_tr} is a high-resolution image for training, I _{LR_tr} is a low-resolution image for training, paramsD is an artificial neural network parameter constituting the discriminator, paramsG is an artificial neural network parameter constituting the generator, and D _paramsD (I) is a parameter value named paramsD. is the output value for the input I of the Discriminator, G _paramsG (I) is the output value for the input I of the Generator with the parameter value named paramsG, E[ ] is the expected value, and N_featuremap is the feature map included in all convolutional layers in the Discriminator. The total number of, featuremap_Discriminator(n) is the nth featuremap among each featuremap existing in the Discriminator, and featuremap_ReferenceModel(n) is the nth featuremap among each featuremap existing in the Reference Model.

Optimization objective function for generator


According to clause 11,
In step (E),
A method of generating a super-resolution image for a target of interest in a SAR image, characterized by training the Reference Model using the following optimization objective function:
Optimization objective function for reference model

here, is a super-resolution video for training; is the noise image, paramsR is the artificial neural network parameter constituting the Reference Model, and R _paramsR (I) is the output value for the input I of the Reference Model with the parameter value named paramsR.

According to clause 11,
(J) after the step (I), inputting a target image of interest in the actual SAR image as an input to the trained generator, and outputting the generator output thereto as a super-resolution image for the input target image;
A method of generating a super-resolution image for a target of interest in a SAR image, further comprising:
According to clause 11,
(J) after step (I), receiving a target image of interest and target identification information in the actual SAR image;
(K) selecting a dictionary corresponding to the target identification information from a database using the target identification information;
(L) Check the similarity between the low-resolution image for training stored in the Key value of the selected dictionary and the target image of interest received in step (J), and save Value data corresponding to the Key value with the highest similarity (J). ) step of outputting the image of the target of interest received in step as a super-resolution image;
A method of generating a super-resolution image for a target of interest in a SAR image, further comprising:

In a device for generating a super-resolution image of a target of interest in a Synthetic Aperture Radar (SAR) image,
A self-training data generator that generates training data consisting of a high-resolution (HR) image and a low-resolution (LR) image for the target of interest;
a database unit that creates and manages a database containing the generated training data as a component;
A super-resolution image generation deep learning model training unit that generates a deep learning model for generating super-resolution (Super-Resolution) images for targets of interest in the SAR image, and trains the deep learning model using training data in the database;
A device for generating ultra-high resolution images of targets of interest in SAR images, including a device.
According to clause 16,
The self-training data generator,
A 3D modeling unit that generates a 3D target model for the target of interest;
a model pose setting unit that sets the pose of the 3D target model in various directions based on the direction in which the SAR looks at the target of interest;
a high-resolution image generator for training that converts the 3D target model with the set posture into a high-resolution 2D image and generates a high-resolution image for training for the target model;
A SAR simulated received signal generator that generates a simulated received signal for the 3D target model by simulating a received signal input to the SAR by reflecting the transmitted signal of the SAR from the 3D target model;
a low-resolution image generator for training that performs SAR processing from the generated SAR simulated reception signal to generate a low-resolution image for training on a target of interest;
A super-resolution image generating device for a target of interest in a SAR image, comprising:

According to clause 17,
The SAR simulated received signal generator,
a waveform generator that generates a transmission waveform of the SAR transmission signal;
a cell division unit that projects the 3D target model generated by the 3D modeling unit at the location of the SAR and divides it into a plurality of cells;
a calculation unit that calculates electromagnetic wave reflectivity for each of the plurality of divided cells and a cell distance, which is the distance between the SAR and the cell, at each transmission time;
a simulation signal unit that generates a SAR simulated reception signal for the 3D target model using the electromagnetic wave reflectivity, cell distance, and transmission waveform calculated for each of the plurality of cells;
Including,
The simulated signal unit,
A super-resolution image generating device for a target of interest in a SAR image, characterized in that it generates a SAR simulated reception signal for the entire simulated movement path of the SAR.

According to clause 18,
The calculation unit,
A super-resolution image generating device for a target of interest in a SAR image, characterized in that the reflectivity of the cell is calculated using a change in the distance between a plurality of specific points located in the cell and the center of the SAR antenna.

According to clause 19,
The calculation unit,
A super-resolution image generating device for a target of interest in a SAR image, characterized in that the reflectivity of the cell is calculated using the following equation:

Here, Reflectivity is the reflectivity at one of a number of specific points (point_K) located in the cell corresponding to (i, j) at a random transmission point, is a positive arbitrary constant, point_K0 means the same point as point_K selected from among the plurality of specific points, point_K1 and point_K2 are the surrounding points of the specific point (point_K), d_cell is the distance between SAR and the specific point or surrounding points lim.

According to clause 18,
The simulated signal unit,
a specific point simulated signal generator that generates a simulated received signal (hereinafter referred to as a 'specific point simulated received signal') at a random transmission point for each of a plurality of specific points located in the cell;
a cell simulated signal generator that generates a cell simulated received signal (hereinafter referred to as a 'cell simulated received signal') by combining the generated specific point simulated received signals;
a 3D target model simulated signal generator that generates a simulated received signal for the 3D target model at a random transmission point by combining cell simulated received signals generated for each of the plurality of cells;
A super-resolution image generating device for a target of interest in a SAR image, comprising:

According to clause 21,
The specific point simulation signal generator,
A super-resolution image generating device for a target of interest in a SAR image, characterized in that it generates a simulated received signal at one of a plurality of specific points located in the cell using the following equation:


Here, rx_timed_cell is a simulated reception signal at one of a number of specific points (A0 ~ D0) (point_K, K = A0, B0, C0, D0) located in the cell corresponding to (i, j) at a random transmission point. , Reflectivity is the electromagnetic wave reflectivity at one (point_K, K=A0, B0, C0, D0) of multiple specific points (A0~D0), d_cell is a specific point of the i and jth cells at the antenna center of the SAR (point_K ) or means the distance between surrounding points, t is time, pi is the pi, f0 is the center frequency of the SAR transmission signal, c is the speed of light, and Kr is the chirp rate, which represents the change in frequency per unit time within the chirp waveform. means Hz/sec, exp means Euler's formula expressed as exp(jХArg) = cos(Arg) + jХsin(Arg) .
In addition, the transmission waveform of the transmission signal is assumed to be pulse_shape(t) ^* × exp(j ^* × pi ^* × Kr ^* × t^2), and pulse_shape(t) is the envelope function of the transmission waveform as follows. Has value.



According to clause 18,
A super-resolution image of a target of interest in a SAR image, characterized in that one training data is generated each time the posture is adjusted by the model posture setting unit, and a plurality of training data are generated for the target of interest while adjusting the posture. Generating device.

According to clause 23,
The 3D modeling unit generates models for multiple targets of interest and generates multiple training data for each target of interest.

According to clause 16,
The database part,
For low-resolution images for training and high-resolution images for training generated for a specific target of interest, a dictionary is created that sets the low-resolution image for training as Key and sets the corresponding high-resolution image for training as Value;
A super-resolution image generating device for a target of interest in a SAR image, characterized in that the dictionary is generated for each target of interest, and a target identifier is assigned to each generated dictionary to form a database with the following structure:
Start database [
Target_A: Dictionary
Target_B: Dictionary
Target_C: Dictionary
] Database shutdown
Here, the structure of the dictionary is as follows.
{key: low-resolution image for training, value: high-resolution image for training,
key: low-resolution image for training, value: high-resolution image for training,
key: low-resolution image for training, value: high-resolution image for training}

According to clause 16,
The super-resolution image generation deep learning model training unit,
Generator with a multi-layer artificial neural network structure including an upsampling layer and a convolutional layer that can increase the resolution of the input image and express image information;
In order to extract image features for input image data, a discriminator with a multi-layer artificial neural network structure including at least one feature extractor including a convolutional layer;
Includes an optimization unit that optimizes the artificial neural network parameters (weight, bias) of the generator and discriminator through training,
The optimization unit includes an optimization calculation unit, a Reference Model, and a Reference Model optimization unit,
The Reference Model is created with a multi-layer artificial neural network structure that includes at least one feature extractor including a convolutional layer to extract image features for the input image data,
Each feature extractor included in the Discriminator has the same structure as each feature extractor included in the Reference Model,
The Reference Model is optimized using high-resolution images and noise images for training as training data,
A super-resolution image generation device for a target of interest in a SAR image, wherein the optimization unit trains a generator and a discriminator using an optimization objective function.

According to clause 26,
The optimization unit,
In order to optimize the artificial neural network parameters (weight, bias) of the generator and discriminator, a super-resolution image generation device for the target of interest in the SAR image is characterized by using the following equation as the optimization objective function:
Optimization objective function for discriminator

Here, I _{HR_tr} is a high-resolution image for training, I _{LR_tr} is a low-resolution image for training, paramsD is an artificial neural network parameter constituting the discriminator, paramsG is an artificial neural network parameter constituting the generator, and D _paramsD (I) is a parameter value named paramsD. is the output value for the input I of the Discriminator, G _paramsG (I) is the output value for the input I of the Generator with the parameter value named paramsG, E[ ] is the expected value, and N_featuremap is the feature map included in all convolutional layers in the Discriminator. The total number of, featuremap_Discriminator(n) is the nth featuremap among each featuremap existing in the Discriminator, and featuremap_ReferenceModel(n) is the nth featuremap among each featuremap existing in the Reference Model.

Optimization objective function for generator


According to clause 26,
The optimization unit,
A super-resolution image generating device for a target of interest in a SAR image, characterized by training the Reference Model using the following optimization objective function:
Optimization objective function for reference model

here, is a super-resolution video for training; is the noise image, paramsR is the artificial neural network parameter constituting the Reference Model, and R _paramsR (I) is the output value for the input I of the Reference Model with the parameter value named paramsR.
According to clause 27,
The optimization unit,
Input the high-resolution training image output from the database into the discriminator, observe the discriminator's output value, and Update the artificial neural network parameters of the discriminator so that is the maximum;
After inputting the output high-resolution training image into the Reference Model, check each feature map value calculated by the Reference Model;
After inputting the output high-resolution image for training into the discriminator, updating the artificial neural network parameters of the discriminator in such a way that each feature map value existing in the discriminator follows the feature map value of the corresponding reference model;
Input the low-resolution image for training output from the database into the generator, input the output of the generator into the discriminator, and observe the output of the discriminator, Update the artificial neural network parameters of the generator to minimize Update the artificial neural network parameters of the discriminator so that is maximized;
A super-resolution image generation device for a target of interest in a SAR image, characterized in that it optimizes artificial neural network parameters of the generator and discriminator by repeating the output of the high-resolution image for training and the output of the low-resolution image for training.
According to clause 26,
A super-resolution image generator that applies the trained generator to receive target-of-interest images in the actual SAR image and outputs a super-resolution image for the received target-of-interest image;
A super-resolution image generating device for a target of interest in a SAR image, further comprising:

According to clause 26,
Receive a target image of interest and target identification information in the actual SAR image, select a dictionary corresponding to the target identification information from the database using the received target identification information, and then store a low-resolution image for training in the key value of the selected dictionary. and a super-resolution image generator that checks the similarity with the received target image of interest and outputs value data corresponding to the key value with the highest similarity calculated as a super-resolution image for the received target image of interest.
A super-resolution image generating device for a target of interest in a SAR image, further comprising: