KR20210045839A - Scene-based nonuniformity correction method and apparatus using deep neural network - Google Patents

Abstract

According to an embodiment of the present invention, through a method and a device for training a deep learning neural network that outputs an offset for scene-based non-uniformity correction, a final offset with high quality and accuracy can be obtained, and a scene-based non-uniformity corrected image can be obtained through the offset. The method for training a deep learning neural network comprises the following steps of: outputting a first offset; calculating a similar image roughness cost function and a spatial noise cost function; obtaining a final cost function; and training the deep neural network.

Description

Scene-based nonuniformity correction method and apparatus using deep neural network

The present invention relates to a scene-based non-uniformity correction method and apparatus using a deep learning neural network.

Infrared sensors, which are currently applied in various fields such as medical, military, and science, are developed from conventional 1D sensors and then use a focal plane array (FPA) 2D sensor to detect infrared radiation energy of an object. Infrared sensor FPA has a different gain and offset for each device due to the non-uniformity generated in the manufacturing process, which results in fixed pattern noise (FPN). Methods for correcting this include Calibration-based Nonuniformity Correction (CBNUC) using a blackbody and Scene-based Nonuniformity Correction (SBNUC) using an image generated by FPA.

CBNUC is a method of obtaining and applying gain and offset by using correction equipment such as blackbody. Two point NUC that corrects the gain and offset of FPA using blackbody of two different temperatures is typically used. CBNUC is easier to implement than SBNUC and has the advantage of obtaining accurate correction tables, but it cannot cope with changes in the surrounding environment outside of the home, and periodic correction is required to compensate for changes in FPA characteristics that occur after a long period of time after initial non-uniformity correction. It must be accompanied. In particular, there is a disadvantage in that it is difficult to cope with the temperature drift phenomenon occurring in the FPA after the sensor is powered on. As a method to overcome this, a shutter is mounted on the front of the sensor to perform shutter compensation to compensate for the drift phenomenon according to temperature, but if the shutter is operated periodically, the sensor surface is covered and continuous image acquisition is impossible.

SBNUC is a method of acquiring and applying the gain and offset of the FPA based on the image formed on the FPA, and does not require correction equipment such as black bodies or chambers for non-uniformity correction. Since it is based on an image, there is an advantage in that it is possible to acquire images continuously without interruption because there is no need for shutter compensation to cope with FPA and surrounding changes. However, compared to the CBNUC method, there is a disadvantage that real-time implementation is limited within limited system resources because a higher amount of computation is required.

SBNUC can be largely divided into statistics based SBNUC and registration based SBNUC. Statistics-based SBNUC is applied by probabilistically estimating the gain and offset under the assumption that the output average of each detector element is temporarily the same due to the radiation of the scene and the variance of the output change is the same. Statistics-based SBNUC is advantageous for real-time application because it requires relatively low computational amount and memory compared to matching-based SBNUC. However, unlike in the home, if various scenes are not input, ghosting is likely to occur. In addition, it has a slower convergence speed compared to the matching-based SBNUC, and has a disadvantage that ghosting is likely to occur if the learning rate is increased to overcome the slower convergence speed.

Matching-based SBNUC assumes that the scenes between successive frames are the same, detects signals of the same scene through different devices when scene movement occurs due to camera motion, and estimates the gain and offset by using the signal difference between the two devices. . Compared to probability-based SBNUC, this method has the advantage of fast convergence speed and little ghost phenomenon. However, matching-based SBNUC requires a relatively large amount of computation and memory compared to statistics-based SBNUC, so real-time implementation is difficult compared to probability-based SBNUC. Particularly, since it is based on image registration, if the correction factor is not updated for the part that does not overlap with the previous frame, or if image registration is unfavorable due to image movement such as rotation, reduction, and enlargement of the image, additional correction is not performed or correction error rate There is a downside to this increase. In addition to the probability and matching-based SBNUC algorithm, there is a method using a neural network, Two Frame NUC (TFNUC) using two frames, and an algorithm that can correct into a single image frame by applying midway equalization in column units.

[Prior patent document number]

Priority 1: Korean Patent Registration No. 10-1854355

Prior 2: Korean Patent Registration No. 10-1910083

[Non-patent literature]

1. D A Scribner, M Kruer, and J Killiany, “Infrared focal plane array technology,” Proc IEEE, vol 79, no 1, pp 66-85, Jan 1991

2. A Friedenberg and I Goldbatt, “Nonuniformity two-point linear correction errors in infrared focal plane arrays,” Opt Eng 37(4), pp 1251-1253, April, 1998

3. O Riou, S Berrebi, and P Bremond, “Nonuniformity correction and thermal drift compensation of thermal infrared camera,” Proc SPIE 5405, pp 294-302, April, 2004

4. R Hardie, F Baxley, B Brys, and P Hytla, “Scene-based nonuniformity correction with reduced ghosting using a gated LMS algorithm,” Opt Express 17, pp 14918-14933, 2009

5. J Harris and Y Chiang, “Minimizing the ‘ghosting’ artifact in scene-based nonuniformity correction,” Proc SPIE 3377, pp 106-113, 1998

6. Russell C Hardie, Majeed M Hayat, Earnest Armstrong, and Brian Yasuda, “Scene-based nonuniformity correction with video sequences and registration,” Applied Optics Vol 39, Issue 8, pp 1241-1250, 2000

7. Chao Zuo, Qian Chen, Guohua Gu and Xiubao Sui, Scene-Based Nonuniformity Correction Algorithm Based on Interframe Registration,” J Opt Soc Am A Opt Image Sci Vis 28 (6), pp 1164-1176, Jun, 2011

8. Chao Zuo, Qian Chen, Guohua Gu, Xiubao Sui, and Jianle Ren, “Improved interframe registration based nonuniformity correction for focal plane arrays,” ELSEVIER Infrared Physics & Technology Vol 55, Issue 4, pp 263-269, July 2012

An embodiment provides a scene-based non-uniformity correction method and apparatus using a deep learning neural network.

Another embodiment provides a method and apparatus for training a deep learning neural network that outputs an offset for scene-based non-uniformity correction.

A method for training a deep learning neural network that outputs an offset for scene-based non-uniformity correction, according to an embodiment, comprising the steps of: outputting a first offset through a deep learning neural network to which an image to which non-uniformity correction is not applied; Calculating a similar image roughness cost function and a spatial noise cost function from the image to which the non-uniformity correction was not applied and the first non-uniformity-corrected image obtained by summing the outputted first offset; Obtaining a final cost function from the sum of the calculated similar image roughness cost function and spatial noise cost function; And providing the obtained final cost function to the deep learning neural network as feedback information to train the deep neural network.

The pseudo image roughness cost function is characterized in that the learning speed of the deep neural network is adjusted in proportion to the roughness of the image.

The similar image roughness cost function is a similar image roughness function obtained by averaging a result of a convolution operation of a horizontal difference filter and a vertical difference filter on the first non-uniformity correction applied image, and each of the horizontal difference filter and the vertical difference filter It is characterized in that the result of the convolution operation is squared.

The pseudo image roughness cost function is characterized in that it is calculated by the following equation.

[Equation]

는 유사영상거칠기, h1은 수평차이필터, h2는 수직차이필터, I는 영상, M은 영상 가로 픽셀수, N은 영상 세로 픽셀수,

는 L1 norm, *는 discrete convolution 이다.here,

Is the pseudo-image roughness, h1 is the horizontal difference filter, h2 is the vertical difference filter, I is the image, M is the number of horizontal pixels in the image, N is the number of vertical pixels in the image,

Is L1 norm, * is discrete convolution.

The spatial noise cost function is characterized by being calculated by the following equation.

[Equation]

는 영상 평균,

는 영상 공간잡음,

는 영상 공간 잡음 비용함수이고, I는 영상, M은 영상 가로 픽셀수, N은 영상 세로 픽셀 수이다.here,

Is the image mean,

Is the image spatial noise,

Is the image spatial noise cost function, I is the image, M is the number of horizontal pixels in the image, and N is the number of vertical pixels in the image.

A scene-based non-uniformity correction method using a deep learning neural network according to another embodiment includes: outputting a first offset through a deep learning neural network to which an image to which the non-uniformity correction is not applied is input; Calculating a similar image roughness cost function and a spatial noise cost function from the image to which the non-uniformity correction was not applied and the first non-uniformity-corrected image obtained by summing the outputted first offset; Obtaining a final cost function from the calculated similar image roughness cost function and spatial noise cost function; Providing the obtained final cost function to the deep learning neural network as feedback information for training; And summing the final offset output from the learned deep learning neural network and the input original image to output a non-uniformly corrected corrected image.

A scene-based non-uniformity correction method using a deep learning neural network according to another embodiment includes the steps of inputting an original image to a deep learning neural network learned by a learning method of the deep learning neural network; And summing the final offset output from the learned deep learning neural network and the original image to output a non-uniformly corrected corrected image.

According to another embodiment, an apparatus for training a deep learning neural network that outputs an offset for scene-based non-uniformity correction, comprising: a memory; And a processor,

The processor outputs a first offset through a deep learning neural network into which a non-uniformity-corrected image is input, and a similar image roughness cost from a first non-uniformity-corrected-applied image obtained by summing the non-uniformity-corrected image and the output first offset. Calculate a function and spatial noise cost function, obtain a final cost function from the sum of the calculated similar image roughness cost function and spatial noise cost function, and provide the obtained final cost function to the deep learning neural network as feedback information Thus, the deep neural network is trained.

The pseudo image roughness cost function is characterized in that the learning speed of the deep neural network is adjusted in proportion to the roughness of the image.

The similar image roughness cost function is a similar image roughness function obtained by averaging a result of a convolution operation of a horizontal difference filter and a vertical difference filter on the first non-uniformity correction applied image, and each of the horizontal difference filter and the vertical difference filter It is characterized in that the result of the convolution operation is squared.

A scene-based non-uniformity correction apparatus using a deep learning neural network according to another embodiment, comprising: a memory; And a processor,

The processor outputs a first offset through a deep learning neural network into which a non-uniformity-corrected image is input, and a similar image roughness cost from a first non-uniformity-corrected-applied image obtained by summing the non-uniformity-corrected image and the output first offset. Learning by calculating a function and spatial noise cost function, obtaining a final cost function from the calculated similar image roughness cost function and spatial noise cost function, and providing the obtained final cost function to the deep learning neural network as feedback information Then, the final offset output from the learned deep learning neural network and the input original image are summed to output a corrected image corrected for non-uniformity.

It includes a recording medium recording a program for executing the method according to another embodiment on a computer.

Scene-based non-uniformity correction (SBNUC) based on statistics based on existing statistics and registration based on scene-based non-uniformity correction (SBNUC) using a deep learning neural network according to an embodiment is a non-uniformity of at least 50 to 300 frames. Compared to the need for an uncorrected image, it is possible to obtain a high-quality non-uniform correction offset with a small number of frames of 2 to 10.

In addition, the scene-based non-uniformity correction method and apparatus using the deep learning neural network according to the embodiment is not affected by rotation, enlargement, and reduction of the vulnerable image in the registration-based scene-based non-uniformity correction (SBNUC). There is almost no ghost image phenomenon compared to the scene-based non-uniformity correction (SBNUC) of statistics based and registration based.

1 is a schematic diagram of a scene-based non-uniformity correction apparatus 100 according to an exemplary embodiment.
2 is a schematic diagram of training a deep learning neural network that outputs an offset for scene-based non-uniformity correction according to another embodiment.
3 and 4 are exemplary diagrams for explaining a similar image roughness cost function.
5 are exemplary diagrams for explaining a spatial noise cost function.
6 and 7 are exemplary diagrams for describing a deep learning neural network according to an embodiment.
8 is a flowchart illustrating a scene-based non-uniformity correction method using a deep learning neural network according to another embodiment.
9 to 11 are exemplary diagrams for explaining a scene-based non-uniformity correction method using a deep learning neural network according to an embodiment.

The terms used in the present embodiments have selected general terms that are currently widely used as possible while considering the functions in the present embodiments, but this may vary depending on the intention or precedent of a technician working in the art, the emergence of new technologies, etc. . In addition, in certain cases, there are terms that are arbitrarily selected, and in this case, the meaning will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents of the present embodiments, not a simple name of the term.

In the description of the embodiments, when a certain part is connected to another part, this includes not only a case in which it is directly connected, but also a case in which it is electrically connected with another component interposed therebetween. In addition, when a certain part includes a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, the term “... unit” described in the embodiments refers to a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

Terms such as “consisting of” or “comprising” used in the present embodiments should not be construed as necessarily including all of the various constituent elements or various steps described in the specification, and some constituent elements or some of them It should be construed that the steps may not be included, or may further include additional elements or steps.

The description of the following embodiments should not be construed as limiting the scope of the rights, and what those skilled in the art can easily infer should be construed as belonging to the scope of the rights of the embodiments. Hereinafter, embodiments for illustration only will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a scene-based non-uniformity correction apparatus 100 according to an exemplary embodiment.

In an embodiment, in the scene-based non-uniformity correction, the offset is estimated by removing and simplifying the gain from the basic model of the relationship between infrared radiation and the detector output as shown in Equation 1 below.

[Equation 1]

는 n번째 프레임의 오프셋,

는 n번째 프레임의 실제 방사 받은 적외선량,

는 n번째 프레임에서 적외선 센서에서 출력된 전압 값이다.here,

Is the offset of the nth frame,

Is the actual amount of infrared radiation received in the nth frame,

Is the voltage value output from the infrared sensor in the nth frame.

을 추정하기 위해 적외선 센서 출력

을 이용하므로 다음 수학식 2와 같이 변환하여 사용한다.The non-uniformity correction

Infrared sensor output to estimate

Is used, so it is converted as shown in Equation 2 below.

[Equation 2]

과

의 관계를 나타내고 있다.Equation 2 is

and

Shows the relationship of.

The scene-based non-uniformity correction apparatus 100 according to the embodiment performs non-uniformity correction in a new method different from the existing scene-based non-uniformity correction (SBNUC) technique, two statistics-based SBNUC and registration-based SBNUC. The deep learning neural network's learning function is used to learn, and the similar image roughness cost function and spatial noise cost function are used as learning feedback. By repeating the learning process and the feedback process, the final converged offset can be obtained. At this time, non-uniformity correction can be performed using the obtained offset.

The scene-based non-uniformity correction apparatus 100 receives an original image and outputs a corrected non-uniformity image. The scene-based non-uniformity correction apparatus 100 learns an offset using a neural network. Similar image roughness cost function and spatial noise cost function from the first non-uniformity-corrected image obtained by summing the non-uniformity-corrected image and the output first offset by outputting the first offset through the deep learning neural network into which the non-uniformity-corrected image is input Calculate A final cost function obtained by combining the similar image roughness cost function and spatial noise cost function is obtained, and the obtained final cost function is provided to the deep learning neural network as feedback information to learn. The scene-based non-uniformity correction apparatus 100 outputs a non-uniformly corrected corrected image by summing the final offset output from the learned deep learning neural network and the input original image.

A structure for learning an offset in such a neural network will be described with reference to FIG. 2. In an embodiment, the deep learning neural network scene-based non-uniformity correction algorithm is different from the existing probability and matching-based non-uniformity correction algorithm. In general, in neural network learning, a user defines the output result of a neural network targeting the neural network learning goal in the form of a label. However, in the case of scene-based non-uniformity correction, there is no corrected image or offset that can be defined as a target. For this reason, it is necessary to estimate the correction coefficient offset by a rule through the cost function using only the uncorrected image used as an input.

2 is a schematic diagram of training a deep learning neural network that outputs an offset for scene-based non-uniformity correction according to another embodiment.

In an embodiment, a scene-based non-uniformity correction structure using a deep learning neural network is as shown in FIG. 2. There is a deep learning neural network 102 to be used when estimating the non-uniformity correction offset, and the deep learning neural network receives an image 101 to which non-uniformity correction is not applied as an input and outputs an assumed offset 103. The assumed offset 103 output from the neural network is combined with the non-uniformity correction unapplied image to output the hypothesized non-uniformity correction applied image 105. The similar image roughness cost function 106 and the spatial noise cost function 107 are calculated based on the assumed non-uniformity correction applied image 105, and the results of the two are summed and used as the final cost function 108. This cost function is used as feedback when learning a neural network.

In an embodiment, uncorrected image stream data output from an image sensor is used as input, and the neural network inference result is assumed to be an offset. Since the neural network inference result is assumed to be an offset, it can be assumed that the uncorrected image used as an input of the neural network and the offset operation result are a corrected image to which non-uniformity correction has been performed. In the hypothesized non-uniformity correction corrected image, the total cost function is calculated using the similar image roughness cost function and the spatial noise cost function, and this is used as feedback when learning a neural network.

With reference to FIGS. 3 and 4, a similar image roughness cost function will be described. The similar image roughness function 201 is obtained by averaging the results obtained through the convolution operation on the image by using the horizontal difference filter 203 shown in FIG. 3A and the vertical difference filter 204 shown in FIG. 3B on an image. There is a characteristic that the value increases as the difference between the values increases.

The pseudo image roughness cost function is created based on the similar image roughness function, and is a similar image roughness cost function using the square of the convolution result of each of the horizontal difference filter 203 and the vertical difference filter 204 of the similar image roughness function. The square operation applied to the pseudo image roughness cost function gives strong feedback to the neural network when it moves away from the lowest cost function when training a deep learning neural network and enables a faster neural network learning speed. As shown in FIGS. 4A and 4B, when the image roughness is high (205 ), the image exhibits a texture similar to that of sandpaper, and this is usually the case in which the difference between the values of the adjacent pixels is large. When the image roughness is low (206), the image has a relatively smooth texture, which is usually the case where the difference between the values of the adjacent pixels is small.

The similar image roughness cost function can be defined according to Equations 3 to 5 below.

는 다음 수학식 3과 같이 정의되어 있다. First, the existing image roughness function

Is defined as in Equation 3 below.

[Equation 3]

Here, the image roughness function normalizes the absolute value of the difference from the pixels to its own pixel value. The existing image roughness function includes a part that is normalized to an image value. The similar image roughness function is defined as in Equation 4 below by removing the normalized part from the image value.

[Equation 4]

Through this, it is possible to reduce the amount of computation when learning a neural network, and to achieve a balance with the spatial noise cost function. If the pseudo-roughness function is used as it is, the image roughness will increase.

In case the value is also increased, it can be used without modification for learning, but if it is far from the global lowest value, the slope of the cost function used for learning is constant, which causes the learning speed to slow down. In order to solve this disadvantage, MSE (Mean Squared Error) is applied by adding a square to the difference value from the adjacent pixel as shown in Equation 5 below. This makes it possible to quickly converge when learning a neural network by increasing the slope value of the function as the image roughness increases. Through this configuration, the learning speed is adjusted in proportion to the roughness of the image.

[Equation 5]

는 유사영상거칠기, h1은 수평차이필터, h2는 수직차이필터, I는 영상, M은 영상 가로 픽셀수, N은 영상 세로 픽셀수,

는 L1 norm, *는 discrete convolution 이다.here,

Is the pseudo-image roughness, h1 is the horizontal difference filter, h2 is the vertical difference filter, I is the image, M is the number of horizontal pixels in the image, N is the number of vertical pixels in the image,

Is L1 norm, * is discrete convolution.

In an embodiment, the spatial noise cost function is a value indicating how distributed each pixel is compared to an average of an image. It is an average of the square of the difference value of each pixel compared to the average of the image. As each pixel is further away from the average, the size of the squared value increases. This feature gives strong feedback to the neural network when it moves away from the lowest cost function and enables a faster neural network learning speed. As shown in FIG. 5A, when the spatial noise is high, each of the pixels exhibits a non-uniform image due to a large difference in value compared to the average. As shown in FIG. 5B, when the spatial noise is low (303), the average values of all pixels represent almost the same value. It usually appears as a uniform gray image.

The spatial noise cost function is defined by the following equations 6 and 7.

[Equation 6]

[Equation 7]

는 영상 평균,

는 영상 공간잡음,

는 영상 공간잡음 비용함수,

은 L1 norm, M은 영상 가로 픽셀수, N은 영상 세로 픽셀 수를 나타낸다.here,

Is the image mean,

Is the image spatial noise,

Is the image spatial noise cost function,

Is L1 norm, M is the number of horizontal pixels in the image, and N is the number of vertical pixels in the image.

The spatial noise function is defined as the average of the absolute value of the difference value of each pixel compared to the image average as shown in Equation 6 above. The spatial noise function also has a property that the larger the value of each pixel compared to the image average, the larger the value, so it can be used for learning as it is. There is a tendency to make. In order to overcome this, MSE is applied by taking the square of the existing spatial noise as shown in Equation 7 above to enable rapid convergence when learning a neural network.

6 and 7 are exemplary diagrams for describing a deep learning neural network according to an embodiment.

As illustrated in FIG. 6, the neural network according to the embodiment may be composed of three layers including one input layer, one hidden layer, and one output layer. This is a result of tuning as it is shallower than a general deep learning neural network that uses at least two or three layers of hidden layers. This, in turn, helps with less memory usage and faster learning speed. The neural network consists of 4800 nodes in both the input/output and hidden layers to perform input/output of an uncorrected image of 80 pixels horizontally and 60 pixels vertically.

In the embodiment, although one hidden layer of the neural network is configured, the present invention is not limited thereto, and a larger number of hidden layers may be used as a matter of course.

As for the activity function, as shown in FIG. 7, ReLU (Rectified Linear Unit) was applied. The ReLU activation function is effective in solving the Vanish Gradient problem. And when the neural network was initialized, Xavier initialization was applied. The sum of the similar image roughness cost function and the spatial noise cost function was used as the final cost function.

8 is a flowchart illustrating a scene-based non-uniformity correction method using a deep learning neural network according to another embodiment.

Referring to FIG. 8, in step 800, a deep learning neural network structure is constructed assuming an offset of the neural network output.

In step 802, image data to which non-uniformity correction is not applied is input.

In step 804, deep learning neural network training is performed.

In step 806, an assumed deep learning neural network output offset is obtained.

In step 808, the image data to which non-uniformity correction is not applied and the assumed offset obtained in step 806 are summed to obtain an assumed non-uniformity correction image.

In steps 810 and 812, a similar image roughness cost function and a spatial noise cost function are obtained from the assumed non-uniformity corrected image, respectively.

In step 814, a final cost function is obtained by summing the similar image roughness cost function and the spatial noise cost function.

The final cost function obtained in step 814 is provided as feedback information of deep learning neural network training in step 804.

After repeating steps 802 to 814 a predetermined number of times, the offset output in step 806 is obtained as the final offset in step 816.

In step 818, non-uniformity correction is performed using the final offset.

In an embodiment, images to which non-uniformity correction is not applied are input to the neural network after constructing a deep learning neural network structure assuming the neural network output as an offset. The neural network output is an assumed offset output, which is summed with the non-uniformity correction unapplied images used for input, and outputs an assumed non-uniformity correction applied image. The similar image roughness cost function and spatial noise cost function are calculated based on the image to which the non-uniformity correction is applied, and these are used as feedback of the deep learning neural network. When this learning process is repeatedly converged, a stabilized offset is finally obtained. This obtained offset is used as an offset when correcting non-uniformity.

An example of a scene-based non-uniformity correction result according to an embodiment is shown in FIG. 9. Using the non-uniformity correction unapplied image 501 data shown in FIG. 9A as an input of a deep learning neural network and using a deep learning neural network learning structure that assumes the neural network output as an offset, a similar image roughness cost function, and a spatial noise cost function. As a result of training the deep learning neural network, a finally generated offset 503 shown in FIG. 9C may be obtained. The obtained offset may be applied to an image to which the non-uniformity correction is not applied to obtain an image 502 to which the non-uniformity correction is applied as shown in FIG. 9B.

In the embodiment, “Tensorflow 180 for CPU” is used as a neural network library, “Python 35” is used for writing the code, “Intel i5-3470” is used for the CPU, 16GB of memory is applied, and an 80×60 14bit image is used. Dropout was not applied for learning of all nodes. Adam optimizer was applied as an optimization algorithm, and a total of 200 training was performed with a learning rate of 0001.

A scene-based non-uniformity correction learning process using a deep learning neural network according to an embodiment is as follows. As shown in FIG. 10, it can be confirmed that when the neural network is trained 200 times at a learning rate of 0.001, the cost function 601 converges to learn about 50 times. Referring to FIG. 11, a process 602 in which an offset is estimated and converged for each learning can be confirmed. The estimated non-uniformity correction offset image of the neural network output according to the learning stage is shown. From the first step of learning, it can be seen that the estimated offset is gradually completed. In Figure 10, it can be seen that the predicted offset image hardly changes from about 50 times as the final cost function converged at about 50 training steps.

Deep learning scene-based non-uniformity correction using the similar image roughness cost function and spatial noise cost function according to the embodiment showed a maximum image quality of 703 dB based on PSNR, and about 80 dB higher image quality than the existing improved IRLMS algorithm. This may vary depending on the characteristics of each frame or continuous characteristics of the uncorrected image, but it can be seen that the performance improved significantly compared to the existing improved IRLMS algorithm.

In an embodiment, an offset used for non-uniformity correction is finally obtained by using a deep learning neural network, a deep learning neural network learning structure that assumes the output of the neural network as an offset, a similar image roughness cost function, and a spatial noise cost function.

From this configuration, the nonuniformity correction offset can be finally obtained using a deep learning neural network learning structure that assumes the neural network output as an offset, and the offset can be derived from the neural network output using a similar image roughness cost function. The noise cost function can be used to derive the offset from the neural network output.

In addition, a final offset of higher quality and accuracy may be obtained by using a final cost function obtained by combining a similar image roughness cost function and a spatial noise cost function.

Conventional statistics-based, registration-based scene-based non-uniformity correction (SBNUC) requires at least 50-300 frames of non-uniformity-corrected uncorrected images, whereas in the embodiment 2 to 10 fewer frames It is also possible to obtain a high-quality non-uniformity correction offset in the water channel.

In the embodiment, the registration-based scene-based non-uniformity correction (SBNUC) is not affected by the rotation, enlargement, and reduction of the weak image, and the scene-based non-uniformity based on existing statistics (Statistics-based) and registration-based (Registration-based) There is almost no ghost image phenomenon compared to correction (SBNUC).

The embodiments may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery media.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily transformed into other specific forms without changing the technical spirit or essential features of the present invention. . Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (12)

As a method of training a deep learning neural network that outputs an offset for scene-based non-uniformity correction,
Outputting a first offset through a deep learning neural network into which an image to which non-uniformity correction is not applied is input;
Calculating a similar image roughness cost function and a spatial noise cost function from the image to which the non-uniformity correction was not applied and the first non-uniformity-corrected image obtained by summing the outputted first offset;
Obtaining a final cost function from the sum of the calculated similar image roughness cost function and spatial noise cost function; And
And training the deep neural network by providing the obtained final cost function to the deep learning neural network as feedback information. The method of claim 1,
The pseudo image roughness cost function is,
A deep learning neural network learning method, characterized in that the learning speed of the deep neural network is adjusted in proportion to the roughness of the image. The method of claim 2,
The pseudo image roughness cost function is,
In a similar image roughness function obtained by averaging the convolution operation results of the horizontal difference filter and the vertical difference filter on the first non-uniformity correction applied image, the result of the convolution operation of the horizontal difference filter and the vertical difference filter is squared. Deep learning neural network learning method, characterized in that. The method of claim 3,
The pseudo image roughness cost function is,
Deep learning neural network learning method, characterized in that calculated by the following equation.
[Equation]

(here,

Is the pseudo-image roughness, h1 is the horizontal difference filter, h2 is the vertical difference filter, I is the image, M is the number of horizontal pixels in the image, N is the number of vertical pixels in the image,

Is L1 norm, * is discrete convolution) The method of claim 1,
The spatial noise cost function is,
Deep learning neural network learning method, characterized in that calculated by the following equation.
[Equation]

(here,

Is the image mean,

Is the image spatial noise,

Is the image spatial noise cost function, I is the image, M is the number of horizontal pixels in the image, and N is the number of vertical pixels in the image) As a scene-based non-uniformity correction method using a deep learning neural network,
Outputting a first offset through a deep learning neural network into which an image to which non-uniformity correction is not applied is input;
Calculating a similar image roughness cost function and a spatial noise cost function from the image to which the non-uniformity correction was not applied and the first non-uniformity-corrected image obtained by summing the outputted first offset;
Obtaining a final cost function from the calculated similar image roughness cost function and spatial noise cost function;
Providing the obtained final cost function to the deep learning neural network as feedback information for training; And
And outputting a non-uniformly corrected corrected image by summing a final offset output from the learned deep learning neural network and an input original image. As a scene-based non-uniformity correction method using a deep learning neural network,
Inputting an original image into a deep learning neural network learned by the learning method according to any one of claims 1 to 5; And
And outputting a non-uniformly corrected corrected image by summing a final offset output from the learned deep learning neural network and the original image. A recording medium on which a program for executing the method according to any one of claims 1 to 6 on a computer is recorded. As a device that learns a deep learning neural network that outputs an offset for scene-based non-uniformity correction,
Memory; And
Including a processor,
The processor,
Similar image roughness cost function and spatial noise from the first non-uniformity-corrected image obtained by summing the non-uniformity-corrected image and the output first offset through the deep learning neural network into which the non-uniformity-corrected image is input Calculate a cost function, obtain a final cost function from the sum of the calculated similar image roughness cost function and spatial noise cost function, and provide the obtained final cost function as feedback information to the deep learning neural network to provide the deep neural network. Deep learning neural network learning device that trains the network. The method of claim 9,
The pseudo image roughness cost function is,
Deep learning neural network learning apparatus, characterized in that adjusting the learning speed of the deep neural network in proportion to the roughness of the image. The method of claim 10,
The pseudo image roughness cost function is,
In a similar image roughness function obtained by averaging the convolution operation results of the horizontal difference filter and the vertical difference filter on the first non-uniformity correction applied image, the result of the convolution operation of the horizontal difference filter and the vertical difference filter is squared. Deep learning neural network learning device, characterized in that. As a scene-based non-uniformity correction device using a deep learning neural network,
Memory; And
Including a processor,
The processor,
Similar image roughness cost function and spatial noise from the first non-uniformity-corrected image obtained by summing the non-uniformity-corrected image and the output first offset through the deep learning neural network into which the non-uniformity-corrected image is input Calculate a cost function, obtain a final cost function from the calculated similar image roughness cost function and spatial noise cost function, provide the obtained final cost function as feedback information to the deep learning neural network to learn, and the learning Scene-based non-uniformity correction device that outputs a non-uniformly corrected corrected image by summing the final offset output from the deep learning neural network and the input original image.

Images