KR101854354B1 - Method for Image correction based on using multi-sensor - Google Patents

Abstract

The present invention relates to a multi-sensor based image correcting method capable of improving matching precision. The method comprises the following steps: performing non-uniformity correction; estimating image movement and magnification variations of a current frame compared with a previous frame by using a multi-sensor; correcting the previous frame based on the estimated image movement and magnification variations; performing image conversion for the corrected previous frame based on a current frame image, and matching the same with the current frame; and updating a corrected table.

Description

[0001] The present invention relates to a multi-sensor based image correction method,

The present invention relates to a multi-sensor-based image correction method, and more particularly, to an image correction method capable of performing image correction using a multi-sensor for a non-uniform image whose degree of unevenness of the image is equal to or higher than a threshold value.

As a general image correction method, calibration based nonuniformity correction (CBNUC) is a method of acquiring and applying a gain and an offset using a correction device such as a blackbody, and using a black body having two different temperatures, The two-point NUC, which corrects the gain and offset of the focal plane array, is mainly used

 Conventional matching scene-based nonuniformity correction (SBNUC) is based on image information to perform matching between frames, using algorithm such as simple image matching repetition, row and column projection matching, and phase correlation registration Matching was performed after motion compensation.

Conventional calibration-based nonuniformity correction (CBNUC) is easier to implement than image-based nonuniformity correction (SBNUC) and has the advantage of obtaining accurate calibration tables. However, it can not cope with changes in the surrounding environment outside the home, and periodic non-uniformity correction should be performed due to the FPA change occurring after a long time after the non-uniformity correction. In addition, there is a disadvantage that it is difficult to cope with the temperature drift phenomenon generated in the FPA after power application. To overcome this, shutter correction is performed. However, in this case, since the shutter periodically covers the image, continuous image acquisition is not possible.

 Conventional matching scene based nonuniformity correction (SBNUC) uses an image based algorithm to perform matching between frames. However, the motion estimation for matching with only image information requires a lot of system resources, which leads to problems such as an increase in system manufacturing cost and an increase in manufacturing difficulty.

In addition, when the video signal is smaller than the signal generated due to the non-uniformity of the image sensor, there is a disadvantage that it is impossible to perform the matching-based non-uniformity correction.

Korean Patent Laid-Open No. 10-2016-0060747 describes a multiple sensor image matching apparatus. However, this technology merely describes a technique for distinguishing a target when the target and the ambient temperature are similar, which is advantageous to the nighttime environment by matching the CCD image and the IR image, and a solution to the uneven image due to the non- .

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image correction method using multiple sensors for an image having a degree of unevenness of the image equal to or higher than a threshold value.

It is another object of the present invention to provide an image correction method capable of improving registration accuracy by performing image registration based on multiple sensors.

It is another object of the present invention to provide an image correction method capable of enhancing the matching precision and optimizing the speed of execution of the matching-based nonuniformity correction by using multiple sensors.

However, the objects of the present invention are not limited to those mentioned above, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method of compensating for non-uniformity of a current frame and a previous frame of a current frame, And correcting the previous frame based on the estimated image movement and magnification change amount; and a step of performing image conversion on the corrected previous frame with reference to a current frame image and matching the current frame with the current frame image And a step of updating the correction table using a signal difference error based on an image difference value between the previous frame and the current frame after the matching.

The multi-sensor-based image correction method according to the embodiment of the present invention converts the degree of unevenness of the image into the roughness value of the image, and if the roughness value is equal to or more than the threshold value, , It is possible to perform the matching-based non-uniformity correction based on the zoom sensor and the gyro sensor information even when the matching is impossible.

In addition, since the present invention performs image matching based on multiple sensors, it is possible to improve the matching precision and optimize the speed of execution of the matching-based non-uniformity correction.

1 is a block diagram showing a schematic configuration of an image correction system that selectively uses multiple sensors according to an embodiment of the present invention.
2 is a block diagram showing a schematic configuration of an image correction apparatus that selectively uses multiple sensors according to an embodiment of the present invention.
3 is a block diagram showing a detailed configuration of a video processor according to an embodiment of the present invention.
4 is a block diagram showing a schematic configuration of a multi-sensor-based correction table update unit according to an embodiment of the present invention.
FIG. 5 illustrates a horizontal difference filter and a vertical difference filter used for determining a degree of unevenness of an image according to an embodiment of the present invention.
6 is photograph data for explaining the image roughness.
7 is a diagram illustrating an algorithm for explaining a matching scene-based non-uniformity correction method according to an embodiment of the present invention.
FIG. 8 shows a signal flow diagram of a matching scene-based non-uniformity correction optical system using multiple sensors according to an embodiment of the present invention.
9 is a flowchart illustrating a frame movement amount estimation method using a gyro sensor according to an embodiment of the present invention.
10 is a diagram illustrating an example of generating a signal difference error by estimating a movement direction and a movement amount of a frame using a gyro sensor according to an embodiment of the present invention.
11 is a flowchart for explaining a method of estimating the amount of enlargement / reduction of a frame, that is, magnification variation using a zoom sensor according to an embodiment of the present invention.
12 is an exemplary diagram for explaining a method of generating a signal difference error by estimating a magnification change amount of a frame using a zoom sensor according to an embodiment of the present invention.
13 is a schematic flowchart for explaining an image correction method based on multiple sensors according to an embodiment of the present invention.
14 is a schematic flowchart for explaining a method of selectively correcting an image using multiple sensors according to the degree of image unevenness according to another embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the preferred embodiments of the present invention will be described below, but it is needless to say that the technical idea of the present invention is not limited thereto and can be variously modified by those skilled in the art.

FIG. 1 is a block diagram illustrating a schematic configuration of an image correction system that selectively uses multiple sensors according to an exemplary embodiment of the present invention. FIG. 2 is a block diagram of an image correction system using selectively multiple sensors according to an exemplary embodiment of the present invention. 3 is a block diagram showing a detailed configuration of an image processor according to an embodiment of the present invention.

Referring to FIG. 1, an image correction system 100 that selectively uses multiple sensors according to an exemplary embodiment of the present invention includes a zoom optical system 120 and an image correction device 110. 2, the image correction apparatus 110 includes an image sensor 130, a zoom sensor 140, a gyro sensor 150, and an image processor 170. [

The image correction system 100 condenses the light energy from the outside to the image sensor 130 through the zoom optical system 120, converts the image into an electric signal, and transmits the electric signal to the image processor 170. The image processor 170 processes the image signal and outputs the corrected image to the outside of the image correction system 100.

When the image signal output from the image sensor 130 is directly output to the outside, the non-uniform fixed pattern noise generated in the process of manufacturing the image sensor is directly output to the outside. In this case, the image quality is poor and it is difficult to distinguish the subject properly. In order to solve this problem, in order to correct the non-uniformity of the image sensor 130, the image processor 170 performs a non-uniformity correction process.

The non-uniformity correction method for the image sensor 130 is roughly classified into calibration-based nonuniformity correction and scene-based nonuniformity correction. The calibration-based nonuniformity correction is a method of correcting an image using a reference input such as a black body or an integral sphere, and the scene-based nonuniformity correction is a method of correcting an uncorrected image input to the image correction system 100 using an image processing process to be.

The scene-based non-uniformity correction method is again a statistical method and a matching method. The present invention discloses an improvement of an optical system using scene-based non-uniformity correction using registration.

In particular, in the present invention, if the degree of unevenness of the image is converted into the roughness value of the image, and if the roughness value is equal to or higher than the threshold value, the conventional image processing based image matching is impossible. Therefore, Based non-uniformity correction can be performed based on the zoom sensor and the gyro sensor information.

The image processor 170 includes an image acquisition unit 171, an image correction unit 172, a corrected image output unit 173, an image unevenness determination unit 174, and a selective correction table update unit 175.

The image acquiring unit 171 acquires an image and divides it into a current frame and a previous frame. The image correction unit 172 performs correction based on the current correction table for the current frame and the previous frame. The correction table is updated with respect to the frame of the input image every time. The method of updating the correction table will be described later.

The image non-uniformity determination unit 174 determines the degree of non-uniformity of the image based on the threshold value to selectively use multiple sensors in the process of correcting the image.

 This is because frame motion estimation using image information is almost impossible when the image sensor 130 is unevenly distributed. For this reason, the image non-uniformity determination unit 174 determines the degree of roughness of the image and determines whether to perform image correction based on image processing or image correction based on multiple sensors. When the image sensor 130 has a large unevenness, the image roughness value is proportionally large.

FIG. 5 illustrates a horizontal difference filter and a vertical difference filter used to determine a degree of image unevenness according to an exemplary embodiment of the present invention, and FIG. 6 illustrates photographs illustrating image roughness.

Specifically, the image unevenness determiner 174 calculates the roughness of the image using the ratio of the sum of the result of applying the horizontal difference filter and the vertical difference filter to the original image as shown in FIG. 5 for one image. The roughness of the image can be calculated as shown in Equation (1).

[Equation 1]

는 영상 거칠기, h₁는 수평차이필터, h₂는 수직차이필터, I는 영상, *는 discrete convolution을 나타낸다.here,

H ₁ is the horizontal difference filter, h ₂ is the vertical difference filter, I is the image, and * denotes the discrete convolution.

As shown in the photograph of FIG. 6, since the image roughness value is higher than the threshold value, a coarse image is displayed because of a large change in brightness between adjacent pixels. When the image roughness value is lower than the threshold value,

The image unevenness determiner 174 compares the roughness values of the image thus calculated on the basis of the threshold values, classifies the roughness values into a first image when the roughness value is less than the threshold value, and classifies the second image if the roughness value is greater than the threshold value .

In the present invention, if the unevenness of the image sensor 130 is excessive, the image correction method based on the multi-sensor can be executed. do.

On the other hand, when the image unevenness due to the image sensor is less than the threshold value, a multi-sensor based image correction method can be used according to the user's selection or designer's setting by using the image processing based image correction technique of the general technology.

The selective correction table update unit 175 performs image change amount estimation based on image processing based or multi-sensor based on the degree of unevenness of the image determined by the image unevenness determination unit 174 described above, and based on the estimated image change amount Update the calibration table.

3, the selective correction table update unit 175 estimates an image change amount based on an image processing based on an operation of comparing a previous frame and a current frame with respect to the first image, Based correction table update unit 180 that updates the correction table. Here, since the image processing-based correction table update unit 180 is the same technology as the conventional image processing technology, a detailed description will be omitted.

The selective correction table update unit 175 estimates the amount of image change based on the multiple sensor based on the second image, that is, the image having the image unevenness due to the image sensor being equal to or more than the threshold value, Sensor-based correction table update unit 190 that updates the multi-sensor-based correction table.

4, the multi-sensor-based correction table update unit 190 includes an image change amount estimation unit 191, a previous frame correction unit 192, a matching unit 193, a signal difference error generation unit 194 And a correction table updating unit 195. The correction table updating unit 195 may be configured to update the correction table.

The image change amount estimating unit 191 uses the gyro sensor A to estimate the movement amount and magnification change amount of the frame based on the multiple sensors and uses the gyro sensor A to estimate the movement change amount of the current frame with respect to the previous frame, The zoom sensor B is used to estimate the magnification variation of the frame.

Specifically, the image change amount estimating unit 191 accumulates the yaw axis angular velocity output and the pitch axis angular velocity output based on the image sensor frame rate using a gyro sensor to obtain the yaw axis angular movement amount and the pitch axis angular movement amount.

The image change amount estimating unit 191 converts the yaw axis angular movement amount and the pitch axis angular movement amount into the horizontal axis image movement amount and the vertical axis image movement amount based on the zoom optical system information and the image sensor information.

Subsequently, the image change amount estimation unit 191 estimates the image shift amount of the current frame with respect to the previous frame by performing the movement amount conversion of the previous frame using the estimated horizontal axis image movement amount and the vertical axis image movement amount.

Then, the signal difference error generator 194 estimates the image movement amount by combining the angular velocity information obtained from the gyro sensor, the frame rate of the image sensor, the image sensor information, and the zoom optical system information. Then, the estimated moving image amount is applied to the previous frame to generate a previous frame in which the image movement amount is corrected, and the difference between the previous frame and the current frame in which the image movement amount is corrected is calculated to generate the signal difference error.

On the other hand, the image change amount estimating unit 191 grasps the zoom state of the current zooming optical system through the encoder information of the zooming sensor, and uses the magnification information of the zooming optical system and the focal length information of the zooming optical system, (Focal plane array) plane of the image. Then, the image change amount estimating unit 191 estimates the image magnification of the previous frame based on the actual frame image using the pixel pitch of the image sensor and the number of pixels of the image sensor in the horizontal and vertical directions, And estimates the magnification variation of the frame.

Then, the signal difference error generating unit 194 obtains the previous frame zoomed in and the current frame reduced in zoom from the image sensor, and obtains the magnification and focal length information of the zoom sensor and the zoom optical system, Obtain the zoom magnification.

Then, the signal difference error generating unit 194 estimates the image magnification of the previous frame with respect to the current frame by combining the obtained zoom magnification with the pixel pitch of the image sensor and the information of the horizontal and vertical pixels.

Then, the signal difference error generating unit 194 obtains a previous frame in which the image magnification is corrected by modifying the previous frame based on the estimated image enlargement / reduction amount, and outputs the difference between the previous frame and the current frame, The difference is then sought to produce a signal difference error.

The correction table update unit 195 updates the offset and gain of the correction table using the signal difference error and transmits the updated offset and gain to the image correction unit 172.

Then, the correction table update unit 195 repeats the update of the correction table for the frame of the input image every time. Such a process is repeated to gradually update the correction table, thereby obtaining a correction table that enables the final high-quality nonuniform correction to be performed.

The corrected video output unit 173 externally outputs the current frame corrected by the video correction unit as a corrected video.

7 is a diagram illustrating an algorithm for explaining a matching scene-based non-uniformity correction method according to an embodiment of the present invention.

Referring to FIG. 7, a matching scene-based non-uniformity correction method according to an embodiment of the present invention stores an original image S201 output from an image sensor 130 in a buffer (S202) And performs non-uniformity correction for correcting the gain and the offset based on the correction table information (S203).

The corrected current frame is outputted to the outside as a corrected image (S207).

The previous frame and the current frame, which have been calibrated, are estimated based on the frame movement amount and the amount of expansion / reduction (magnification variation) based on the current frame image, and then the estimated frame movement amount and magnification variation amount are applied to the previous frame, The image is converted based on the current frame, and is matched with the current frame (S204).

After performing the matching, the difference between the corrected current frame and the previous frame is obtained (S205) and used as a signal difference error. The variable update is executed based on this signal difference error (S206). At this time, the gain and the offset of the correction table used in the nonuniform correction are updated.

The updating of the correction table is repeated in correspondence with the frame input every time this process is performed to finally obtain a stable correction table.

FIG. 8 shows a signal flow diagram of a matching scene-based non-uniformity correction optical system using multiple sensors according to an embodiment of the present invention.

Referring to FIG. 8, a signal flow of a matching scene-based non-uniformity correction optical system using multiple sensors is as follows. The image signal S301 obtained by the image sensor is stored in the buffer after receiving the image signal S302 in the image processor 170 in step S303 and if the buffer is divided into the current frame and the previous frame in steps S304 and S305, Non-uniformity correction is performed on the basis of the same current correction table for each frame (S306).

The corrected current frame is output to the outside as a corrected image (S311).

Thereafter, the corrected previous frame is corrected for the movement amount and magnification change amount of the previous frame (S307). However, in the present invention, the zoom sensor information of the zoom optical system (S312) is processed (S313), the zoom amount is estimated (S314), and the gyro sensor information is processed (S315), the rotation speed and the rotation direction of the optical system are estimated (S316). Based on the information of the image sensor specification (S317) and the zoom optical system specification (S318) The variation is estimated (S319).

The previous frame corrected based on the image change amount estimation information is converted into an image based on the current frame image, and the previous frame and the current frame are matched (S308). After the matching, a signal difference error is generated using the image difference value between the previous frame and the current frame (S309). The correction table update is performed based on the signal difference error (S310). By repeating the above process and gradually updating the correction table, it is possible to obtain a correction table capable of finally performing high quality nonuniform correction.

FIG. 9 is a flowchart illustrating a frame movement amount estimation method using a gyro sensor according to an embodiment of the present invention. FIG. 10 is a flowchart illustrating a method of estimating a frame movement amount using a gyro sensor according to an embodiment of the present invention. And generates a difference error.

Referring to FIGS. 9 and 10, a method of estimating a frame movement amount using a gyro sensor includes the steps of: outputting a gyro yaw axis angular velocity output S401 and a gyro pitch axis angular velocity output S402 based on a frame rate of the image sensor 130; (Step S405), the gyro yaw axis angular movement amount is obtained (S403), and the gyro pitch angular movement amount is obtained (S404). The obtained angular displacement is converted into an image horizontal axis movement amount based on the zoom optical system information S408 and the image sensor information S409 (S406), and converted into the image vertical axis movement amount (S407). Then, the movement amount conversion of the previous frame is performed using the estimated horizontal axis and vertical axis frame movement amounts.

A method of generating a signal difference error by estimating a frame movement amount using a gyro sensor is as follows. The previous frame and the current frame are acquired from the image sensor 130 (S501, S502). The frame movement amount is estimated by combining the angular velocity information obtained from the gyro sensor, the frame rate of the image sensor 130, the specification information of the image sensor, and the specification information of the zoom optical system, and then applied to the previous frame to generate a previous frame (S503). The difference between the previous frame and the current frame is corrected (S504), and a signal difference error is obtained (S505).

FIG. 11 is a flowchart for explaining a method of estimating an amount of enlargement / reduction of a frame, that is, a magnification variation using a zoom sensor according to an embodiment of the present invention. FIG. And a signal difference error is generated.

Referring to FIGS. 11 and 12, a frame enlargement / reduction amount estimation method using a zoom sensor is as follows. The zoom state of the current zoom optical system is grasped by using the encoder information for the zoom sensor of the zoom optical system (S601). After the zoom state is grasped, the magnification of the image on the FPA plane of the image sensor is estimated in real time (S602, S603) using the zoom optical system magnification information and the zoom optical system focal length information. (S604 and S605) using the pixel pitch of the image sensor and the horizontal and vertical pixels of the image sensor to estimate how much the image is enlarged or reduced with respect to the previous frame based on the actual frame image (S606 ).

12, an example of a process of generating a signal difference error by estimating an amount of enlargement / reduction of a frame using a zoom sensor is as follows. The previous frame zoomed in and the current frame zoomed in are obtained from the image sensor (S701, S702).

The zoom magnification of the current frame with respect to the previous frame is obtained from the zoom sensor and the zoom optical system magnification and the focal length information, and the pixel pitch of the image sensor and the information of the number of pixels in the horizontal and vertical directions are combined to estimate the amount of enlargement / The previous frame in which the image enlargement / reduction amount is corrected is obtained by changing the previous frame (S703). A difference between the current frame and the current frame is obtained (S704), and a signal difference error is obtained (S705).

13 is a schematic flowchart for explaining an image correction method based on multiple sensors according to an embodiment of the present invention. The detailed description of the same configurations and functions as those described above with reference to Figs. 1 to 12 will be omitted since they are duplicated.

Referring to FIG. 13, in the multi-sensor based image correction method according to the embodiment of the present invention, an image is acquired in an image acquisition unit (S110), and a current frame and a previous frame Non-uniformity correction is performed based on the correction table (S120).

Next, the image change estimating unit estimates the image shift and magnification change amount of the current frame with respect to the previous frame using multiple sensors (S130).

Next, the previous frame correcting unit corrects the previous frame based on the estimated image shift and magnification change amount (S140).

Next, the previous frame corrected by the image matching unit is converted into an image based on the current frame image and matched with the current frame (S150).

Next, the correction table updating unit updates the correction table using the signal difference error based on the image difference value between the previous frame and the current frame after the matching (S160).

14 is a schematic flowchart for explaining a method of selectively correcting an image using multiple sensors according to the degree of image unevenness according to another embodiment of the present invention.

Referring to FIG. 14, in order to selectively use multiple sensors according to the degree of image unevenness according to another embodiment of the present invention, after the step of S130 in FIG. 13, the image non- (S210).

 If the roughness value is less than the threshold value, the roughness value is classified as the first image. If the roughness value is greater than the threshold value, the roughness value is classified as the second image.

In the case of unevenness of the image sensor, it is impossible to match the image correction method based on the conventional image processing. Therefore, in the present invention, the image correction method based on the multi-sensor is executed when the unevenness of the image sensor is equal to or more than the threshold value.

On the other hand, when the unevenness of the image due to the image sensor is less than the threshold value, the image processing based image correction technique of the general technology (S225) may be used or a multi-sensor based image correction method may be used according to the user's selection or designer's setting .

In step S240, the image variation is estimated based on the multi-sensor for the second image, that is, the image having the image unevenness due to the image sensor is equal to or more than the threshold value, and the correction table is updated based on the estimated image variation amount in step S240.

That is, in order to estimate the movement amount and magnification change amount of the frame based on the multiple sensors, the gyro sensor A is used to estimate the movement change amount of the current frame with respect to the previous frame, The zoom sensor B is used.

Specifically, the yaw axis angular velocity and the pitch axis angular velocity are obtained by accumulating the yaw axis angular velocity output and the pitch axis angular velocity output based on the image sensor frame rate using a gyro sensor. Then, the yaw axis angular movement amount and the pitch axis angular movement amount are converted into the horizontal axis image movement amount and the vertical axis image movement amount based on the zoom optical system information and the image sensor information. Subsequently, the amount of image movement change of the current frame is estimated by performing the movement amount conversion of the previous frame using the estimated horizontal axis image movement amount and vertical axis image movement amount.

Then, the image movement amount is estimated by combining the angular velocity information obtained from the gyro sensor, the frame rate of the image sensor, the image sensor information, and the zoom optical system information. Then, the estimated moving image amount is applied to the previous frame to generate a previous frame in which the image movement amount is corrected, and the difference between the previous frame and the current frame in which the image movement amount is corrected is calculated to generate the signal difference error.

In addition, it is possible to grasp the zoom state of the current zoom optical system through the encoder information of the zoom sensor, to register the focal plane array (FPA) of the image sensor using the magnification information of the zoom optical system and the focal length information of the zoom optical system The image magnification is estimated in real time. Then, the magnification variation of the current frame with respect to the previous frame is estimated by estimating the magnification of the previous frame with reference to the actual frame image using the pixel pitch of the image sensor and the number of pixels of the image sensor in the horizontal and vertical directions.

Then, a previous zoom magnified frame and a zoom reduced current frame are obtained from the image sensor, and the zoom magnification of the current frame is obtained from the magnification and focal length information of the zoom sensor and the zoom optical system with respect to the previous frame.

Then, the obtained zoom magnification is combined with the pixel pitch of the image sensor and the horizontal and vertical pixel count information to estimate the image magnification of the previous frame with respect to the current frame.

Then, based on the estimated image enlargement / reduction amount, a previous frame having the corrected image magnification is obtained by modifying the previous frame, and a signal difference error is generated by obtaining the difference between the previous frame and the current frame in which the image magnification is corrected do.

Then, by using the signal difference error, the offset and gain of the correction table are updated and the image correction is executed (S130).

At this time, the update execution of the correction table is repeated for the frame of the input image every time. Such a process is repeated to gradually update the correction table, thereby obtaining a correction table that enables the final high-quality nonuniform correction to be performed. The corrected image frame is output to the outside as a corrected image.

As described above, the image correction apparatus that selectively uses multiple sensors according to the embodiment of the present invention converts the degree of unevenness of the image into the roughness value of the image, and if the roughness value is equal to or larger than the threshold value, Therefore, by performing image correction using multiple sensors selectively, registration-based non-uniformity correction can be performed based on the zoom sensor and gyro sensor information even when matching is impossible.

In addition, since the present invention performs image matching based on multiple sensors, it is possible to improve the matching precision and optimize the speed of execution of the matching-based non-uniformity correction.

It is to be understood that the present invention is not limited to these embodiments, and all elements constituting the embodiment of the present invention described above are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer to implement an embodiment of the present invention. As the recording medium of the computer program, a magnetic recording medium, an optical recording medium, or the like can be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Commonly used terms, such as predefined terms, should be interpreted to be consistent with the contextual meanings of the related art, and are not to be construed as ideal or overly formal, unless expressly defined to the contrary.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate and not to limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (9)

Performing non-uniformity correction on the basis of a correction table for a current frame and a previous frame of the image;
Estimating an image shift and magnification change amount of a current frame with respect to the previous frame using multiple sensors;
Correcting the previous frame based on the estimated image movement and magnification change amount;
Converting the corrected previous frame to a current frame image and matching the current frame with the current frame image; And
And updating the correction table using a signal difference error due to an image difference value between the previous frame and the current frame after the matching,
Wherein the step of estimating the image movement variation of the current frame with respect to the previous frame using the multiple sensors comprises:
Accumulating the yaw axis angular velocity output and the pitch axis angular velocity output based on the image sensor frame rate using a gyro sensor to obtain a yaw axis angular movement amount and a pitch axis angular movement amount;
Converting the yaw axis angular movement amount and the pitch axis angular movement amount into a horizontal axis image movement amount and a vertical axis image movement amount based on the zoom optical system information and the image sensor information; And
And performing a movement amount conversion of the previous frame using the estimated horizontal axis image movement amount and vertical axis image movement amount. delete The method according to claim 1,
Wherein the step of using the signal difference error comprises:
Estimating an image movement amount by combining angular velocity information obtained from a gyro sensor and frame rate, image sensor information, and zoom optical system information of the image sensor;
Generating a previous frame in which the image movement amount is corrected by applying the estimated image movement amount to a previous frame; And
And generating the signal difference error by obtaining a difference between a previous frame and a current frame in which the image movement amount is corrected, and generating the signal difference error. The method according to claim 1,
Estimating a magnification change amount of a current frame with respect to the previous frame using the multiple sensors,
Grasping the zoom state of the current zoom optical system through the encoder information of the zoom sensor;
Estimating an image magnification of a focal plane array (FPA) of the image sensor in real time using magnification information of the zoom optical system and focal length information of the zoom optical system; And
And estimating an image magnification of the previous frame based on the actual frame image using the pixel pitch of the image sensor and the number of pixels of the image sensor in the horizontal and vertical directions. 5. The method of claim 4,
Wherein generating the signal difference error comprises:
Obtaining a previous frame zoomed in and a current frame zoomed in from the image sensor;
Acquiring a zoom magnification of the current frame with respect to the previous frame using magnification and focal length information of the zoom sensor and the zoom optical system;
Estimating an image magnification of the previous frame with respect to the current frame by combining the obtained zoom magnification with pixel pitch of the image sensor and information of horizontal and vertical pixel numbers;
Obtaining a previous frame in which the image magnification is corrected by modifying the previous frame based on the estimated image enlargement / reduction amount; And
And generating the signal difference error by obtaining a difference between the current frame and a previous frame in which the image magnification is corrected. The method according to claim 1,
Wherein the step of generating the signal difference error for each frame of the input image and the step of updating the correction table are repeated. The method according to claim 1,
Wherein updating the correction table comprises:
And the gain and offset of the correction table are updated using the signal difference error. The method according to claim 1,
After performing the non-uniformity correction on the basis of the correction table for the current frame and the previous frame of the image,
And outputting the corrected current frame to the outside as a corrected image. A computer-readable recording medium storing a program for causing a computer to execute the method according to any one of claims 1 to 9.

Images