KR102083926B1 - Image fusion system and image fusion method - Google Patents

Abstract

The present invention relates to an image fusion system and an image fusion method.
The image fusion system of the present invention divides each of the visible image and the thermal image into a plurality of local regions, calculates weights for each local region by using pixel values average and standard deviation of each local region, and calculates the weights for each local region. A weight determination unit determining weights of pixels of the visible image and the thermal image based on the weight; And a post-processing unit generating a fusion image in which the visible image and the thermal image are fused by applying the weights.

Description

Image fusion system and its method {IMAGE FUSION SYSTEM AND IMAGE FUSION METHOD}

The present invention relates to an image fusion system and an image fusion method.

Visible light cameras for general surveillance systems, such as CCD cameras, have reached a technically high level compared to other cameras and have a wide range of applications. However, since the CCD camera can implement the monitoring function only in the presence of light and it is impossible to implement the monitoring function in the absence of light, it is difficult to use the CCD camera in a surveillance system that does not use a separate light. Infrared thermal cameras can be used for surveillance in low light environments. Infrared thermal imaging cameras are used in various areas related to heat dissipation such as detection of heat loss of a subject and analysis of a circuit board as well as realizing a surveillance function in the absence of lighting.

However, since thermal imaging cameras are equipment for the purpose of detecting heat emitted from a subject, that is, infrared rays, performance of the thermal imaging camera is lower than that of a conventional surveillance camera using a CCD in terms of image quality and resolution. That is, the infrared thermal camera alone may be suitable for detecting heat emission characteristics of the subject, but not suitable for distinguishing the detailed form of the subject.

Accordingly, by combining the visible image obtained from the visible light camera and the thermal image obtained from the thermal camera, the development of an algorithm that combines the advantages of the two images and expresses them as one image is being actively made.

The present invention provides an image fusion system capable of generating a fusion image that properly reflects information of a visible image and a thermal image.

In an image fusion system according to an embodiment of the present invention, a visible image and a thermal image are each divided into a plurality of local regions, the weight of each local region is calculated by using a mean and standard deviation of pixel values of each local region, and A weight determination unit configured to determine pixel weights of the visible image and the thermal image based on weights of local regions; And a post-processing unit generating the fusion image by fusing the visible image and the thermal image by applying the weight.

The weight determiner may correct a deviation between the weights of the regions.

The system may further include a fusion region detector for detecting a fusion region from the visible image and the thermal image.

The system may further include a user input unit configured to receive pixel weights of the visible image and the thermal image of the user designated area and the user specified area from a user.

In an image fusion method according to an embodiment of the present invention, each of a visible image and a thermal image is divided into a plurality of local regions, a weight for each local region is calculated by using a mean and standard deviation of pixel values of each local region, Determining pixel weights of the visible image and the thermal image based on weights of local regions; And generating a fusion image in which the visible image and the thermal image are fused by applying the weights.

The method may further include receiving a user-designated region and pixel weights of the user-designated region from a user, and changing fusion information of the user-designated region in the fusion image.

According to an embodiment of the present invention, it is possible to generate a converged image that more effectively reflects visible and thermal information.

1 is a block diagram schematically illustrating an image fusion system according to an embodiment of the present invention.
FIG. 2 is a block diagram schematically illustrating the image fusion unit shown in FIG. 1.
3 is a diagram illustrating region division for determining a weight according to an embodiment of the present invention.
4 is a diagram illustrating a user designation method according to an embodiment of the present invention.
5 is a flowchart schematically illustrating an image fusion method according to an embodiment of the present invention.
6 is an exemplary view illustrating a fusion image according to an image fusion according to an embodiment of the present invention and a fusion image according to a comparative example.

Hereinafter, with reference to the accompanying drawings will be described in detail the present invention.

1 is a block diagram schematically illustrating an image fusion system according to an embodiment of the present invention. FIG. 2 is a block diagram schematically illustrating the image fusion unit shown in FIG. 1. 3 is a diagram illustrating region division for determining a weight according to an embodiment of the present invention.

Referring to FIG. 1, the image fusion system 1 of the present invention includes a first image sensor 10, a second image sensor 20, an image matching unit 30, an image fusion unit 40, and a display unit 50. ) And a user input unit 60.

The first image sensor 10 and the second image sensor 20 may be cameras having different characteristics that provide image information by photographing the same scene. The first image sensor 10 and the second image sensor 20 may have a pan tilt zoom (PTZ) function, and may be panned and tilted together to acquire images of the same point at each zoom magnification. The first image sensor 10 and the second image sensor 20 are installed in and out of an office, a house, a hospital, as well as a bank or a public building requiring security, and are used for access control or crime prevention. Depending on the location and purpose of use, it can have various shapes such as straight and dome shaped.

In the present embodiment, the first image sensor 10 is a visible light camera, and acquires image information by detecting light to generate a first image, which is a visible image according to a luminance distribution of an object. For example, the visible camera may be a camera using a CCD or a CMOS as an image pickup device. Visible cameras have the advantage of being most intuitive to understand because they obtain information in the visible wavelength range. However, it is difficult to monitor the night environment or detect hidden objects due to the weakness of the information in the low light environment and the inability to recognize hidden objects such as humans hidden by grasses and shadows that are invisible to the human eye. have.

In the present embodiment, the second image sensor 20 is an infrared camera (or a thermal camera), which detects radiant energy (thermal energy) emitted by an object, detects it in the form of infrared wavelength, which is a kind of electromagnetic waves, and measures the intensity of thermal energy. The second image may be a thermal image having different colors according to the intensity. The thermal image may be expressed by mapping an object surface to different colors according to the temperature of the object. Since the thermal image obtained from the thermal camera represents the scene using the heat information of the object, the thermal image is not affected by the lighting environment, and the hidden object may also be recognized.

The image matching unit 30 performs image registration for matching the positional relationship of two or more images obtained from different sensors with the same scene in one coordinate system. In the field of surveillance system and medical imaging, image matching using two or more sensors to generate a single image is required.

The image matching unit 30 registers the first image photographed by the first image sensor 10 and the second image photographed by the second image sensor 20. To this end, the image matching unit 30 may estimate the conversion function based on the feature point information extracted from the first image and the second image. The transform function is a matrix representing a correspondence relationship between feature point information of each of the first and second images. When the feature point information is not extracted, the image matching unit 30 may extract motion information from the first image and the second image, and estimate a conversion function based on the extracted motion information. When the zoom change of at least one of the first image sensor 10 and the second image sensor 20 is detected, the image matching unit 30 may estimate the conversion function again.

The image matching unit 30 matches the first image and the second image by applying the estimated transform function. The image matching unit 30 applies a temporary conversion function reflecting the zoom information of the first image sensor 10 and the second image sensor 20 to the estimated conversion function while estimating the conversion function again by the zoom change. By matching the first image and the second image.

The image fusion unit 40 fuses the matched first image and the second image. For example, an infrared camera may well indicate the thermal distribution of an object, but the shape of the measured object is not clear, and a visible light camera may clearly indicate the shape of the object but not the thermal distribution of the object. The image fusion unit 40 generates a fusion image expressing optimal information by displaying an image of an object and clearly displaying a thermal distribution state of the object by appropriately using mutual strengths and weaknesses of a visible light camera and an infrared light camera. Can be. The image fusion unit 40 performs signal processing to output the received image signal as a signal conforming to the display standard.

2, the image fusion unit 40 may include a preprocessor 401, a fusion region detector 403, a weight determiner 405, and a post processor 407.

The preprocessor 401 performs pre-processing for image processing such as normalization on the first image and the second image. The preprocessor 401 may reduce noise with respect to the first image and the second image. The preprocessor 401 performs gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement on the first image, which is a visible image. image signal processing for image quality improvement such as enhancement). The preprocessor 401 may perform signal processing to emphasize a region where the intensity of the thermal energy is relatively high and to emphasize the region where the intensity of the thermal energy is relatively low with respect to the second image which is the thermal image.

The fusion region detector 403 may detect a fusion region with respect to the preprocessed first and second images. The fusion region detector 403 may detect a predetermined region centered on the entire image or a portion of the image, for example, a motion detection region, a preset ROI, or the like as the fusion region. Also, the fusion region detector 403 may detect a user-designated region designated by the user as the fusion region.

The weight determiner 405 may determine a weight for each pixel in the fusion region of the first image and the second image. To this end, the weight determiner 405 may divide the first image and the second image into one or more local areas. The weight determiner 405 may determine the size of the local area according to the photographing environment, and the detailed expression power and the amount of calculation may be determined according to the local area size.

FIG. 3 illustrates an example of dividing each of the first image (a) and the second image (b) having an M × N size into a plurality of local regions including ixj pixels. The weight (w_CCD_xy) of the pixels in each of the local regions c11, c12, ..., c21, c22, ..., cmn of the first image (a) is equal to the local regions c11, c12, It can be calculated using the mean (m_CCD_mn) and standard deviation (std_CCD_mn) of ..., c21, c22, ..., cmn. The weights w_IR_xy of the pixels in the local regions i11, i12,..., I21, i22,..., And imn of the second image b are equal to each local region i11, i12, It can be calculated using the mean (m_IR_mn) and standard deviation (std_IR_mn) of ..., i21, i22, ..., imn). Here, a, b, c, and d are coefficients for adjusting the degree of reflection of the mean and standard deviation of each local area of the first image and the second image. For example, by adjusting a, b, c, and d, the first image is the threshold (a = 1, b = 0) and the second image is the threshold (c = 0, d = 1). Can be calculated as a, b, c and d can be determined by experimentally optimized values according to the system or environment.

...(1)

...(One)

...(2)

...(2)

If the weight variation of each local region is large, the boundary of the local region may be reflected in the fusion image. Accordingly, the weight determiner 405 may apply a weight low pass filter (WLPF) to the pixel weights to correct the weight deviation of each local area. Here, the size of the mask MASK is set larger than that of the local area, and an optimized value may be set by an experiment. The corrected pixel weights w'_CCD_xy and w'_IR_xy are shown in Equation 3 below.

...(3)

... (3)

The weight determination unit 405 may first apply the user-specified weight to the user-designated areas of the first and second images when there is a user-specified weight input from the user.

The post-processing unit 407 may generate a fusion image F by fusion of the first image and the second image by applying respective pixel weights in the fusion region of the first image and the second image. The pixel value x_F of each pixel of the fused image is a weight (w'_CCD, w'_IR) corrected to the pixel value x_CCD of the first image and the pixel value x_IR of the second image, respectively, as shown in Equation 4. It can be calculated by applying.

...(4)

...(4)

In addition, the post processor 407 may generate an image file by compressing the data of the fusion image by processing an image signal for improving image quality, or may restore the image data from the image file. The compressed format of the image may include a reversible format or an irreversible format. In addition, the post processing unit 407 can also perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, and the like. Face recognition, scene recognition processing, and the like can be performed by the image recognition processing.

The display unit 50 provides the user with the fusion image output from the image fusion unit 40 so that the user can monitor the displayed image. The display apparatus 50 may display a fusion image in which the first image and the second image are overlapped. The display device 50 may be formed of a liquid crystal display panel (LCD), an organic light emitting display panel (OLED), an electrophoretic display panel (EPD), or the like. The display device 50 may be provided in the form of a touch screen to receive an input through a user's touch and operate as the user input unit 60. The display unit 50 may display a user interface (UI) or a graphic user interface (GUI) through an on screen display (OSD) function.

The user input unit 60 is connected to the image fusion system 1 by wire or wirelessly to generate a user input data for controlling the operation of the image fusion system 1. The user input unit 60 may include a key pad, a dome switch, a touch pad (contact capacitive type, pressure resistive layer type, infrared sensing type, surface ultrasonic conduction type, integral tension measuring type, Piezo effect method), a mouse, a remote controller, a jog wheel, a jog switch, and the like.

4 is a diagram illustrating a user designation method according to an embodiment of the present invention. According to an embodiment of the present disclosure, as shown in FIG. 4, the user sets one or more user-specified areas through the OSD menu in the fused image F displayed on the display unit 50 through the user input unit 60. In addition, a user-specified weight may be input for each user-specified area. Although FIG. 4 illustrates an example in which the user designated area is in the form of a box, the embodiment of the present invention is not limited thereto and may be set in various shapes such as a circle shape and a triangular shape. In FIG. 4, the pixel weight of the first image is 0.8 and the pixel weight of the second image is 0.8, and the pixel weight of the first image is 0.9 and the pixel of the second image is designated as the pixel weight of the second image. The example which specified the weight as 0.1 is shown.

5 is a flowchart schematically illustrating an image fusion method according to an embodiment of the present invention.

Referring to FIG. 5, the image fusion unit of the image fusion system detects a fusion region from the first image and the second image (S51). The first image and the second image may be matched and preprocessed by the transform function estimation. The fusion region may be an entire image or a part of the image. When the fusion region is a part of the image, the image fusion unit may generate a fusion image in which only the fusion region overlaps the thermal image information while the background is a visible image. Hereinafter, an image fusion method will be described with a first image as a visible image and a second image as a thermal image.

The image fusion unit determines a weight for each pixel in the fusion region of the visible image and the thermal image (S53). The image fusion unit may divide the visible image and the thermal image into a plurality of local regions, respectively, and determine the weight of each pixel by determining the weight for each local region by using the average and standard deviation of pixel values for each local region. Also, the image fusion unit may correct a weight deviation of each local region by using a low pass filter.

The image fusion unit may generate a fusion image in which the visible image and the thermal image are fused using the weight of each pixel or the corrected weight of each pixel (S55). The image fusion unit may output the fusion image to the display unit (S59). The image fusion unit may output a fusion image having the visible image as a background and displaying only thermal image information of the thermal image only in the fusion region.

The image fusion unit detects the user-specified area as a fusion area when a weight of the user-defined area and the user-defined area is input from the user, and applies a user-specified weight to each pixel in the fusion area of the visible image and the thermal image, thereby performing The convergence information may be changed (S57). The fusion information corresponds to pixel values of the fusion image. The pixel value of the fusion image is changed by the user adjusting the fusion region weight of the visible image and the fusion region weight of the thermal image. Since the weight of the visible image and the thermal image can be adjusted by the user, the visible image information and the thermal image information can be freely added or decreased.

The image fusion unit may output the fusion image or the fusion image whose fusion information is changed by the user input (S59).

6 is an exemplary view illustrating a fusion image according to an image fusion according to an embodiment of the present invention and a fusion image according to a comparative example.

Referring to FIG. 6, the fusion image d is a fusion image in which the visible image a and the thermal image b are fused using weights for each local area according to an exemplary embodiment of the present invention.

The fusion image (c) is a fusion image as a comparative example generated by the existing image fusion method. Conventionally, the global weight of the visible image and the global weight of the thermal image are calculated by using the mean and standard deviation of pixel values for the entire area of each of the visible and thermal images, and the global and thermal images are calculated using the global weights. Images were fused.

Accordingly, in the fusion image c, the regions A and B of the plurality of fusion regions A, B and C effectively display object information of the thermal column, but in the case of the region C, the object of the visible image. Although the information is clearer than the object information of the fused image (c), it is whitened by increasing the luminance value of the area (C).

On the other hand, the converged image (d) generated according to the embodiment of the present invention effectively reflects the object information of the visible image in the region (C) and is effectively displayed without whitening. The fusion image e is a diagram in which a portion of the fusion image d in which information of the thermal image is reflected is displayed in red.

Meanwhile, the present invention may set a desired fusion region according to various situations and environments, and add a weight of the fusion region to a manual.

The image fusion system may provide an auto mode, a manual mode, and an auto-manual mode. In the automatic mode, the image fusion system may generate a fusion image by detecting a fusion region and determining a weight according to a predetermined algorithm. In the manual mode, the image fusion system allows a user to specify a fusion area and weights. In the auto-manual mode, the image fusion system is based on the automatic mode, but the user can manually specify a specific area and determine the weight.

The image fusion system may receive a user input command by wire or wirelessly, and output a fusion image by applying an input fusion region and a weight.

In the above-described embodiment, the first image is described as a visible image and the second image as an example of a thermal image. However, embodiments of the present invention are not limited thereto, and the first image and the second image are different from each other. The same may be applied to the case of an image obtained from a sensor having different characteristics other than a time or visible light camera and an infrared light camera.

The above-described embodiments are applicable to boundary area monitoring such as GOP, surveillance requiring 24-hour real-time monitoring such as forest fire monitoring, detection of building and residential invasion in a no-light or low light environment, tracking missing persons and criminals in places such as mountains, medical imaging, etc. Can be.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

Claims (6)

Each of the visible image and the thermal image is divided into a plurality of local regions, and the weights of the local regions are calculated using a linear combination of pixel values and standard deviations of each local region, and the visible and A weight determiner configured to determine pixel weights of the thermal images; And
And a post-processing unit which generates a fusion image in which the visible image and the thermal image are fused by applying the pixel weight of the visible image and the pixel weight of the thermal image. The method of claim 1, wherein the weight determination unit,
The image fusion system for correcting the deviation between the weight for each area. The method of claim 1,
And a fusion region detector for detecting a fusion region from the visible image and the thermal image.
And the post processor is configured to generate a fusion image of the fusion region with the visible image as a background. delete delete delete

Images