KR20130054281A - Infrared resolution and contrast enhancement with fusion - Google Patents

Abstract

The present invention relates to the combination of images. A method according to an embodiment comprises the steps of receiving a visual image and an infrared (IR) image of a scene, extracting high spatial frequency content from the visual image, and extracting the extracted high spatial frequency content and the IR from the visual image. Combining the images, wherein the resolutions for the visual and IR images are substantially the same to produce the combined image.

Description

INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION}

The present invention relates to methods, imaging devices, software, and systems for enhancing infrared ("IR") images.

Within the image processing area, IR images of scenes containing one or more objects can be enhanced by combining with image information from the visual image, which combination is known as fusion. Many technical problems arise in attempting to perform such combinations and enhancements.

In general, an imaging device in the form of a camera is provided for capturing visual and IR images and processing these images so that they can be displayed together. The combination is advantageous for identifying temperature changes in an object using IR data from the IR image while simultaneously displaying enough data from the visual image to simplify the orientation and recognition of the object in the final generated image for the user using the imaging device. .

Since capturing IR and visual images can be performed by other components of the imaging device, the optical axes between the imaging components can be separated from each other and an optical phenomenon known as parallax occurs. In order to eliminate this and errors caused by the angle between the optical axes, the images must be aligned.

When combining an IR image with a visual image, many other methods are known. The most commonly used methods are known as critical fusion and picture-in-picture fusion.

In a method for performing critical fusion of images, visual and IR images of the same scene are captured. In an IR image, a temperature interval is selected and only those pixels of the image corresponding to the temperature within the selected interval are selected and displayed with information data from all other pixels. The final resulting combined image represents a visual image except those areas where a temperature within a selected interval can be detected and instead displays data from the IR image in these pixels. For example, if a moist spot on the wall is detected, critical fusion can be used to determine the humidity range by setting the temperature threshold at intervals around the temperature of the liquid that made the spot. The other parts of the wall will be more similar to room temperature and appear as visual data on the screen, so the exact location of the spot can be determined. By looking at the texture of the wall, such as the pattern of the wallpaper, the location of the stain can be further determined very accurately.

When performing PIP fusion, a visual image and an IR image representing the same scene including one or more objects are captured, and pixels in a predetermined area, often in a square shape, are displayed from the IR image while the rest of the combination image is represented by visual data. do. For example, when detecting deviations in a series of objects that are assumed to be approximately equal in temperature, a square is formed around many objects and can be moved until a failed object is captured except for a functioning object, the difference being It will be easy to find. By displaying elements from a visual image outside of this square, for example text or a pattern, the exact location of the objects according to a particular temperature can be determined more easily and reliably.

The critical fusion and PIP fusion methods both display selected portions of the image combined as IR data, while the rest are represented as visual data. This is a drawback of missing details that can be seen in the visual image when representing IR data for the same area. Similarly, temperature data from IR images cannot be represented with the shapes and tissues given by visual images of the same area.

There are several ways to blend IR data and visual data in the same image. However, the results are generally difficult to interpret and can be confusing to the user because the temperature data is blended with the color data of the visual image from IR images displayed in different colors of the palette or at different gray scale levels. As a result, for example, a difference between a red object and a hot object or between a blue object and a low temperature object may not be identified. In general, the importance of color from the radiometer or other IR related aspects of the image, i. E. Palette or gray scale level, is lost when blending the IR image with the visual image.

Therefore, there is a need for an improved method of providing a combinatorial method of comparing data from an IR image and data from a visual image together.

The following publications show examples of related technologies:

A method of aligning an IR image with a visual optical image is disclosed in WO2009 / 008812 Al (Flir Systems AB), in which the images specify the position of the focus motor for determining the distance to the object captured by the image parts. Sorted on the basis.

Reference is made to US20020113210 Al discloses chemical imaging systems and methods using near infrared radiation microscopy.

Xue Z. et al., "Fusion of Visual and IR Images for Concealed Weapon Detection," describes a method of image fusion of visual and IR images.

None of the above related techniques teaches features of the present invention and novel techniques.

One or more embodiments of the present invention may solve or at least minimize the problems described above. This is achieved by the method according to the claims, the imaging device and / or the permanent computer program product, wherein the IR image is combined with the high spatial frequency content of the visual image. The combination is performed through the superposition of the high spatial frequency content of the visual image and the IR image, or alternatively through the superposition of the IR image on the high spatial frequency content of the visual image. As a result, a contrast from the visual image can be inserted into the IR image representing the temperature change, combining the advantages of the two image types without losing the clarity and image interpretation capability of the final combination image.

More specific aspects of embodiments of the invention are described below.

The method according to an embodiment of the present invention includes the step of ensuring that the resolution of the combined image, that is, the resolution of the visual and IR images, is substantially the same. In a first exemplary embodiment, this can be done by configuring an imaging device having an IR sensor and a visual image sensor, such that the IR sensor and the visual image sensor have substantially the same resolution. In another embodiment, the resolution of the imaging sensor is not known to be substantially the same, for example different from the preset difference threshold. In another embodiment, if the resolution of the imaging sensor is not previously known, the method may include checking whether the resolution of the received image is substantially the same. The inspecting step can be performed through a resolution comparison of the received image, and information about the resolution of the image can be calculated based on the image received or retrieved from the received image or used from separate imaging sensors. If the resolutions are not substantially the same, ensuring that the resolutions are substantially the same, either through previous knowledge of the sensor resolution or resolution checks, further comprises resampling of at least one of the received images.

Since the resolution of the IR image is generally much lower than the resolution of the visual image due to the properties of the IR imaging device compared to the visual imaging device, the resolution of the IR image can be upsampled to be substantially equal to the resolution of the visual image. As a result, higher levels of detail can be achieved and combined images that are easier to analyze can be presented to the user. As another example, the visual image may be downsampled to be substantially the same as the resolution of the IR image.

As another example, both images can be sampled to fit the third resolution, if appropriate. Both images may originally have substantially the same resolution, or the resolution of the images may be different. However, after resampling for the third resolution, both images will have substantially the same resolution. This may allow the images to be combined in a convenient and appropriate manner regardless of how the images are displayed. In one example, the third resolution may be the resolution of the display screen on which the combined image is displayed.

In addition, extraction of high spatial frequency content from the visual image and removal of noise and / or blurring of the IR image may be preferably performed. In general, this is accomplished by high pass filtering of the visual image and low pass filtering of the IR image by use of a spatial filter that is moved across the image on a pixel-by-pixel basis. It is apparent to one skilled in the art that other well known image processing methods can be used to produce the same result. As a result of the filtering performed on the IR image, the IR image may contain an amount of noise that is flattened and / or reduced relative to the original IR image. In addition, the high-space frequency content extracted from the visual image includes information about a large contrast in the visual image, that is, information in which a sharp edge such as an object outline is located in the visual image. Performing filtering of the IR image is optional. The method for an embodiment of the present invention provides an advantageous effect on the final generated image presented to the user even without filtering of the IR image, so that the user can clearly know one or more objects in the scene and temperature information associated with the image scene. do. However, since the sharp edges and the noise seen in the original IR image are removed but at least reduced by the filtering process, the field of view in the final generated image can be improved by filtering the IR image, and the IR and visual images are aligned. The risk of double edges appearing in non-combination images is reduced.

In addition to high-pass filtering, methods for extracting high-spatial frequency content from an image may include extracting a difference (usually called a difference image) between two images representing the same scene. The first image is captured at a first time instance and the second image is preferably captured at a second time instance at a time close to the first time instance. The two images may generally be two consecutive image frames in the image frame sequence. High-space frequency content representing the edges and outlines of objects in the scene will appear as a difference image unless the image scene is completely immutable from the first time instance to the second time instance and the imaging sensor is not completely stationary. The scene may change from one frame to the next, for example, due to a change in light or a movement of the indicated object in the image scene. Also, in almost all cases, the imaging sensor does not stop completely. It is apparent that if the imaging device is to be carried, movement will be caused by the user of the imaging device. If the camera is fixed to a stand, for example, the imaging sensor or the vibration of the surroundings may move the imaging sensor. Vibration of the imaging device may be generated, for example, by an image stabilization system, which is commonly used in visual imaging devices to compensate for movement of the imaging device. Other methods of performing image stabilization are well known in the art. In an imaging device having an image stabilization system, the imaging sensor may be placed in a device capable of moving the imaging sensor in response to the measured movement of the imaging device. This configuration can be used to capture the edge / outer in the difference image when the movement of the imaging sensor is controlled according to some predefined difference between successive image frames. In this case, the difference may further correspond to a predetermined width of the edge / outer of the difference image, which width is selected according to the situation. Another way to obtain an image from which the difference image can be derived is to use the focus motor of the imaging device to move one or more lenses of the imaging device. The use of focus motors to move lenses in imaging devices is well known in the art. In this case, when slightly out of focus, the image captured by the imaging device becomes a flattened and noise-free image that can directly match the low pass image. After the focus of the imaging device is reset, the focus image can be captured and the high spatial frequency content of the focus image can be obtained by subtracting the defocus image from the focus image. Refocusing of one or more lenses in an imaging device An approach using vibration of an imaging sensor is also not necessary for any digital image processing and thus can be used in conjunction with an analog imaging device. As will be apparent to those skilled in the art, by subtracting the high spatial frequency content obtained by any of the methods described above from the image, the corresponding low pass filter form of the image is obtained since only the low spatial frequency content remains after subtraction. to be. When combining images, adding the high pass filter of the visual image or the extracted high spatial frequency content to the IR image adds an edge and contrast to the IR image but does not alter the IR image. As a result, the boundaries or edges of the object captured by the image can be clearly seen in the combined image while maintaining high radiometric or other related IR information simultaneously.

In one example, only the component of the filtered visual image may be added to the IR image to preserve the color or gray scale palette of the IR image. As a result, the colors remain unchanged and at the same time add some contrast while maintaining the original IR palette's properties. It is advantageous to maintain the IR palette through all steps of processing and display, because radiometric or other related IR information can be maintained throughout the process and interpretation of the combination image can thereby facilitate the user.

When combining the brightness of the visual image with the IR image, an alpha factor can be used to determine the balance between the two images. This factor may be determined by the imaging device or the imaging system itself using appropriate parameters to determine the required level of outline from the visual image to produce a good image, but may also be determined by the user by providing input to the device or imaging system. have. The argument may also be changed at a later stage, such as when the images are stored in a system or PC, etc., and may be adjusted to suit any needs from the user.

Before displaying the resulting combination image to the user, high resolution noise may be added to the image to create high resolution effects and increased details and to make the image easier to interpret by the user.

Included in the context of the present invention.

1 shows a flowchart of a method according to an exemplary embodiment.
2 shows a schematic diagram of a method according to images of other steps of the method according to an exemplary embodiment.
3A shows the IR image in halftones.
3B shows the image of FIG. 3A in halftone after low pass filtering.
FIG. 3C shows the extracted high spatial frequency content of the visual image in halftone in this image obtained by high pass filtering.
FIG. 3D illustrates the combination of the lowpass filtered IR image of FIG. 3B and the highpass filtered visual image of FIG. 3C in halftones.
4 illustrates an example embodiment of an image processing system for performing a method according to an example embodiment.
FIG. 5A shows the IR image of FIG. 3A with different temperature zones indicated in different patterns.
FIG. 5B shows the IR image of FIG. 3B with different temperature zones indicated in different patterns.
5C shows the image of FIG. 3C.
FIG. 5d shows the image of FIG. 3d with different temperature zones indicated in different patterns.
6A depicts a low resolution IR image of a scene, with the image having a resolution of 32 × 32 pixels.
FIG. 6B illustrates the IR image of FIG. 6A after the IR image has been resampled, processed, and combined with the extracted high spatial frequency content of the visual image representing the same scene.

In FIG. 1, an exemplary method according to an embodiment of the present invention can be seen. In block 101, a visual image is captured and in block 102 an IR image is captured. The visual image and the IR image can be captured by the light sensor and the IR sensor respectively. After capture, the visual image and the IR image are aligned at block 103 due to mechanical tolerances that generally prevent them from being mounted exactly parallel with the parallax between the optical axes caused by the difference in the arrangement of the sensors for capturing the image. Compensate the angle generated between these axes.

Blocks 101 and 102 may be performed simultaneously or sequentially. In one example, the images can be captured simultaneously or with as little time difference as possible, because this reduces the risk for alignment differences due to the movement of the imaging device unit capturing visual and IR images.

After alignment at block 103, a guarantee is made at block 110 that the visual image resolution and the IR image resolution are substantially the same. In a first exemplary embodiment, this can be done by constructing an imaging device having an IR sensor and a visual image sensor, so that the IR sensor and the visual image sensor have approximately the same resolution. In another exemplary embodiment, the resolution of the imaging sensors is not known to be substantially the same, for example, differently than the preset difference threshold. In another exemplary embodiment, if the resolution of the imaging sensor is not previously known, the method may include checking whether the resolution of the received image is substantially the same at block 109. The inspecting step may be performed through comparison of the resolution of the received image, and information about the resolution of the image may be available based on the image received or retrieved from the received image or available from a separate imaging sensor. If it is found that the resolutions are not substantially the same, then inspection of the information or resolution for the previous sensor resolution, ensuring that the resolutions are substantially the same includes resampling of at least one of the images received at block 104.

In one exemplary embodiment, the IR image may be resampled to increase or decrease the resolution. In general, the resolution of the captured IR image has a resolution different from that of the captured visual image; Usually the IR image has a lower resolution than the resolution of the visual image. The typical resolution of the IR image may be, for example, 320 × 240 pixels, while the typical resolution of the visual image may be about 5M pixels. If the resolution of the recombinant image is not substantially the same, at least one of these may have a resolution changed to match the other to compensate for the difference and more successfully combine the images. In one example, this can be done by upsampling the IR image to the resolution of the visual image via interpolation. It may also constitute an imaging device in which the IR sensor and the visual image sensor have substantially the same resolution.

As an alternative to upsampling the IR image, the visual image may be sampled to match the resolution of the IR image or both images may be sampled to match the third resolution. Both images may originally have substantially the same resolution or the image resolution may be different. However, after resampling to match the third resolution, both images will have substantially the same resolution. This allows the images to be combined in a comfortable and appropriate manner regardless of how the images are displayed.

If, for example, a combined image is stored and displayed by an IR camera, PC or other device having a high resolution in the image data structure and / or image display means, upsample the IR image to fit a higher resolution visual image generally. It can be convenient to do. However, if the combined image is displayed by a much lower resolution system, it may be more appropriate to downsample the visual image to this requirement. According to an exemplary embodiment, the third resolution may be selected as the resolution of the display screen on which the combined image appears. Both images may have substantially the same resolution or the resolution of the image may be different. However, it is advantageous if the resolutions of the visual and IR images are substantially the same before each image is combined, so that appropriate matching of data for each pixel of the images can be performed.

At block 105, high spatial frequency content of the visual image may be extracted, such as by high band filtering the visual image using a spatial filter. In addition to highband filtering, examples of methods for extracting high spatial frequency content from an image include extracting a difference (usually called a difference image) between two images representing the same scene, wherein the first time The first image is captured at an instance of time and preferably the second image is captured at a second time instance of a time close to the first time instance. The two images may generally be two consecutive frames in an image frame sequence. High-space frequency content representing the edges and outlines of objects in the scene will appear as a difference image unless the imaged scene changes completely from the first time instance to the second time instance and the image sensor is not completely stationary. The scene may change from one frame to the next, for example, due to changes in light and movement of the indicated objects in the imaged scene. Also, in almost all cases, the imaging sensor does not stop completely. It is clear that when the imaging device is carried, movement will be caused by the user of the imaging device. When the camera is fixed to a stand, for example, movement of the imaging sensor may occur due to vibration of the imaging device or surroundings. Vibration of the imaging device may be caused, for example, by an image stabilization system, which is typically used in the visual imaging device to compensate for the movement of the imaging device. Other ways of achieving image stabilization are well known in the art. In an imaging device having an imaging stabilization system, the imaging sensor may be placed in a device capable of moving the imaging sensor in response to the measurement movement of the imaging device. This configuration can be used to capture the edge / outer in the difference image if the movement of the image sensor is controlled to conform to a predetermined predefined difference between successive image frames. In this case, the difference may further correspond to a predetermined width of the edge / outer of the difference image, the width being selected according to the situation. Another method of obtaining an image from which the difference image can be derived is to use the focus motor of the imaging device to move one or more lenses of the imaging device. The use of focus motors for moving lenses in imaging devices is well known in the art. In this case, when slightly out of focus, the image captured by the imaging device becomes a flattened and noise canceled image that can directly match the lowpass filtered image. After the focus of the imaging device is reset, the focus image can be captured and the high spatial frequency content of the focus image can be obtained by subtracting the defocus image from the focus image. Approaches using vibration of the imaging sensor of refocusing one or more lenses in the imaging device require no digital image processing at all and can therefore be used in conjunction with an analog imaging device. As will be apparent to those skilled in the art, by subtracting the extracted high spatial frequency content obtained by any of the methods described above from the image, the corresponding low pass filter form of the image is obtained since only the low spatial frequency content remains from the image.

At block 106, the IR image may reduce and / or reduce noise in the image and blur the image. Low-pass filtering can be performed by placing a spatial core over each pixel of the image and calculating a new value for the pixel by using the values in adjacent pixels and the coefficients of the spatial core. In another example, images can be filtered using only software.

The spatial low pass filter core may be a 3x3 filter core with a coefficient of 1 at every position, and the filter value of the pixel is calculated by multiplying the original pixel value and the eight adjacent pixels by their filter coefficients and adding them together to divide by 9 Can be.

After performing this operation on each pixel in the IR image, a smoother low pass filter image can be created. In order to high pass filter an IR image, the same filter coefficients can be used to form the high pass filter image in a manner well known in the art by subtracting the low pass filter image from the original image one pixel at a time. However, it should be noted that the coefficients of the filter core may be set to other values and the size of the filter core may be different from the 3 × 3 filter core described above.

The resulting processed visual value and possible processed IR image may be combined at block 107. High resolution noise, such as high resolution temporal noise, may be added to block 108 before displaying the resulting combination image.

Performing filtering of the IR image at block 106 is optional. The method for an embodiment of the present invention provides an advantageous effect on the final image presented to the user without filtering the IR image, and the user can clearly identify temperature information of the image scene as well as objects in the image scene. The purpose of the low pass filtering performed at block 106 is to mitigate non-uniformities in the IR image from the noise appearing in the original IR image captured at block 102. Because the sharp edges and noise seen in the original IR image are removed or at least reduced by filtering, the field of view of the final image is further enhanced by filtering the IR image and the IR and visual images are not aligned. The risk of double edges in the combined image is reduced.

In order to extract the high spatial frequency content from the image, in other words, high pass filtering is performed to locate the contrast area, i.e., the area where adjacent pixel values show a large difference, such as a sharp edge. The resulting high pass filter image can be achieved by subtracting the low pass filter image from the original image computed pixel by pixel, which will also be described in detail below.

FIG. 2 illustrates exemplary embodiments of images generated in other blocks of the method shown in FIG. 1. The visual image 301 captured at block 101 and the IR image 302 captured at block 102 are used for upsampling and filtering during processing 303 corresponding to blocks 103, 104, 105, 106. Used as input

After processing 303, the extracted high spatial frequency content 304 of the visual image appears, and the outlines of the objects that were originally shown in the visual image 301 can be seen. According to an exemplary embodiment of the present invention, the IR image 302 is processed with a low pass filter and an upsample image 305. Upsampling has increased the resolution of the image and now each object in the imaged scene can be seen more clearly without showing much noise in the form of blur or graininess in the low pass filter image 305. Arrows from the extracted high-spatial frequency content of the visual image 304 and low pass filter IR image 305, which can now be described as processed images 304, 305, indicate a processed IR image that exhibits a gentle temperature distribution in the imaged scene. 305 represents a combination of these images 304, 305 to form a combined image 307 that is combined with the processed visual image 304 where an outline or edge also appears from the objects of the original visual image 301. Thus, the combined image 307 takes advantage of the IR image 302, which also shows any temperature differences across the objects along with the outlines from the visual image 304 processed to more clearly represent the shape of each object. Display. The combination is preferably performed by superimposing the high spatial frequency content of the visual image on the IR image or alternatively by superimposing the IR image on the high spatial frequency content of the visual image.

According to an exemplary embodiment of the present invention, an IR image can be captured with a very low resolution IR imaging device, the resolution being as low as 64 × 64 or 32 × 32 pixels, but many other resolutions are readily understood by those skilled in the art. The same can be applied. In itself, a 32x32 pixel IR image contains very little information and it is difficult for an observer to interpret the information in the image. An example of an IR image with 32x32 pixel resolution is shown in FIG. 6A. When the edge and outer (high spatial frequency) information is added to the combined image from the visual image, the inventors clearly distinguish between the objects shown and the temperature or other IR information associated with them even with the use of very low resolution IR images. It was found that it would still provide a combined image to do. FIG. 6B shows the IR image of FIG. 6A after the IR image has been resampled and combined with the extracted high spatial frequency content of the visual image representing the same scene. This allows the present method for one embodiment to still provide very advantageous results and to be used in combination with very small and inexpensive image detectors.

According to another example embodiment, the IR image may be captured with a high resolution IR imaging device. As technology advances, IR imaging devices continue to have higher resolution. Current high resolution IR imaging devices have a resolution of, for example, 640 × 640 pixels. The IR image captured with this high resolution imaging device may possibly be sufficient to show the edge and outline information to the viewer by itself. By combining this high resolution IR image with the high spatial frequency content of that visual image, the viewer can see more details of the visual image that do not appear in the IR image. For example, areas where water damage has been identified may be drawn or outlined on the wall using a pen. This information can be advantageous in conjunction with the measured temperature information. As another example, it may be a serial number or other identifying letter or number in the image that may help identify the scene or objects shown in the scene. High resolution IR images can be more advantageously downsampled and / or highly noise-reduced / low-pass filtered such that the resulting processed IR image contains a very low noise level, but still very high in terms of temperature information of the scene shown. Sensitivity is obtained.

The high resolution noise 306 blocks to provide a more clearly final generated image to the observer and to reduce the appearance of speckles or the like that may appear due to noise in the original IR image 302 preserved during low pass filtering of the IR image 302. It may be added to the combination image 307 corresponding to 108.

3A shows the IR image 302 immediately after capture in block 102. The imaged scene represents a bookcase with binders arranged in rows and shelves fitted to a predetermined height. As can be seen, the objects in the scene are at different temperatures represented by different sections, and the binders in the highest part and the middle shelf of the image are warmer than the bottom shelf or the areas next to and above the binders. The actual shape of the objects shown is difficult to discern because no outlines of the object other than the lines between different temperatures are displayed at all. Therefore, it may be very difficult for a user facing only with this image to identify a particular object at a certain temperature. The IR image can be colored according to the selected color space (described further below) by adding color to the signal after filtering.

In an exemplary embodiment of the invention, the captured IR image is processed via low pass filtering. 3B shows a low pass filtered IR image 305. Spatial filtering removed the non-uniformity in the captured IR image 302, making it easier to distinguish different objects in the scene. Filtering also removed noise from the image 302. Also, the edges between these objects were smoothed. This can be done because the outlines are added from the filtered visual image 304, or they can be double outlined with some alignment error between the images, distracting the observer's attention.

In an exemplary embodiment of the invention, the high spatial frequency content of the captured visual image is extracted by high pass filtering of the visual image. This high pass filtered visual image 304, which is the result of high pass filtering of the captured visual image 301, is shown in FIG. 3C. In the high pass filtered visual image 304, the outlines and edges of the objects of the scene originally imaged in the visual image 301 can be seen. The edges of the edges can be seen between the object as well as lines such as text for the pattern due to the binder or book.

3D shows the combined image 307 after the original IR image 302 is upsampled and lowpass filtered, and combined with a highpass visual image of the same scene. Other temperature zones are still visible, but the boundaries between them have become clearer and the outlines of the binder and shelf have been added, resulting from high-pass visual images and revealing details that cannot be seen in IR images such as text or other visual patterns. . The increased clarity is also due to the low pass filtering of the IR image, where the noise pixels within the larger other temperature field are removed to form a larger area that is more similar. As a result, at least some of the noise that may arise from the condition in which the original image was captured can be removed.

5A-5D show the image of FIGS. 3A-3D described above in such a way that different temperature regions are displayed in a different pattern instead of halftones, all of which are shown in FIGS. 5A-5D with reference to FIGS. 3A-3D. Each can be acted directly.

The low pass filtering performed on the IR image 302 may be performed using a spatial filter with an appropriate filter core to calculate the new value for each pixel and the value of the surrounding pixels according to the previous value. High pass filtering is generally performed by applying a low pass filter and subtracting the resulting low pass filtered image from the original image, leaving only lines and edges visible in the high pass filtered image. As mentioned above, methods of applying spatial filters are well known in the art and any such method may be used.

For example, when selecting a palette according to the YCbCr family of color spaces, the Y component (i.e. luminance) may be selected as a constant over the entire palette. In one example, the Y component may be selected to be 0.5 times the maximum luminance. As a result, upon combining the IR image according to the selected palette with the visual image, the Y component of the processed visual image 304 may be added to the processed IR image 305 and the color of the processed IR image 305 may not change. Without causing a certain contrast. This maintains the importance of the particular nuances of color during the processing of the original IR image 302.

When calculating the color components, the following equations are used to produce the component Y for the image 307 combined with the Y component from the high-pass filtered visual image 304 and Cr from the signal of the IR image 305 and Cb: , Cr, and Cb) can be determined.

hp_y_vis = highpass (y_vis)

(y_ir, cr_ir, cb_ir) = colored (lowpass (ir_signal_linear))

Another notation can be written as:

hp _yvis = highpass (y _vis )

(y _ir , cr _ir , cb _ir ) = colored (lowpass (ir _{signal linear} ))

Of course YCbCr and other color spaces can also be used with embodiments of the present invention. The conversion between color spaces as well as the use of other color spaces such as, for example, RGB, YCbCr, HSV, CIE 1931 XYZ or CIELab are well known to those skilled in the art. For example, using the RGB color model, the luminance can be calculated as the average of all color components, and by converting a formula that calculates the luminance from one color space to another, a new representation for determining the luminance is obtained. Determined for color space.

In one embodiment, block 107 combining the processed visual image 304 with the processed IR image 305 may be performed using only the luminance component Y from the processed visual image 304.

It should be noted that the blocks of the method described above may be performed in a different order as appropriate in accordance with one or more embodiments.

4 shows a schematic diagram of an embodiment of an image processing system for performing a method according to the invention. The imaging device unit 1 may comprise a vision imaging device 11 with a vision sensor and an IR imaging device 12 with an IR sensor, such that the optical axis of the vision sensor of the vision imaging device 11 is an IR imaging device. And mounted at a distance d from the IR sensor of (12). The visual imaging device can be any known type of visual imaging device, such as a CCD imaging device, an EMCCD imaging device, a CMOS imaging device or an sCMOS imaging device. The IR imaging device may be any kind of imaging device capable of detecting electromagnetic radiation at least, eg, at intervals of 0.7 to 20 μm. The visual imaging device has a field of view α of approximately 53 ° while the IR imaging device has a field of view β of approximately 24 °. It should be appreciated by those skilled in the art that other viewing angles may be used, for example, through the use of interchangeable optics or lenses including optics. For IR imaging devices, interchangeable optics or lenses including optics may provide a field of view of 15-45 °, for example. The capturing of the blocks 101, 102, ie the visual image 301 and the IR image 302, can be performed by the imaging device unit 1, the captured images being sent to a processing unit 2 also called a processor. , Where the remaining blocks are also performed. According to another embodiment, the selection step in block 106 of FIG. 1, ie, reducing noise and blurring the captured IR image may be performed by image processing means or image processor included in the imaging device, After the visual image and the processed IR image are transmitted to the processing unit 2, the method steps of the remaining blocks of FIG. 1 are performed. The processing unit 2 includes, for example, a microprocessor, microcontroller or other portion of code stored on a computer readable storage medium adapted to perform a predetermined task or other code portions stored on a computer readable storage medium that can be changed during use. It can be a processor such as a general purpose or special processing engine such as control logic or a field-programmable gate array (FPGA) unit. These modifiable parts provide inputs for various tasks such as adjustment of the IR imaging device 12, alignment to the visual imaging device 11 and the IR imaging device 12, in particular filters or sample rates for spatial filtering of images. It may include parameters that can be used as.

 In this document, the terms "computer program product" and "computer-readable storage medium" generally refer to a permanent medium, such as a memory 41, a storage medium of the processing unit 2, or a storage medium of the control unit 42. Can be used to speak. These and other forms of computer readable storage media can be used to provide instructions to the processing unit 2 for execution. Such instructions, commonly referred to as "computer program code" or computer program code portion (which may be grouped in the form of computer programs or other groupings), are used to control the data processing system to perform the steps and functions of the method of the present invention. Is formed. Thus, in execution, the computer program code portion may cause the imaging device 1 or the computer to perform the features or functions of the embodiments of the present technology. Further, as used herein, processing logic or logic may be hardware, software, or the like. , Firmware or a combination thereof.

The processing unit 2 is in communication with the memory 41, in which these parameters are maintained ready for use by the processing unit 2, and if the user wishes the image processed by the processing unit 2 can be stored. have. The memory 41 may be RAM, register memory, processor cache, hard disk drive, floppy disk drive, magnetic tape drive, optical disk drive, CD or DVD drive (R or RW), or other removable or fixed media drive. have. The memory 41 in turn communicates with the control unit 42, in which the parameters are passed from the imaging device unit 1 to the object from which the image is captured via input from an adjustment file 43, which can be supplied, for example, from the manufacturer. It is generated by parameters provided by the imaging processing system itself, such as data from a sensor or the like for distance, or by parameters provided by a user. The control unit 42 may be a programmable unit and determines the parameters needed to perform the exemplary methods, and how these parameters interact with the processing unit 2 and are easily retrieved by the processing unit 2 in order to retrieve them. Can determine whether these parameters should be stored.

The processing unit 2 aligning the image (block 103), upsampling the original IR image 302 to generate an upsampled IR image (block 104), generating a processed visual image 304 High pass filtering of the original visual image 301 (block 105), low pass filtering of the upsampled IR image (block 106) to generate the processed IR image 305, to generate a combined image 307 Combining the processed visual image 304 with the processed IR image 305 (block 107), and adding high frequency noise to the combined image 307 (block 108). The last generated image is shown on the display unit 3 to be visible to the user of the image processing system. If necessary, the user can store either the combination image 307 or other images corresponding to different method steps in the memory 41 for later viewing or transfer to another unit such as a computer for further analysis and storage. have.

In another embodiment, the disclosed methods may in particular comprise the functionality of an FPGA unit configured to perform the method steps for one or more embodiments of the invention or may comprise a general purpose processing unit 2 according to the description in connection with FIG. 4. May be executed by a computing device such as a PC. The computing device may further comprise a memory 41 and a control unit 42 and also a display unit 3. It can be used vividly for the set of streams of filtered and combined images in real time, for example at 30 Hz, which can be recorded and replayed as moving pictures, but can also use still photos.

In one example, the user may be allowed to change the positive factor alpha to determine how much of the luminance from the visual image 301, 304, which may be used to combine with the IR image 302, 305, for example, by using the equation below. Can be. The luminance Y of the combined image 307 is achieved by adding the luminance of the processed IR image 305 to the luminance of the high pass filter visual image multiplied by the factor alpha. The combined components Cr and Cb are taken directly from the IR images 302 and 305 and are therefore not affected by this process. If another color space is used, the equation is of course transformed before use.

comb_y = y_ir + alpha x hp_y_vis

comb_cr = cr_ir

comb_cb = cb_ir

Another notation can be written as:

comb _y = y _ir + alpha * hp _yvis

comb _cr = cr _ir

comb _cb = cb _ir

Thus, the change in alpha gives the user the opportunity to determine how much contrast is needed for the combination image. If the alpha is close to zero, only the IR image will appear, but if the alpha is very large, very sharp outlines can be seen in the combination image. Theoretically, alpha can be an infinitely large number, but in practice a restriction will probably be needed to limit the size of alpha that can be chosen as convenient in the current application.

As long as the upsampling of the resolution of the IR image 302 in block 104 matches the resolution of the IR image 302, or indeed the result is that the IR image 302 and the visual image 301 have the same resolution after the sampling step. Alternately may be performed as downsampling of visual image 301 to match a combination of upsampling of IR image 302 and downsampling of visual image 301 to a resolution that none of the images 301, 302 have inherently. . It may be convenient to determine the resolution according to the display area, such as the display unit 3 where the combined image 307 is displayed to match the resolution that is most suitable for the display unit 3 and sample the image or images 301, 302. .

For clarity, it will be appreciated that the above description has described embodiments of the technology for other functional units and processors. However, it is apparent that any suitable functional distribution may be used between other functional units, processors, or domains without departing from the technology. For example, functionality illustrated to be performed by separate processors or controllers may be performed by the same processor or controller. Thus, references to particular functional units should not be considered strict logical or physical structures or configurations, but merely reference to appropriate means for providing the aforementioned functionality.

The present invention is not limited to the above-described embodiments but may be modified within the scope of the claims as those skilled in the art will readily appreciate.

Claims (25)

Receiving a visual image and an infrared (IR) image of the scene,
Extracting high spatial frequency content from the visual image;
Combining the extracted high spatial frequency content with the IR image from the visual image,
The resolution for visual and IR images is substantially the same way to generate a combination image. The method of claim 1,
Ensuring that the resolution for the visual image and the IR image are substantially the same. 3. The method according to claim 1 or 2,
Checking whether the resolutions for the visual and IR images are substantially the same;
If the resolution for the visual image and the IR image are not substantially the same, changing the resolution of the visual image and / or the resolution of the IR image such that the resolution for the visual image and the IR image is substantially the same. The method according to any one of claims 1 to 3,
Changing at least one of the resolution of the visual image and / or the resolution of the IR image such that the visual image and the IR image have substantially the same resolution. The method according to any one of claims 1 to 4,
Upsampling the resolution of the IR image to substantially the resolution of the visual image and / or downsampling the resolution of the visual image to substantially the resolution of the IR image. 6. The method according to any one of claims 1 to 5,
Changing the resolution of the visual image and the resolution of the IR image to be substantially equal to a third resolution. 7. The method according to any one of claims 1 to 6,
The resolution of the received IR image is 64 × 64 pixels or less. The method according to any one of claims 1 to 7,
The resolution of the received IR image is 32x32 pixels or less. The method according to any one of claims 1 to 8,
Extracting high spatial frequency content from the visual image is performed by high pass filtering using at least one spatial filter. 10. The method according to any one of claims 1 to 9,
Processing the IR image to remove noise from the IR image and / or to flatten the IR image. 11. The method according to any one of claims 1 to 10,
Lowpass filtering the IR image prior to combining. 12. The method according to any one of claims 1 to 11,
Low pass filtering the IR image using a spatial filter prior to combining. 13. The method according to any one of claims 1 to 12,
Combining includes superimposing the extracted high spatial frequency content from the visual image on the IR image. 14. The method according to any one of claims 1 to 13,
Combining includes superimposing the IR image on the extracted high spatial frequency content from the visual image. 15. The method according to any one of claims 1 to 14,
Adding the luminance component of the visual image to the IR image. The method according to any one of claims 1 to 15,
Adding a luminance component of the visual image multiplied by a factor to the IR image. 17. The method according to any one of claims 1 to 16,
The argument is a variable based on a control parameter received from a user input. 18. The method according to any one of claims 1 to 17,
Adding high resolution noise to the combined image. 19. The method according to any one of claims 1 to 18,
Maintaining an IR palette for the IR image and / or the combination image. 20. The method according to any one of claims 1 to 19,
Capturing visual and IR images of the scene;
The method of claim 1 wherein the method of claim 1 is performed by an image device having an IR image capturing device and a visual image capturing device. 21. The method according to any one of claims 1 to 20, executed on a computer configured to receive a visual image and an IR image of a scene. A permanent computer program product for combining IR and visual images,
Receiving visual and IR images of the scene,
Extracting high spatial frequency content from the visual image;
Computer program code portion designed to control a data processing system to perform an operation comprising combining the extracted high spatial frequency content with the IR image from the visual image,
Permanent computer program products that are substantially the same for generating visual images and resolution for combination images. 23. The method of claim 22,
22. A permanent computer program product further comprising a computer program code portion designed to control the data processing system to perform the step or function of any one of claims 2 to 21. A visual image sensor for capturing visual images,
An IR image sensor for capturing IR images,
Extracts high spatial frequency content from the visual image
A processing unit configured to combine the extracted high spatial frequency content with the IR image from the visual image,
Imaging devices that are substantially the same for generating visual images and resolutions for IR images. 25. The method of claim 24,
22. An imaging device, further comprising a processor configured to perform the steps or functions of any of claims 2 to 21.

Images