EP1287684A1

EP1287684A1 - Method and system for fusing images

Info

Publication number: EP1287684A1
Application number: EP01927025A
Authority: EP
Inventors: Timothy Ostromek
Original assignee: Litton Systems Inc
Current assignee: Northrop Grumman Guidance and Electronics Co Inc
Priority date: 2000-04-27
Filing date: 2001-04-13
Publication date: 2003-03-05
Also published as: CA2404654A1; IL151574A0; EP1287684A4; AU2001253518A1; WO2001084828A1

Abstract

A system for fusing images comprises sensors (102 and 104) for generating sets of image data. An information processor (108) receives and samples the sets of image data to generate sample data for computing a fused image array. A display (110) receives the fused image array and displays a generated fused image. Step 1 of 4 for fusing images receives sets of image data generated by sensors (102 and 104). Step 2 samples the sets of image data to produce sample data. Step 3 computes a fused image array from the sample data. Step 4 displays a fused image generated from the fused image array on a display (110). Step 1 of 4 for computing a fused image array samples sets of image data generated from sensors (102 and 104) to produce sample data. Image fusion metrics from the sample data are determined in step 2. Step 3 calculates weighting factors from the image fusion metrics. Step 4 computes a fused image array from the weighting factors, wherein the fused image array is used to generate the fused image.

Description

METHOD AND SYSTEM FOR FUSING IMAGES

TECHNICAL FIELD OF THE INVENTION

This invention relates generally to the field of electro-optical systems and more specifically to a method and system for fusing images.

BACKGROUND OF THE INVENTION

Image fusion involves combining two or more images produced by two or more image sensors into one single image. Producing one image that mitigates the weak aspects of the individual images while retaining the strong ones is a complicated task, often requiring a mainframe computer. Known approaches to image fusion have not been able to produce a small, lightweight system that consumes minimal power.

Known approaches to fusing images require a great deal of computing power. To illustrate, suppose that two image sensors each have the same pixel arrangement. Let N„ be the number of horizontal pixels, and let N_v be the number of vertical pixels, such that the total number of pixels per sensor is N_/, • N_v. Let the frame rate of the display be Ωj, expressed in Hz. The time τ_d allowed for processing each frame is given by: τ_d = \I Ω_d All processing for each displayed image must be done within this time to keep the system operating in real time. To calculate the processing time per pixel, first compute the total number of pixels of both sensors. Given that each image sensor has the same pixel arrangement, the total number of pixels for both sensors is:

2 • N_/.N The processing time τ_p per pixel is given by:

2N,N Ω, * 2N_ΛN_V

The processing time τ_p per pixel is the maximum amount of time allotted per pixel to calculate a display pixel from the two corresponding sensor pixels, while allowing for real time processing. For example, given an average system where Ω^ = 30 Hz, Ν/, = 640, and Ν_y = 480, the processing time r^per pixel is: τ_p = 108.5 ns

For handheld or portable applications, processor speed Ω_p is limited to about 150 MHz. A maximum of approximately two instructions per cycle is allowed in current microprocessors and digital signal processors. The time required per instruction r; is given by: ι £ — = 3.33 ns 2Ω _P

The number of instruction cycles allowed for each pixel in real time processing is given by: r N, = — - = 32 instruction cycles ^τ, Thirty-two instruction cycles per pixel is often not a sufficient number of cycles, especially considering the fact that a simple "divide, floating point" could easily require 10 to 100 instruction cycles to complete. Practical image fusion systems generally require over 100 instruction cycles per pixel, and sophisticated image fusion algorithms often require over 1,000 instruction cycles per pixel. Consequently, current image fusion systems are confined to mainframe computers.

While known approaches have not been applied to handheld or portable applications, the challenges in the field of image fusion have continued to increase with demands for small, lightweight systems that consume minimal power. Therefore, a need has arisen for a new method and system for fusing images.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method and system for fusing images are provided that substantially eliminate or reduce disadvantages and problems associated with previously developed systems and methods. A system for fusing images is disclosed. The system comprises two or more sensors for generating two or more sets of image data. An information processor receives and samples the sets of image data to generate sample data and computes a fused image array from the sample data. A display receives the fused image array and displays a fused image generated from the fused image array. A four-step method for fusing images is disclosed. Step one calls for receiving sets of image data generated by sensors. Step two provides for sampling the sets of image data to produce sample data. In step three, the method provides for computing a fused image array from the sample data. The last step calls for displaying a fused image generated from the fused image array. A four-step method for computing a fused image array is disclosed. Step one calls for sampling sets of image data generated from sensors to produce sample data. Step two provides for determining image fusion metrics from the sample data. Step three calls for calculating weighting factors from the image fusion metrics. The last step provides for computing a fused image array from the weighting factors, wherein the fused image array is used to generate the fused image.

A technical advantage of the present invention is that it computes the fused image from sampled sensor data. By sampling the sensor data, the invention reduces the number of instruction cycles required to compute a fused image. Reducing the number of instruction cycles allows for smaller, lightweight image fusion systems that consume minimal power. BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and for further features and advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which: FIGURE 1 is a system block diagram of one embodiment of the present invention;

FIGURE 2 is a flowchart demonstrating one method of fusing images in accordance with the present invention;

FIGURE 3 A illustrates sampling with a fixed array pattern in accordance with the present invention;

FIGURE 3B illustrates sampling with a varied array pattern in accordance with the present invention;

FIGURE 3C illustrates sampling randomly in accordance with the present invention; and FIGURE 4 illustrates a method of computing weighting factors in accordance with the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

FIGURE 1 is a system block diagram of one embodiment of the present invention. In this embodiment, a sensor A 102 and a sensor B 104 detect one or more physical objects 106 in order to generate image data to send to an information processor 108, which fuses the sets of image data to produce a single fused image to be displayed by a display 110. The sensor A 102 detects the physical objects 106 and generates sensor data, which is sent to an amplifier A 112. Amplifier 1 12 amplifies the sensor data and then sends it to an analog-to-digital converter A l \4. The analog- to-digital converter y-1 1 14 converts the analog sensor data to digital data, and sends the data to an image buffer A 116 to store the data. The sensor B operates in a similar fashion. The sensor B 104 detects the physical objects 106 and sends the data to amplifier B 118. The amplifier B 118 sends amplified data to an analog-to-digital converter B 120, which sends converted data to an image buffer B 122. A field programmable gate array 124 receives the data generated by the sensor 102 and the sensor B 104. The information processor 108 receives the data from the field programmable gate array 124. The information processor 108 generates a fused image from the sets of sensor data, and uses an information processor buffer 126 to store data while generating the fused image. The information processor 108 sends the fused image data to a display buffer 128, which stores the data until it is to be displayed on the display 110.

FIGURE 2 is a flowchart demonstrating one method of image fusion in accordance with the present invention. The following steps may be performed automatically using an information processor 108. The method begins with step 202, where two or more image sensors generate two or more sets of image data. As above, suppose that there are two image sensors, each with the same pixel arrangement. Let

N_h be the number of horizontal pixels, and let N_v be the number of vertical pixels, such that the total number of pixels per sensor is N/, • N_v. The sensors may comprise, for example, visible light or infrared light image detectors. Assume that detectable variations in the proportion of the fused image computed from one set of image data and from the other set of image data occur in time τ_s, where: τ_s > \IΩ_d. Hence, the computation of the proportion does not need to be calculated at each frame. Also, assume that the information required to form a metric that adjusts the system to a given wavelength λ proportion can be derived from fewer than N_/-»N_V pixels.

The method then proceeds to step 204 where the sets of image data are sampled to produce sample data. FIGURES 3A, 3B, and 3C illustrate three methods of sampling image data in accordance with the present invention. FIGURE 3A illustrates sampling with a fixed array pattern. The sampled pixels (/, j) 302, 304, 306, 308, 310, 312, 314, 316, and 318 may be described by:

/ where/? = 1, 2 and

j = qΔ_v , where q =

One possible arrangement is to have Δ_/, = 2 for the horizontal difference between one sampled pixel to the next sampled pixel, and Δ_v = 2 for the vertical difference between one sampled pixel to the next sampled pixel. The groups of pixels 320 and

322, each with 2 sampled pixels 302 and 304, and 308 and 310, respectively, are sampling blocks. FIGURE 3B illustrates sampling with a varied array pattern. FIGURE 3C illustrates random sampling. A sequence of sampling patterns may also be used, repeating at any given number of sampling cycles, or never repeating, as in a random pattern for each continued sampling cycle.

Referring again to FIGURE 2, in steps 206 to 210, a fused image array is computed from the sample data. In step 206, image fusion metrics are calculated from the sample data. The image fusion metrics are values assigned to the pixels of the sample data. These values, for example, may give the relative weight of the data from each sensor, such that the data from the sensor that produces the better image is given more weight. Or, these values may be used to provide a control for the production of, for example, a false color image. All the pixels may be assigned the same metric, β, or each sample pixel may assigned its own metric, β_y, where the subscript if designates the pixel in the rth row andy^'th column.

In step 208, weighting factors _υ, where the subscript if designates the pixel in the rth row and y^'th column, are calculated from the image fusion metrics. The weighting factors are values assigned to the pixels of the fused image. The weighting factors may be computed by, for example, linear interpolation of the image fusion metrics.

FIGURE 4 illustrates a method of computing weighting factors in accordance with the present invention. For example, suppose that the sample data was sampled using a fixed array pattern, where every fifth point 402, 404, 406, and 408 is sampled in both the horizontal and vertical direction, that is, Δ_/7 = Δ_v = Δ = 5. A sampling block 410 comprises to two sampled points 402 and 404. The weighting factors α₀ of the first row may be computed in the following manner. First, an incremental value for the first row in the horizontal direction δ_/7] is calculated using the following formula: Then, the weighting factors between βn and β_i6 in the horizontal direction are calculated using the following formula: The weighting factors in the vertical direction between βn and β₆ι are calculated in a similar manner, using the following equations: α,ι = βn + δ_vι ( - 1)

Referring again to FIGURE 2, the method then proceeds to step 210, where a fused image array, which is used to generate a fused image, is computed from the weighting factors. An array of weighting factors _υ generates the following fused image array:

V, = V_l/^A)-a_lJ + V_l/^BHl - a_v) where / e {1, ..., N_n},j e {1, ..., N_v}, the superscripts (d) denotes display, A) denotes sensor A, and (B) denotes sensor B, and V_υ corresponds to the voltage at pixel (zJ). The fused image array describes the relative weights of the data from each sensor. Weighting factor _υ gives the relative weight of the voltage from sensor A at pixel (/J); weighting factor (l- _y) gives the relative weight of the voltage from sensor B at pixel (z^' ). This example shows a linear weight; other schemes, however, can be used. The method then proceeds to step 212, where the fused image generated from the fused image array is displayed on a display 1 10.

By sampling the image data, this embodiment allows for more instruction cycles to calculate β_y for each sampled pixel. To calculate the number of instruction cycles available for each sampled pixel, first calculate the total number of instruction cycles per sampled pixel, and then subtract number of cycles per pixel needed to sample the pixels and to compute the fused image metrics and the fused image array. For example, assume that data is sampled using fixed array sampling. The total number of instructions for each sampled pixel is given by:

X ^τ, where x_s is the processing time per sampled pixel, which is given by:

1 Ω_d - 2n_hn_v where n„ and n_v are the number of sampled pixels in the horizontal direction and in the vertical direction, respectively. Sampling each sampling block, without double counting borders, requires about (Δ + 1)[2(Δ - 1) + 6] instruction cycles. Each sampling block contains two sampled pixels, so each sampled pixel loses 1/2(Δ +

1)[2(Δ - 1) + 6] instruction cycles per pixel. Computing the fused image array from the weighting factors requires approximately four instruction cycles for each calculation, that is: ^,N_V Therefore, the time left per pixel for calculating the image fusion metrics β_y is: τ 4N N

Ν. = -s- 1/2 (Δ + 1) [2(Δ-1) + 6] - — ^h—*- Using the values given above: Ω_d = 30 Hz, N_h = 640, N_v = 480, A_h = Δ_v = 5, n_h = 128, and n_v = 96, the number of instruction cycles is computed to be:

N, = 269 instruction cycles. This is a dramatic improvement compared with the 32 cycles allotted in conventional methods. The extra cycles may be used for more complex calculations of β_y or other features. Moreover, if β_y is assumed to be the same for all pixels, even more additional cycles may be available to determine β_y, allowing for a more sophisticated manipulation.

Although an embodiment of the invention and its advantages are described in detail, a person skilled in the art could make various alternations, additions, and omissions without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

WHAT IS CLAIMED IS:

1. A system for fusing images, the system comprising: a. two or more sensors for generating two or more sets of image data; b. an information processor for receiving and sampling the sets of image data to generate sample data and for computing a fused image array from the sample data; and c. a display for receiving the fused image array and displaying a fused image generated from the fused image array.

2. The system of Claim 1 further comprising a field programmable gate array coupled to the sensors and the information processor.

3. The system of Claim 1 further comprising one or more converters operable to convert analog signals from the sensors to digital data for use by the information processor.

4. The system of Claim 1 wherein the fused image array assigns a value to one or more pixels of the fused image, wherein the value describes the relative weights of the sets of image data.

5. A method for fusing images, the method comprising: a. receiving two or more sets of image data generated by two or more sensors; b. sampling the sets of image data to produce sample data; c. computing a fused image array from the sample data; and d. displaying a fused image generated from the fused image array.

6. The method of Claim 5, wherein the sampling step further comprises sampling with a fixed array pattern.

7. The method of Claim 5, wherein the sampling step further comprises sampling with a varied array pattern.

8. The method of Claim 5, wherein the sampling step further comprises sampling randomly.

9. The method of Claim 5, wherein the computing step further comprises determining one or more image fusion metrics, wherein the image fusion metrics are values assigned to one or more pixels of the sample data.

10. The method of Claim 5, wherein the computing step further comprises calculating one or more weighting factors from the image fusion metrics, wherein the weighting factors are values assigned to one or more pixels of the fused image.

11. The method of Claim 10, wherein the computing step further comprises calculating the weighting factors by interpolation of the image fusion metrics.

12. The method of Claim 10, wherein the computing step further comprises calculating the weighting factors by linear inteφolation of the image fusion metrics.

13. The method of Claim 5, further comprising performing the foregoing steps automatically using an information processor.

14. A method for computing a fused image array, the method comprising: a. sampling the sets of image data generated from two or more sensors to produce sample data; b. determining one or more image fusion metrics from the sample data, wherein the image fusion metrics are values assigned to one or more pixels of the sample data; c. calculating one or more weighting factors from the image fusion metrics, wherein the weighting factors are values assigned to one or more pixels of a fused image; and d. computing a fused image array from the weighting factors, wherein the fused image array is used to generate the fused image.

15. The method of Claim 14, wherein the fused image array describes the relative weights of the sets of image data at one or more pixels of the fused image.

16. The method of Claim 14, wherein the sampling step further comprises sampling with a fixed array pattern.

17. The method of Claim 14, wherein the sampling step further comprises sampling with a varied array pattern.

18. The method of Claim 14, wherein the sampling step further comprises sampling randomly.

19. The method of Claim 14, wherein the calculating step further comprises calculating the weighting factors by inteφolation of the image fusion metrics.

20. The method of Claim 14, wherein the calculating step further comprises calculating the weighting factors by linear inteφolation of the image fusion metrics.

21. The method of Claim 14, further comprising performing the foregoing steps automatically using an information processor.