CN115018889A - Short wave infrared and visible light image registration and fusion device and method - Google Patents
Short wave infrared and visible light image registration and fusion device and method Download PDFInfo
- Publication number
- CN115018889A CN115018889A CN202210883550.7A CN202210883550A CN115018889A CN 115018889 A CN115018889 A CN 115018889A CN 202210883550 A CN202210883550 A CN 202210883550A CN 115018889 A CN115018889 A CN 115018889A
- Authority
- CN
- China
- Prior art keywords
- image
- visible light
- wave infrared
- registration
- short
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 title claims abstract description 10
- 238000005314 correlation function Methods 0.000 claims description 6
- 238000001228 spectrum Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000003708 edge detection Methods 0.000 claims description 3
- 238000007500 overflow downdraw method Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000001419 dependent effect Effects 0.000 claims description 2
- 238000001514 detection method Methods 0.000 abstract description 3
- 238000003384 imaging method Methods 0.000 abstract description 3
- 238000007499 fusion processing Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/10—Image enhancement or restoration using non-spatial domain filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
- Photometry And Measurement Of Optical Pulse Characteristics (AREA)
Abstract
The invention discloses a short wave infrared and visible light image registration fusion device and a method, wherein a short wave infrared detector and a visible light detector are assembled on the same side, are linked through an angle adjustable mechanism, and are assembled with a horizontal calibrator on the rear side to calibrate shooting levelness; the device can adjust the azimuth angle of the detector according to the distance of the target to be detected, and carry out coarse coincidence registration of the collected images. The rear end of the device is provided with an FPGA module with 2 paths of image input interfaces and 1 path of image output interfaces, the image registration algorithm based on the combination of image texture information and phase correlation is originally invented to reside in the module, the two paths of detectors synchronously input images into the module, and the input images are precisely registered through algorithm software so as to complete image fusion processing. The invention can realize the real-time acquisition and registration fusion of the short wave infrared and visible light images, has simple operation and portable equipment, and is suitable for target imaging and detection in complex weather such as rain, fog and the like.
Description
Technical Field
The invention relates to the technical field of image fusion and image processing, in particular to a short wave infrared and visible light image registration fusion device and method.
Background
When the visible light image is sampled, the visible light image is easily limited by factors such as weather and environment, for example, fog weather and cloudy weather, so that the imaging effect is poor, or the target is hidden and the rear side of the shielding object is shielded. The short-wave infrared image has good fog-penetrating and cloud-penetrating effects and the like, and still has certain detection capability in some extreme environments. Therefore, the characteristics of visible light imaging and short wave infrared imaging are combined at present, and the fusion of image characteristics is a popular technology in the fields of target detection and image enhancement.
Traditional visible light equipment and short wave infrared equipment independence are great, and the image angle difference of gathering is great, and difficult quantization relies on the algorithm to carry out registration fusion completely and avoids appearing the error, influences image quality on the contrary. Secondly, the traditional equipment is inconvenient to carry, cannot realize real-time effect fusion, can only collect data and then guide the data into a computer for fusion processing, and the realization effect is slow.
Disclosure of Invention
In order to solve the problems, the invention discloses a short wave infrared and visible light image registration and fusion device and a method, and firstly solves the problems that the traditional acquisition equipment is large in independence and inconvenient to carry and use outdoors. And secondly, the problems of large error and poor real-time performance of a registration algorithm caused by overlarge image difference between visible light and an infrared sensor are solved. The invention can independently realize synchronous acquisition, registration and fusion and real-time output of the short wave infrared image and the visible light image. The equipment is portable, simple in operation and convenient for the user to operate outdoors.
A short wave infrared and visible light image registration fusion device comprises a left end support seat, an FPGA image processing module and a right end support seat which are sequentially arranged; the visible light sensor and the short wave infrared sensor are positioned at the same side, and are symmetrically arranged at the left end, the right end and the two ends; the left end supporting seat is rotatably connected with the right end supporting seat through an angle adjusting mechanism; a visible light lens and a visible light sensor are respectively arranged on the left end supporting seat; a short-wave infrared lens and a short-wave infrared sensor are respectively arranged on the right support seat; the level meter is positioned in the middle of the device and is arranged at the front end of the FPGA image processing module; one end of the visible light image input interface is connected with the visible light sensor, and the other end of the visible light image input interface is connected with the FPGA image processing module; one end of the short wave infrared image input interface is connected with the short wave infrared sensor, and the other end of the short wave infrared image input interface is connected with the FPGA image processing module.
The angle adjusting mechanism comprises an angle adjusting rotating shaft and a fixed angle knob; the angle adjustment of the left and right end supporting seats is realized through the angle adjusting rotating shaft.
The visible light sensor is connected with the short wave infrared sensor through an angle rotatable device, the dispensing angle of the rotating shaft is 90-180 degrees, and a tightening fixed angle knob is arranged in the middle.
A registration and fusion method for short wave infrared and visible light images comprises the following steps:
s1: adjusting the angle according to the distance of the target to be detected, tightening and fixing to complete the primary registration of image acquisition;
s2: the visible light sensor and the FPGA image processing module are linked, and the short wave infrared detector and the FPGA image processing module are linked;
s3: registering and fusing the two paths of images input into the FPGA by combining an embedded phase correlation algorithm, and calculating a transformation matrix between the two paths of images;
s4: continuously compensating and registering the video sequence images according to the output transformation matrix result;
s6: when the photographing scene is changed, the steps of S1 to S4 are repeated.
S7: and storing/outputting the short wave infrared and visible light registration fused image.
Further, the phase-dependent registration algorithm of S3 includes the following sub-steps:
s31: preprocessing an input image based on an edge detection operator to enhance image texture information;
s32: performing FFT on the two preprocessed images in S31 to transform the two preprocessed images to a frequency domain;
N(u,v)＝M(u,v)e -j2π(u△x+v△y) (1)
wherein N (u, v) and M (u, v) are Fourier transforms of an infrared image N (x, y) and a visible light image M (x, y), respectively;
s33: obtaining two paths of image normalized cross-power spectrum phases according to the frequency domain result of S32;
wherein M is * (u, v) is the complex conjugate of M (u, v);
s34: further carrying out inverse Fourier transform on the phases of the two paths of image normalized cross-power spectrums to obtain a phase correlation function;
c(x,y)＝δ(x-△x,y-△y) (3)
s35: calculating the impulse peak position of the phase correlation function, solving the offset of the two paths of images, and registering;
s36: carrying out weighted processing on the registration image pixel points based on a weighted average method according to the registration result to complete image fusion;
wherein α is the corresponding weight, and
f is the fused image.
The invention has the beneficial effects that:
1) the short wave infrared detector and the visible light detector are connected through the angle adjustable mechanism, so that two paths of images can be simultaneously and stably shot, and manual adjustment and registration are realized; 2) the FPGA image processing module can acquire and process the two-way video and output a registration fusion result in real time; 3) the image sub-pixel level registration is realized by embedding a uniquely realized image texture phase correlation registration processing algorithm in an FPGA module; fusion enhancement of the visible light image and the short wave infrared image can be completed based on a weighted average algorithm; the image processing speed meets the real-time output requirement of 25 fps; 4) the device has the advantages of light volume, portability, simple structure and operation, and is very friendly to users and convenient for outdoor use.
Drawings
FIG. 1 is a flow chart of an embodiment;
FIG. 2 is a flow chart of an algorithm implementation of the present invention;
FIG. 3 is a schematic view of the overall structure of the apparatus of the present invention;
fig. 4 is a side view of the whole structure of the device of the invention.
FIG. 5 is a first axial view of the apparatus of the present invention;
FIG. 6 is a second axis view of the apparatus of the present invention;
FIG. 7, axial three of the inventive apparatus;
fig. 8 shows the axis of the device of the present invention measured four.
List of reference numerals:
the device comprises a 1-visible light lens, a 2-visible light sensor, a 3-visible light image input interface, a 4-level meter, a 5-short wave infrared lens, a 6-short wave infrared sensor, a 7-1-left end support seat, a 7-2-right end support seat, an 8-short wave infrared image input interface, a 9-image processing module and a 10-angle adjusting rotating shaft.
Detailed Description
The present invention will be further illustrated with reference to the accompanying drawings and specific embodiments, which are to be understood as merely illustrative of the invention and not as limiting the scope of the invention. It should be noted that the terms "front," "back," "left," "right," "upper" and "lower" used in the following description refer to directions in the drawings, and the terms "inner" and "outer" refer to directions toward and away from, respectively, the geometric center of a particular component.
As shown in fig. 3-8, the short-wave infrared and visible light fusion image registration and fusion device of the embodiment is characterized by comprising a left support 7-1, an FPGA image processing module 9 and a right support 7-2 which are sequentially arranged; the visible light sensor 2 and the short-red wave external sensor 6 are positioned at the same side, and are symmetrically arranged at the left end, the right end and the two ends; wherein the left end supporting seat 7-1 is rotationally connected with the right end supporting seat 7-2 through an angle adjusting mechanism; the angle adjusting mechanism comprises an angle adjusting rotating shaft 10 and a fixed angle knob.
A visible light lens 1 and a visible light sensor 2 are respectively arranged on the left end supporting seat 7-1; a short red wave external lens 5 and a short red wave external sensor 6 are respectively arranged on the right support seat 7-2; the level meter is positioned in the middle of the device and is arranged at the front end of the FPGA image processing module 9; one end of the visible light image input interface 3 is connected with the visible light sensor 2, and the other end is connected with the FPGA image processing module 9; one end of the short wave infrared image input interface 8 is connected with the short wave infrared sensor 6, and the other end is connected with the FPGA image processing module 9.
As shown in fig. 1-2, in this embodiment, an operator first places the equipment device at the image acquisition site and adjusts the levelness of the mechanism by observing the level.
And secondly, connecting the detector with the FPGA module, connecting the FPGA module with the video monitor, and starting the infrared detector and the visible light detector.
Thirdly, adjusting the angle of the connecting mechanism according to the image sequence returned by the detector, manually carrying out image coarse registration, and finishing manual adjustment when the two paths of images are approximately consistent;
and fourthly, starting an FPGA image processing function, and finishing image registration and fusion based on an FPGA embedded phase correlation algorithm and a weighted average method.
Fifthly, when image acquisition scenes are switched, the sampling contents of the two paths of detectors need to be manually adjusted again, the angle adjusting step is repeated, coarse image registration is carried out, and the fourth step is further repeated, so that the requirements of image acquisition, registration and fusion of different scenes are met.
As shown in fig. 2, the algorithm flow of the present embodiment includes the following sub-steps:
s31: preprocessing an input image based on an edge detection operator to enhance image texture information;
s32: performing FFT on the two preprocessed images in S31 to transform the two preprocessed images to a frequency domain;
N(u,v)＝M(u,v)e -j2π(u△x+v△y) (1)
wherein N (u, v) and M (u, v) are Fourier transforms of an infrared image N (x, y) and a visible light image M (x, y), respectively;
s33: obtaining two paths of image normalized cross-power spectrum phases according to the frequency domain result of S32;
wherein M is * (u, v) is the complex conjugate of M (u, v);
s34: further carrying out inverse Fourier transform on the phases of the two paths of image normalized cross-power spectrums to obtain a phase correlation function;
c(x,y)＝δ(x-△x,y-△y) (3)
s35: calculating the impulse peak position of the phase correlation function, solving the offset of the two paths of images, and registering;
s36: registering image pixel points based on a weighted average method according to the registration result to perform weighted processing, and performing image fusion;
wherein α is the corresponding weight, and
f is the fused image.
The technical means disclosed in the invention scheme are not limited to the technical means disclosed in the above embodiments, but also include the technical scheme formed by any combination of the above technical features.
Claims (5)
1. A short wave infrared and visible light image registration and fusion device is characterized by comprising a left end support seat (7-1), an FPGA image processing module (9) and a right end support seat (7-2) which are sequentially arranged; the visible light sensor (2) and the short wave infrared sensor (6) are positioned at the same side, and are symmetrically arranged at the left end, the right end and the two ends; wherein the left end supporting seat (7-1) is rotationally connected with the right end supporting seat (7-2) through an angle adjusting mechanism; a visible light lens (1) and a visible light sensor (2) are respectively arranged on the left end supporting seat (7-1); a short-wave infrared lens (5) and a short-wave infrared sensor (6) are respectively arranged on the right support seat (7-2); the level meter (4) is positioned in the middle of the device and is arranged at the front end of the FPGA image processing module (9); one end of the visible light image input interface (3) is connected with the visible light sensor (2), and the other end is connected with the FPGA image processing module (9); one end of the short wave infrared image input interface (8) is connected with the short wave infrared sensor (6), and the other end is connected with the FPGA image processing module (9).
2. The short-wave infrared and visible image registration fusion device according to claim 1, characterized in that said angle adjustment mechanism comprises an angle adjustment shaft (10) and a fixed angle knob.
3. The short-wave infrared and visible light image registration and fusion device according to claim 2, characterized in that the adjustment angle of the angle adjustment rotating shaft (10) is 90-180 °.
4. The short-wave infrared and visible light image registration and fusion method according to claim 1, comprising the steps of:
s1: adjusting the angle according to the distance of the target to be detected, tightening and fixing to complete the primary registration of image acquisition;
s2: the visible light sensor (2) and the FPGA image processing module (9) are linked, and the short wave infrared detector and the FPGA image processing module are linked;
s3: registering and fusing the two paths of images input to the FPGA by combining an embedded phase correlation algorithm, and calculating a transformation matrix between the two paths of images;
s4: continuously compensating and registering the video sequence images according to the output transformation matrix result;
s6: when the shooting scene changes, the steps S1 to S4 are repeated.
S7: and storing/outputting the short wave infrared and visible light registration fused image.
5. The short-wave infrared and visible light image registration and fusion method of claim 1, wherein the phase-dependent registration algorithm of S3 comprises the following sub-steps:
s31: preprocessing an input image based on an edge detection operator to enhance image texture information;
s32: performing FFT on the two preprocessed images in S31 to transform the two preprocessed images to a frequency domain;
N(u,v)＝M(u,v)e -j2π(u△x+v△y) (1)
wherein N (u, v) and M (u, v) are Fourier transforms of an infrared image N (x, y) and a visible light image M (x, y), respectively;
s33: obtaining two paths of image normalized cross-power spectrum phases according to the frequency domain result of S32;
wherein M is * (u, v) is the complex conjugate of M (u, v);
s34: further carrying out inverse Fourier transform on the phases of the two paths of image normalized cross-power spectrums to obtain a phase correlation function;
c(x,y)＝δ(x-△x,y-△y) (3)
s35: calculating the impulse peak position of the phase correlation function, solving the offset of the two paths of images, and registering;
s36: performing weighting processing on the registration image pixel points based on a weighted average method according to the registration result to complete image fusion;
wherein α is the corresponding weight, and
f is the fused image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210883550.7A CN115018889B (en) | 2022-07-26 | 2022-07-26 | Short wave infrared and visible light image registration fusion device and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210883550.7A CN115018889B (en) | 2022-07-26 | 2022-07-26 | Short wave infrared and visible light image registration fusion device and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115018889A true CN115018889A (en) | 2022-09-06 |
| CN115018889B CN115018889B (en) | 2024-07-12 |
Family
ID=83081918
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210883550.7A Active CN115018889B (en) | 2022-07-26 | 2022-07-26 | Short wave infrared and visible light image registration fusion device and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115018889B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100045809A1 (en) * | 2008-08-22 | 2010-02-25 | Fluke Corporation | Infrared and visible-light image registration |
| CN102982518A (en) * | 2012-11-06 | 2013-03-20 | 扬州万方电子技术有限责任公司 | Fusion method of infrared image and visible light dynamic image and fusion device of infrared image and visible light dynamic image |
| CN106500852A (en) * | 2016-09-28 | 2017-03-15 | 北方夜视技术股份有限公司 | Infrared and visible light image registration and the System and method for for merging |
| CN109410159A (en) * | 2018-09-11 | 2019-03-01 | 上海创客科技有限公司 | Binocular visible light and infrared thermal imaging complex imaging system, method and medium |
| CN111273134A (en) * | 2020-01-21 | 2020-06-12 | 北方夜视技术股份有限公司 | Ultraviolet and short wave infrared band fused corona monitoring device |
-
2022
- 2022-07-26 CN CN202210883550.7A patent/CN115018889B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100045809A1 (en) * | 2008-08-22 | 2010-02-25 | Fluke Corporation | Infrared and visible-light image registration |
| CN102982518A (en) * | 2012-11-06 | 2013-03-20 | 扬州万方电子技术有限责任公司 | Fusion method of infrared image and visible light dynamic image and fusion device of infrared image and visible light dynamic image |
| CN106500852A (en) * | 2016-09-28 | 2017-03-15 | 北方夜视技术股份有限公司 | Infrared and visible light image registration and the System and method for for merging |
| CN109410159A (en) * | 2018-09-11 | 2019-03-01 | 上海创客科技有限公司 | Binocular visible light and infrared thermal imaging complex imaging system, method and medium |
| CN111273134A (en) * | 2020-01-21 | 2020-06-12 | 北方夜视技术股份有限公司 | Ultraviolet and short wave infrared band fused corona monitoring device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115018889B (en) | 2024-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Schechner et al. | Generalized mosaicing: Wide field of view multispectral imaging | |
| CN106525238B (en) | A kind of satellite-borne multispectral imaging system design method based on super-resolution rebuilding | |
| CN105447850B (en) | A kind of Panoramagram montage synthetic method based on multi-view image | |
| CN106500852B (en) | It is infrared with visible light image registration with the System and method for that merges | |
| Suzuki et al. | Initial inflight calibration for Hayabusa2 optical navigation camera (ONC) for science observations of asteroid Ryugu | |
| Anderson et al. | Ground-based CCD astrometry with wide field imagers-I. Observations just a few years apart allow decontamination of field objects from members in two globular clusters | |
| CN113701885B (en) | Off-axis three-reflector full-spectrum polarized spectrum imaging detection device | |
| CN109919911B (en) | Mobile three-dimensional reconstruction method based on multi-view photometric stereo | |
| US20030103063A1 (en) | Panoramic imaging and display system with canonical magnifier | |
| WO2022042350A1 (en) | Target sky area image acquisition method and astrophotography device | |
| CN111612693B (en) | Method for correcting rotary large-width optical satellite sensor | |
| CN105182678A (en) | System and method for observing space target based on multiple channel cameras | |
| CN109618087A (en) | A kind of infrared and low-light fusion night vision device having precision target positioning function | |
| CN103905746A (en) | Method and device for localization and superposition of sub-pixel-level image offset and video device | |
| CN111413279B (en) | Video processing method and device for multispectral detection and multispectral detection terminal | |
| CN107135336B (en) | A kind of video camera array | |
| CN111524196B (en) | In-orbit geometric calibration method for sweep large-width optical satellite | |
| CN114331851B (en) | Method for generating simulated airborne fire control radar SAR image based on DEM data | |
| CN115018889B (en) | Short wave infrared and visible light image registration fusion device and method | |
| CN106558036B (en) | A kind of spaceborne super-resolution imaging design method | |
| KR101830296B1 (en) | System for drawing digital map | |
| Hsu et al. | CrossbowCam: a handheld adjustable multi-camera system | |
| Hagolle et al. | POLDER level-1 processing algorithms | |
| JP5499363B2 (en) | Image input device, image input method, and image input program | |
| Abramov et al. | Ground testing of the landing platform television system of the ExoMars-2022 spacecraft |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |