CN117036219A - Ultrahigh-definition visible and infrared light image real-time fusion system and method based on ZYNQ MPSoC - Google Patents
Ultrahigh-definition visible and infrared light image real-time fusion system and method based on ZYNQ MPSoC Download PDFInfo
- Publication number
- CN117036219A CN117036219A CN202311011170.5A CN202311011170A CN117036219A CN 117036219 A CN117036219 A CN 117036219A CN 202311011170 A CN202311011170 A CN 202311011170A CN 117036219 A CN117036219 A CN 117036219A
- Authority
- CN
- China
- Prior art keywords
- image
- module
- infrared light
- visible light
- visible
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000006243 chemical reaction Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims abstract description 10
- 238000001931 thermography Methods 0.000 claims abstract description 10
- 239000000872 buffer Substances 0.000 claims description 17
- 238000001514 detection method Methods 0.000 claims description 8
- 238000005070 sampling Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 6
- 238000013461 design Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000007500 overflow downdraw method Methods 0.000 claims 4
- 238000009825 accumulation Methods 0.000 claims 1
- 230000000007 visual effect Effects 0.000 claims 1
- 230000008901 benefit Effects 0.000 abstract description 3
- 230000005855 radiation Effects 0.000 abstract 1
- 238000001914 filtration Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 3
- 230000003139 buffering effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000630 rising effect Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000000087 stabilizing effect Effects 0.000 description 2
- XHCLAFWTIXFWPH-UHFFFAOYSA-N [O-2].[O-2].[O-2].[O-2].[O-2].[V+5].[V+5] Chemical compound [O-2].[O-2].[O-2].[O-2].[O-2].[V+5].[V+5] XHCLAFWTIXFWPH-UHFFFAOYSA-N 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 229910001935 vanadium oxide Inorganic materials 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20032—Median filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a ZYNQ MPSoC-based ultra-high definition visible and infrared light image real-time fusion system and a method thereof, wherein the fusion system comprises a 4k ultra-high definition visible light acquisition module, a micro thermal imaging module and an image processing module. The 4k ultra-high definition visible light acquisition module is used for acquiring visible light image data and outputting 4k@60fps ultra-high definition video. The miniature thermal imaging module is a long-wave infrared miniature thermal imaging module and can convert the thermal radiation of an object into image and temperature data. The image processing module processes and fuses the acquired visible and infrared light image data. The invention adopts the FPGA parallel realization and the infrared high-resolution conversion of the fusion algorithm, so that the system has the advantages of ultra-high resolution and real-time.
Description
Technical Field
The invention relates to a ZYNQ MPSoC-based ultra-high definition visible and infrared light image real-time fusion system and method, and belongs to the technical field of multispectral fusion imaging.
Background
The visible and infrared light image fusion technology makes full use of the advantages and complementarity of the infrared and visible light imaging technologies, effectively improves the space range and detection probability of target detection, becomes the research focus of each country in the military and civil fields, and makes breakthrough progress in the research work of fusion systems. With the rapid development of image sensor technology, the quality requirements of various fields on video image information are gradually strict. The need for high resolution, high fidelity, high frame rate, high dynamic range video images is increasing.
Most of the existing visible light and infrared light fusion systems are dual-processor fusion systems based on ARM and FPGA, although the defects of a single-processor scheme can be overcome. But the integration of the system, the PCB design, and the data transfer between the multiple processors present challenges. Meanwhile, due to the limitation of transmission bandwidth and the complexity of an image fusion algorithm, most of the systems have the defects of low resolution and frame rate, poor real-time performance and the like, and cannot meet the requirements of various fields on high resolution, high frame rate and real-time video images.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the system and the method for fusing the ultra-high definition visible and infrared light images in real time based on ZYNQ MPSoC are provided, the FPGA of the fusion algorithm is adopted to realize parallel and infrared high-resolution transformation, the pixel-level real-time fusion of the visible and infrared light images under the ultra-high definition resolution is realized, and the video image output requirements of ultra-high definition, high frame rate and real-time are met.
The invention adopts the following technical scheme for solving the technical problems:
the system comprises a visible light acquisition module, an infrared light acquisition module and an image processing module; the image processing module comprises a video acquisition IP core module and a video processing IP core module; the video acquisition IP core module comprises a visible light video acquisition IP core module and an infrared video image acquisition IP core module, wherein the visible light video acquisition IP core module comprises a CMOS control module, an LVDS deserializing module and a RAW2RGB conversion module, and the infrared video image acquisition IP core module comprises an I2C control module, a YUV2RGB color gamut conversion module and an infrared image super-resolution module; the video processing IP core module comprises a video frame buffer module and an image fusion module;
the visible light acquisition module is used for acquiring a visible light image video of the target detection area, and the infrared light acquisition module is used for acquiring an infrared light image video of the target detection area;
the CMOS control module is used for configuring a register in the visible light acquisition module, the LVDS deserializing module is used for converting serial image data signals output by the visible light acquisition module into parallel signals through an anti-deflection algorithm, and the RAW2RGB conversion module is used for converting gray image information output by the visible light acquisition module into color image information through a bilinear interpolation algorithm;
the I2C control module is used for configuring a register in the infrared light acquisition module, the YUV2RGB color gamut conversion module is used for converting YUV422 format pixel information output by the infrared light acquisition module into RGB888 format, and the infrared image super-resolution module is used for converting the resolution of an image output by the YUV2RGB color gamut conversion module from 256×192 to 3840×2160 with ultra-high definition resolution;
the video frame buffer module buffers the image data acquired by the visible light video acquisition IP core module and the infrared video image acquisition IP core module into an external memory DDR, and reads out the buffered image data in the DDR through a frame synchronizing signal; the image fusion module is used for fusing the visible light image and the infrared light image read out by the video frame buffer module to obtain a fused image.
As a preferable scheme of the system, the visible light acquisition module comprises a CMOS image sensor with a target surface size of 1 inch and an effective pixel of 2000 ten thousand, and is used for providing working voltage and a printed board for level conversion for the CMOS image sensor, wherein the CMOS image sensor adopts a SONYIMX183CQJ sensor, and the printed board adopts a 4-layer board design and performs impedance matching.
As a preferable scheme of the system, the infrared light acquisition module adopts a Tiny1-C miniature thermal imaging module, and the resolution of an output image is 256 multiplied by 192.
As a preferable scheme of the system, the working process of the visible light video acquisition IP core module is as follows:
(1) The CMOS control module configures a register of the CMOS image sensor through the SPI bus, so that the CMOS image sensor is converted into a normal working mode from a power-on standby mode;
(2) After the CMOS image sensor works normally, the LVDS deserializing module deserializes the output serial data into parallel 10bit RAW data through an anti-skew algorithm; the specific implementation process is as follows:
step1, a differential clock output by the CMOS image sensor is routed to an internal clock area of the FPGA core through an input pin pair with a clock function in the FPGA core I/O bank without using input delay, and then the differential clock is output in two ways: one path of the clock is output to a mixed mode clock manager in the LVDS deserializing module and is used as a reference clock of a subsequent module; one path of the initial training data is output to a delay unit and a clock serial-parallel conversion unit in the LVDS deserializing module to be used as initial delay setting required by a data bit receiver;
step2, inputting differential data signals output by the CMOS image sensor into an LVDS deserializing module, respectively inputting the differential data signals into a master serial-to-parallel conversion unit and a slave serial-to-parallel conversion unit after passing through a differential input and differential output buffer and a delay unit, inputting parallel data output by the master serial-to-parallel conversion unit into a user logic and control state machine as output of the LVDS deserializing module, and only using the parallel data output by the slave serial-to-parallel conversion unit by the state machine;
step3, transmitting the initial delay calculated in Step1 to a delay unit on the data line, ensuring that the sampling point is positioned near the center of the eye diagram, and enabling an anti-skew algorithm to only be used for fine-tuning each data line from the sampling point, positioning the acquisition point of the main data line near the center of the eye diagram by setting the initial delay of the main data line and the auxiliary data line, wherein the acquisition point of the auxiliary data line is different from the acquisition point of the main data line by a half bit period;
the Step4 anti-skew algorithm increases or reduces delay by comparing data acquired from the same time of the main data line and the slave data line, and when the data acquired from the main data line and the slave data line which differ by half a bit period are the same, the acquisition point is too late, and input delay is required to be reduced; when the acquired data are different, the acquisition point is too early, and input delay is required to be increased;
(3) The RAW2RGB conversion module adopts bilinear interpolation algorithm to convert the RAW data output by the LVDS deserializing module into RGB data.
As a preferable scheme of the system, the working process of the infrared video image acquisition IP core module is as follows:
(1) The I2C control module configures registers in Tiny1-C through an I2C bus to enable the Tiny1-C to enter a working mode;
(2) The YUV2RGB color gamut conversion module converts the YUV422 format of the Tiny1-C output to the RGB888 format by the following conversion formula:
wherein,values representing R, G, B three channels contained in one pixel in the RGB gamut space, +.>A value representing luminance information Y and color difference information U, V contained in one pixel in the YVU color gamut space;
(3) The infrared image super-resolution module expands the image resolution output by the YUV2RGB color gamut conversion module from 256×192 to 3840×2160 through bilinear interpolation algorithm.
The method for fusing the ultra-high definition visible and infrared light images based on ZYNQ MPSoC in real time is realized based on the fusion system and comprises the following steps:
step1, respectively acquiring visible light images and infrared light images of a target detection area at the same moment and preprocessing the visible light images and the infrared light images so that the sizes of the preprocessed visible light images and infrared light images are the same;
step2, decomposing the visible light image and the infrared light image with the same size into a basic layer containing large scale change and a detail layer containing small scale change through an average filter;
step3, taking the visible light image and the infrared light image with the same size as a source image, smoothing the source image through a mean value filter, removing noise and artifacts in the source image through a median value filter, and finally obtaining the saliency information of edges and lines through taking the difference between the mean value and the median value filter;
step4, calculating a fusion weight coefficient of the detail layer according to the significance information obtained in the step3, and linearly fusing the detail layers of the visible light image and the infrared light image according to the calculated fusion weight coefficient to obtain a fused detail layer image;
and 5, fusing the base layers of the visible light image and the infrared light image by adopting an average fusion rule to obtain a fused base layer image, and linearly combining the fused detail layer image and the fused base layer image to obtain a fused image.
As a preferable scheme of the method, the specific process of the step2 is as follows:
the visible light image and the infrared light image with the same size are decomposed into a base layer containing large-scale variation by adopting the following formula:
wherein (x, y) represents the coordinates of the pixel point on the two-dimensional image plane, phi 1 (x, y) is a visible light image, phi 2 (x, y) is an infrared light image,is a base layer of visible light, +.>For the base layer of infrared light, μ (x, y) represents the window size w μ Represents a convolution operation;
since the value of the average filter template is 1, it is taken into the above equation:
after the mean filter template is slid to the next pixel, the above equation is expressed as:
wherein,for the sum of all visible pixels in the template before sliding the mean filter template, +.>For the summation of the first row of visible pixels in the filter template sliding front template, +.> For the summation of the last row of visible pixels in the template after the filter template is slid,for the sum of all infrared pixels in the template before the mean filter template is slid,for the summation of the first row of infrared pixels in the filter template before sliding the template,the sum of the last row of infrared pixels in the template after the filter template slides is added;
the detail layer is extracted by subtracting the base layer from the source image using the following formula:
wherein,is a detail layer of visible light, +.>Is a detail layer of infrared light.
As a preferred scheme of the method of the present invention, the saliency information of the edge and the line in the step3 is as follows:
wherein, the absolute value of the difference value is taken, and the value is xi 1 (x, y) is the saliency information of the edges and lines of the visible image, ζ 2 (x, y) is saliency information of the infrared light image edges and lines,the size of the passing window for visible light image is w μ Is filtered by an averaging filter, +.>For infrared light image, the size of the passing window is w μ Is filtered by an averaging filter, +.>The size of the passing window for visible light image is w η Is filtered by a median filter, +.>The size of the passing window for visible light image is w η (x, y) representing coordinates of pixel points on the two-dimensional image plane.
As a preferred scheme of the method of the present invention, the specific formula of the fusion weight coefficient in the step4 is as follows:
wherein, psi is 1 (x, y) is the fusion weight coefficient of the detail layer of visible light, ψ 2 (x, y) is the fusion weight coefficient of the detail layer of infrared light, ζ 1 (x, y) is the saliency information of the edges and lines of the visible image, ζ 2 (x, y) is saliency information of the infrared light image edges and lines;
the specific formula of the linear fusion is as follows:
wherein phi is D (x, y) is the fused detail layer image,is a detail layer of visible light, +.>Is a detail layer of infrared light.
As a preferred scheme of the method of the present invention, the specific formula of the fused base layer image in the step5 is as follows:
wherein phi is B (x, y) is the fused base layer image,is a base layer of visible light, +.>A base layer that is infrared light; the specific formula of the fused image is as follows:
γ(x，y)＝φ B (x，y)+φ D (x，y)
wherein, gamma (x, y) is a fusion image phi D And (x, y) is the fused detail layer image.
Compared with the prior art, the technical scheme provided by the invention has the following technical effects:
aiming at the problems of poor real-time performance of the existing visible and infrared light image fusion system and transmission bandwidth limitation of a multiprocessor system, the invention adopts the FPGA parallel implementation of a fusion algorithm to solve the problem of poor real-time performance of the fusion system, solves the problems of insufficient single processor scheme and complexity and data bandwidth limitation of multiprocessor scheme design by selecting a ZYNQ MPSoC chip, carries out pixel-level real-time fusion on visible light and infrared light images under ultra-high definition resolution, and realizes the video image output with ultra-high definition, high frame rate and real-time performance.
Drawings
FIG. 1 is a schematic diagram of the fusion system of the present invention;
FIG. 2 is a three-dimensional view of a PCB of the ultra-high definition visible light acquisition module of the present invention, wherein (a) is the front side of the PCB, including an IMX183CQJ image sensor, and (b) is the back side of the PCB, including three paths of linear voltage stabilizing circuits;
FIG. 3 is a flow chart of an anti-skew algorithm in the present invention;
FIG. 4 is a flow chart of a fusion algorithm implemented by the FPGA kernel in the invention;
FIG. 5 is a flow chart of the mean value filtered data buffering in the FPGA core of the present invention;
FIG. 6 is a flow chart of mean filter computation in the FPGA core in the present invention;
FIG. 7 is a fused image of the FPGA core output in the present invention.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below by referring to the drawings are exemplary only for explaining the present invention and are not to be construed as limiting the present invention.
The invention provides a ZYNQ MPSoC-based visible light and infrared ultra-high definition real-time image fusion system, which comprises an ultra-high definition visible light acquisition module, a micro thermal imaging module and an image processing module as shown in figure 1:
(1) The ultra-high definition visible light acquisition module is designed independently, and adopts a CMOS image sensor developed by SONY, and the specific model is IMX183CQJ. The target surface size of IMX183CQJ was 1 inch, the pixel size was 2.4 μm×2.4 μm, the maximum resolution that could be supported at 20.48M effective pixels was 5472×3648@25fps, while 4096×2160@60fps could be achieved at 9.03M effective pixels and various resolution sizes could be achieved by clipping. The camera uses a single power +3.3V input with three linear voltage regulator modules (LDOs) inside to generate the CMOS operating voltage. The PCB was of a 4-layer design and impedance matched, 40mm by 40mm in size. And the power supply and the signal transmission are connected with the FPGA core through a 50pin FPC flexible flat cable. As shown in fig. 2 (a) and (b), a PCB is three-dimensional. In the figure, the serial number 1 is an IMX183CQJ image sensor, and the serial numbers 2, 3 and 4 are linear voltage stabilizing modules (LDOs). Wherein sequence number 2 is used to convert 3.3V voltage to 2.8V; sequence number 3 is used to convert 3.3V to 1.2V. The serial numbers 2 and 3 are MP20051DN, which is an adjustable low noise LDO produced by MPS company. Serial No. 4 is used to convert 3.3V voltage to 1.8V, model SPX3819M5, a low noise LDO manufactured by Ai Kejia (EXAR).
(2) The miniature thermal imaging module is a Tiyl-C miniature thermal imaging module which is researched and developed by Ai Ruiguang electricity. The module is a long-wave infrared micro thermal imaging module, adopts a domestic vanadium oxide uncooled infrared focal plane detector, has the pixel spacing of 12 mu m, the spectrum range of 8-14 mu m and the resolution of 256 multiplied by 192@25fps.
(3) And the image processing module is a Zynq UltraScale+MPSoC processor manufactured by Xilinx company. The processor adopts a heterogeneous architecture, integrates an ARM core and an FPGA core into one chip, performs data interaction between the ARM core and the FPGA core through an AXI4 bus, and is used for running a Linux operating system to transmit acquired videos through a UDP protocol. With the advantage, high-bandwidth data transmission between ARM and FPGA cores can be realized through an internal bus, and the limitation of data bandwidth among processors in the multiprocessor fusion system is solved.
The ultra-high definition visible light acquisition module and the micro thermal imaging module are mounted on an FPGA kernel of the image processing module. The FPGA kernel comprises a video acquisition IP kernel module and a video processing IP kernel module. The video acquisition IP core module is used for acquiring image data output by the visible light sensor and the infrared light sensor and converting the image data into RGB format; the video processing IP core module is used for caching, frame synchronization and pixel-level fusion of the image data output by the video acquisition IP core module. The video acquisition IP core module comprises a visible light video acquisition IP core module and an infrared video image acquisition IP core module.
The FPGA kernel visible light video acquisition IP core comprises: the device comprises a CMOS control module, an LVDS deserializing module and a RAW2RGB conversion module. The CMOS control module is used for configuring a register in the CMOS sensor to enter a normal working state. The LVDS deserializing module converts serial image data signals output by the CMOS into parallel signals through an anti-skew algorithm. The anti-skew algorithm adjusts the sampling delay by comparing two master and slave sample data that differ by half a bit period, thereby achieving real-time correction of the sampling time. The RAW2RGB conversion module converts the gray image information output by the CMOS into color image information through a bilinear interpolation algorithm. The bilinear interpolation algorithm obtains color components represented by each pixel point in the 3×3 matrix by selecting a 3×3 matrix on the image plane and then obtaining R, G, B three-channel information of the middle pixel point by adding and averaging the same color components according to the arrangement mode of the CMOS color filters.
The visible light video acquisition IP core work flow is as follows:
(1) The CMOS control module configures the register of the CMOS through the SPI bus to enable the register to be converted into a normal working mode from a power-on standby mode.
(2) When the CMOS works normally, the LVDS deserializing module deserializes the output serial data into parallel 10bit RAW data through an anti-skew algorithm. The LVDS deserializing module deserializes the output serial data into parallel 10bit RAW data through an anti-skew algorithm, and the specific implementation steps are shown in fig. 3:
step1 routes the differential clock output by the CMOS to the internal clock area of the FPGA core without using input delay through the input pin pair with clock function in the FPGA core I/O bank. And then output in two paths. One path is output to a Mixed Mode Clock Manager (MMCM) in the LVDS deserializing module and is used as a reference clock of a subsequent module. One way is output to a delay unit (IDELAYE 3) and a serial-parallel conversion unit (ISERDSE 3) inside the LVDS deserializing module as initial training data to determine the initial delay setting required by the data bit receiver.
Step2 inputs the differential data signal outputted by the CMOS into the LVDS deserializing module, and then respectively inputs the differential data signal into the master and slave isdse 3 through the differential input and differential output buffer (ibufds_diff_out) and the delay unit. The parallel data output by the main ISERDSE3 is input into a user logic and control state machine as the output of the LVDS deserializing module, and the parallel data output from the ISERDSE3 is only used by the state machine.
Step3 transmits the initial delay calculated in Step1 to IDELAYE3 on the data line. This ensures that the sampling point is located near the center of the eye and the deskewing algorithm is used only to fine tune each data line starting from that point. By setting the initial delay of the master and slave data lines, the acquisition point of the master data line is positioned near the center of the eye pattern, and the acquisition point of the slave data line is different from the acquisition point of the master data line by a half bit period.
The Step4 deskew algorithm increases or decreases the delay by comparing the data acquired from the main data line and the slave data line at the same time. When the data collected from the data lines are the same, which differ by half a bit period, it is indicated that the collection point is too late, and the input delay should be reduced; when the acquired data is different, indicating that the acquisition point is too early, the input delay should be increased.
(3) The RAW2RGB conversion module adopts bilinear interpolation algorithm to convert the RAW data output by the LVDS deserializing module into RGB data.
The infrared light video acquisition IP core in the FPGA core comprises: a 12C control module, a YUV2RGB color gamut conversion module and an infrared image super-resolution module. The 12C control module is used for configuring a register in the infrared sensor to enter a normal working state. The YUV2RGB color gamut conversion module is used for converting YUV422 format image output by the infrared sensor into RGB888 format image. The infrared image super-resolution module is used for stretching the video image with the resolution of 256 multiplied by 192 output by the YUV2RGB color gamut conversion module into 3840 multiplied by 2160 through a bilinear interpolation algorithm.
The work flow of the infrared light video acquisition IP core is as follows:
(1) The I2C control module configures registers in Tiny1-C through the I2C bus to enter a working mode.
(2) The YUV2RGB color gamut conversion module converts the YUV422 format to the RGB format output by Tiny1-C by the following conversion formula:
(3) The infrared image super-resolution module expands the resolution of the Tiny1-C output into 3840×2160 images with 256×192 by bilinear interpolation (Bilinear Interpolation) algorithm.
The video processing IP core module in the FPGA core comprises: and the video frame buffer module and the image fusion module. The video frame buffer module is used for buffering the image data acquired by the visible light video acquisition IP core module and the infrared video image acquisition IP core module into the DDR, and then reading out the image data buffered in the DDR through the frame synchronizing signal. The image fusion module performs pixel-level image fusion on the two images through a double-scale image fusion algorithm.
The video processing IP core module work flow is as follows:
the video frame buffer module buffers visible light and infrared light images into an external DDR memory through an AXI bus, and the specific flow is as follows:
after the rising edge of the video frame synchronization signal of the Step1 write channel arrives, the write channel first-in first-out queue (First Input First Output, FIFO) is reset, waiting for valid pixel data to be written.
Step2, when the length of the data written in the FIFO is greater than the first BURST of the AXI bus, the AXI bus starts to transmit, and the image data is cached in the DDR.
After the writing of one frame of data is finished, the writing channel enters a waiting state, waits for the rising edge of a frame synchronizing signal of the next frame to come, and starts the buffer operation of the next frame.
The Step4 read channel and the write channel are operated in parallel, when the rising edge of the video frame synchronizing signal of the read channel arrives, the read channel FIFO is reset, and when the data length in the FIFO is smaller than one BURST of the AXI bus, the buffered image data in the DDR is read. After one frame of data is read, waiting for the arrival of a frame synchronizing signal of the next frame, and starting the reading operation of the next frame.
The fusion module is concretely implemented in the FPGA kernel as follows:
step1 is a detailed description of one way because the visible and infrared images are symmetrical when running the two-scale image fusion algorithm. Fig. 4 is a flow chart of one path of image data, wherein pix_data is pixel data output by the video buffering module. The pixel data can obtain gray information of the pixel after passing through a module for converting RGB into YUV, and the gray information is denoted by Y.
After Step2 obtains the gray information Y, the gray information Y is input into a line_buffer, where 34 synchronous FIFO primitive modules are instantiated by using the UltraRAM resources in the FPGA kernel, and a specific implementation flow is shown in fig. 5.
The Step3 mean filter template is shown in fig. 6, with a window size of 35 and the filter values of all 1. In the solving process of realizing the mean filtering, the FPGA kernel firstly accumulates and registers pixel values in a window according to rows, then accumulates and registers the sum of each row, and finally carries out division operation on the sum. After the filter translates one pixel, the red box area in fig. 6, where the first line moves out of the window and the 36 th line moves into the window. At this time, only the sum of 36 th row is calculated, the sum of 1 st row is subtracted from the registered sum, and the sum of 36 th row is added, namely the sum of all pixels in the current filtering window, and finally division operation is performed to obtain the current filtering result.
Step4 obtains pixel information (b_layer) of the base layer through mean value filtering. The detail layer data (d_layer) is obtained by subtracting the base layer data (b_layer) from the current pixel data Y. For the median filter, the window size is 3, so that only the first three data of the first three rows are needed to be taken for summation and average, and finally the fusion weight (a) of the detail layer is obtained by making a difference with the median filter. Because the FPGA kernel is complex in realizing floating point operation, the weight value a does not perform normalization operation.
Step5 obtains the b_layer, d_layer and the weight value a of the current input pixel through the above three steps, and then calculates the fusion result of the current pixel, as shown in fig. 7.
The image fusion module performs pixel-level image fusion on the two images through a double-scale image fusion algorithm. The method comprises the following specific steps:
step1 firstly, solving two visible and infrared light images with the same size into a basic layer containing large scale change and a detail layer containing small scale change through an average filter, wherein the following formula is adopted:
wherein (x, y) represents the coordinates of the pixel point on the two-dimensional image plane, phi 1 (x, y) is a visible light image, phi 2 (x, y) is an infrared light image,is a base layer of visible light, +.>For the base layer of infrared light, μ (x, y) represents the window size w μ Represents a convolution operation.
Since the value of the average filter template is 1, it is taken into the above equation:
after the mean filter template is slid to the next pixel, the above equation can be expressed as:
wherein,for the sum of all pixels before the filter template is slid,for the first row of summation before the filter template is slid,/->Which is the sum of the last row after the filter template slides. When the FPGA design is carried out, the filter is arranged in the templateThe sum and the sum of each row of the filter template are buffered, and the base layer can be obtained only by calculating the sum of the last row of the filter template after the filter template slides.
The detail layer is extracted by subtracting the base layer from the source image using the following formula:
wherein the method comprises the steps ofIs a detail layer.
Step2 performs smoothing processing on each source image through a mean filter, and simultaneously removes noise and artifacts in the source image through a median filter. Finally, the difference between the average value and the median filtering is taken to obtain the saliency information of the edge and the line, and the specific formula is as follows:
ξ(x，y)＝|φ μ (x，y)-φ η (x，y)|
where || denotes the absolute value of the difference value, φ μ For a window size of w μ And phi is the output of the averaging filter η For a window size of w η Is provided for the output of the median filter of (c).
Step3 calculates a fusion weight coefficient of the detail layer according to the significance information obtained in Step2, wherein the specific formula is as follows:
wherein psi is 1 Is thatIs a fusion weight coefficient of phi 2 Is->Is used for fusing the weight coefficients.
Step4, carrying out linear fusion on the detail layers according to the fusion weight system calculated in Step3, wherein the formula is as follows:
wherein phi is D And the detail layer images are fused.
Step5 adopts an average fusion rule for the base layer, and the formula is as follows:
wherein phi is B Is the fused base layer image.
Step6 finally obtains a fused image through linear combination of the base layer and the detail layer, and the formula is as follows:
γ(x，y)＝φ B (x，y)+φ D (x，y)
the above embodiments are only for illustrating the technical idea of the present invention, and the protection scope of the present invention is not limited thereto, and any modification made on the basis of the technical scheme according to the technical idea of the present invention falls within the protection scope of the present invention.
Claims (10)
1. The ZYNQ MPSoC-based ultra-high definition visible and infrared light image real-time fusion system is characterized by comprising a visible light acquisition module, an infrared light acquisition module and an image processing module; the image processing module comprises a video acquisition IP core module and a video processing IP core module; the video acquisition IP core module comprises a visible light video acquisition IP core module and an infrared video image acquisition IP core module, wherein the visible light video acquisition IP core module comprises a CMOS control module, an LVDS deserializing module and a RAW2RGB conversion module, and the infrared video image acquisition IP core module comprises an I2C control module, a YUV2RGB color gamut conversion module and an infrared image super-resolution module; the video processing IP core module comprises a video frame buffer module and an image fusion module;
the visible light acquisition module is used for acquiring a visible light image video of the target detection area, and the infrared light acquisition module is used for acquiring an infrared light image video of the target detection area;
the CMOS control module is used for configuring a register in the visible light acquisition module, the LVDS deserializing module is used for converting serial image data signals output by the visible light acquisition module into parallel signals through an anti-deflection algorithm, and the RAW2RGB conversion module is used for converting gray image information output by the visible light acquisition module into color image information through a bilinear interpolation algorithm;
the I2C control module is used for configuring a register in the infrared light acquisition module, the YUV2RGB color gamut conversion module is used for converting YUV422 format pixel information output by the infrared light acquisition module into RGB888 format, and the infrared image super-resolution module is used for converting the resolution of an image output by the YUV2RGB color gamut conversion module from 256×192 to 3840×2160 with ultra-high definition resolution;
the video frame buffer module buffers the image data acquired by the visible light video acquisition IP core module and the infrared video image acquisition IP core module into an external memory DDR, and reads out the buffered image data in the DDR through a frame synchronizing signal; the image fusion module is used for fusing the visible light image and the infrared light image read out by the video frame buffer module to obtain a fused image.
2. The real-time ultra-high definition visible and infrared light image fusion system based on ZYNQ MPSoC as claimed in claim 1, wherein the visible light acquisition module comprises a CMOS image sensor with a target surface size of 1 inch and an effective pixel of 2000 ten thousand, and is used for providing working voltage and a printed board for level conversion for the CMOS image sensor, wherein the CMOS image sensor adopts a SONY IMX183CQJ sensor, and the printed board adopts a 4-layer board design and performs impedance matching.
3. The ZYNQ MPSoC-based ultra-high definition visual and infrared light image real-time fusion system according to claim 1, wherein the infrared light acquisition module is a Tiny1-C micro thermal imaging module, and the resolution of the output image is 256×192.
4. The real-time fusion system of ultra-high definition visible and infrared light images based on ZYNQ MPSoC as claimed in claim 2, wherein the working process of the visible light video acquisition IP core module is as follows:
(1) The CMOS control module configures a register of the CMOS image sensor through the SPI bus, so that the CMOS image sensor is converted into a normal working mode from a power-on standby mode;
(2) After the CMOS image sensor works normally, the LVDS deserializing module deserializes the output serial data into parallel 10bit RAW data through an anti-skew algorithm; the specific implementation process is as follows:
step1, a differential clock output by the CMOS image sensor is routed to an internal clock area of the FPGA core through an input pin pair with a clock function in the FPGA core I/O bank without using input delay, and then the differential clock is output in two ways: one path of the clock is output to a mixed mode clock manager in the LVDS deserializing module and is used as a reference clock of a subsequent module; one path of the initial training data is output to a delay unit and a clock serial-parallel conversion unit in the LVDS deserializing module to be used as initial delay setting required by a data bit receiver;
step2, inputting differential data signals output by the CMOS image sensor into an LVDS deserializing module, respectively inputting the differential data signals into a master serial-to-parallel conversion unit and a slave serial-to-parallel conversion unit after passing through a differential input and differential output buffer and a delay unit, inputting parallel data output by the master serial-to-parallel conversion unit into a user logic and control state machine as output of the LVDS deserializing module, and only using the parallel data output by the slave serial-to-parallel conversion unit by the state machine;
step3, transmitting the initial delay calculated in Step1 to a delay unit on the data line, ensuring that the sampling point is positioned near the center of the eye diagram, and enabling an anti-skew algorithm to only be used for fine-tuning each data line from the sampling point, positioning the acquisition point of the main data line near the center of the eye diagram by setting the initial delay of the main data line and the auxiliary data line, wherein the acquisition point of the auxiliary data line is different from the acquisition point of the main data line by a half bit period;
the Step4 anti-skew algorithm increases or reduces delay by comparing data acquired from the same time of the main data line and the slave data line, and when the data acquired from the main data line and the slave data line which differ by half a bit period are the same, the acquisition point is too late, and input delay is required to be reduced; when the acquired data are different, the acquisition point is too early, and input delay is required to be increased;
(3) The RAW2RGB conversion module adopts bilinear interpolation algorithm to convert the RAW data output by the LVDS deserializing module into RGB data.
5. The system for fusing ultra-high definition visible and infrared light images in real time based on ZYNQ MPSoC as claimed in claim 3, wherein the working process of the infrared video image acquisition IP core module is as follows:
(1) The I2C control module configures registers in Tiny1-C through an I2C bus to enable the Tiny1-C to enter a working mode;
(2) The YUV2RGB color gamut conversion module converts the YUV422 format of the Tiny1-C output to the RGB888 format by the following conversion formula:
wherein,values representing R, G, B three channels contained in one pixel in the RGB gamut space, +.>Representing a pixel in YVU gamut spaceThe values of the luminance information Y and the color difference information U, V included;
(3) The infrared image super-resolution module expands the image resolution output by the YUV2RGB color gamut conversion module from 256×192 to 3840×2160 through bilinear interpolation algorithm.
6. The method for fusing the ultra-high definition visible and infrared light images in real time based on ZYNQ MPSoC is realized based on the fusion system of any one of claims 1 to 5, and is characterized by comprising the following steps:
step1, respectively acquiring visible light images and infrared light images of a target detection area at the same moment and preprocessing the visible light images and the infrared light images so that the sizes of the preprocessed visible light images and infrared light images are the same;
step2, decomposing the visible light image and the infrared light image with the same size into a basic layer containing large scale change and a detail layer containing small scale change through an average filter;
step3, taking the visible light image and the infrared light image with the same size as a source image, smoothing the source image through a mean value filter, removing noise and artifacts in the source image through a median value filter, and finally obtaining the saliency information of edges and lines through taking the difference between the mean value and the median value filter;
step4, calculating a fusion weight coefficient of the detail layer according to the significance information obtained in the step3, and linearly fusing the detail layers of the visible light image and the infrared light image according to the calculated fusion weight coefficient to obtain a fused detail layer image;
and 5, fusing the base layers of the visible light image and the infrared light image by adopting an average fusion rule to obtain a fused base layer image, and linearly combining the fused detail layer image and the fused base layer image to obtain a fused image.
7. The real-time fusion method of ultra-high definition visible and infrared light images based on ZYNQ MPSoC as claimed in claim 6, wherein the specific process of the step2 is as follows:
the visible light image and the infrared light image with the same size are decomposed into a base layer containing large-scale variation by adopting the following formula:
wherein (x, y) represents the coordinates of the pixel point on the two-dimensional image plane, phi 1 (x, y) is a visible light image, phi 2 (x, y) is an infrared light image,is a base layer of visible light, +.>For the base layer of infrared light, μ (x, y) represents the window size w μ Represents a convolution operation;
since the value of the average filter template is 1, it is taken into the above equation:
after the mean filter template is slid to the next pixel, the above equation is expressed as:
wherein,for the sum of all visible pixels in the template before sliding the mean filter template, +.>For the accumulation of the first row of visible light pixels in the filter template sliding front template, for the summation of the last row of visible pixels in the template after the filter template is slid,for the sum of all infrared pixels in the template before the mean filter template is slid,for the summation of the first row of infrared pixels in the filter template before sliding the template,the sum of the last row of infrared pixels in the template after the filter template slides is added;
the detail layer is extracted by subtracting the base layer from the source image using the following formula:
wherein,is a detail layer of visible light, +.>Is a detail layer of infrared light.
8. The real-time fusion method of ultra-high definition visible and infrared light images based on ZYNQ MPSoC as claimed in claim 6, wherein the saliency information of the edges and lines in the step3 is as follows:
wherein, the absolute value of the difference value is taken, and the value is xi 1 (x, y) is the saliency information of the edges and lines of the visible image, ζ 2 (x, y) is saliency information of the infrared light image edges and lines,the size of the passing window for visible light image is w μ Is filtered by an averaging filter, +.>For infrared light image, the size of the passing window is w μ Is filtered by an averaging filter, +.>Through window for visible light imageSize w η Is filtered by a median filter, +.>The size of the passing window for visible light image is w η (x, y) representing coordinates of pixel points on the two-dimensional image plane.
9. The real-time fusion method of ultra-high definition visible and infrared light images based on ZYNQ MPSoC as claimed in claim 6, wherein the fusion weight coefficient in the step4 has the following specific formula:
wherein, psi is 1 (x, y) is the fusion weight coefficient of the detail layer of visible light, ψ 2 (x, y) is the fusion weight coefficient of the detail layer of infrared light, ζ 1 (x, y) is the saliency information of the edges and lines of the visible image, ζ 2 (x, y) is saliency information of the infrared light image edges and lines;
the specific formula of the linear fusion is as follows:
wherein phi is D (x, y) is the fused detail layer image,is a detail layer of visible light, +.>Is a detail layer of infrared light.
10. The real-time fusion method of ultra-high definition visible and infrared light images based on ZYNQ MPSoC according to claim 6, wherein the fused base layer image in step5 has the following specific formula:
wherein phi is B (x, y) is the fused base layer image,is a base layer of visible light, +.>A base layer that is infrared light;
the specific formula of the fused image is as follows:
γ(x,y)＝φ B (x,y)+φ D (x,y)
wherein, gamma (x, y) is a fusion image phi D And (x, y) is the fused detail layer image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311011170.5A CN117036219A (en) | 2023-08-10 | 2023-08-10 | Ultrahigh-definition visible and infrared light image real-time fusion system and method based on ZYNQ MPSoC |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311011170.5A CN117036219A (en) | 2023-08-10 | 2023-08-10 | Ultrahigh-definition visible and infrared light image real-time fusion system and method based on ZYNQ MPSoC |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117036219A true CN117036219A (en) | 2023-11-10 |
Family
ID=88622288
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311011170.5A Pending CN117036219A (en) | 2023-08-10 | 2023-08-10 | Ultrahigh-definition visible and infrared light image real-time fusion system and method based on ZYNQ MPSoC |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117036219A (en) |
-
2023
- 2023-08-10 CN CN202311011170.5A patent/CN117036219A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107249101B (en) | High-resolution image acquisition and processing device | |
| WO2019085605A1 (en) | Automatic optical detection system based on cpu+gpu+fpga architecture | |
| US11323628B2 (en) | Field of view adjustment | |
| CN109690628B (en) | Image processing apparatus | |
| WO2019000676A1 (en) | Method and device for achieving three-dimensional image by using dual cameras | |
| CN103595924B (en) | A kind of image fusion system based on Cameralink and method thereof | |
| CN109688382B (en) | Underwater image processing system of underwater robot | |
| TW201445505A (en) | Mobile device and system for generating panoramic video | |
| CN102202171A (en) | Embedded high-speed multi-channel image acquisition and storage system | |
| CN106817545B (en) | A kind of fast multiresolution video image mirror image rotation processing system | |
| CN113194269B (en) | Image output system and method | |
| CN110620885B (en) | Infrared low-light-level image fusion system and method and electronic equipment | |
| CN208956216U (en) | Based on the long large car viewing system away from transmission of low-light (level) camera synchronous acquisition | |
| WO2020006078A1 (en) | Entropy maximization based auto-exposure | |
| CN109951617A (en) | A kind of double spectrum high speed cameras based on fpga | |
| US9530173B2 (en) | Information processing device, imaging device, and information processing method | |
| Yan et al. | Design of CMOS image acquisition system based on FPGA | |
| US9013556B2 (en) | 3D image capturing device for generating a 3D image based on two 2D images and controller chip thereof | |
| CN103716544A (en) | Rapid and continuous focusing method and system for high-resolution module, | |
| CN103369338B (en) | Based on image processing system and the method for the nearly eye binocular imaging system of FPGA | |
| CN117036219A (en) | Ultrahigh-definition visible and infrared light image real-time fusion system and method based on ZYNQ MPSoC | |
| CN106686323B (en) | Imaging method and system of high-speed CMOS camera | |
| CN109089048A (en) | More camera lens full-view cooperative device and methods | |
| TWI475875B (en) | Three-dimensional image genetating device | |
| CN104463774A (en) | Designing method of DM642-based three-path image fusion processor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |