WO2017113540A1 - 一种气动光学效应校正识别一体化实时处理的系统和方法 - Google Patents

一种气动光学效应校正识别一体化实时处理的系统和方法 Download PDF

Info

Publication number
WO2017113540A1
WO2017113540A1 PCT/CN2016/079133 CN2016079133W WO2017113540A1 WO 2017113540 A1 WO2017113540 A1 WO 2017113540A1 CN 2016079133 W CN2016079133 W CN 2016079133W WO 2017113540 A1 WO2017113540 A1 WO 2017113540A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image data
correction
fpga module
asic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/079133
Other languages
English (en)
French (fr)
Inventor
张天序
侯旋
章川
刘立
陈泉
钟奥
许明星
周雨田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huazhong University of Science and Technology
Original Assignee
Huazhong University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huazhong University of Science and Technology filed Critical Huazhong University of Science and Technology
Priority to US15/576,847 priority Critical patent/US10297013B2/en
Publication of WO2017113540A1 publication Critical patent/WO2017113540A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/10—Image enhancement or restoration using non-spatial domain filtering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]

Definitions

  • the invention belongs to the field of cross-scientific technology combining aerospace technology and image processing, and particularly relates to an integrated real-time processing system and method for aero-optical effect correction recognition.
  • Pneumatic optical effects mainly include pneumatic optical transmission effects and aerodynamic thermal radiation effects.
  • Pneumatic optical transmission effect usually refers to the formation of a complex flow field between the optical hood and the incoming flow when a high-speed aircraft with an optical imaging detection system is flying in the atmosphere, causing optical wavefronts other than thermal radiation to the optical imaging detection system.
  • Transmission distortion or transmission interference causing the image of the observed object to shift, shake, and blur;
  • the aerodynamic thermal radiation effect means that under the influence of high temperature and high pressure gas, the window shape with uniform optical characteristics will be distorted and formed inside.
  • the refractive index gradient distribution when the plane wave is incident on the distortion window, the evanescent wavefront after the exit deviates from the ideal wavefront, and the wavefront is distorted, resulting in imaging blur.
  • the present invention proposes an integrated real-time processing system and method for aero-optical effect correction and identification, which effectively solves the problem that the processor requires a processor to complete a short detection time interval under high-speed flight conditions.
  • the invention simultaneously has the structural design of the hardware system and the flow design of the algorithm software system, and mainly aims at denoising, aerodynamic thermal radiation correction, aerodynamic optical transmission effect correction and real-time processing of target detection and recognition for the degraded image of the aero-optical effect, and adopts Task allocation and multi-core parallel processing, so that the real-time performance of the entire system is greatly improved.
  • the present invention provides an integrated real-time processing system for aero-optical effect correction recognition, characterized in that the system comprises an FPGA module, a multi-core main processor DSP, a plurality of coprocessor ASICs, and infrared image non-uniformity.
  • Supreme Correction System-on-Chip SoC :
  • the FPGA module is connected to the multi-core main processor DSP for receiving and buffering the pneumatic optical effect degradation image data and receiving the processed image data and corresponding image information;
  • the multi-core main processor DSP is connected to the FPGA module, and is configured to read the degraded image information of the FPGA module end and receive the degraded image data, and correct the on-chip system SoC and the plurality of coprocessor ASICs by calling the infrared image non-uniformity correction.
  • the image data is subjected to full-image thermal radiation correction, denoising, transmission effect correction, target detection and recognition, and the processed image data and corresponding image information are sent to the FPGA module;
  • the plurality of coprocessor ASICs are used for dividing according to their functions, respectively, for preprocessing of an image, acceleration processing of an algorithm, and target detection and recognition;
  • the infrared image non-uniformity correction system on chip SoC is used to perform non-uniformity correction on the degraded infrared image data.
  • the multi-core main processor DSP includes Core0 ⁇ Core7:
  • the Core7 is connected to the FPGA module, and is used for reading image information of the FPGA module and creating an image buffer of a corresponding size, and sending the buffer information to the FPGA module; Degraded image data of the image buffer and notifying Core0 for subsequent image data processing; also for receiving processed image data and image information And send it to the FPGA module;
  • the Core0 is used to perform non-uniformity correction on the received degraded image data by calling the correction SoC, and then performing full-image thermal radiation correction, denoising, and transmission effects by calling multiple coprocessor ASICs. Correction and target detection.
  • the Core0 can guide Core0 to Core6 to perform parallel processing on the degraded image data through OpenMP.
  • the plurality of coprocessor ASICs specifically include an FFT operation ASIC, a rotation ASIC, a multi-stage filter ASIC, and a mark ASIC.
  • the FPGA module, the multi-core main processor DSP, the plurality of coprocessor ASICs, and the infrared image non-uniformity correction system-on-chip SoC may be equipped with an extended memory, the FPGA module, the multi-core main processor DSP, and Infrared image non-uniformity correction
  • the on-chip SoC can also be configured with a power-down non-volatile FLASH device.
  • an integrated real-time processing method for aero-optical effect correction recognition is proposed, characterized in that the method comprises:
  • the multi-core main processor DSP reads the degraded image information of the FPGA module end and receives the degraded image data
  • Multi-core main processor DSP calls infrared image non-uniformity correction on-chip system SoC and multiple coprocessor ASICs perform full-image thermal radiation correction, denoising, transmission effect correction and target detection on image data;
  • the multi-core main processor DSP sends the processed image data and image information to the FPGA module.
  • the step (3) specifically includes:
  • (3.1) Core7 of the multi-core main processor DSP reads the degraded image information of the FPGA module and creates an image buffer of a corresponding size, and sends the information of the buffer to the FPGA module;
  • the Core 7 receives the degraded image data written by the FPGA module into the image buffer and Notifying Core0 to perform subsequent image data processing;
  • the Core0 first performs non-uniformity correction on the received degraded image data by calling the calibration SoC, and then performs full-image thermal radiation correction, denoising, transmission effect correction, and the like by calling multiple coprocessor ASICs.
  • the Core 7 sends the processed image data and image information to the FPGA module.
  • the Core0 can guide Core0 to Core6 to perform parallel processing on the image data through OpenMP.
  • the plurality of coprocessor ASICs specifically include an FFT operation ASIC, a rotation ASIC, a multi-stage filter ASIC, and a mark ASIC.
  • the target detection specifically includes:
  • the multi-core main processor DSP transmits the processed image data to the multi-stage filter ASIC through the FPGA module for multi-stage filtering operation, and then transmits the filtered result to the mark ASIC to mark the image, and finally returns the mark result to the DSP.
  • Feature extraction and target centroid tracking processing
  • the system of the present invention includes non-uniformity correction, rotation, image denoising, aerodynamic thermal radiation effect correction, aero-optical transmission effect correction, and target recognition of an infrared image, wherein image non-uniformity correction, rotation, and target recognition are in progress
  • image non-uniformity correction, rotation, and target recognition are in progress
  • the multi-stage filtering and marking are processed by a dedicated ASIC designed and developed by the independent research and development, which significantly improves the real-time performance of the entire system;
  • the multi-core main processor DSP dominates the execution of the degraded image data processing process, and the tasks are reasonably allocated and executed in a multi-core parallel manner, which greatly shortens the processing time of the degraded image;
  • the FPGA module in the invention acts as a system interconnect, and the various units are reasonably connected to a closed loop that forms the entire system, further improving the stability of the system.
  • the integrated real-time processing system and method for aero-optical effect correction and recognition according to the present invention
  • the method can effectively solve the problem of aero-optical effect and the problem that the processor requires a short time interval between detection and processing under high-speed flight conditions of the aircraft, and has no excessive computational complexity and is easy to control, thus having certain implementability and Practical promotion value.
  • FIG. 1 is a structural block diagram of a hardware system of a pneumatic correction recognition integrated processing system
  • FIG. 2 is an overall block diagram of a dedicated FFT operation ASIC (FFT 8092V1.0);
  • Figure 3 is a system block diagram of a labeled ASIC
  • FIG. 4 is a system block diagram of a multi-stage filter ASIC
  • Figure 5 is a system block diagram of a calibration SoC
  • Figure 6 is a block diagram of the interconnection of the rotating ASIC
  • Figure 7 is a block diagram of the pneumatic optical effect correction software
  • Figure 8 is a flow chart of the heat radiation effect correction algorithm and a task assignment diagram
  • Figure 10 (a) is an external iterative flow chart of the pneumatic optical transmission effect correction algorithm
  • Figure 10 (b) is a flow chart of a point spread function (PSF) estimation of a pneumatic optical transmission effect correction algorithm
  • Figure 10 (c) is a flow chart of target image estimation of a pneumatic optical transmission effect correction algorithm
  • Figure 10 (d) is a task assignment diagram of a pneumatic optical transmission effect correction algorithm
  • Figure 11 is a DSP multi-core scheduling diagram
  • Figure 12 (a) is a DSP Core7 processing flow chart
  • Figure 12 (b) is a flow chart of DSP Core0 ⁇ Core6 processing
  • Figure 13 is a flow chart of target recognition.
  • the core device of the hardware system is a TMS320C6678 multi-core main processor DSP produced by TI Company, a Kintex7 FPGA module produced by XILINX Company; the ASIC adopts self-developed chips, including a dedicated FFT (Fast Fourier) Transform) ASIC (FFT 8192V1.0), an infrared image non-uniformity correction system-on-chip SoC, a rotating ASIC, a multi-stage filtering AISC, and a tag ASIC.
  • FFT Fast Fourier
  • SoC a dedicated FFT (Fast Fourier) Transform) ASIC
  • SoC rotating ASIC
  • multi-stage filtering AISC multi-stage filtering AISC
  • tag ASIC tag ASIC.
  • Each device is equipped with extended memory according to requirements.
  • the FPGA module, DSP and SoC are also equipped with FLASH devices that are not lost when power is off, which is convenient for curing the program.
  • the DSP is used as the main processor to control the algorithm flow;
  • the ASIC is used as the coprocessor, the correction SoC and the rotary ASIC are used for image preprocessing, and the multistage filter ASIC and the mark ASIC are used for object detection and recognition.
  • the dedicated FFT operation ASIC is used to calculate the FFT operation that consumes a large amount;
  • the FPGA is mainly used for interface design.
  • a set of 4 ⁇ SRIO Serial Rapid I/O
  • parallel input and output bus is used between DSP and FFT operation ASIC for data transmission and working by I2C bus or synchronous serial port. Make settings.
  • the image data enters the FPGA module through the LVDS (Low Voltage Differential Signaling) signal interface, and the FPGA module first buffers the data. Due to the imaging factor of the infrared camera, the non-uniformity correction of the infrared image needs to be performed by calling the calibration SoC;
  • the DSP reads the data into the DSP's memory through SRIO and processes the data. When the algorithm involves the FFT or IFFT operation part, the DSP will send the data to the ASIC for calculation, and then read the data back to the DSP for subsequent processing. Since the aircraft will have a rotating process during the flight, the DSP will correct the image by calling the rotating ASIC through the FPGA.
  • LVDS Low Voltage Differential Signaling
  • the DSP After completing one frame processing, the DSP calls the multi-stage filtering ASIC and the tag ASIC for target recognition, and notifies the FPGA of the corrected image and the target position information, and the FPGA transmits the data after receiving the message, thereby completing the entire aero-optical effect correction and Target recognition algorithm.
  • the software system includes aerodynamic thermal radiation effect correction, image denoising, pneumatic optical transmission effect correction, target detection and recognition.
  • the framework of the whole calibration process is: firstly, the aerodynamic thermal radiation correction is performed, then the denoising process is performed, then the aero-optical transmission effect correction is performed, the image recovery is completed by the above processing, and finally the target detection is performed. And identification.
  • DSP Core7 communicates with the FPGA through SRIO, reads the degraded image information (resolution, image format) of the FPGA through the NREAD transaction to create an image buffer of corresponding size, and then sends the buffer information to the FPGA through the NWRITE_R transaction.
  • DSP Core7 informs the FPGA that the address information has been sent through the Doorbell transaction, indicating that the FPGA can write the degraded image data to the buffer, and then the FPGA writes the degraded image data to the corresponding buffer through the NWRITE_R transaction according to the address mapping table.
  • the DSP Core7 is notified of the current degraded image data integrity through the Doorbell transaction.
  • Core0 After receiving the first frame of degraded image data, Core0 first calls the calibration SoC pair.
  • the image is non-uniformly corrected, as shown in Figure 5 for a system block diagram for correcting the SoC. Then the image is subjected to full-image thermal radiation correction, denoising and transmission effect correction, then the whole image is pre-detected, and the region of interest with a resolution of 64 ⁇ 64 is extracted, and then corrected in the region, which will be corrected.
  • the subsequent subgraph is restored to the detection target position in the original image, and the precise position of the target can be determined at this time.
  • the full-image thermal radiation correction method uses a gradient-fitted aerodynamic thermal radiation correction algorithm by fitting a gray-scale offset field to a degraded image of aero-optical effect, and then subtracting the degraded image from the degraded image.
  • the estimated grayscale offset field is obtained, thereby obtaining a pneumatic thermal radiation corrected image, the flow chart of which is shown in FIG.
  • the full-picture denoising method uses the frequency domain filtering method using the target frequency domain characteristics.
  • the point source target is taken as an example for specific description.
  • the Gaussian ring filter with different radii is constructed according to the point source target and the spectral distribution characteristics of the noise, and then The image with noise is subjected to Fourier transform (this part of the calculation is implemented in the FFT operation AISC), and then it is multiplied by the filter and then subjected to the inverse Fourier transform operation to obtain the denoised image, as shown in FIG. .
  • the transmission effect correction method uses the maximum likelihood estimation algorithm under Hu moment constraint.
  • the characteristic of the algorithm is to recover the image by iterative iteration of the point spread function (PSF) and the image.
  • PSF point spread function
  • the specific process of target detection is as follows: the DSP transmits the processed image data to the multi-stage filter ASIC through the FPGA for multi-stage filtering operation, and then transmits the filtered result to the mark ASIC to mark the image, and finally returns the mark result to the DSP.
  • the process of feature extraction and target centroid tracking is performed. The whole process is dominated by DSP, and the processing flow is shown in Figure 13.
  • Figure 3 and Figure 4 show the block diagrams of the tag ASIC and multi-stage filter ASIC, respectively.
  • Core7 sends the image data and image information to the FPGA through SRIO. After receiving the data, the FPGA finally transmits the data to the next processing system through the LVDS signal interface to complete the entire aero-optical effect correction recognition process.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

一种气动光学效应校正识别一体化实时处理系统，包括FPGA模块、多核主处理器DSP、多个协处理器ASIC及红外图像非均匀性校正片上系统SoC，通过上述系统完成气动光学效应退化图像的全图热辐射校正、去噪、传输效应校正及目标检测过程。相应的，提出了一种与之对应的方法。该系统能够有效解决气动光学效应问题以及飞行器高速飞行条件下要求处理器在完成探测处理时间间隔短的问题，通过采用自主研发的专用ASIC，使得整个系统的实时性大大提升；对各任务进行合理分配并采用多核并行的方式，大大缩短了图像的处理时间；同时，FPGA模块将各单元连接成闭环系统，进一步提高了系统的稳定性。

Description

一种气动光学效应校正识别一体化实时处理的系统和方法 【技术领域】
本发明属于航天技术与图像处理相结合的交叉科学技术领域，具体涉及一种气动光学效应校正识别一体化实时处理系统和方法。
【背景技术】
二十世界末以来，高超声速飞行器逐渐成为航空航天领域的重要发展方向。无论是在高科技产业，还是在经济领域都具有极高的战略意义，主要体现在洲际飞行器以及空天飞机等飞行器的大力发展。但是由于飞行器的高速性带来的严酷的成像环境使得成像传感器成像品质劣化，信噪比、信杂比大幅降低，探测距离大幅下降；此外飞行器的高速运动导致处理器在完成探测处理时间间隔更短，对有限的处理能力提出了更加严峻的挑战。所以超高声速飞行器的发展还面临着这样一系列的气动光学效应问题以及对应的校正算法计算相当复杂的难题。
气动光学效应主要包括气动光学传输效应和气动热辐射效应。气动光学传输效应通常是指带有光学成像探测系统的高速飞行器在大气层内飞行时，光学头罩与来流之间形成复杂的流场，对光学成像探测系统造成除热辐射外的光学波前传输畸变或传输干扰，引起被观测对象图像的偏移、抖动、模糊；而气动热辐射效应指的是在高温高压气体的影响下，原本光学特性均匀的窗口外形将产生畸变，并且在内部形成了折射率梯度分布，当平面波入射到畸变窗口时，出射后的等光程波面偏离了理想波面，波面发生畸变，从而导致成像模糊。
目前已经具备相应的气动光学传输效应校正算法以及气动热辐射效应校正算法，但还没有同时具备这两种校正算法，带有识别功能并且具有体积小、重量轻、功耗低、处理速度快以及结构明确等特点的实时处理系统。
【发明内容】
针对现有技术的以上缺陷或不足，本发明提出了一种气动光学效应校正识别一体化实时处理系统及方法，有效解决了飞行器高速飞行条件下要求处理器在完成探测处理时间间隔短的问题。本发明同时具备硬件系统的结构设计和算法软件系统的流程设计，主要针对气动光学效应退化图像的去噪、气动热辐射校正、气动光学传输效应校正以及目标检测与识别的实时处理展开，并采取任务分配和多核并行的处理方式，使得整个系统的实时性大大提升。
为实现上述目的，本发明提出了一种气动光学效应校正识别一体化实时处理系统，其特征在于，所述系统包括FPGA模块、多核主处理器DSP、多个协处理器ASIC及红外图像非均匀性校正片上系统SoC：
所述FPGA模块，与所述多核主处理器DSP相连，用于接收并缓存气动光学效应退化图像数据及接收处理后的图像数据以及对应的图像信息；
所述多核主处理器DSP，与FPGA模块相连，用于读取FPGA模块端退化图像信息及接收退化图像数据，并通过调用红外图像非均匀性校正片上系统SoC及多个协处理器ASIC对退化图像数据进行全图热辐射校正、去噪、传输效应校正及目标检测、识别，并将处理后的图像数据以及对应的图像信息发送给FPGA模块；
所述多个协处理器ASIC，用于对按其功能划分，分别用于对图像的预处理、算法的加速处理以及目标检测与识别；
所述红外图像非均匀性校正片上系统SoC，用于对退化的红外图像数据进行非均匀性校正。
作为进一步优选的，所述多核主处理器DSP包括Core0～Core7：
所述Core7，与FPGA模块相连，一方面用于读取FPGA模块图像信息并创建相应大小的图像缓冲区，并将缓冲区的信息发送至FPGA模块；另一方面用于接收FPGA模块写入相应图像缓冲区的退化图像数据并通知Core0进行后续的图像数据处理；还用于接收处理后的图像数据及图像信息 并将其发送给FPGA模块；
所述Core0，一方面用于对接收到的退化图像数据，先通过调用校正SoC对其进行非均匀性校正，然后通过调用多个协处理器ASIC进行全图热辐射校正、去噪、传输效应校正以及目标检测。
作为进一步优选的，所述Core0可通过OpenMP来引导Core0～Core6对退化图像数据进行并行处理。
作为进一步优选的，所述多个协处理器ASIC具体包括FFT运算ASIC、旋转ASIC、多级滤波ASIC及标记ASIC。
作为进一步优选的，所述FPGA模块、多核主处理器DSP、多个协处理器ASIC及红外图像非均匀性校正片上系统SoC均可配有扩展存储器，所述FPGA模块、多核主处理器DSP以及红外图像非均匀性校正片上系统SoC还可配置有掉电不易失FLASH器件。
按照本发明的另一方面，提出了一种气动光学效应校正识别一体化实时处理方法，其特征在于，所述方法包括：
(1)利用FPGA模块接收并缓存气动光学效应退化图像数据；
(2)多核主处理器DSP读取FPGA模块端退化图像信息，并接收退化图像数据；
(3)多核主处理器DSP调用红外图像非均匀性校正片上系统SoC及多个协处理器ASIC对图像数据进行全图热辐射校正、去噪、传输效应校正及目标检测；
(4)多核主处理器DSP将处理后的图像数据以及图像信息发送给FPGA模块。
作为进一步优选的，所述步骤(3)具体包括：
(3.1)所述多核主处理器DSP的Core7，读取FPGA模块退化图像信息并创建相应大小的图像缓冲区，并将缓冲区的信息发送至FPGA模块；
(3.2)所述Core7接收FPGA模块写入图像缓冲区的退化图像数据并 通知Core0进行后续的图像数据处理；
(3.3)所述Core0对接收到的退化图像数据，先通过调用校正SoC对其进行非均匀性校正，然后通过调用多个协处理器ASIC进行全图热辐射校正、去噪、传输效应校正以及目标检测；
(3.4)所述Core7将处理后图像数据及图像信息发送给FPGA模块。
作为进一步优选的，所述Core0可通过OpenMP来引导Core0～Core6对图像数据进行并行处理。
作为进一步优选的，所述多个协处理器ASIC具体包括FFT运算ASIC、旋转ASIC、多级滤波ASIC及标记ASIC。
作为进一步优选的，所述目标检测具体包括：
首先，多核主处理器DSP通过FPGA模块将处理后的图像数据传送到多级滤波ASIC进行多级滤波操作，然后将滤波后的结果传送到标记ASIC对图像进行标记，最终将标记结果送回DSP进行特征提取以及目标形心跟踪处理。
总体而言，按照本发明点的以上技术方案与现有技术相比，主要具备以下的技术优点：
1、本发明系统包括红外图像的非均匀性校正、旋转、图像去噪、气动热辐射效应校正、气动光学传输效应校正以及目标识别等过程，其中图像非均匀性校正、旋转以及目标识别过程中的多级滤波和标记均采用自主研发设计的专用ASIC来处理，显著提升了整个系统的实时性；
2、此外，本发明系统及方法中多核主处理器DSP主导退化图像数据处理过程的执行，且对任务进行合理分配并采用多核并行的方式执行，大大缩短了退化图像的处理时间；同时，本发明中的FPGA模块充当系统互连的角色，将各个单元合理的连接到一块形成整个系统的闭环，进一步提高了系统的稳定性。
3、按照本发明提出的气动光学效应校正识别一体化实时处理系统和方 法，能有效的解决气动光学效应问题以及飞行器高速飞行条件下要求处理器在完成探测处理时间间隔短的问题，且未存在过多的计算复杂度，便于操控，因而具有一定的可实施性及实用推广价值。
【附图说明】
图1为气动校正识别一体化处理系统的硬件系统结构框图；
图2为专用FFT运算ASIC(FFT8092V1.0)的整体框图；
图3为标记ASIC的系统框图；
图4为多级滤波ASIC的系统框图；
图5为校正SoC的系统框图；
图6为旋转ASIC的互连框图；
图7为气动光学效应校正软件框图；
图8为热辐射效应校正算法流程及任务分配图；
图9为图像去噪算法流程及任务分配图；
图10(a)为气动光学传输效应校正算法的外迭代流程图；
图10(b)为气动光学传输效应校正算法的点扩展函数(PSF)估计流程图；
图10(c)为气动光学传输效应校正算法的目标图像估计流程图；
图10(d)为气动光学传输效应校正算法任务分配图；
图11为DSP多核调度图；
图12(a)为DSP Core7处理流程图；
图12(b)为DSP Core0～Core6处理流程图；
图13为目标识别流程图。
【具体实施方式】
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
硬件系统：
(1)硬件系统的核心器件为一颗TI公司生产的TMS320C6678多核主处理器DSP，一颗XILINX公司生产的Kintex7 FPGA模块；ASIC均采用的是自主研发的芯片，包括一颗专用FFT(Fast Fourier Transform)运算的ASIC(FFT8192V1.0)、一颗红外图像非均匀性校正片上系统SoC、一颗旋转ASIC、一颗多级滤波AISC以及一颗标记ASIC。每颗器件根据需求均配有扩展存储器，此外FPGA模块、DSP以及SoC还配置有掉电不易失的FLASH器件，便于程序的固化。
(2)在本发明中DSP作为主处理器，控制算法流程；ASIC作为协处理器，校正SoC和旋转ASIC用在图像预处理方面，多级滤波ASIC和标记ASIC用在目标检测和识别方面，专用FFT运算ASIC则用来计算消耗较大的FFT运算；FPGA主要用来做接口设计。DSP与FPGA之间采用一组4×SRIO(Serial Rapid I/O)进行数据传输；DSP与FFT运算ASIC之间采用并行输入输出总线进行数据传输并通过I2C总线或同步串口的方式对其工作方式进行设置。
(3)图像数据通过LVDS(Low Voltage Differential Signaling)信号接口进入FPGA模块，FPGA模块先对对数据进行缓存，由于红外相机的成像因素，需要通过调用校正SoC对红外图像进行非均匀性校正；然后DSP通过SRIO将数据读入到DSP的存储器中然后对数据进行处理。当算法涉及到FFT或IFFT运算部分时，DSP会将该数据送入到ASIC进行计算，然后再将数据读回DSP进行后续处理。由于飞行器在飞行过程中会有旋转的过程，此时DSP会通过FPGA调用旋转ASIC对图像进行校正。完成一帧处理后DSP调用多级滤波ASIC和标记ASIC进行目标识别，并将校正后的图像以及目标位置信息通知FPGA，FPGA收到消息后在将数据传送出去，至此完成整个气动光学效应校正以及目标识别算法。
软件系统：
(1)软件系统包括气动热辐射效应校正、图像去噪、气动光学传输效应校正，目标检测与识别等过程。根据气动光学效应退化机理，整个校正过程的框架为：首先进行气动热辐射校正，然后进行去噪处理，之后在做气动光学传输效应校正，经以上处理完成对图像的恢复，最后在进行目标检测与识别。
(2)针对气动光学效应的复杂性，通过对算法有效性的研究，对以上三种处理过程分别提出了适合并有效的校正算法分别为：基于梯度拟合的气动热辐射校正算法、利用目标频域特性的频域滤波方法和Hu矩约束下的最大似然估计算法。由于以上算法中涉及到大量的FFT/IFFT运算，并且这部分运算消耗较大，所以在本发明中使用了自主研发的专用FFT运算ASIC来对算法进行加速，并且将相对成熟的非均匀性校正、旋转、标记、多级滤波等算法做成ASIC来使用，进而提升处理的实时性。
基于ASIC+FPGA+DSP结构的气动校正识别一体化实时处理系统的硬件框图如图1所示，软件流程如图7所示。
具体实现过程为：
(1)气动光学效应退化图像数据通过LVDS信号接口传输至FPGA，FPGA对数据进行缓存。
(2)DSP Core7通过SRIO与FPGA进行通信，通过NREAD事务读取FPGA端退化图像信息(分辨率、图像格式)创建相应大小的图像缓冲区，然后将缓冲区的信息通过NWRITE_R事务发送给FPGA。
(3)DSP Core7通过Doorbell事务通知FPGA地址信息已发送完成表示FPGA可以向该缓冲区写入退化图像数据，然后FPGA根据地址映射表将退化图像数据通过NWRITE_R事务写入相应的缓冲区，完成后通过Doorbell事务通知DSP Core7当前退化图像数据完整。
(4)DSP Core7通知Core0退化图像数据已就绪可以进行后续的处理；
(5)Core0在接收到第一帧退化图像数据后，先通过调用校正SoC对 图像进行非均匀性校正，如图5所示为校正SoC的系统框图。然后对图像进行全图热辐射校正、去噪以及传输效应校正，然后对全图进行预检测，并提取出分辨率为64×64大小的感兴趣区域，之后在对该区域进行校正，将校正后的子图还原到原图中检测目标位置，此时可以确定目标的精确位置。
具体而言，全图热辐射校正方法采用的是基于梯度拟合的气动热辐射校正算法，该方法是通过对气动光学效应退化图像进行灰度偏移场拟合，然后用该退化图像减去估计出的灰度偏移场，从而得到气动热辐射校正图像，其流程图如图8所示。全图去噪采用的是利用目标频域特性的频域滤波方法，以点源目标为例进行具体说明，根据点源目标以及噪声的频谱分布特性构建不同半径的高斯环状滤波器，然后对含有噪声的图像进行傅里叶变换(这部分计算在FFT运算AISC中实现)，然后将其与滤波器作点乘然后进行傅里叶逆变换操作得到去噪后的图像，如图9所示。传输效应校正方法采用的是Hu矩约束下的最大似然估计算法，该算法的特性就是通过对点扩展函数(PSF)和图像进行反复的迭代来进行图像的复原，整个算法流程以及其在硬件系统上实现的分布如图10所示。
(6)在进行连续多帧处理时，由于帧频较快，下一帧和上一帧目标所在的位置偏差很小，所以在对下一帧处理时首先进行全图热辐射校正，然后直接以上一帧目标所在的位置为中心提取出分辨率为64×64大小的感兴趣区，然后直接对该区域进行去噪与校正处理，最终再进行目标检测，所有算法在DSP中执行的具体过程如图12所示。
目标检测的具体过程为：DSP通过FPGA将处理后的图像数据传送到多级滤波ASIC进行多级滤波操作，然后将滤波后的结果传送到标记ASIC对图像进行标记，最终将标记结果送回DSP进行特征提取以及目标形心跟踪等处理，整个过程由DSP作为主导，其处理流程如图13所示。图3和图4分别展示了标记ASIC与多级滤波ASIC的结构框图。
(7)Core0在执行过程中，需要使用FFT/IFFT运算时，将退化图像数据通过FPGA传送到FFT运算ASIC，其系统结构如图2所示，并对该ASIC进行配置，ASIC处理完成后再将数据送回至DSP，Core0继续执行；在需要将图像旋转时，通过调用旋转ASIC进行旋转操作，图6显示了旋转ASCI的典型连接方式；在遇到大量复杂循环操作时，Core0通过调用OpenMP来引导Core0～Core6对这部分循环进行并行处理，多核调度过程如图11所示，处理完成后将图像信息(分辨率，图像格式，存储地址)发送给Core7。
(8)Core7通过SRIO将图像数据以及图像信息发送给FPGA，FPGA收到数据后最终通过LVDS信号接口将数据传输到下一级处理系统，完成整个气动光学效应校正识别过程。
本领域的技术人员容易理解，以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种气动光学效应校正识别一体化实时处理系统，其特征在于，所述系统包括FPGA模块、多核主处理器DSP、多个协处理器ASIC及红外图像非均匀性校正片上系统SoC：
    所述FPGA模块，与所述多核主处理器DSP相连，用于接收并缓存气动光学效应退化图像数据及接收处理后的图像数据以及对应的图像信息；
    所述多核主处理器DSP，与FPGA模块相连，用于读取FPGA模块端退化图像信息及接收退化图像数据，并通过调用红外图像非均匀性校正片上系统SoC及多个协处理器ASIC对退化图像数据进行全图热辐射校正、去噪、传输效应校正及目标检测，并将处理后的图像数据以及对应的图像信息发送给FPGA模块；
    所述多个协处理器ASIC，用于对退化图像数据进行全图热辐射校正、去噪、传输效应校正以及目标检测；
    所述红外图像非均匀性校正片上系统SoC，用于对退化图像数据进行非均匀性校正。
  2. 如权利要求1所述的系统，其特征在于，所述多核主处理器DSP包括Core0～Core7：
    所述Core7，与FPGA模块相连，一方面用于读取FPGA模块图像信息并创建相应大小的图像缓冲区，并将缓冲区的信息发送至FPGA模块；另一方面用于接收FPGA模块写入相应图像缓冲区的退化图像数据并通知Core0进行后续的图像数据处理；还用于接收处理后的图像数据及图像信息并将其发送给FPGA模块；
    所述Core0，一方面用于对接收到的退化图像数据，先通过调用校正SoC对其进行非均匀性校正，然后通过调用多个协处理器ASIC进行全图热辐射校正、去噪、传输效应校正以及目标检测。
  3. 如权利要求1所述的系统，其特征在于，所述Core0可通过OpenMP来引导Core0～Core6对退化图像数据进行并行处理。
  4. 如权利要求1-3任一项所述的系统，其特征在于，所述多个协处理器ASIC具体包括FFT运算ASIC、旋转ASIC、多级滤波ASIC及标记ASIC。
  5. 如权利要求1所述的系统，其特征在于，所述FPGA模块、多核主处理器DSP、多个协处理器ASIC及红外图像非均匀性校正片上系统SoC均可配有扩展存储器，所述FPGA模块、多核主处理器DSP以及红外图像非均匀性校正片上系统SoC还可配置有掉电不易失FLASH器件。
  6. 一种气动光学效应校正识别一体化实时处理方法，其特征在于，所述方法包括：
    (1)利用FPGA模块接收并缓存气动光学效应退化图像数据；
    (2)多核主处理器DSP读取FPGA模块端退化图像信息，并接收退化图像数据；
    (3)多核主处理器DSP调用红外图像非均匀性校正片上系统SoC及多个协处理器ASIC对图像数据进行全图热辐射校正、去噪、传输效应校正及目标检测；
    (4)多核主处理器DSP将处理后的图像数据以及图像信息发送给FPGA模块。
  7. 如权利要求6所述的方法，其特征在于，所述步骤(3)具体包括：
    (3.1)所述多核主处理器DSP的Core7，读取FPGA模块退化图像信息并创建相应大小的图像缓冲区，并将缓冲区的信息发送至FPGA模块；
    (3.2)所述Core7接收FPGA模块写入图像缓冲区的退化图像数据并通知Core0进行后续的图像数据处理；
    (3.3)所述Core0对接收到的退化图像数据，先通过调用校正SoC对其进行非均匀性校正，然后通过调用多个协处理器ASIC进行全图热辐射校正、去噪、传输效应校正以及目标检测；
    (3.4)所述Core7将处理后图像数据及图像信息发送给FPGA模块。
  8. 如权利要求7所述的方法，其特征在于，所述Core0可通过OpenMP来引导Core0～Core6对图像数据进行并行处理。
  9. 如权利要求6-8任一项所述的方法，其特征在于，所述多个协处理器ASIC具体包括FFT运算ASIC、旋转ASIC、多级滤波ASIC及标记ASIC。
  10. 如权利要求6-8任一项所述的方法，其特征在于，所述目标检测具体包括：
    首先，多核主处理器DSP通过FPGA模块将处理后的图像数据传送到多级滤波ASIC进行多级滤波操作，然后将滤波后的结果传送到标记ASIC对图像进行标记，最终将标记结果送回DSP进行特征提取以及目标形心跟踪处理。
PCT/CN2016/079133 2015-12-28 2016-04-13 一种气动光学效应校正识别一体化实时处理的系统和方法 Ceased WO2017113540A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/576,847 US10297013B2 (en) 2015-12-28 2016-04-13 Aerodynamic optical effect correction and identification integrated real-time processing system and method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201511002876.0A CN105469370B (zh) 2015-12-28 2015-12-28 一种气动光学效应校正识别一体化实时处理的系统和方法
CN2015110028760 2015-12-28

Publications (1)

Publication Number Publication Date
WO2017113540A1 true WO2017113540A1 (zh) 2017-07-06

Family

ID=55607031

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/079133 Ceased WO2017113540A1 (zh) 2015-12-28 2016-04-13 一种气动光学效应校正识别一体化实时处理的系统和方法

Country Status (3)

Country Link
US (1) US10297013B2 (zh)
CN (1) CN105469370B (zh)
WO (1) WO2017113540A1 (zh)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10521888B2 (en) * 2015-12-23 2019-12-31 Huazhong University Of Science And Technology Aerothermal radiation effect frequency domain correction method
CN105469370B (zh) * 2015-12-28 2018-05-11 华中科技大学 一种气动光学效应校正识别一体化实时处理的系统和方法
CN108534979A (zh) * 2016-10-08 2018-09-14 哈尔滨理工大学 立式湍流效应模拟装置
CN108288030B (zh) * 2017-12-29 2021-05-18 华中科技大学 一种机载红外小目标检测识别实时处理系统和方法
CN109587380B (zh) * 2018-12-06 2020-04-28 天津津航技术物理研究所 小型化高温工作多接口制冷型红外成像器
EP3764235B1 (en) * 2019-07-12 2022-10-26 Ratier-Figeac SAS Field programmable gate array (fpga) having dissimilar cores
CN111078210B (zh) * 2019-12-11 2023-06-30 中国科学院光电技术研究所 一种用于自适应光学操控系统的QT和Matlab多线程混合编程软件架构
CN111290309A (zh) * 2019-12-26 2020-06-16 深圳市宝捷信科技有限公司 驱控一体控制系统及fpga电流环配置方法
EP3872634A1 (en) * 2020-02-27 2021-09-01 Mitsubishi Electric R&D Centre Europe B.V. Multicore system on chip architecture
CN112256627A (zh) * 2020-10-21 2021-01-22 四川望望科技有限公司 一种光电跟踪系统高清高帧频实时图像处理平台及方法
CN113096041B (zh) * 2021-04-08 2023-03-24 杭州海康消防科技有限公司 图像校正方法及装置
CN113536461B (zh) * 2021-07-15 2023-11-03 北京航空航天大学 用于高超声速强激波流场气动热预测的湍流模型修正方法
CN114139486A (zh) * 2021-11-02 2022-03-04 上海航天控制技术研究所 一种小型化高速图像多核信息处理系统及方法
CN113793284B (zh) * 2021-11-17 2022-03-15 武汉工程大学 气动光学效应空变模糊图像不均匀分块的图像复原方法
CN114492240B (zh) * 2022-01-25 2026-04-21 中国科学院上海技术物理研究所 一种红外气动光学成像仿真方法、存储介质及计算机设备
CN118587134A (zh) * 2024-05-23 2024-09-03 华中科技大学 一种红外成像气动热辐射效应矫正方法、系统及装置
CN119043504B (zh) * 2024-08-08 2025-10-14 中国人民解放军国防科技大学 一种气动光学一维畸变波前的高时空分辨率测量方法及系统
CN121527687B (zh) * 2026-01-19 2026-04-24 湖南长城银河科技有限公司 一种基于国产智能dsp的交通事件检测器系统和检测方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120330633A1 (en) * 2011-06-21 2012-12-27 Lockheed Martin Corporation Scintillation generator for simulation of aero-optical and atmospheric turbulence
CN103544470A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对空目标双色红外异构并行自动目标识别器
CN103544471A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对地固定目标异构并行自动目标识别器
CN103679134A (zh) * 2013-09-09 2014-03-26 华中科技大学 一种对海目标红外成像识别装置
CN105469370A (zh) * 2015-12-28 2016-04-06 华中科技大学 一种气动光学效应校正识别一体化实时处理的系统和方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140105515A1 (en) * 2012-04-04 2014-04-17 The Regents Of The University Of California Stabilizing and Deblurring Atmospheric Turbulence
US9811884B2 (en) * 2012-07-16 2017-11-07 Flir Systems, Inc. Methods and systems for suppressing atmospheric turbulence in images
US10389940B2 (en) * 2015-07-02 2019-08-20 Mission Support and Test Services, LLC Passive method to measure strength of turbulence

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120330633A1 (en) * 2011-06-21 2012-12-27 Lockheed Martin Corporation Scintillation generator for simulation of aero-optical and atmospheric turbulence
CN103544470A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对空目标双色红外异构并行自动目标识别器
CN103544471A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对地固定目标异构并行自动目标识别器
CN103679134A (zh) * 2013-09-09 2014-03-26 华中科技大学 一种对海目标红外成像识别装置
CN105469370A (zh) * 2015-12-28 2016-04-06 华中科技大学 一种气动光学效应校正识别一体化实时处理的系统和方法

Also Published As

Publication number Publication date
US20180293709A1 (en) 2018-10-11
CN105469370A (zh) 2016-04-06
US10297013B2 (en) 2019-05-21
CN105469370B (zh) 2018-05-11

Similar Documents

Publication Publication Date Title
WO2017113540A1 (zh) 一种气动光学效应校正识别一体化实时处理的系统和方法
CN106686278A (zh) 一种面向实时目标识别的异构处理机并行协调处理方法
CN102438118B (zh) 一种运动目标特征高速视觉捕捉装置
WO2019085377A1 (zh) 目标跟踪硬件实现系统和方法
CN102289825B (zh) 一种实时图像边缘检测电路及其实现方法
JP7586909B2 (ja) 画像フレームフリーズ検出のための方法及び装置
CN106204660A (zh) 一种基于特征匹配的地面目标跟踪装置
CN104902167A (zh) 一种高速图像采集处理系统
CN109919825A (zh) 一种orb-slam硬件加速器
CN103544470B (zh) 一种动平台对空目标双色红外异构并行自动目标识别器
CN108288030B (zh) 一种机载红外小目标检测识别实时处理系统和方法
CN111583149B (zh) 一种基于l1-l0范数最小化的气动热辐射图像自动校正方法
Tanabe et al. 18.2 a 1.9 tops and 564gops/w heterogeneous multicore soc with color-based object classification accelerator for image-recognition applications
CN103108109B (zh) 一种数字视频降噪系统与方法
CN118587134A (zh) 一种红外成像气动热辐射效应矫正方法、系统及装置
CN109584143B (zh) 一种航空相机图像增强设备以及方法
CN109873998A (zh) 基于多层次引导滤波的红外视频增强系统
CN103604945A (zh) 一种三通道cmos同步偏振成像系统
CN103544471A (zh) 一种动平台对地固定目标异构并行自动目标识别器
CN107085827A (zh) 基于硬件平台实现的超分辨力图像复原方法
CN106791803B (zh) 一种扰动测量成像系统
CN118279360A (zh) 一种基于fpga的缪勒矩阵偏振成像方法和系统
CN113255538A (zh) 基于fpga的红外弱小目标的检测跟踪装置及方法
CN115018730A (zh) 一种图像条纹噪声的去除方法、装置、设备及介质
CN107945197A (zh) 一种用于图像边缘提取的智能图像处理仪

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16880348

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15576847

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16880348

Country of ref document: EP

Kind code of ref document: A1