CN109995965B

CN109995965B - Ultrahigh-resolution video image real-time calibration method based on FPGA

Info

Publication number: CN109995965B
Application number: CN201910276412.0A
Authority: CN
Inventors: 李华宇; 苗书宇; 范锡添; 冯瑞; 刘妍
Original assignee: Fudan University
Current assignee: Fudan University
Priority date: 2019-04-08
Filing date: 2019-04-08
Publication date: 2021-12-03
Anticipated expiration: 2039-04-08
Also published as: CN109995965A

Abstract

The invention provides an ultra high definition video image real-time calibration method based on an FPGA (field programmable gate array), which is used for carrying out real-time calibration on an ultra high definition video image through the FPGA and is characterized by comprising the following steps of: step S1, establishing an original image data receiving channel, and directly transmitting the original image data of the ultra-high definition video image to the FPGA; step S2, calibrating the original image data to obtain result calibration data; step S3, establishing a calibration result output channel, and returning the result calibration data from the FPGA, wherein the step S2 comprises the following substeps: step S2-1, establishing an image data frame buffer area, and preprocessing the original image data to obtain preprocessed image data; step S2-2, processing the preprocessed image data through an image calibration algorithm to obtain delta X, delta Y and delta theta; and step S2-3, establishing a result memory buffer area, and sorting the obtained delta X, delta Y and delta theta to obtain final calibration result data.

Description

Ultrahigh-resolution video image real-time calibration method based on FPGA

Technical Field

The invention belongs to the field of video image processing, and particularly relates to an ultra-high-definition video image real-time calibration processing method based on an FPGA.

Background

Ultra-high definition video image calibration techniques play an important role in our lives. Plays a significant role in biomedical engineering, industrial engineering, military police and aerospace. With the development of science and technology, the resolution of video images is higher and higher, the information of the brought video images is richer, more useful information is more beneficial to be acquired, and the resolution application of 5Kx5K is gradually common.

However, the large video image resolution results in a greater computational effort required for calibration, i.e., the pressure to increase the hardware processing speed is increasing. On one hand, with the development of computer technology, the processing speed of a CPU has been continuously improved, and the processing speed of unit data volume has been fast; on the other hand, video image calibration processing in many fields requires higher and higher real-time performance, which requires higher hardware computation power per unit time. But the improvement of the computing power of the CPU is not enough to make up for the requirement of the real-time processing of the video image under the larger resolution.

Common processing modes for the calibration processing of video images include a processing mode based on a PC CPU and a processing mode based on a DSP. The CPU-based processing mode is based on a von Neumann architecture computer platform, the mode is executed sequentially in nature, and for a video image with the resolution of 5Kx5K, the processing mode of the CPU is slow in processing speed and low in real-time performance, and the requirement for high processing speed in many practical scenes cannot be met. Based on the DSP processing approach, Digital Signal Processors (DSPs) employ mathematical operations units (MACs) that employ pipeline structures configured in hardware at different levels. The defects brought by the method are obvious, and due to the fixation of a hardware structure and the structure of a production line, the DSP is linear calculation in the video image processing process, so that the method has obvious defects and is not friendly to the development process of a processing system.

The development of technology brings video images with higher and higher resolution, the semantic information contained in the video images is richer, the extraction of the information of the images is more and more beneficial, and the difficulty of image calibration is accompanied, so that the computational load of hardware caused by huge computation amount is not small. In the face of the processing requirement of video image calibration real-time performance, how to quickly and accurately analyze, process and calibrate a target video image in unit time is of great importance to the application of the actual industry landing of the video image.

At present, the increasing speed of the computing power of hardware cannot meet the huge demand of the computing power in the video image calibration algorithm. Therefore, the method has important significance for realizing real-time processing of ultra-high-definition video image calibration aiming at limited hardware conditions.

Disclosure of Invention

In order to solve the problems, the invention provides an FPGA-based accelerated processing method for ultra-high-definition video image calibration, and the method can rapidly process the ultra-high-definition video image calibration based on effective operation of software and hardware, and can remarkably increase the processing speed of the ultra-high-definition video image calibration. The technical scheme of the invention is as follows:

the invention provides an ultra-high resolution video image real-time calibration method based on an FPGA (field programmable gate array), which is used for carrying out real-time calibration on an ultra-high resolution video image through the FPGA and is characterized by comprising the following steps of: step S1, establishing an original image data receiving channel, and directly transmitting the original image data of the ultra-high definition video image to the FPGA; step S2, calibrating the original image data to obtain result calibration data, wherein the result calibration data comprises an offset delta X of the central coordinate of the ultra-high definition video image on an X axis, an offset delta Y on a Y axis and a deflection angle delta theta; step S3, establishing a calibration result output channel, and returning the result calibration data from the FPGA, wherein the step S2 comprises the following substeps: step S2-1, establishing an image data frame buffer area for temporarily storing original image data, and preprocessing the original image data to obtain preprocessed image data, wherein the image data frame buffer area comprises a frame buffer area scheduling controller and an address management module; step S2-2, processing the preprocessed image data through an image calibration algorithm to obtain delta X, delta Y and delta theta; and step S2-3, establishing a result memory buffer area, and sorting the obtained delta X, delta Y and delta theta to obtain final calibration result data.

The method for calibrating the ultrahigh-resolution video image in real time based on the FPGA can also have the technical characteristics that the transmission media of the original image data receiving channel and the calibration result output channel are ten-gigabit network cards, gigabit network cards or PCIE.

The real-time calibration method for the ultra-high resolution video image based on the FPGA can also have the technical characteristics that the number of the FPGAs is one or more.

The real-time calibration method for the ultrahigh-resolution video image based on the FPGA provided by the invention can also have the technical characteristics that the preprocessing comprises the following steps: firstly, the original image data is subjected to preliminary decoding processing to obtain preprocessed image data, then the preprocessed image data is subjected to frame serialization processing, and further personalized ID marking processing is carried out on each preprocessed image data in the frame serialization.

The method for calibrating the ultrahigh-resolution video image in real time based on the FPGA can also have the technical characteristics that the initial decoding processing adopts a linear decoding processing method, the frame serialization processing is to establish one or more first-in first-out queue type buffer areas for the preprocessed image data, and the personalized ID marking processing is to carry out ID marking according to the time stamp of the preprocessed image data or carry out ID marking according to the sequence of the preprocessed image data entering the queue type buffer areas.

The real-time calibration method for the ultra-high resolution video image based on the FPGA provided by the invention can also have the technical characteristics that the frame buffer scheduling controller is used for scheduling the state of the image data frame buffer, and comprises the steps of detecting whether the image data frame buffer is in a non-full state and detecting whether the preprocessing image data processed by the FPGA memory is finished, if the image data frame buffer is not full, the original image data is transmitted to the image data frame buffer, and if the image processed by the FPGA memory is finished, the preprocessing image data in the queue type buffer is transmitted, and the address management module is used for managing the memory address of the FPGA, and comprises the steps of allocating the memory according to the pre-requirement and applying for the dynamic memory according to the processing requirement of part of modules.

According to the FPGA-based real-time calibration method for the ultrahigh-resolution video image, the technical characteristics can be further provided, and the image calibration algorithm processing comprises the steps of processing the preprocessed image data through processes of an integral graph calculation module, convolution difference closed operation processing, a normalization inverse corrosion calculation module, a coordinate set calculation module, a Span matching center coordinate calculation module, an ROI convolution module, Cross coordinate post-processing and the like, and finally obtaining delta X, delta Y and delta theta.

According to the ultrahigh resolution video image real-time calibration method based on the FPGA, the method can also have the technical characteristics that in the process of an integral graph calculation module, an independent memory address of an integral graph cache can be adopted, a coordinate cache generated by calculation of a Track coordinate set can be adopted, an independent memory address of a coordinate cache can be adopted, and the like.

According to the ultrahigh-resolution video image real-time calibration method based on the FPGA, the method can also have the technical characteristics that an integral graph cache generated by an integral graph calculation module and a coordinate cache generated by a Track coordinate set calculation can adopt a form of sharing a memory address.

The method for calibrating the ultrahigh-resolution video image in real time based on the FPGA can also have the technical characteristics that the normalization and inverse corrosion module uses one corrosion or a plurality of iterative corrosion operations.

The real-time calibration method for the ultrahigh-resolution video image based on the FPGA provided by the invention can also have the technical characteristics that the result memory cache region is a fixed memory cache region established according to a preset address or a dynamic memory applied in real time according to the data size of the calibration result data.

The method for calibrating the ultrahigh-resolution video image in real time based on the FPGA can also have the technical characteristics that the process of sorting the calibration result data is to judge and sort according to the ID marks generated during the preprocessing of the preprocessed image data corresponding to each piece of the calibration result data,

action and Effect of the invention

According to the accelerated processing method for ultra-high definition video image calibration, provided by the invention, an end-to-end FPGA processing mode is directly carried out through a data transmission channel during calibration aiming at a large-scale image data set of an ultra-high resolution video, so that the time overhead of transmission, copying and the like through CPU processing is reduced. In the process of running the image calibration algorithm, a deep learning method is used, and compared with the traditional image processing, the method has the advantages that the lack of accuracy caused by manual design of parameters is avoided, and the method has a practical effect on the requirements of gene sequencing on high accuracy. Meanwhile, in the internal processing of calibration, the calibration process from coarse to fine is combined, a coarse offset result is obtained firstly, and then detection and sorting are carried out on the result, so that the final result obtained in the mode has higher accuracy. The invention can complete the real-time calibration task of the ultra-high-definition video image under the condition of ensuring the calibration accuracy.

Drawings

FIG. 1 is a system architecture diagram of ultra high definition video image calibration based on FPGA in an embodiment of the present invention;

FIG. 2 is a flow chart of an image calibration process in an embodiment of the invention;

FIG. 3 is a schematic diagram of data transmission and result collection control logic according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a data interaction control logic in an embodiment of the present invention; and

FIG. 5 is a flow chart of an implementation of single frame image data alignment using an end-to-end FPGA in an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made with reference to the accompanying drawings.

< example >

The software system of the embodiment is implemented on a Windows server 2012, a ten-gigabit network card is adopted on hardware, and the acceleration computing unit uses an FPGA (field programmable gate array), and the number of the FPGAs is 8.

In the embodiment, 4 paths of ultra-high-definition video images are taken as processing objects, a single frame of the video images is 16-bit images, the resolution is 5120 × 5120, and the frame rate is 20 frames/s.

The implementation goal of this example is to directly bypass the processing of the CPU for the original video image data through the end-to-end implementation, and directly send the original video image data to the FPGA through the input channel such as the tera network card for processing, so as to achieve the purpose of eliminating the delay, achieve the real-time processing in the FPGA, and quickly calculate the offset of the central coordinate of the image, and the final obtained results are the offset of the X axis and the Y axis of the central coordinate, and the deflection angles of the image, that is, Δ X, Δ Y, and Δ θ. And then calculating the offset, and returning the result through an output channel such as a tera network card.

For video streaming, due to real-time transmission, the whole process can be divided into three stages:

(1) and a video image input stage. The video image can be directly transmitted to the FPGA through a form of, but not limited to, a ten-gigabit network card without a CPU.

(2) And (5) FPGA processing stage. The FPGA obtains final calibration results delta X, delta Y and delta theta by performing the processes of decoding, preprocessing, queue processing, state marking, an integral graph calculation module, convolution difference closed operation processing, a normalization inverse corrosion calculation module, a coordinate set calculation module, a Span matching center coordinate calculation module, an ROI convolution module, Cross coordinate post-processing and the like of a video image.

(3) And a calibration result output stage. And the FPGA outputs the final calibration results delta X, delta Y and delta theta obtained by calculation in the FPGA in the form of, but not limited to, a tera network card and the like.

Fig. 1 is a system architecture diagram of ultra high definition video image calibration based on FPGA in the embodiment of the present invention.

Fig. 2 is a flow chart of an image calibration process in an embodiment of the invention.

As shown in fig. 1 and fig. 2, the method for calibrating an ultra high definition video image of the present embodiment mainly includes 3 steps, i.e., steps S1 to S3.

Step S1 is to establish a raw image data receiving channel and directly transmit the raw image data of the ultra-high-definition video image from the computer to the FPGA.

In step S1, the transmission process is to directly acquire Raw video image Data Raw Data from the video capture device without time-consuming processing by the CPU, and directly transmit the Raw video image Data Raw Data to the FPGA (i.e., Data- > input- > Data stream transmission process of the FPGA) by the method in step S1, where the intermediate transmission channel (i.e., Raw image Data reception channel) is a PCIE, a gigabit network card, a tera network card, or the like (e.g., a Mellanox connex-3 Pro VPI network card, which supports 40Gbps throughput, and supports both transmission through the switch and transmission in a direct interconnect mode).

In this embodiment, the number of the FPGAs is one, and the FPGAs linearly process the original image data.

Step S2 in fig. 1 is a core processing part accelerated by FPGA, and is used for performing calibration processing on raw image data to obtain resultant calibration data, which mainly includes three sub-steps: preprocessing an S2-1 image, calibrating an S2-2 image, and sorting and outputting an S2-3 result.

And step S2-1, establishing an image data frame buffer area for temporarily storing original image data, and preprocessing the original image data to obtain preprocessed image data.

In step S2-1, preliminary processing of the image data is mainly provided for the process of calibration in the following step S2-2. The processing mainly includes the following points that image data needs to be decoded first, and then subjected to frame serialization processing and ID marking processing.

In this embodiment, a linear decoding processing method is used for decoding the original image data, and the frame serialization processing is to establish a plurality of first-in first-out queue-type buffer areas for the preprocessed image data obtained after decoding, and input the preprocessed image data into the queue-type buffer areas in sequence, and further perform ID marking according to the time stamp of each preprocessed image data. In other embodiments, there can be one queuing buffer, and the preprocessed image data are sequentially ID-tagged according to the order in which they are input into the queuing buffer.

In step S2-1, the image data frame buffer further has an address management module and a buffer scheduling controller. The address management module is a DMA address management module and is responsible for providing guarantee for correct reference of FPGA addresses, so that management of the addresses in the FPGA is facilitated. The buffer scheduling controller is also used for video image data, so that the memory resource of the FPGA is utilized as much as possible. For example, the status of the buffer is scheduled by the buffer scheduling controller: detecting whether the buffer area is not full or not, and whether newly input Raw Data is accepted or not; detecting whether the frame queue is full, and if not, transmitting the image data to the frame queue; and detecting whether the image processed by the FPGA memory is finished, if so, transmitting the image in the queue, and the like.

Step S2-2, the preprocessed image data is processed by the image calibration algorithm in parallel to obtain Δ X, Δ Y, Δ θ.

In step S2-2, mainly the calibration algorithm of the implemented video image works. The input processed image data is processed by an integral diagram calculation module, a convolution difference closed operation processing module, a normalization inverse corrosion module, a coordinate set calculation module, a Span matching center coordinate calculation module, an ROI convolution module, Cross coordinate post-processing and the like to obtain final results delta X, delta Y and delta theta. In this embodiment, two adjacent independent memory addresses are designed in one memory as an integral map cache and a coordinate cache.

And step S2-3, establishing a result memory buffer area, and detecting and sorting the obtained delta X, delta Y and delta theta to obtain final calibration result data.

In step S2-3, the final result Δ X, Δ Y, Δ θ is mainly verified again, because in the process of step S2-2, it cannot be guaranteed that the final output result corresponds to the input ID number one-to-one, and in order to solve the consistency of the numbers, a verification process is required, that is, whether the ID of the calibration result data is consistent with the ID of the pre-processed image data is determined: when the ID of the calibration result data is identical to the ID of the previously input preprocessed image data, continuing the subsequent processing; and when the ID of the calibration result data is inconsistent with the ID of the previously input preprocessed image data, deleting the calibration result data, carrying out image calibration algorithm processing on the corresponding preprocessed image data again to obtain new calibration result data, and further carrying out ID verification on the new calibration result data.

In addition, in the embodiment, the magnitude of the values Δ X, Δ Y, and Δ θ in the calibration result data is determined according to a preset calibration result threshold, so that the problem of an overlarge value is avoided, which causes a large deviation of the calibration result and affects the calibration effect.

In this embodiment, the result memory cache region is used to store the calibration result data and perform detection and sorting on the cached calibration result data, and the result memory cache region is a fixed memory cache region established according to a preset address. In other embodiments, the result memory buffer may also be a dynamic memory applied according to the data size of the calibration result data: when a set of calibration result data is output in step S2-2, the address management module applies a memory with a data size corresponding to the set of calibration result data as a part of the result memory buffer, and when the stored calibration result is sorted and the data is output in step S3, the address management module removes the address of the corresponding part from the result memory buffer.

And step S3, establishing a calibration result output channel, and returning the result calibration data to the computer from the FPGA.

Step S3 of this embodiment corresponds to the effect of step S1, and this step is to send the calibration result obtained by processing in the FPGA to the result receiving place (i.e. the data stream where the FPGA calibration result- > output- > HOST data receiving place) directly through the transmission channel (i.e. the calibration result output channel) without processing by the CPU or the like. The carrier of the transmission channel is PCIE, gigabit network card, ten-gigabit network card, etc. (such as Mellanox ConnetX-3Pro VPI network card).

Fig. 3 is a schematic diagram of HOST end data sending and result collection control logic according to an embodiment of the present invention.

As shown in fig. 3, the whole data flow can be regarded as an end-to-end flow, but in order to speed up the whole running time of the flow, a cache is set in the FPGA. In the process of data transmission and data reception, because of the existence of data transmission time, an input queue is preset in the process of image input, for example, two pieces of data 1 and data 2 are continuously input, and image data 3 can be input into the queue in the process of processing image data 1. By analogy, a processing sequence of input data 1, input data 2, an acceptance result 1, input data 3 and an acceptance result 2 shown in fig. 3 can be obtained, so that the time for masking data sending and receiving is calculated by using the FPGA.

Fig. 4 is a schematic diagram of a data interaction control logic according to an embodiment of the present invention.

As shown in fig. 4, for the FPGA top-level data interaction control logic, ping-pong operation is mainly adopted. That is, for input, two memory spaces are opened up on DDR4, such as I1 and I2; for output, two blocks of output memory space are similarly opened up on DDR4, e.g., O1, O2. Thus, for FPGA computation, the computation is performed under the condition that at least one of I1 and I2 stores data, and at least one of the output spaces O1 and O2 is empty. Thus, the influence of data transmission time can be effectively reduced by the whole calculation in cooperation with the data sending and receiving logic of the HOST end. Of course, for input, each time the corresponding input space is calculated, the space is released, and for output, each time the result is read by the HOST, the corresponding space is also released. It is guaranteed that there is room to continue to store the input data and the resulting results.

The scheme of the data interaction control logic shown in fig. 4 corresponds to step S2 of fig. 1. From fig. 4, it can be seen that the DMA address control unit in step S2-1 can complete the opening up of the storage space of the data interaction control logic unit shown in fig. 4. The condition that at least one block satisfied by the input corresponding I1, I2 storage space is at least one non-empty, and the output space O1 or O2 is at least one empty may be accomplished by the scheduling controller in step S2. As is clear from fig. 4 and fig. 1, in the FPGA, the DMA address control module is separately set up to control the memory address of each storage space in the FPGA, which is convenient for management. On the other hand, the image sequence input to the FPGA can be conveniently regulated and controlled through the scheduling controller. In detail, when the image frame sequence is not full, a new image sequence is input into the queue, and when the queue is full, the new image is prevented from proceeding to the image frame sequence. Meanwhile, it is necessary to regulate and control at least one of the storage spaces I1 and I2 in fig. 4 to be non-empty and at least one of O1 and O2 to be empty, so as to achieve dynamic balance between data transmission and reception, and effectively reduce the influence of data transmission time in the whole calculation.

As shown in fig. 5, the single-frame video image execution flowchart is mainly divided into three parts, an input image and output result part, an input/output channel part, and an FPGA processing part. The input images are from a connected real-time video stream at a rate of 20 frames per second. The output results are in the form of the offset of the image coordinates with respect to the X-axis and Y-axis of the center coordinates of the base coordinate system, and the deflection angles Δ X, Δ Y, Δ θ. The three offsets are transmitted to a console (i.e., a computer) for subsequent operations.

Examples effects and effects

According to the accelerated processing method for ultra-high definition video image calibration provided by the embodiment, an end-to-end FPGA processing mode is directly performed through a data transmission channel during calibration aiming at a large-scale image data set of an ultra-high resolution video, so that the time overhead of transmission, copying and the like through CPU processing is reduced. In the process of running the image calibration algorithm, a deep learning method is used, and compared with the traditional image processing, the method has the advantages that the lack of accuracy caused by manual design of parameters is avoided, and the method has a practical effect on the requirements of gene sequencing on high accuracy. Meanwhile, in the internal processing of calibration, the calibration process from coarse to fine is combined, a coarse offset result is obtained firstly, and then detection and sorting are carried out on the result, so that the final result obtained in the mode has higher accuracy. The invention can complete the real-time calibration task of the ultra-high-definition video image under the condition of ensuring the calibration accuracy.

In the embodiment, the original image data and the calibration result are directly transmitted through the original image data receiving channel and the calibration result output channel, so that the time consumption problem of decoding processing through a CPU in the traditional processing method is reduced, and the image data acquired by the video acquisition device is not subjected to other redundant processing before being transmitted to the FPGA.

In an embodiment, the number of FPGAs is one, and the raw image data is processed linearly by the FPGAs. In addition, the invention can also use a mode of multi-FPGA cooperative work (namely the number of the FPGAs is multiple), so that the original image data receiving channel can be connected with each FPGA, and the original image data is sent to each FPGA to carry out the parallel processing of multi-frame images, thereby accelerating the execution speed of image calibration to a greater extent. Through the image calibration acceleration processing based on the FPGA, in the processing process of gene sequencing, for example, the same data volume and the speed are improved by tens of times, the time cost and the resource cost of calibration can be greatly reduced, and the accumulated time delay in the processing process can be better eliminated.

In the embodiment, in order to prevent the calibration data from causing the information result to be asymmetric with the original image data, the ID encoding is set for each image data in the preprocessing of the original data image.

In the embodiment, the state of the image data frame buffer is scheduled by the frame buffer scheduling controller, and the memory address of the FPGA is managed by the address management module, so that a dynamic balance is formed in the data sending and receiving processes of the FPGA, and the influence of data transmission time in the whole calculation process is effectively reduced.

In the embodiment, the integral map cache and the coordinate cache adopt respective independent memory addresses, and as the calibration processing of the video image in the FPGA is a linear processing process, the time for processing once is limited by the integral operation with a slow speed. In addition, the integral map cache and the coordinate cache can also adopt a form of sharing a memory address, because the hysteresis of a calculation result is realized in the process of large calculation amount, and the utilization rate of the memory can be increased by sharing the form of the memory.

In the embodiment, the result memory cache region is used for caching the calibration result data to be detected and sorted, and the result memory cache region is a fixed memory cache region set according to a preset address, so that the calibration result data can be cached in a more orderly manner. In addition, the result memory cache region can also apply for a dynamic memory according to the data size of the calibration result data in real time, so that the overall memory of the FPGA can be better utilized, and the excessive idle addresses are avoided.

In the embodiment, the ID of the calibration result data and the ID of the pre-processing image data input before are subjected to consistency detection, and the numerical value of the calibration result data is detected and sorted, so that the calibration result data is further checked, and the data inaccuracy caused by the error of the calibration result data is avoided.

The above-described embodiments are merely illustrative of specific embodiments of the present invention, and the present invention is not limited to the description of the above-described embodiments.

In an embodiment, the calibration result data includes Δ X, Δ Y, Δ θ. In other embodiments, the calibration result data of the present invention may also be a permutation and combination of two of Δ X, Δ Y, and Δ θ, or even a fourth result value that is more desirable.

Claims

1. An FPGA-based ultrahigh resolution video image real-time calibration method is used for carrying out real-time calibration on an ultrahigh resolution video image through an FPGA, and is characterized by comprising the following steps:

step S1, establishing an original image data receiving channel, and directly transmitting the original image data of the ultra-high-definition video image to the FPGA;

step S2, calibrating the original image data to obtain result calibration data, wherein the result calibration data comprises an offset delta X of the central coordinate of the ultra-high definition video image on an X axis, an offset delta Y on a Y axis and a deflection angle delta theta;

step S3, establishing a calibration result output channel, returning the result calibration data from the FPGA,

wherein the step S2 includes the following sub-steps:

step S2-1, establishing an image data frame buffer area for temporarily storing the original image data, and preprocessing the original image data to obtain preprocessed image data, wherein the image data frame buffer area comprises a frame buffer area scheduling controller and an address management module;

step S2-2, the preprocessed image data are processed through an image calibration algorithm to obtain delta X, delta Y and delta theta, and the image calibration algorithm processing comprises the steps that the preprocessed image data are processed through an integral graph calculation module, a convolution difference closed operation module, a normalization inverse corrosion module, a coordinate set calculation module, a Span matching center coordinate calculation module, an ROI convolution module and a Cross coordinate post-processing module;

step S2-3, establishing a result memory buffer area, detecting and sorting the obtained delta X, delta Y and delta theta to obtain the final calibration result data,

the pretreatment comprises the following steps:

firstly, the original image data is primarily decoded to obtain preprocessed image data, then the preprocessed image data is subjected to frame sequential processing, and further personalized ID marking processing is carried out on each preprocessed image data in the frame sequential,

the frame buffer scheduling controller is used for scheduling the state of the image data frame buffer, including detecting whether the image data frame buffer is in a state of not being full and detecting whether the preprocessing of the image data processed by the FPGA memory is completed,

if the image data frame buffer is not full, then the raw image data is transferred into the image data frame buffer,

if the image processed by the FPGA memory is finished, the preprocessed image data in the queue type buffer area is transmitted,

the address management module is used for managing the memory address of the FPGA, and comprises memory allocation according to the pre-demand and dynamic memory application according to the processing demand of part of the modules.

2. The real-time calibration method for the ultra-high resolution video image based on the FPGA of claim 1, characterized in that:

the transmission media of the original image data receiving channel and the calibration result output channel are ten-gigabit network cards, gigabit network cards or PCIE.

3. The real-time calibration method for the ultra-high resolution video image based on the FPGA of claim 1, characterized in that:

the number of the FPGAs is one or more.

4. The real-time calibration method for the ultra-high resolution video image based on the FPGA of claim 1, characterized in that:

wherein the preliminary decoding process adopts a linear decoding process method,

the frame serialization process is to create one or more first-in-first-out queuing type buffers for the preprocessed image data,

and the individualized ID marking processing is to mark an ID according to a time stamp of the preprocessed image data or mark an ID according to the sequence of the preprocessed image data entering the queue type buffer area.

5. The real-time calibration method for the ultra-high resolution video image based on the FPGA of claim 1, characterized in that:

the integral graph cache generated in the process of the integral graph calculation module and the coordinate cache generated by the calculation of the Track coordinate set in the process of each coordinate calculation adopt independent memory addresses or common memory addresses,

the normalized inverse erosion module includes using one or more iterative erosion operations.

6. The real-time calibration method for the ultra-high resolution video image based on the FPGA of claim 1, characterized in that:

the result memory cache region is a fixed memory cache region set according to a preset address or a dynamic memory applied in real time according to the data size of the calibration result data.

7. The FPGA-based ultrahigh resolution video image real-time calibration method of claim 6, characterized in that:

wherein the process of detecting and sorting the calibration result data is to judge and sort according to the ID mark generated during the preprocessing of the preprocessed image data corresponding to each calibration result data,

and when the ID of the calibration result data is inconsistent with the ID of the previously input preprocessed image data, deleting the calibration result data, and performing image calibration algorithm processing on the corresponding preprocessed image data again to obtain new calibration result data.