CN114581380A - Dispersion correction method based on FPGA and storage medium - Google Patents

Abstract

An FPGA-based dispersion correction method and storage medium of the present invention includes the following steps: receiving an image data stream, and using an on-chip RAM to store data of four adjacent clock cycles; Select the correction method; process each line of data in segments with the center; use a unified mathematical model for dispersion correction processing; output the corrected image in real time. The FPGA-based dispersion correction method of the present invention uses the hardware algorithm in the FPGA to process the image in parallel, so as to offset the dispersion phenomenon on the photosensitive element after the light passes through the lens during imaging. And the pipeline processing method is used to further improve the efficiency of the algorithm. The algorithm supports real-time processing of images without preset chromatic aberration compensation table. It only needs simple parameter input, and calculates the corresponding offsets of different pixels in real-time processing. This method is more flexible and can be applied to various complex and changing scenes. middle.

Description

An FPGA-based dispersion correction method and storage medium

technical field

The invention relates to the technical field of image processing, in particular to an FPGA-based dispersion correction method.

Background technique

With the prosperity and development of industry, high-speed industrial cameras face more complex scene requirements in use. In actual use, the captured image will appear dispersion phenomenon, which will affect the imaging effect. This is because the refractive index of the lens for different wavelengths of light is different, so when the light is refracted to the sensor through the lens group, different wavelengths of light will converge at different points. This phenomenon is more severe and obvious at places far from the imaging center.

In order to improve the image quality and offset the dispersion phenomenon, there are mainly two existing solutions, but there are some problems: the first one is mainly to eliminate the dispersion through optical methods, such as achromatic lenses to improve or even offset the dispersion, but This approach requires a reconsideration of the optical structure, increases the complexity of the optical and hardware architecture design, and increases the cost. The second is mainly by storing the data and processing it through software algorithms after the image acquisition is completed. The processing efficiency of this method is low, which leads to its use with large scene limitations and is not suitable for high-speed real-time image acquisition systems.

In order to solve the above problems, taking into account the high-speed real-time transmission scene of industrial cameras, and in order to reduce the complexity and cost of industrial camera design as much as possible, it is necessary to adopt an efficient, real-time and low-cost dispersion correction method.

SUMMARY OF THE INVENTION

The FPGA-based dispersion correction method proposed by the present invention can at least solve one of the above technical problems.

To achieve the above object, the present invention has adopted the following technical solutions:

An FPGA-based dispersion correction method, comprising:

Receive the image data stream, and use the on-chip RAM to store the data of the adjacent 4 clock cycles;

Select the correction method according to the boundary offset before and after correction;

Segment each row of data by center;

Dispersion correction processing using unified mathematical model;

The corrected image is output in real time.

Further, described receiving the image data stream, and using the on-chip RAM to store the data of the adjacent 4 clock cycles, specifically including,

The image is acquired under actual lighting conditions, the image is transmitted in the FPGA as streaming data, M data is transmitted in each clock cycle, the data of the adjacent 4 clock cycles is stored in the on-chip RAM, and each line is before correction. The data is recorded as R[n], where n is the column value of the pixel value;

最边界处像素的偏移量视为x，假设该感光元件的水平分辨率为2H_Pixel，伸缩系数表达式如下式：The whole process is regarded as a long strip of length L, keeping the midpoint unchanged, stretched to L', then its expansion coefficient is

The offset of the pixel at the most boundary is regarded as x. Assuming that the horizontal resolution of the photosensitive element is 2H _Pixel , the scaling factor is expressed as follows:

Further, the process of segmenting each row of data with a center specifically includes:

From the data center of each row, it is divided into two halves for segmentation processing, respectively 0≤n≤H _Pixel -1, H _Pixel ≤n≤2H _Pixel -1.

Further, the use of a unified mathematical model to perform dispersion correction processing specifically includes:

Set the offset of the boundary pixels before and after correction as x, and select integer pixel correction or half-integer pixel correction according to the actual situation;

If the offset is an integer, then use the value of the integer pixel before correction for interpolation calculation; if the offset of the boundary pixel is the interpolation of two adjacent integer pixels before, then use half-integer pixel correction;

使coef为整数值来代表每种偏移量，其为奇数时，将对应整数偏移量；若为偶数则对应半整数偏移量。The former means that x is an integer value, and the latter means that x is a half-integer value.

Let coef be an integer value to represent each offset, if it is odd, it will correspond to an integer offset; if it is even, it will correspond to a half-integer offset.

Further, the use of a unified mathematical model to perform dispersion correction processing also includes:

The pixel value is assumed to be R[n] before scaling, and the pixel value after scaling is R'[n]. According to the scaling coefficient and the linear interpolation algorithm, the corrected expression is obtained by induction:

即看到校正后的像素值由两个相关像素值线性插值得到，将计算中的复杂运算分解至多步进行，具体包括以下步骤，When 0≤n≤H _Pixel -1,

That is, it can be seen that the corrected pixel value is obtained by linear interpolation of two related pixel values, and the complex operation in the calculation is decomposed into multiple steps, which specifically includes the following steps:

5.1) Calculate the coordinate index(n) of the corrected pixel to be interpolated, where

5.2) Calculate the linear interpolation coefficient a, and its calculation expression is a=n(coef+1);

5.3) Complete the multiplication of R[index(n)] and R[index(n)+1] and the corresponding linear interpolation coefficient;

5.4) Perform the addition of R[index(n)]×a and R[index(n)+1]×(2H _Pixel -a);

5.5) Complete the division calculation of 2H _Pixel by shifting the calculation result in 5.4).

On the other hand, the present invention also discloses a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, causes the processor to execute the steps of the above method.

It can be seen from the above technical solutions that the FPGA-based dispersion correction method of the present invention uses the hardware algorithm in the FPGA to perform parallel processing on the image to offset the dispersion phenomenon on the photosensitive element after the light passes through the lens during imaging. And the pipeline processing method is used to further improve the efficiency of the algorithm.

The invention provides an FPGA-based chromatic aberration correction algorithm, which avoids chromatic dispersion from affecting the imaging effect of an industrial camera, and utilizes the parallelism and pipeline of the FPGA to complete efficient real-time image processing.

Compared with the prior art, the advantages and positive effects of the present invention are:

1. This algorithm supports real-time processing of images without preset chromatic aberration compensation table, only simple parameter input is required, and the corresponding offsets of different pixels are calculated in real-time processing. This method is more flexible and can be applied to various complex changes. in the scene.

2. Using pipeline and parallel processing methods, disassemble complex operations into multi-step pipelines to improve module processing efficiency.

3. Use a unified mathematical model to deal with complex dispersion scenes, and support half-pixel and integer-pixel dispersion correction.

4. Less computing resources are required, and the entire image is stored and processed without large storage resources.

5. It avoids the complex optical and hardware structure correction, and uses the existing FPGA to perform hardware algorithm processing on the image.

Description of drawings

Fig. 1 is the method principle schematic diagram of the present invention;

Figure 2 is a flow chart of the method of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments.

The embodiment of the present invention provides an FPGA-based chromatic aberration correction algorithm, which avoids the chromatic dispersion from affecting the imaging effect of an industrial camera, utilizes the parallelism and pipeline of the FPGA to complete efficient real-time processing of images, and realizes real-time chromatic dispersion correction for high-resolution cameras The algorithm provides the possibility, and can be applied to both line scan and area scan cameras.

最边界处像素的偏移量视为x。After being refracted through the lens, the lengths of different RGB components on the photosensitive element change. Considering the imaging characteristics of light, at the imaging center, the light converging points of different wavelengths are the same; when the converging point is far away from the imaging center, the converging points are offset. The degree of shift is increased. Therefore, in order to complete the dispersion correction, the image after dispersion needs to be stretched symmetrically around the imaging center. As shown in Figure 1, if it is further quantified, it can be regarded as a long strip of length L, keeping the midpoint unchanged, stretched to L', then its expansion coefficient is

The offset of the pixel at the most border is taken as x.

Specifically, as shown in FIG. 2 , the present invention provides an FPGA-based chromatic aberration correction algorithm. The specific steps are as follows:

Receive the image data stream, and use the on-chip RAM to store the data of the adjacent 4 clock cycles;

Select the correction method according to the boundary offset before and after correction;

Segment each row of data by center;

Dispersion correction processing using unified mathematical model;

The corrected image is output in real time.

The following specific instructions:

1) The image is obtained under the actual lighting conditions, the image is transmitted in the FPGA as streaming data, M data is transmitted in each clock cycle, and the on-chip RAM is used to store the data of the adjacent 4 clock cycles. The data of each row before correction is denoted as R[n], where n is the column value of the pixel value. Assume that the horizontal resolution of the sensor is 2H _Pixel .

2) From the data center of each row, it is equally divided into two halves for segmentation processing, respectively 0≤n≤H _Pixel -1, H _Pixel ≤n≤2H _Pixel -1.

最边界处像素的偏移量视为x。伸缩系数表达式如下式：3) The whole process is regarded as a long strip of length L, keeping the midpoint unchanged, stretched to L', then its expansion coefficient is

The offset of the pixel at the most border is taken as x. The expansion coefficient expression is as follows:

使coef为整数值来代表每种偏移量，其为奇数时，将对应整数偏移量；若为偶数则对应半整数偏移量。4) Set the offset of the boundary pixels before and after correction as x, and select integer pixel correction or half-integer pixel correction according to the actual situation. If the offset is an integer, then the value of the integer pixel before correction is used for interpolation calculation; if the offset of the boundary pixel is the interpolation of two adjacent integer pixels before, then half-integer pixel correction is used later. The former is an integer value such as x=1, 2, 3, and the latter is a half-integer value such as x=0.5, 1.5, 2.5. Considering that integer operations in hardware consume less resources than floating-point operations, set the offset

Let coef be an integer value to represent each offset, if it is odd, it will correspond to an integer offset; if it is even, it will correspond to a half-integer offset.

5) According to the expansion coefficient and the linear interpolation interpolation algorithm, the corrected expression is obtained by induction:

When 0≤n≤H _Pixel -1,

, it can be seen that the corrected pixel value is obtained by linear interpolation of two related pixel values. Taking into account the operation characteristics in the FPGA, step 5 uses the pipeline method shown in Figure 1 to decompose the complex operation in the calculation into multiple steps to avoid the timing tension problem caused by the high clock evaluation rate.

5.1) Calculate the coordinate index(n) of the corrected pixel to be interpolated, where

5.2) Calculate the linear interpolation coefficient a, and its calculation expression is a=n(coef+1);

5.3) Complete the multiplication of R[index(n)] and R[index(n)+1] and the corresponding linear interpolation coefficient;

5.4) Perform the addition of R[index(n)]×a and R[index(n)+1]×(2H _Pixel -a);

5.5) Complete the division calculation of 2H _Pixel by shifting the calculation result in 5.4).

6) Continue to transmit the real-time corrected data to other image processing modules.

使coef为整数值来代表每种偏移量，其为奇数时，将对应整数偏移量；若为偶数则对应半整数偏移量。Specifically, in the embodiment of the present invention, considering that the actual imaging situation is relatively complicated, there are many possibilities for the offset x of the boundary pixels in the correction process, and the correction is mainly abstracted into two corrections: integer pixel correction and half pixel correction. Integer pixel correction refers to setting the offset of the boundary pixel in the correction to the value of a previous integer pixel. Half-pixel correction means that the offset of the boundary pixel in the correction is the interpolation of some two adjacent integer pixels before. The former is an integer value such as x=1, 2, 3, and the latter is a half-integer value such as x=0.5, 1.5, 2.5. Considering that integer operations in hardware consume less resources than floating-point operations, set the offset

Let coef be an integer value to represent each offset, if it is odd, it will correspond to an integer offset; if it is even, it will correspond to a half-integer offset.

Assuming that the horizontal resolution of the photosensitive element is 2H _Pixel , each pixel can be abstracted as n. Considering the actual imaging process, the image is centrally symmetric about the imaging center, so the model needs to be segmented. That is, the pixels need to be divided into two segments: 0≤n≤H _Pixel -1, H _Pixel ≤n≤2H _Pixel -1. The pixel value is assumed to be R[n] before scaling, and the pixel value after scaling is R'[n]. Using the first paragraph as an example, use induction to calculate the relationship between R[n] and R'[n].

Assuming that after correction, the expansion coefficient according to the formula is

Considering the boundary conditions of the model, assuming that the pixel whose coordinate is x, after stretching, the coordinate becomes 0, that is, R'[0]=R[x], correspondingly,

Infer from this:

带入if the

bring in

It can be seen that the final formula can be regarded as

in

Now consider the section H _Pixel ≤n≤2H _Pixel -1, and also consider the boundary conditions, there is R'[2H _Pixel -1]=R[2H _Pixel -1-x], so it can be calculated according to the same method as above. the relationship between R[n] and R'[n], and write it as

的形式。在半像素情况中，也使用相同的方式进行归纳，最终都可以写为相同的表达

form. In the half-pixel case, the same way of induction is also used, and in the end both can be written as the same expression

After abstracting it into a unified calculation formula, it can be seen that complex situations can be abstracted into a unified processing method. At the same time, according to the expression of R'[n[, in the calculation, only two adjacent data Rpindex(n) ] is related to R[index(n)+1], which means that the calculation only requires a small amount of storage resources to store adjacent data, and does not need to store the entire image for calculation, which ensures good real-time processing performance of the algorithm. In the calculation, the calculation of multiple pixels is carried out at the same time, making full use of the parallelism of the FPGA and improving the processing efficiency.

In order to further improve the processing performance of the algorithm and increase the maximum clock frequency when the algorithm is running, the pipeline method shown in Figure 1 is used to decompose the complex operations in the calculation into multiple steps, which will effectively avoid the timing caused by the high clock evaluation rate. tension issues. Calculate index(n) first, calculate a after the second clock cycle, and complete R[index(n)] and R[index(n)+1] in the third clock cycle

Multiplying the corresponding coefficient, the addition of R[index(n)]×a and R[index(n)+1×(2HPixel-a) is completed in the fourth clock cycle, and the fifth clock cycle will use the above calculation result The division calculation of the 2H _Pixel is done by shifting.

In general, after abstracting it into a unified calculation formula in the embodiment of the present invention, it can be seen that complex situations can be abstracted into a unified processing method, and the calculation only requires a small amount of storage resources to store adjacent data, without the need for The whole image is stored for calculation, which ensures the good real-time processing performance of the algorithm. In the calculation, the calculation of multiple pixels is carried out at the same time, making full use of the parallelism of the FPGA and improving the processing efficiency.

In order to further improve the processing performance of the algorithm and increase the maximum clock frequency when the algorithm is running, the pipeline method is used to decompose the complex operations in the calculation into multiple steps, which will effectively avoid the timing tightness caused by the high clock evaluation rate.

To sum up, the technical features of the present invention are as follows:

An FPGA-based chromatic aberration correction algorithm according to the embodiment of the present invention uses a unified method to calculate different chromatic dispersion scenes, and adopts a real-time image processing method, and the characteristics are as follows:

方式进行计算。1. Use the offset image

way to calculate.

2. Using pipeline and parallel processing methods, disassemble complex operations into multi-step pipelines to improve module processing efficiency.

3. It can cope with complex dispersion scenes, and supports half-pixel and integer-pixel dispersion correction.

4. Less computing resources are required, and the entire image is stored and processed without large storage resources.

In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to execute the steps of the above method.

In another aspect, the present invention also discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to execute the steps of the above method.

In yet another embodiment provided by the present application, a computer program product including instructions is also provided, which, when running on a computer, enables the computer to execute any of the FPGA-based dispersion correction methods in the foregoing embodiments.

It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and reference may be made to the corresponding part of the above-mentioned method for the explanation, examples and beneficial effects of the related content.

Embodiments of the present application further provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus,

memory for storing computer programs;

The processor is used to implement the above-mentioned FPGA-based dispersion correction method when executing the program stored in the memory, and the method includes:

Receive the image data stream, and use the on-chip RAM to store the data of the adjacent 4 clock cycles;

Select the correction method according to the boundary offset before and after correction;

Segment each row of data by center;

Dispersion correction processing using unified mathematical model;

The corrected image is output in real time.

The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (English: Peripheral Component Interconnect, abbreviated: PCI) bus or an Extended Industry Standard Architecture (English: Extended Industry Standard Architecture, abbreviated: EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include random access memory (English: Random Access Memory, short: RAM), or may include non-volatile memory (English: Non-Volatile Memory, short: NVM), for example, at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (English: Central Processing Unit, referred to as: CPU), a network processor (English: Network Processor, referred to as: NP), etc.; or a digital signal processor (English: Digital Signal Processing, referred to as: DSP), application specific integrated circuit (English: ApplicationSpecific Integrated Circuit, referred to as: ASIC), Field Programmable Gate Array (English: Field-Programmable Gate Array, referred to as: FPGA) or other programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), among others.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (6)

1. A dispersion correction method based on FPGA is characterized by comprising the following steps,

receiving an image data stream, and storing data of 4 adjacent clock periods by using an on-chip RAM;

selecting a correction mode according to the boundary offset before and after correction;

carrying out segmentation processing on each line of data by using a center;

carrying out dispersion correction processing by using a unified mathematical model;

and outputting the corrected image in real time.

2. The FPGA-based dispersion correction method of claim 1, wherein: the receiving of the image data stream and the storing of the data of adjacent 4 clock cycles using the on-chip RAM specifically include,

obtaining an image under an actual lighting condition, transmitting the image in a FPGA by streaming data, transmitting M data in each clock period, storing the data of 4 adjacent clock periods by using an on-chip RAM, and marking the data of each row as R [ n ] before correction, wherein n is a column value of the pixel value;

the whole process is regarded as that the strip with the length L is stretched into L' under the condition that the middle point is kept unchanged, and the expansion coefficient is

The shift of the pixels at the extreme boundary is taken as x, and the horizontal resolution of the photosensitive element is assumed to be 2H_PixelThe coefficient of expansion is expressed by the following formula:

3. the FPGA-based dispersion correction method of claim 2, wherein: the step of performing the segmentation processing on each line of data with the center specifically comprises,

dividing each row of data center into two halves for segmented processing, wherein n is more than or equal to 0 and less than or equal to H_Pixel-1,H_Pixel≤n≤2H_Pixel-1。

4. The FPGA-based dispersion correction method of claim 3, wherein: the dispersion correction process using the unified mathematical model includes, in particular,

setting the offset of the boundary pixel before and after correction as x, and selecting integer pixel correction or half-integer pixel correction according to the actual situation;

if the offset is an integer, then performing interpolation calculation by using the value of the integer pixel before correction; if the offset of the boundary pixel is the interpolation of a certain two adjacent integer pixels, then correcting by using a half integer pixel;

the former, x, is an integer value, the latter, x, is a half-integer value, and the offset is set

Let coef be an integer value to represent each offset, and if it is an odd number, it will correspond to the integer offset; if the number is even, the offset corresponds to a half integer offset.

5. The FPGA-based dispersion correction method of claim 4, wherein: the dispersion correction process using the unified mathematical model, further comprising,

assuming that the pixel value is R [ n ] before stretching, the pixel value after stretching is R' n, and obtaining a corrected pixel value expression by a reduction method according to the stretching coefficient and a linear interpolation algorithm, wherein the corrected pixel value expression is as follows:

when n is more than or equal to 0 and less than or equal to H_PixelWhen the reaction temperature is 1, adding a catalyst,

that is, the corrected pixel value is obtained by linear interpolation of two related pixel values, the complex operation in the calculation is decomposed into multiple steps, and the method specifically comprises the following steps,

5.1) calculating the coordinates index (n) of the corrected interpolated pixel, where

5.2) calculating a linear interpolation coefficient a, wherein the calculation expression is that a is n (coef + 1);

5.3) completing the multiplication of R [ index (n) ] and R [ index (n)) +1] and the corresponding linear interpolation coefficient;

5.4) execute R [ index (n)]X a and R [ index (n) +1]×(2H_Pixel-addition of a);

5.5) to 5.4) the computation results are shifted to complete the pair 2H_PixelAnd (4) calculating the division of (1).

6. A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 5.

Images