CN114581380A

CN114581380A - Dispersion correction method based on FPGA and storage medium

Info

Publication number: CN114581380A
Application number: CN202210142266.4A
Authority: CN
Inventors: 刘若琳; 杨晨飞; 王淑文
Original assignee: Hefei Eko Photoelectric Technology Co ltd
Current assignee: Hefei Eko Photoelectric Technology Co ltd
Priority date: 2022-02-16
Filing date: 2022-02-16
Publication date: 2022-06-03
Anticipated expiration: 2042-02-16
Also published as: CN114581380B

Abstract

The invention relates to a dispersion correction method and a storage medium based on FPGA, wherein the method comprises the following steps of receiving image data flow and storing data of adjacent 4 clock periods by using an on-chip RAM; selecting a correction mode according to the boundary offset before and after correction; carrying out segmentation processing on each line of data by using a center; carrying out dispersion correction processing by using a unified mathematical model; and outputting the corrected image in real time. The dispersion correction method based on the FPGA uses a hardware algorithm in the FPGA to perform parallel processing on the image and counteract the dispersion phenomenon generated on the photosensitive element after light passes through the lens in imaging. And the algorithm efficiency is further improved by adopting a pipeline processing mode. The algorithm supports real-time processing of images, a chromatic aberration compensation table does not need to be preset, only simple parameter input is needed, and corresponding offsets of different pixels are calculated in real-time processing.

Description

Dispersion correction method based on FPGA and storage medium

Technical Field

The invention relates to the technical field of image processing, in particular to a dispersion correction method based on an FPGA (field programmable gate array).

Background

With the industry flourishing, high-speed industrial cameras face more complex scene requirements in use. In actual use, the shot image has dispersion phenomenon, which affects the imaging effect. This is because the refractive index of the lens for light with different wavelengths is different, so when the light is refracted to the sensor through the lens group, the phenomenon that the light with different wavelengths converges at different points occurs. This phenomenon is more severe and noticeable at a distance from the center of the image.

In order to improve the imaging quality and counteract the chromatic dispersion, there are two main solutions, but there are some problems: the first is mainly to improve or even cancel the dispersion by optical methods such as achromatic lens, etc., but this method needs to reconsider the optical structure, which increases the complexity of the optical and hardware architecture design and increases the cost. The second method is mainly to store data and perform processing by software algorithm or the like after the image acquisition is finished. The method has low processing efficiency, which causes the use of the method to have large scene limitation and is not suitable for a high-speed real-time image acquisition system.

In order to solve the above problems, while considering a high-speed real-time transmission scenario of an industrial camera, and to reduce complexity and cost in industrial camera design as much as possible, it is necessary to adopt a dispersion correction method having high efficiency, real-time performance, and low cost.

Disclosure of Invention

The invention provides a dispersion correction method based on FPGA, which can at least solve one of the technical problems.

In order to achieve the purpose, the invention adopts the following technical scheme:

an FPGA-based dispersion correction method comprises the following steps:

receiving an image data stream, and storing data of 4 adjacent clock periods by using an on-chip RAM;

selecting a correction mode according to the boundary offset before and after correction;

carrying out segmentation processing on each line of data by using a center;

carrying out dispersion correction processing by using a unified mathematical model;

and outputting the corrected image in real time.

Further, the receiving the image data stream and using the on-chip RAM to store the data of 4 adjacent clock cycles specifically includes,

obtaining an image under an actual lighting condition, transmitting the image in a FPGA by streaming data, transmitting M data in each clock period, storing the data of 4 adjacent clock periods by using an on-chip RAM, and marking the data of each row as R [ n ] before correction, wherein n is a column value of the pixel value;

the whole process is regarded as that the strip with the length L is stretched into L' under the condition that the middle point is kept unchanged, and the expansion coefficient is

The shift of the pixels at the extreme boundary is taken as x, and the horizontal resolution of the photosensitive element is assumed to be 2H_PixelThe coefficient of expansion is expressed by the following formula:

further, the performing, by the center, the segmentation processing on each line of data specifically includes,

dividing each row of data center into two halves for segmented processing, wherein n is more than or equal to 0 and less than or equal to H_Pixel-1,H_Pixel≤n≤2H_Pixel-1。

Further, the dispersion correction process using the unified mathematical model specifically includes,

setting the offset of the boundary pixel before and after correction as x, and selecting integer pixel correction or half integer pixel correction according to the actual situation;

if the offset is an integer, then performing interpolation calculation by using the value of the integer pixel before correction; if the offset of the boundary pixel is the interpolation of a certain two adjacent integer pixels, then correcting by using a half integer pixel;

the former, x, is an integer value, the latter, x, is a half-integer value, and the offset is set

Let coef be an integer value to represent each offset, and if it is an odd number, it will correspond to the integer offset; if the number is even, the offset corresponds to a half integer offset.

Further, the performing dispersion correction processing using the unified mathematical model further includes,

assuming that the pixel value is R [ n ] before expansion and contraction and the pixel value is R' n after expansion and contraction, obtaining a corrected expression by a reduction method according to the expansion coefficient and a linear interpolation algorithm:

when n is more than or equal to 0 and less than or equal to H_PixelWhen the reaction temperature is 1, adding a catalyst,

namely, the corrected pixel value is obtained by linear interpolation of two related pixel values, the complex operation in the calculation is decomposed into multiple steps to be carried out, and the method specifically comprises the following steps,

5.1) calculating the coordinates index (n) of the corrected interpolated pixel, where

5.2) calculating a linear interpolation coefficient a, wherein the calculation expression is that a is n (coef + 1);

5.3) completing the multiplication of R [ index (n) ] and R [ index (n)) +1] and the corresponding linear interpolation coefficient;

5.4) Perform R [ index (n)]X a and R [ index (n) +1]×(2H_Pixel-addition of a);

5.5) to 5.4) the computation results are shifted to complete the pair 2H_PixelAnd (4) calculating the division of (1).

In another aspect, the present invention also discloses a computer readable storage medium storing a computer program, which when executed by a processor causes the processor to perform the steps of the method as described above.

According to the technical scheme, the dispersion correction method based on the FPGA uses a hardware algorithm in the FPGA to perform parallel processing on the image, and counteracts the dispersion phenomenon generated on the photosensitive element after the light passes through the lens in imaging. And the algorithm efficiency is further improved by adopting a pipeline processing mode.

The invention provides a color difference correction algorithm based on an FPGA (field programmable gate array), which avoids the influence of a dispersion phenomenon on the imaging effect of an industrial camera and utilizes the parallelism and the production line of the FPGA to finish the high-efficiency real-time processing of an image.

Compared with the prior art, the invention has the advantages and positive effects that:

1. the algorithm supports real-time processing of images, a color difference compensation table does not need to be preset, only simple parameter input is needed, and corresponding offset of different pixels is calculated in real-time processing.

2. And a flow line and a parallel processing mode are adopted, complex operation is disassembled into a multi-step flow line for carrying out, and the module processing efficiency is improved.

3. The use of a unified mathematical model can cope with complex dispersion scenes, and support dispersion correction of half-pixel and integer-pixel.

4. The required computing resources are few, and the whole image is processed after being stored without larger storage resources.

5. The method avoids the correction of complex optical and hardware structures, and utilizes the existing FPGA to perform hardware algorithm processing on the image.

Drawings

FIG. 1 is a schematic diagram of the method of the present invention;

fig. 2 is a flow chart of the method of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.

The embodiment of the invention provides a chromatic aberration correction algorithm based on an FPGA (field programmable gate array), which avoids the influence of chromatic dispersion on the imaging effect of an industrial camera, utilizes the parallelism and the production line of the FPGA to finish the high-efficiency real-time processing of an image, provides possibility for a high-resolution camera to realize a real-time chromatic dispersion correction algorithm, and can be applied to an area-array camera and a line-array camera.

After refraction through the lens, the lengths of different components of RGB on the photosensitive element are changed, and light convergence points with different wavelengths are the same at the imaging center in consideration of the imaging characteristics of light; when the convergent point is far away from the imaging center, the convergent point is shifted to a larger extent. Therefore, in order to complete dispersion correction, the dispersed image needs to be stretched around the imaging center. As shown in FIG. 1, when the length of a strip L is further quantified, the stretching of the strip L' with the midpoint being constant will result in a coefficient of expansion and contraction of L

The offset of the pixel at the extreme boundary is considered x.

Specifically, as shown in fig. 2, the invention provides an FPGA-based chromatic aberration correction algorithm, which comprises the following specific steps:

carrying out segmentation processing on each line of data by using a center;

and outputting the corrected image in real time.

The following is a detailed description:

1) the method comprises the steps of obtaining images under actual lighting conditions, transmitting the images in a FPGA in streaming data, transmitting M data in each clock period, and storing the data of 4 adjacent clock periods by using an on-chip RAM. The data for each line before correction is recorded as R [ n ]]Where n is the column value of the pixel value. Assuming that the horizontal resolution of the photosensitive element is 2H_Pixel。

2) Dividing each row of data center into two halves for segmented processing, wherein n is more than or equal to 0 and less than or equal to H_Pixel-1,H_Pixel≤n≤2H_Pixel-1。

3) The whole process is regarded as that the strip with the length L is stretched into L' under the condition that the middle point is kept unchanged, and the expansion coefficient is

The offset of the pixel at the extreme boundary is taken as x. The coefficient of expansion is expressed as follows:

4) and setting the offset of the boundary pixel before and after correction as x, and selecting integer pixel correction or half integer pixel correction according to the actual situation. If the offset is an integer, then performing interpolation calculation by using the value of the integer pixel before correction; if the offset of the boundary pixel is the interpolation of some two adjacent integer pixels before, then the half integer pixel correction is used afterwards. The former is an integer value such as x being 1,2,3, and the latter is a half-integer value such as x being 0.5,1.5, 2.5. Considering that integer arithmetic consumes less resources in hardware than floating-point arithmetic, the offset is set

5) Obtaining a corrected expression through a induction method according to the expansion coefficient and a linear interpolation algorithm:

it can be seen that the corrected pixel value is obtained by linear interpolation of the two correlated pixel values. Considering the operation characteristics in the FPGA, the step five uses a pipeline mode as shown in figure 1, and the complex operation in the calculation is decomposed into multiple steps to be carried out, so that the problem of time sequence tension caused by high clock evaluation rate is avoided.

5.2) calculating a linear interpolation coefficient a, wherein the calculation expression is that a is equal to n (coef + 1);

5.4) execute R [ index (n)]X a and R [ index (n) +1]×(2H_Pixel-addition of a);

6) And continuously transmitting the data after real-time correction to other image processing modules.

Specifically, in the embodiment of the present invention, considering that the actual imaging is complex, there are many possibilities for the offset x of the boundary pixel in the correction process, and the main abstraction in the correction is two corrections: integer pixel and half-pixel correction. Integer pixel correction refers to the offset of the center boundary pixel in the correction to the value of some integer pixel before. Half-pixel correction is the interpolation of some two adjacent integer pixels before the offset of the boundary pixel in correction. The former is an integer value such as x being 1,2,3, and the latter is a half-integer value such as x being 0.5,1.5, 2.5. Considering that integer arithmetic consumes less resources in hardware than floating-point arithmetic, the offset is set

Assuming the photosensitive elementThe horizontal resolution of the member is 2H_PixelEach pixel can be abstracted as n, and the image is centrosymmetric about the imaging center in consideration of the actual imaging process, so that the model needs to be segmented. I.e. the pixel needs to be divided into two segments: n is more than or equal to 0 and less than or equal to H_Pixel-1,H_Pixel≤n≤2H_Pixel-1. Assume its pixel value to be R [ n ] before scaling]The pixel value after expansion and contraction is R' [ n ]]. Taking the first paragraph as an example, R [ n ] is calculated using the induction method]And R' [ n ]]The relationship between them.

Assuming that after correction, the coefficient of expansion is given by the formula

Considering the boundary condition of the model, assuming that a pixel with coordinate x is stretched, the coordinate becomes 0, i.e. R' 0 ═ R x, accordingly,

it follows from this that:

if it will be

Bringing in

It can be seen that the final formula can be viewed as

Wherein

Now consider H_Pixel≤n≤2H_PixelParagraph 1, also taking into account the boundary conditions, has R' [2H ]_Pixel-1]＝R[2H_Pixel-1-x]Thus, R [ n ] can be calculated also according to the same method as described above]And R' [ n ]]And write it as

In the form of (1). In the half-pixel case, generalizations are also made in the same way, and eventually can be written with the same expression

After the method is abstracted into a uniform calculation formula, the complex situations can be abstracted into a uniform processing mode, and meanwhile, according to the expression of R' [ n ], the complex situations are only related to two adjacent data Rindex (n) and R [ index (n) +1] in the calculation, which means that the calculation only needs a small amount of storage resources to store adjacent data, the whole image does not need to be stored for calculation, and the good real-time processing performance of the algorithm is guaranteed. In the calculation, the calculation of a plurality of pixels is carried out simultaneously, the parallelism of the FPGA is fully utilized, and the processing efficiency is improved.

In order to further improve the processing performance of the algorithm and the highest clock frequency during the operation of the algorithm, a pipeline mode as shown in fig. 1 is used, and complex operations in the calculation are divided into multiple steps to be carried out, so that the problem of time sequence tension caused by high clock rate is effectively avoided. First, calculate index (n), calculate a after the second clock cycle, finish R [ index (n) ] and R [ index (n) +1] in the third clock cycle

Multiplying the corresponding coefficient, and finishing R [ index (n) at the fourth clock period]Addition of x a and R [ index (n) +1 × (2HPixel-a), the fifth clock cycle will calculate the aboveResults complete Pair 2H by Shift_PixelAnd (4) calculating the division of (1).

In summary, after the embodiment of the invention abstracts the data into a unified calculation formula, it can be seen that complex situations can be abstracted into a unified processing mode, and meanwhile, the calculation only needs a small amount of storage resources to store adjacent data, and does not need to store the whole image for calculation, thereby ensuring good real-time processing performance of the algorithm. In the calculation, the calculation of a plurality of pixels is carried out simultaneously, the parallelism of the FPGA is fully utilized, and the processing efficiency is improved.

In order to further improve the processing performance of the algorithm and the highest clock frequency during the operation of the algorithm, a pipeline mode is used for decomposing complex operation in calculation into multiple steps, and the problem of time sequence tension caused by high clock evaluation rate is effectively avoided.

In summary, the technical features of the present invention are as follows:

the color difference correction algorithm based on the FPGA of the embodiment of the invention calculates different dispersion scenes in a unified mode, adopts a real-time image processing mode and has the following characteristics:

1. biased image usage

The mode is calculated.

3. The method can cope with complex dispersion scenes and support dispersion correction of half-pixel and integer-pixel.

4. The required computing resources are few, and the whole image is stored and then processed without large storage resources.

In yet another aspect, the present invention also discloses a computer readable storage medium storing a computer program, which when executed by a processor causes the processor to perform the steps of the method as described above.

In yet another aspect, the present invention also discloses a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the above method.

In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the above-described FPGA-based dispersion correction methods.

It can be understood that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and for the explanation, examples and beneficial effects of the relevant contents, reference may be made to the corresponding parts in the above method.

The embodiment of the application also provides an electronic device, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus,

a memory for storing a computer program;

a processor, configured to implement the dispersion correction method based on the FPGA when executing a program stored in a memory, where the method includes:

receiving an image data stream, and storing data of adjacent 4 clock periods by using an on-chip RAM;

carrying out segmentation processing on each line of data by using a center;

and outputting the corrected image in real time.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, or discrete hardware components.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A dispersion correction method based on FPGA is characterized by comprising the following steps,

carrying out segmentation processing on each line of data by using a center;

and outputting the corrected image in real time.

2. The FPGA-based dispersion correction method of claim 1, wherein: the receiving of the image data stream and the storing of the data of adjacent 4 clock cycles using the on-chip RAM specifically include,

3. the FPGA-based dispersion correction method of claim 2, wherein: the step of performing the segmentation processing on each line of data with the center specifically comprises,

4. The FPGA-based dispersion correction method of claim 3, wherein: the dispersion correction process using the unified mathematical model includes, in particular,

setting the offset of the boundary pixel before and after correction as x, and selecting integer pixel correction or half-integer pixel correction according to the actual situation;

5. The FPGA-based dispersion correction method of claim 4, wherein: the dispersion correction process using the unified mathematical model, further comprising,

assuming that the pixel value is R [ n ] before stretching, the pixel value after stretching is R' n, and obtaining a corrected pixel value expression by a reduction method according to the stretching coefficient and a linear interpolation algorithm, wherein the corrected pixel value expression is as follows:

that is, the corrected pixel value is obtained by linear interpolation of two related pixel values, the complex operation in the calculation is decomposed into multiple steps, and the method specifically comprises the following steps,

5.4) execute R [ index (n)]X a and R [ index (n) +1]×(2H_Pixel-addition of a);

6. A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 5.