CN111199520A - FPGA implementation method for color image scale expansion based on cubic convolution algorithm - Google Patents

Abstract

The invention discloses a color image scale expansion FPGA implementation method based on a cubic convolution algorithm, which optimizes the algorithm according to the basic principle of a cubic convolution interpolation algorithm, and optimizes floating-point matrix operation into parameter template M operation with integer coefficients; and reading the color image file by using the matlab, and processing the FPGA image according to the optimized cubic convolution interpolation algorithm. The processed image has clear edge and fine gray level change, and the defects of fuzzy edge and discontinuous gray level of the image processed by the nearest neighbor interpolation method and the bilinear interpolation method are overcome.

Description

FPGA implementation method for color image scale expansion based on cubic convolution algorithm

Technical Field

The invention belongs to the field of digital signal processing, and particularly relates to an FPGA (field programmable gate array) implementation method for color image scale expansion based on a cubic convolution algorithm.

Background

In the process of identifying the target of a human in a loop, when a pilot or an operator selects a tracking target, the selection is difficult due to the fact that the distance is long and the target is small. If the scale of the small target image shot in a long distance is enlarged by 4 times or more, the resolution of the image is improved, the target of the image is clearly visible and is easy to distinguish, and the difficulty of selecting a tracking target is greatly reduced.

One of the image scaling is to change the size of an image as needed to reduce or enlarge the image in a certain ratio. In the digital geometric transformation, the image scale transformation belongs to a more complex operation, different algorithms have a great influence on the quality of the processed image, and the pixels of the processed new image can not be found in the original image and can only be approximately processed by an interpolation method. Common scale conversion algorithms include nearest neighbor interpolation, bilinear interpolation, and cubic convolution interpolation. Wherein, the nearest neighbor interpolation method has serious sawtooth phenomenon at the edge of the processed image; the image edge after bilinear interpolation has obvious grid phenomenon, the edge gray scale is discontinuous, and the resolution of the image can not be effectively improved. The cubic convolution interpolation method has large operation amount, complex floating point type operation exists, and template control operation is not facilitated.

Disclosure of Invention

In view of the above problems, the present invention needs to provide a method for enlarging the scale of a color image, which can effectively improve the resolution of the image, and the processed image has clear edges and fine gray scale change.

In order to solve the technical problem, the invention provides an FPGA implementation method for color image scale expansion based on a cubic convolution algorithm, which comprises the following steps:

step 1, optimizing an algorithm according to the basic principle of a cubic convolution interpolation algorithm, and optimizing floating-point type matrix operation into parameter template M operation with integral coefficients;

and 2, reading the color image file by using matlab, and processing the FPGA image according to the cubic convolution interpolation algorithm optimized in the step 1.

Further, the step 1 is realized by the following method:

if the gray value f' (x, y) of any point (x, y) of the output image corresponds to the gray value f (i + u, j + v) with the input source image coordinate (i + u, j + v), wherein i, j, u and v are shown in formula (1):

i＝[x/2],j＝[y/2],u＝x/2-i,v＝y/2-j (1)

f(i+u,j+v)＝A×B×C (2)

in the formula (2), A, B and C are all matrixes.

A＝[S(1+u) S(u) S(1-u) S(2-u)](3)

Wherein S (w) is an approximation of sin (w)/w;

substituting equations (3), (4) and (5) into equation (2) to obtain f (i + u, j + v) ═ axbxc ═ mxb,

wherein M is shown in the formula (7),

further, the step 2 performs the FPGA image processing according to the optimized cubic convolution interpolation algorithm, and includes the following steps:

the method comprises the following steps that (1) according to input image parameters, the line cache depth in an FPGA and the serial connection framework setting between a line cache and a trigger are carried out;

step (2) controlling the reading time of the line cache;

and (3) calculating the gray level of the processed image.

Preferably, the step (1) sets the depth of the line cache in the FPGA and sets the serial and parallel connection architecture between the line cache and the flip-flop, and the specific implementation method is as follows:

4 FIFOs with the depth of 1024 bytes are generated in the FPGA, the 4 FIFOs are connected end to end, 3 triggers are connected behind each FIFO, and the depth of the FIFOs is larger than the number of pixels of each line of the input source image.

The step (2) of controlling the read timing of the line cache comprises the following processes:

when reading FIFO, reading every 3 clock cycles, keeping the data of the output end of the trigger at 3 clock cycles, performing serial processing of RGB three-color information, and in the serial processing process, sequentially starting R, G, B interpolation operation of single-path data by 3 clock edges;

when processing a certain line of data of an output image, the line coordinate x is not changed, the column coordinate y is changed alternately even and odd, the parallel processing simultaneously starts the interpolation operation of which y is odd and y is even, and the parameter template M also needs to be changed according to the parity of y during the operation.

The step (3) calculates the processed image gray scale, specifically as follows:

if the row coordinates x and y are both odd numbers, the parameter template M is shown as a formula (8), the diagonal angles of the parameters are consistent, by utilizing the characteristic, the elements of the corresponding positions of the 1 st column and the 4 th column, the 2 nd column and the 3 rd column of the output end of the trigger are added during calculation to obtain a 4 x 2 matrix B1, the parameter template corresponding to the B1 is a matrix M1, then the corresponding elements of the 1 st row and the 4 th row, the 2 nd row and the 3 rd row in the matrix B1 are added to obtain a 2 x 2 matrix B2, and the B2 corresponds to the moment thereof

The matrix M2, M1 and M2 are shown in formula (9); adding element B2(1,2) and B2(2,1) in matrix B2, and multiplying by coefficient M2(2,1) to obtain sum 1; multiplying B2(2,2) by a parameter matrix M2(2,2), and adding B2(1,1) to obtain sum 2; finally, subtracting sum1 from sum2, and dividing by 64 to obtain the final result f' (x, y).

The invention has the following beneficial effects:

the method optimizes the algorithm according to the basic principle of the cubic convolution interpolation algorithm, optimizes the floating-point matrix operation with large calculation amount into the parameter template operation with integer coefficients, and adapts to FPGA transplantation of the algorithm. The processing characteristics of the FPGA are fully utilized, a serial connection framework between the FIFO and the trigger in the FPGA and the flow calculation of the parameter template are designed, and the real-time processing of color image scale enlargement is realized. The gray level of the processed image edge is excessively natural, the outline is fine and smooth, and no obvious saw tooth or net phenomenon exists. The invention can enlarge the global or local target of the image, and is beneficial to the selection of the moving target image when a pilot or an operator tracks the target in the process of identifying the target of the loop.

Drawings

FIG. 1 is a diagram of the FIFO connection architecture of the present invention;

FIG. 2 is a schematic diagram of the calculation process of cubic convolution interpolation according to the present invention;

FIG. 3 is a comparison of image edges after processing by three interpolation algorithms in accordance with the present invention;

Detailed Description

The specific implementation mode of the invention provides an FPGA implementation method for color image scale expansion based on a cubic convolution algorithm, which comprises the following steps: step 1: basic principle and optimization of cubic convolution; step 2: determining the depth of a line cache and a connection framework of the line cache according to the input image parameters; and step 3: controlling the reading time of the line cache according to the step 1 and the step 2; step 4, calculating the gray level of the processed image; and 5, realizing the real-time performance of the algorithm by the FPGA.

The steps of the above method will be described in detail with reference to fig. 1 to 3.

Step 1: the basic principle of cubic convolution and optimization. The cubic convolution is also called an inner cubic interpolation method, and the basic principle is as follows: if the gray value f' (x, y) of any point (x, y) of the output image corresponds to the gray value f (i + u, j + v) with the input source image coordinate (i + u, j + v), wherein i, j, u and v are shown in formula (1):

i＝[x/2],j＝[y/2],u＝x/2-i,v＝y/2-j (1)

f(i+u,j+v)＝A×B×C (2)

in the formula (2), A, B and C are all matrixes.

A＝[S(1+u) S(u) S(1-u) S(2-u)](3)

Where S (w) is an approximation of sin (w)/w.

The equations (3), (4) and (5) are substituted into the equation (2) to obtain f (i + u, j + v) ═ a × B × C ═ M × B, where M is represented by the equation (7).

Substituting u and v into formula (6) according to the parity of any point coordinate (x, y) of the output image to obtain the specific coefficient corresponding to the matrix M, multiplying M and the element at the corresponding position of B, and summing to obtain the gray value f (i + u, j + v)

Step 2: and determining the depth of the line cache and the connection architecture of the line cache according to the input image parameters. The input data is color RGB with 24 bits, image size 960x540, i.e. 960 RGB data per line, 540 lines per frame, and the input clock frequency is 40 Mhz.

In fpga, a series-parallel combined processing mode is adopted to perform real-time processing of enlarging the scale by 2 times on RGB three-way information of a color digital image. Firstly, 4 FIFOs with the depth of 1024 bytes are generated in the FPGA, the 4 FIFOs are connected end to end, and 3 triggers are connected behind each FIFO so as to register 4 columns of data of each row, wherein the connection mode is shown in figure 1. The depth of the FIFO is greater than the number of pixels per line of the input source image. The structure is that the data read from the previous FIFO is written into the next FIFO, 4 rows of data are stored simultaneously, the read enable of 4 FIFOs is consistent, so that 4 rows and 4 columns of data appear at the output end of a trigger simultaneously, and interpolation operation is carried out

And step 3: according to the step 1 and the step 2, the reading time of the line cache is controlled. When reading the FIFO, reading once every 3 clock cycles, keeping the data of the output end of the trigger for 3 clock cycles, and performing serial processing of RGB three-color information. In the serial processing process, 3 clock edges sequentially start R, G, B interpolation operation of single-path data, so that registers of each path of data can be multiplexed, and logic resources of fpga are saved; when processing a certain row of data of an output image, the row coordinate x is unchanged and the column coordinate y is odd-even alternation. Therefore, the parallel processing means that the interpolation operation in which y is an odd number and y is an even number is started at the same time, and the parameter template M also needs to be changed according to the parity of y during the operation.

And 4, step 4: calculating the gray level of the processed image; if the row coordinates x and y are both odd numbers, the parameter template M has the same diagonal as the parameter shown in equation (8). By using the characteristic, the elements at the corresponding positions of the 1 st column, the 4 th column, the 2 nd column and the 3 rd column of the output end of the trigger are added during calculation to obtain a 4 x 2 matrix B1, and a parameter template corresponding to B1 is a matrix M1. Adding corresponding elements of the 1 st row and the 4 th row, the 2 nd row and the 3 rd row in the matrix B1 to obtain a 2 x 2 matrix B2, wherein the B2 corresponds to the matrixes M1 and M2 which are as common

Formula (9). Since the elements M2(2,1) and M2(1,2) of the parameter template M2 are identical and are both 5, the elements B2(1,2) and B2(2,1) in the matrix B2 are added and multiplied by the coefficient M2(2,1) to obtain sum 1; multiplying B2(2,2) by a parameter matrix M2(2,2), and adding B2(1,1) to obtain sum 2; finally, subtracting sum1 from sum2, and dividing by 64 to obtain the final result f' (x, y), the calculation process is shown in FIG. 2. Only 2 fpga multipliers are used in the calculation process, and because RGB three-color information is processed in series and the registers are also multiplexed, the calculation process is delayed by 6 clock cycles

And 5: and the FPGA realizes the real-time performance of the algorithm. According to the characteristic of the parameter template M, when any one of the row coordinate x and the column coordinate y is an odd number, the processing process only uses 1 multiplier of fpga, and 4 clock cycles are delayed; when both the row coordinate x and the column coordinate y are even, the processing is delayed by only 1 clock cycle. The whole calculation process is the processing of a pipeline, and the processing time is not additionally occupied. During board level testing, the TXT file written by matlab is used as a data source and is solidified into the on-chip ROM for actual board level testing. And in the testing process, Chipscope software is used for randomly monitoring the processed image data of a certain line, and the image data is compared with the simulated color image data, so that the calculation result is consistent.

With the present invention, a color image (fig. 3(a)) is processed, and the result after the processing is shown in fig. 3 (d). Compared with the image processed by the nearest neighbor interpolation method (figure 3(b)) and the image processed by the bilinear interpolation method (figure 3(c)), the method overcomes the defects of the two interpolation methods, and has natural image edge gray scale process, fine outline and no obvious sawtooth and net phenomenon.

Claims (6)

1. An FPGA implementation method for color image scale expansion based on a cubic convolution algorithm is characterized by comprising the following steps:

step 1, optimizing an algorithm according to the basic principle of a cubic convolution interpolation algorithm, and optimizing floating-point type matrix operation into parameter template M operation with integral coefficients;

and 2, reading the color image file by using matlab, and processing the FPGA image according to the cubic convolution interpolation algorithm optimized in the step 1.

2. The FPGA implementation method for color image scale expansion based on the cubic convolution algorithm as claimed in claim 1, wherein: the step 1 is realized by the following method:

if the gray value f' (x, y) of any point (x, y) of the output image corresponds to the gray value f (i + u, j + v) with the input source image coordinate (i + u, j + v), wherein i, j, u and v are shown in formula (1):

i＝[x/2],j＝[y/2],u＝x/2-i,v＝y/2-j (1)

f(i+u,j+v)＝A×B×C (2)

in the formula (2), A, B and C are all matrixes.

A＝[S(1+u) S(u) S(1-u) S(2-u)](3)

Wherein S (w) is an approximation of sin (w)/w;

substituting equations (3), (4) and (5) into equation (2) to obtain f (i + u, j + v) ═ axbxc ═ mxb,

wherein M is shown in the formula (7),

3. the FPGA implementation method for color image scale expansion based on the cubic convolution algorithm as claimed in claim 1, wherein: the step 2 of processing the FPGA image according to the optimized cubic convolution interpolation algorithm comprises the following steps:

the method comprises the following steps that (1) according to input image parameters, the line cache depth in an FPGA and the serial connection framework setting between a line cache and a trigger are carried out;

step (2) controlling the reading time of the line cache;

and (3) calculating the gray level of the processed image.

4. The FPGA implementation method for color image scale expansion based on the cubic convolution algorithm as claimed in claim 3, wherein: the step (1) of setting the line cache depth in the FPGA and setting the serial connection architecture between the line cache and the trigger comprises the following specific implementation methods:

4 FIFOs with the depth of 1024 bytes are generated in the FPGA, the 4 FIFOs are connected end to end, 3 triggers are connected behind each FIFO, and the depth of the FIFOs is larger than the number of pixels of each line of the input source image.

5. The FPGA implementation method for color image scale expansion based on the cubic convolution algorithm as recited in claim 4, wherein: the step (2) of controlling the read timing of the line cache comprises the following processes:

when reading FIFO, reading every 3 clock cycles, keeping the data of the output end of the trigger at 3 clock cycles, performing serial processing of RGB three-color information, and in the serial processing process, sequentially starting R, G, B interpolation operation of single-path data by 3 clock edges;

when processing a certain line of data of an output image, the line coordinate x is not changed, the column coordinate y is changed alternately even and odd, the parallel processing simultaneously starts the interpolation operation of which y is odd and y is even, and the parameter template M also needs to be changed according to the parity of y during the operation.

6. The FPGA implementation method for color image scale expansion based on the cubic convolution algorithm as recited in claim 3 or 5, wherein: the step (3) calculates the processed image gray scale, specifically as follows:

if the row coordinates x and y are both odd numbers, the parameter template M is shown as a formula (8), the diagonal angles of the parameters are consistent, by utilizing the characteristic, the elements of the corresponding positions of the 1 st column and the 4 th column, the 2 nd column and the 3 rd column of the output end of the trigger are added during calculation to obtain a 4 x 2 matrix B1, the parameter template corresponding to the B1 is a matrix M1, then the corresponding elements of the 1 st row and the 4 th row, the 2 nd row and the 3 rd row in the matrix B1 are added to obtain a 2 x 2 matrix B2, and the B2 corresponds to the moment thereof

The matrix M2, M1 and M2 are shown in formula (9); adding element B2(1,2) and B2(2,1) in matrix B2, and multiplying by coefficient M2(2,1) to obtain sum 1; multiplying B2(2,2) by a parameter matrix M2(2,2), and adding B2(1,1) to obtain sum 2; finally, subtracting sum1 from sum2, and dividing by 64 to obtain the final result f' (x, y).

Images