CN115601223B - Image preprocessing device, method and chip - Google Patents
Image preprocessing device, method and chip Download PDFInfo
- Publication number
- CN115601223B CN115601223B CN202211094583.XA CN202211094583A CN115601223B CN 115601223 B CN115601223 B CN 115601223B CN 202211094583 A CN202211094583 A CN 202211094583A CN 115601223 B CN115601223 B CN 115601223B
- Authority
- CN
- China
- Prior art keywords
- control module
- module
- image
- adjacent source
- coordinates
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007781 pre-processing Methods 0.000 title claims abstract description 63
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000004364 calculation method Methods 0.000 claims abstract description 71
- 230000004927 fusion Effects 0.000 claims abstract description 10
- 239000000872 buffer Substances 0.000 claims description 61
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 26
- 230000003139 buffering effect Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/60—Memory management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
- G06T3/606—Rotation of whole images or parts thereof by memory addressing or mapping
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image preprocessing device, an image preprocessing method and a chip, wherein the device comprises the following components: the system comprises a main control module, a reading module, a cache control module and an interpolation calculation module; the main control module is used for calculating a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters, wherein the current pixel coordinates in the target image correspond to the adjacent source pixel coordinates in the original image; the cache control module sends a corresponding back pressure instruction to the main control module, so that the main control module judges whether to perform the related calculation of the next pixel coordinate according to the back pressure instruction; the interpolation calculation module performs fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinates and the difference weight parameters to obtain a target pixel value; the invention solves the problem that the image preprocessing in the prior art uses a larger storage space, adopts a register pipelining mode to cache interpolation weight data, saves two lookup table type storage units and reduces the consumption of on-chip storage units.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to an image preprocessing device, an image preprocessing method, and an image preprocessing chip.
Background
In the field of computer vision, particularly in the direction of target detection, the quality of the image acquired from the camera directly affects the accuracy and performance of target detection, so that preprocessing is generally required for the acquired image, wherein scaling and affine transformation of the image are important links in image preprocessing, and the image can be scaled or rotated to a required size so as to facilitate further processing and identification by the neural network accelerator.
At present, a bilinear interpolation algorithm is generally adopted to realize scaling or rotation calculation of an image, namely, two storage units are used as lookup tables to respectively store row and column direction original image pixel point coordinates and corresponding interpolation coefficients, the original image pixel point coordinates are used for acquiring source pixel values, and the interpolation coefficients are read to perform interpolation calculation after the source pixel values are acquired, so that a larger storage space is wasted, higher read-write delay is provided, and the area and the power consumption of a chip are directly increased. It can be seen that the image preprocessing in the prior art has a problem of using a large storage space.
Disclosure of Invention
Aiming at the defects in the related art, the image preprocessing device, the method and the chip provided by the invention solve the problem that a larger storage space is used in the image preprocessing in the prior art, and the interpolation weight data is cached in a register pipelining mode, so that two lookup table type storage units are saved, and the consumption of the on-chip storage units is reduced.
In a first aspect, the present invention provides an image preprocessing apparatus, the apparatus comprising: the system comprises a main control module, a reading module, a cache control module and an interpolation calculation module; the main control module is used for calculating a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in the original image corresponding to the current pixel coordinates in the target image according to the current preprocessing configuration parameters; wherein the current preprocessing configuration parameters comprise a scaling factor or an affine factor; the reading module is respectively connected with the main control module, the cache module and the external image memory, and is used for acquiring a plurality of corresponding adjacent source pixel values from the external image memory according to the plurality of adjacent source pixel coordinates and storing the plurality of adjacent source pixel values in the cache module; the buffer control module is respectively connected with the main control module, the buffer module and the interpolation calculation module, and is used for storing the plurality of difference weight parameters through a register pipeline structure and reading the plurality of adjacent source pixel values from the buffer module; the main control module is further used for sending corresponding back pressure instructions to the main control module according to the calculation progress of the difference calculation module, the storage capacity of the cache module or/and the storage capacity of the register pipeline structure, so that the main control module judges whether to calculate the adjacent source pixel coordinates of the next pixel coordinate and interpolation weight parameters according to the back pressure instructions; the interpolation calculation module is used for carrying out fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinates and the difference weight parameters to obtain target pixel values corresponding to the current pixel coordinates in the target image.
Optionally, the reading module includes: the read control module is connected with the main control module and is used for generating a read instruction according to the coordinates of the plurality of adjacent source pixels; and the DMA controller is respectively connected with the read control module and the external image memory and is used for reading a plurality of corresponding adjacent source pixel values from the external image memory in a DAM mode according to the read instruction.
Optionally, when the scaling factor is greater than or equal to a preset threshold, the read control module is further configured to generate a Burst read instruction according to the coordinates of the multiple adjacent source pixels, so that the DMA controller reads the corresponding whole row of source pixel values from the external image memory in an AXI Burst read mode according to the Burst read instruction.
Optionally, when the scaling factor is greater than or equal to a preset threshold, the read control module is further configured to determine whether an ordinate in the current adjacent source pixel coordinates is the same as the target ordinate, and if it is determined that the current adjacent source pixel coordinates are not the same as the target ordinate, then generate a corresponding Burst read instruction; the ordinate of the target is the ordinate in the adjacent source pixel coordinates for generating the last Burst read instruction.
Optionally, the interpolation calculation module includes: the Booth multiplication unit is connected with the cache control module and is used for carrying out multiplication operation on the interpolation weight parameters to obtain a plurality of difference operation results; the FIFO buffer unit is connected with the Booth multiplication unit and used for buffering the multiple difference operation results; and the mixed operation unit is connected with the FIFO buffer unit and the buffer control module and is used for carrying out mixed operation on the plurality of adjacent source pixel values read by the buffer control module and the plurality of difference operation results to obtain a target pixel value corresponding to the current pixel coordinate.
Optionally, the apparatus further comprises: and the pixel value writing module is respectively connected with the interpolation calculation module and the DMA controller and is used for writing the target pixel value into a corresponding position in the external image memory through the DMA controller.
Optionally, the buffer module includes a Ping-Pong buffer.
Optionally, the main control module is configured to obtain, according to a current preprocessing configuration parameter, a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in an original image corresponding to a current pixel coordinate in a target image, where the main control module includes: obtaining original coordinates of the original image according to the scaling factors in the current preprocessing configuration parameters and the current pixel coordinates in the target image, or obtaining original coordinates of the original image according to affine factors in the current preprocessing configuration parameters and the current pixel coordinates in the target image; obtaining a plurality of adjacent source pixel coordinates adjacent to the original coordinates according to the integer part of the original coordinates; and obtaining the interpolation weight parameters according to the decimal part of the original coordinates.
In a second aspect, the present invention provides an image preprocessing method, the method comprising: the main control module calculates a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in the original image corresponding to the current pixel coordinates in the target image according to the current preprocessing configuration parameters; wherein the current preprocessing configuration parameters comprise a scaling factor or an affine factor; the reading module acquires a plurality of corresponding adjacent source pixel values from an external image memory according to the adjacent source pixel coordinates, and stores the adjacent source pixel values in the caching module; the buffer control module stores the multiple difference weight parameters through a register pipeline structure, and reads the multiple adjacent source pixel values from the buffer module; sending a corresponding back pressure instruction to the main control module according to the calculation progress of the difference calculation module, the storage capacity of the cache module or/and the storage capacity of the register pipeline structure, so that the main control module judges whether to calculate the adjacent source pixel coordinate and interpolation weight parameter of the next pixel coordinate according to the back pressure instruction; and the interpolation calculation module performs fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinate and the difference weight parameter to obtain a target pixel value corresponding to the current pixel coordinate in the target image.
In a third aspect, the present invention provides a chip comprising the image preprocessing device described above.
Compared with the related art, the invention has the following beneficial effects:
1. the method comprises the steps that a main control module traverses all pixel coordinates of a target image, and simultaneously calculates adjacent source pixel coordinates and corresponding interpolation weight coefficients corresponding to an original image, so that a reading module reads corresponding adjacent source pixel values from an external image memory according to the adjacent source pixel coordinates and temporarily stores the adjacent source pixel values in a cache control module; the buffer control module buffers the interpolation weight coefficient in a register pipelining mode, and the buffer control module sends the self-stored interpolation weight coefficient and the corresponding adjacent source pixel value read from the buffer module to the interpolation calculation module for fusion calculation, so that a target pixel value corresponding to each pixel point in the target image is obtained. Therefore, the embodiment adopts a register pipelining mode to cache interpolation weight data, saves two lookup table type storage units and reduces the consumption of on-chip storage units.
2. Because the data transmission paths of the adjacent source pixel values and the interpolation weight parameters are different in the embodiment, the back pressure control of the buffer control module on the main control module can realize the dynamic balance adjustment of the interpolation weight calculation in the main control module and the bilinear interpolation calculation in the interpolation calculation module, thereby ensuring the data processing efficiency and ensuring that the data cannot be lost.
Drawings
Fig. 1 is a schematic structural diagram of an image preprocessing device according to the present embodiment;
fig. 2 is a schematic diagram of image scaling according to the present embodiment;
fig. 3 is a schematic flow chart of image scaling according to the present embodiment;
fig. 4 is a schematic structural diagram of another image preprocessing device according to the present embodiment;
fig. 5 is a schematic structural diagram of an interpolation calculation module according to the present embodiment;
fig. 6 is a schematic diagram of a state machine of a main control module according to the present embodiment;
fig. 7 is a flowchart of an image preprocessing method according to the present embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the accompanying claims.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The features of the examples and embodiments described below may be combined with each other without conflict. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In a first aspect, the present invention provides an image preprocessing method, which specifically includes the following embodiments:
example 1
Fig. 1 is a schematic structural diagram of an image preprocessing device according to the present embodiment, and as shown in fig. 1, an image preprocessing device 100 provided in the present invention specifically includes:
the device comprises a main control module 110, a reading module 120, a cache module 130, a cache control module 140 and an interpolation calculation module 150;
the main control module 110 is configured to calculate, according to the current preprocessing configuration parameter, a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in the original image corresponding to the current pixel coordinates in the target image; wherein the current preprocessing configuration parameters comprise a scaling factor or an affine factor;
the reading module 120 is respectively connected to the main control module 110, the buffer module 130 and the external image memory 200, and is configured to obtain a plurality of corresponding neighboring source pixel values from the external image memory 200 according to the plurality of neighboring source pixel coordinates, and store the plurality of neighboring source pixel values in the buffer module 130;
the buffer control module 140 is respectively connected to the main control module 110, the buffer module 130, and the interpolation computation module 150, and is configured to store the plurality of difference weight parameters through a register pipeline structure, and further configured to read the plurality of adjacent source pixel values from the buffer module 130; the method is further used for sending a corresponding back pressure instruction to the main control module 110 according to the calculation progress of the difference calculation module, the storage capacity of the buffer module 130 or/and the storage capacity of the register pipeline structure, so that the main control module 110 can judge whether to calculate the adjacent source pixel coordinate and the interpolation weight parameter of the next pixel coordinate according to the back pressure instruction;
the interpolation calculation module 150 is configured to perform fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinate and the difference weight parameter, so as to obtain a target pixel value corresponding to the current pixel coordinate in the target image.
It should be noted that, in this embodiment, the preprocessing of the image includes image scaling and affine transformation, and before explaining the working principle of the image preprocessing device in this embodiment, the processing procedure of the image scaling and affine transformation in this embodiment is described:
the image scaling aims at converting an image with a certain size into an image with other sizes, such as an enlarged or reduced image, 2a in fig. 2 is an original image, 2b in fig. 2 is a scaled target image, pixel coordinates si [ x ] [ y ] corresponding to the original image can be reversely deduced according to scaling factors and pixel coordinates di [ i ] [ j ] of the target image, the obtained original image pixel coordinates si [ x ] [ y ] are not necessarily integers, but are certainly between four adjacent pixels si [ ix ] [ iy ], si [ ix+1] [ iy ], si [ ix ] [ iy+1], si [ ix+1] [ iy+1] (ix is integral to x), and the pixel values corresponding to the four adjacent pixel coordinates can be interpolated to obtain the pixel value of the target image. As shown in fig. 3, the original image is si (wide ws, high hs), the target image is di (wide wd, high hd), the line scaling factor is (ws/wd), the column scaling factor is (hs/hd), the calculation method is to traverse all pixel coordinates [ i, j ] of the target image, obtain pixel coordinates [ x, y ] of the corresponding original image according to the scaling factor, the coordinates are floating point numbers, the integer part of [ x, y ] can be obtained [ ix, iy ], the fractional part of [ x, y ] can be obtained by multiplying 2048 (left shift 11 bit), and the coordinates of four adjacent points around the pixel of the original image can be known as siix [ iy ], siix+1 ] [ iy+1], and the four pixel weights (2048-u, 2048-v), (u, 2048-v), 2048-u, 204v (u, 2048-v). And carrying out weighted summation according to the pixel values of four adjacent points of the original image and the corresponding weights to obtain the pixel value di [ i ] [ j ] of the final target image.
Affine transformation is a simple transformation of images, comprising rotation and translation, namely, one image is subjected to linear transformation, and the images before and after affine transformation have the characteristics of convexity, collinearity, parallelism, invariance of collinearity proportion and the like. Assuming that the original image is si and the target image is di, according to the rotation and translation invariance principle of affine transformation, the following can be obtained: [ X ] s ,Y s ]Is the coordinates of pixel points in the original image, [ X ] d ,Y d ]For the coordinates of the corresponding pixel point in the target image after affine transformation, [ t ] x ,t y ]Offset in the horizontal direction and offset in the vertical direction, respectively, and θ is the rotation angle of affine transformation, can be obtained:
and (3) transforming to obtain:
X s ＝cosθ*X d -sinθ*Y d -(t x *cosθ-t y *sinθ)
Y s ＝sinθ*X d +cosθ*Y d -(t x *sinθ+t y *cosθ)
original image pixel corresponding to the calculated target pixel pointPoint coordinates si [ X ] s ][Y s ]，si[X s ][Y s ]Is not necessarily an integer, for [ X ] s ,Y s ]The integer part can be taken to obtain [ iX ] s ,iY s ]Take [ X ] s ,Y s ]By multiplying 2048 (11 bits shifted left) the fractional part of [ u, v ] can be obtained]Four points around the original image pixel point have coordinates si [ iX ] s ][iY s ]、si[iX s +1][iY s ]、si[iX s ][iY s +1]、si[iX s +1][iY s +1]And four adjacent pixel weights (2048-u, 2048-v), (u, 2048-v), (2048-u, v), and (u, v). The pixel values di [ i ] of the final target image can be obtained by weighting and summing the pixel values of four adjacent points of the original image and the corresponding weights][j]。
In this embodiment, the working principle of the image preprocessing device is as follows: the main control module traverses all coordinates in the target image, and obtains four adjacent pixel point coordinates and four interpolation weight parameters around the original coordinates in the corresponding original image according to the scaling factors or affine factors in the current preprocessing configuration parameters and the image scaling or affine transformation formulas; in addition, the target image and the current preprocessing configuration parameters can be set in the main control module in a parameter configuration mode, and can also be sent in the main control module in an instruction mode.
Further, the main control module sends the calculated four adjacent pixel point coordinates to the reading module, so that the reading module reads corresponding four adjacent source pixel values from the external image memory according to the four adjacent source pixel point coordinates calculated by the main control module, and then the read data are written into the cache module for storage; meanwhile, the main control module also sends the calculated four interpolation weight parameters to the cache control module, so that the cache control module stores the four interpolation weight parameters through a register flow structure. The four interpolation weight parameters are registered in the buffer control module as a set of weight parameter running water, the four adjacent source pixel values are also buffered in the buffer module as a set of pixel values, the buffer control module reads a set of pixel values and a corresponding set of weight parameters from the buffer module and simultaneously sends the pixel values and the corresponding set of weight parameters to the interpolation calculation module, so that the interpolation calculation module performs fusion calculation on the set of pixel values and the corresponding weight parameters to obtain a target pixel value corresponding to the current pixel coordinate in the target image.
In this embodiment, the back pressure instruction includes an abnormal instruction and a normal instruction, and when the storage capacity of the buffer module is full or the storage of a register running water structure in the reading module is full or the calculation progress of the interpolation calculation module is abnormal, the buffer control module sends the abnormal back pressure instruction to the main control module, so that the main control module pauses the related calculation of the next pixel coordinate; when the storage capacity of the buffer module is not full, the storage of the register pipeline structure is not full, and the calculation progress of the interpolation calculation module is normal, the buffer control module sends a normal back pressure instruction to the main control module, so that the main control module continues to perform the related calculation of the next pixel coordinate, and the dynamic adjustment of data processing is realized.
Compared with the related art, the invention has the following beneficial effects:
1. the method comprises the steps that a main control module traverses all pixel coordinates of a target image, and simultaneously calculates adjacent source pixel coordinates and corresponding interpolation weight coefficients corresponding to an original image, so that a reading module reads corresponding adjacent source pixel values from an external image memory according to the adjacent source pixel coordinates and temporarily stores the adjacent source pixel values in a cache control module; the buffer control module buffers the interpolation weight coefficient in a register pipelining mode, and the buffer control module sends the self-stored interpolation weight coefficient and the corresponding adjacent source pixel value read from the buffer module to the interpolation calculation module for fusion calculation, so that a target pixel value corresponding to each pixel point in the target image is obtained. Therefore, the invention adopts a register pipelining mode to cache interpolation weight data, saves two lookup table type storage units and reduces the consumption of on-chip storage units.
2. Because the data transmission paths of the adjacent source pixel values and the interpolation weight parameters are different, the dynamic balance adjustment of the interpolation weight calculation in the main control module and the bilinear interpolation calculation in the interpolation calculation module can be realized through the back pressure control of the buffer control module on the main control module, thereby ensuring the data processing efficiency and ensuring that the data is not lost.
Example two
Fig. 4 is a schematic structural diagram of another image preprocessing device according to the present embodiment; as shown in fig. 4, in this embodiment, the reading module 120 includes:
the read control module 121 is connected with the main control module 110 and is used for generating a read instruction according to the coordinates of the plurality of adjacent source pixels;
the DMA controller 122 is respectively connected to the read control module 121 and the external image memory 200, and is configured to read a corresponding plurality of adjacent source pixel values from the external image memory 200 by means of a DAM according to the read command.
Further, when the scaling factor is greater than or equal to the preset threshold, the read control module 121 is further configured to generate a Burst read instruction according to the coordinates of the plurality of adjacent source pixels, so that the DMA controller 122 reads the corresponding whole row of source pixel values from the external image memory 200 in an AXI Burst read mode according to the Burst read instruction.
Optionally, when the scaling factor is greater than or equal to a preset threshold, the read control module 121 is further configured to determine whether an ordinate in the current neighboring source pixel coordinates is the same as the target ordinate, and if it is determined that the current neighboring source pixel coordinates are not the same as the target ordinate, generate a corresponding Burst read instruction; the ordinate of the target is the ordinate in the adjacent source pixel coordinates for generating the last Burst read instruction.
It should be noted that, through the above description of the image scaling principle, in the scene of image magnification, when all pixel coordinates of the target image are traversed, several adjacent pixel coordinate values may be mapped to the same pixel value in the original image, that is, the original image pixel value has relatively high utilization rate, but if the pixel value is read frequently in a single reading point manner, the efficiency is definitely low, so in this case, the embodiment selects to use an AXI Burst reading manner to continuously read pixels of a certain line in the original image, and stores the pixels in the buffer module, and then if a certain pixel value needs to be repeatedly used to directly read data from the buffer module, the problem that the system bus is frequently occupied to cause low efficiency is avoided, thereby improving the system bus utilization rate.
In this embodiment, the apparatus further includes: and a pixel value writing module 160, respectively connected to the interpolation calculation module 150 and the DMA controller 121, for writing the target pixel value into a corresponding position in the external image memory 200 through the DMA controller.
In this embodiment, the buffer module includes a Ping-Pong buffer, which includes 2 sets of RAMs, each set of RAMs including two true dual-port RAMs having a depth of 128 and a width of 32 bits.
Example III
Fig. 5 is a schematic structural diagram of an interpolation calculation module according to the present embodiment; as shown in fig. 5, the interpolation calculation module 150 includes:
the Booth multiplication unit 151 is connected to the cache control module 140, and is configured to perform multiplication operation on the interpolation weight parameters to obtain a plurality of difference operation results;
the FIFO buffer unit 152 is connected to the Booth multiplication unit 151, and configured to buffer the multiple difference operation results;
and a mixing operation unit 153, connected to the FIFO buffer unit 152 and the buffer control module 140, configured to perform a mixing operation on the plurality of adjacent source pixel values and the plurality of difference operation results read by the buffer control module 140, to obtain a target pixel value corresponding to the current pixel coordinate.
It should be noted that, the interpolation calculation module mainly completes the calculation shown in the following formula, the structural block diagram of which is shown in fig. 5,
for(i＝0；i<W；i++){for(j＝0；j<H；j++){di[i][j]＝(si[iy][ix]*(2048-u)*(2048-v)+si[iy][ix+1]*u*(2048-v)+si[iy+1][ix]*(2048-u)*v+si[iy+1][ix+1]*u*v)>>22}}
the four multiplication calculations of (2048-u), (2048-u) v and u are completed by the Booth multiplication unit, so that the time sequence can be better improved, and the calculation efficiency can be improved; these four multiplication results are then buffered in a FIFO buffer element of depth 4 and width 88. When a plurality of adjacent source pixel values in the original image are read back from the buffer module, the multiplication result of the FIFO buffer unit is read at the same time, the data are aligned at the front end of the hybrid operation unit, multiplication calculation is carried out, then the multiplication results are added, and the target pixel value of the corresponding point of the target image is obtained through bit cutting processing after the addition result is shifted.
Example IV
In this embodiment, the main control module is configured to obtain, according to a current preprocessing configuration parameter, a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in an original image corresponding to a current pixel coordinate in a target image, where the main control module includes: obtaining original coordinates of the original image according to the scaling factors in the current preprocessing configuration parameters and the current pixel coordinates in the target image, or obtaining original coordinates of the original image according to affine factors in the current preprocessing configuration parameters and the current pixel coordinates in the target image; obtaining a plurality of adjacent source pixel coordinates adjacent to the original coordinates according to the integer part of the original coordinates; and obtaining the interpolation weight parameters according to the decimal part of the original coordinates.
It should be noted that, the calculation process of the main control module calculating the coordinates of the adjacent source pixels and the interpolation weight parameters according to the scaling factor or the affine factor may be described with reference to the principles of image scaling and affine transformation in the above embodiment.
Fig. 6 is a schematic diagram of a state machine of a main control module according to the present embodiment; the main function of the main control module is to traverse all pixel coordinates of a target image under different scaling multiple or rotation angle and translation configuration according to an image scaling or affine transformation mode, calculate original pixel coordinates in the original image and obtain four adjacent pixel points around the corresponding original pixel coordinates, the state machine of the main control module is shown in fig. 6, the IDLE is in an initial state, after the configuration is completed and a START instruction is received, the main control module jumps to an ar_start state, then unconditionally jumps to an ar_y_cnt state, when a reading module can receive a data reading command, the state jumps to an ar_x_cnt state, at this time, the horizontal coordinate of the target image is counted w_cnt, when the current line is counted, the state jumps to an ar_y_cnt state, and at the same time, the vertical coordinate h_cnt of the target image is added with 1, and then jumps to an ar_x_cnt state until the pixel coordinates of the target image jump to the IDLE state at this time. When traversing coordinates, four adjacent source pixel coordinates in the original image corresponding to the current coordinates and 4 corresponding interpolation weight parameters are synchronously generated, the pixel coordinates are sent to a reading module so as to read image data, and the 4 interpolation weight parameters are sent to a cache control module so as to carry out pipeline registering.
It should be noted that, after the affine transformation algorithm is transformed, the main processes of image scaling and affine transformation are all to calculate the pixel coordinates and interpolation weights of the corresponding original image according to the pixel coordinates of the traversing target image, but the calculation methods are slightly different, and the whole data path is an external image memory-reading module-caching module-interpolation calculation module, so that the logic and hardware modules of the data path, the caching and interpolation calculation part can be completely multiplexed. Therefore, the affine transformation algorithm is skillfully adjusted to enable the affine transformation algorithm to have higher similarity with image scaling, and the cache unit and the computing resource can be reused as much as possible on hardware implementation.
In a second aspect, the present invention provides an image preprocessing method, which specifically includes the following embodiments:
example five
Fig. 7 is a schematic flow chart of an image preprocessing method according to the present embodiment; as shown in fig. 7, the image preprocessing method provided in this embodiment specifically includes the following steps:
step S101, a main control module calculates a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in an original image corresponding to the current pixel coordinates in a target image according to the current preprocessing configuration parameters; wherein the current preprocessing configuration parameters comprise a scaling factor or an affine factor;
step S102, a reading module acquires a plurality of corresponding adjacent source pixel values from an external image memory according to the adjacent source pixel coordinates, and stores the adjacent source pixel values in the buffer module;
step S103, according to the calculation progress of the difference calculation module, the storage capacity of the buffer module or/and the storage capacity of the register pipeline structure, a corresponding back pressure instruction is sent to the main control module, so that the main control module judges whether to calculate the adjacent source pixel coordinate and interpolation weight parameter of the next pixel coordinate according to the back pressure instruction;
step S104, the interpolation calculation module performs fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinates and the difference weight parameters to obtain target pixel values corresponding to the current pixel coordinates in the target image.
In this embodiment, the buffer control module further stores the plurality of difference weight parameters through a register pipeline structure, and reads the plurality of adjacent source pixel values from the buffer module.
It should be noted that, the image preprocessing method provided in this embodiment is the same as the principle and the processing procedure in the first embodiment, and will not be described herein.
In a third aspect, the present invention provides a chip comprising the image preprocessing apparatus according to any one of the first to fourth embodiments.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (8)
1. An image preprocessing apparatus, characterized in that the apparatus comprises:
the system comprises a main control module, a read control module, a DMA controller, a cache module, a cache control module and an interpolation calculation module;
the main control module is used for calculating a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in the original image corresponding to the current pixel coordinates in the target image according to the current preprocessing configuration parameters; wherein the current preprocessing configuration parameters comprise a scaling factor or an affine factor;
the read control module is connected with the main control module and is used for generating a read instruction according to the coordinates of the plurality of adjacent source pixels; the DMA controller is respectively connected with the read control module, the external image memory and the cache module and is used for reading a plurality of corresponding adjacent source pixel values from the external image memory in a DAM mode according to the read instruction and storing the adjacent source pixel values in the cache module; when the scaling factor is greater than or equal to a preset threshold, the read control module is further used for judging whether the ordinate in the current adjacent source pixel coordinates is the same as the target ordinate, and if yes, a corresponding Burst read instruction is generated again; the ordinate of the target is the ordinate in the adjacent source pixel coordinates for generating the previous Burst reading instruction;
the buffer control module is respectively connected with the main control module, the buffer module and the interpolation calculation module, and is used for storing the interpolation weight parameters through a register pipeline structure and reading the adjacent source pixel values from the buffer module; the system is also used for sending corresponding back pressure instructions to the main control module according to the calculation progress of the interpolation calculation module, the storage capacity of the cache module or/and the storage capacity of the register pipeline structure, so that the main control module judges whether to calculate the adjacent source pixel coordinates of the next pixel coordinate and interpolation weight parameters according to the back pressure instructions;
the interpolation calculation module is used for carrying out fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinates and the interpolation weight parameters to obtain target pixel values corresponding to the current pixel coordinates in the target image.
2. The image preprocessing device as recited in claim 1, wherein the DMA controller reads corresponding whole row of source pixel values from the external image memory in an AXI Burst read mode according to the Burst read instruction.
3. The image preprocessing apparatus according to claim 1, wherein said interpolation computation module includes:
the Booth multiplication unit is connected with the cache control module and is used for carrying out multiplication operation on the interpolation weight parameters to obtain a plurality of interpolation operation results;
the FIFO buffer unit is connected with the Booth multiplication unit and used for buffering the interpolation operation results;
and the mixed operation unit is connected with the FIFO buffer unit and the buffer control module and is used for carrying out mixed operation on the plurality of adjacent source pixel values read by the buffer control module and the plurality of interpolation operation results to obtain a target pixel value corresponding to the current pixel coordinate.
4. The image preprocessing apparatus according to claim 1, wherein said apparatus further comprises:
and the pixel value writing module is respectively connected with the interpolation calculation module and the DMA controller and is used for writing the target pixel value into a corresponding position in the external image memory through the DMA controller.
5. The image preprocessing device as recited in any one of claims 1-4, wherein said buffering module comprises a Ping-Pong buffer.
6. The image preprocessing device as set forth in claim 1, wherein the main control module is configured to obtain, according to a current preprocessing configuration parameter, a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters corresponding to a current pixel coordinate in a target image to an original image, and the main control module includes:
obtaining original coordinates of the original image according to the scaling factors in the current preprocessing configuration parameters and the current pixel coordinates in the target image, or obtaining original coordinates of the original image according to affine factors in the current preprocessing configuration parameters and the current pixel coordinates in the target image;
obtaining a plurality of adjacent source pixel coordinates adjacent to the original coordinates according to the integer part of the original coordinates;
and obtaining the interpolation weight parameters according to the decimal part of the original coordinates.
7. An image preprocessing method, characterized by being applied to the image preprocessing apparatus as claimed in any one of claims 1 to 6, comprising:
the main control module calculates a plurality of adjacent source pixel coordinates and a plurality of interpolation weight parameters in the original image corresponding to the current pixel coordinates in the target image according to the current preprocessing configuration parameters; wherein the current preprocessing configuration parameters comprise a scaling factor or an affine factor;
the reading module acquires a plurality of corresponding adjacent source pixel values from an external image memory according to the adjacent source pixel coordinates, and stores the adjacent source pixel values in the caching module;
the buffer control module stores the interpolation weight parameters through a register pipeline structure, and reads the adjacent source pixel values from the buffer module; sending a corresponding back pressure instruction to the main control module according to the calculation progress of the interpolation calculation module, the storage capacity of the cache module or/and the storage capacity of the register pipeline structure, so that the main control module judges whether to calculate the adjacent source pixel coordinate of the next pixel coordinate and the interpolation weight parameter according to the back pressure instruction;
and the interpolation calculation module performs fusion calculation on the plurality of adjacent source pixel values corresponding to the current pixel coordinate and the interpolation weight parameter to obtain a target pixel value corresponding to the current pixel coordinate in the target image.
8. A chip, characterized in that it comprises the image preprocessing device according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211094583.XA CN115601223B (en) | 2022-09-08 | 2022-09-08 | Image preprocessing device, method and chip |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211094583.XA CN115601223B (en) | 2022-09-08 | 2022-09-08 | Image preprocessing device, method and chip |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115601223A CN115601223A (en) | 2023-01-13 |
| CN115601223B true CN115601223B (en) | 2023-08-04 |
Family
ID=84843463
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211094583.XA Active CN115601223B (en) | 2022-09-08 | 2022-09-08 | Image preprocessing device, method and chip |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115601223B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117057982B (en) * | 2023-10-11 | 2024-01-26 | 北京地平线信息技术有限公司 | Integrated circuit for performing distortion transformation on image and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107851214A (en) * | 2015-07-23 | 2018-03-27 | 米雷普里卡技术有限责任公司 | For the performance enhancement of two-dimensional array processor |
| CN114092338A (en) * | 2022-01-20 | 2022-02-25 | 长沙金维信息技术有限公司 | Image zooming fast calculation method |
| CN114463178A (en) * | 2022-02-09 | 2022-05-10 | 浙江派大芯集成电路有限公司 | Image processing chip, method and equipment |
| CN114707649A (en) * | 2022-03-28 | 2022-07-05 | 北京理工大学 | General convolution arithmetic device |
-
2022
- 2022-09-08 CN CN202211094583.XA patent/CN115601223B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107851214A (en) * | 2015-07-23 | 2018-03-27 | 米雷普里卡技术有限责任公司 | For the performance enhancement of two-dimensional array processor |
| CN114092338A (en) * | 2022-01-20 | 2022-02-25 | 长沙金维信息技术有限公司 | Image zooming fast calculation method |
| CN114463178A (en) * | 2022-02-09 | 2022-05-10 | 浙江派大芯集成电路有限公司 | Image processing chip, method and equipment |
| CN114707649A (en) * | 2022-03-28 | 2022-07-05 | 北京理工大学 | General convolution arithmetic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115601223A (en) | 2023-01-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5437311B2 (en) | Image correction method, image correction system, angle estimation method, and angle estimation device | |
| US9280810B2 (en) | Method and system for correcting a distorted input image | |
| EP2347385A1 (en) | Method and system for image resizing based on interpolation enhanced seam operations | |
| US7149355B2 (en) | Image processing apparatus, image processing method, image processing program, and computer-readable record medium storing image processing program | |
| CN107749987B (en) | Digital video image stabilization method based on block motion estimation | |
| WO2014005783A1 (en) | A method and system for correcting a distorted input image | |
| CN109886866B (en) | Self-adaptive affine transformation processing method and device | |
| CN115601223B (en) | Image preprocessing device, method and chip | |
| CN110958362A (en) | Image correction system and method based on block table look-up | |
| CN114913076A (en) | Image scaling and rotating method, device, system and medium | |
| Zhang et al. | Optical flow reusing for high-efficiency space-time video super resolution | |
| US5870105A (en) | System and method for local storage of image data during object to image mapping | |
| CN107645634A (en) | A kind of undistorted wide angle network video camera and safety defense monitoring system | |
| CN115631223A (en) | Multi-view stereo reconstruction method based on self-adaptive learning and aggregation | |
| CN113538237A (en) | Image splicing system and method and electronic equipment | |
| Tian et al. | A fast edge-directed interpolation algorithm | |
| Yang et al. | FPGA Implementation of Image Super-Resolution Based on Bicubic Interpolation and CNN | |
| CN116071245B (en) | Image processing method, device and system | |
| CN117057982B (en) | Integrated circuit for performing distortion transformation on image and electronic equipment | |
| CN112508790B (en) | Image interpolation method, device, equipment and medium | |
| CN114741352B (en) | FPGA-based bilinear interpolation resampling implementation method and device | |
| CN114820289A (en) | Fisheye image correction method based on radial symmetric projection model | |
| CN117710201A (en) | Image scaling method and device, electronic equipment and computer storage medium | |
| CN115187448A (en) | Chip-based image affine transformation method and chip | |
| Xing-Yuan et al. | An improved image interpolation algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |