CN109785265B - Distortion correction image processing method and image processing apparatus - Google Patents
Distortion correction image processing method and image processing apparatus Download PDFInfo
- Publication number
- CN109785265B CN109785265B CN201910040943.XA CN201910040943A CN109785265B CN 109785265 B CN109785265 B CN 109785265B CN 201910040943 A CN201910040943 A CN 201910040943A CN 109785265 B CN109785265 B CN 109785265B
- Authority
- CN
- China
- Prior art keywords
- coordinates
- data
- image
- buffer
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Image Processing (AREA)
Abstract
The invention discloses a distortion correction image processing method and a device, wherein the method comprises the steps of obtaining coordinates of a plurality of pixel points from a mapping relation corresponding table, wherein the mapping relation corresponding table is a mapping relation corresponding table of pixel point coordinates in an output image and pixel point coordinates in an input image; wherein the acquired coordinates are coordinates obtained by vertical scanning in 16-line units; and judging whether the coordinates of a plurality of reference pixel points used for carrying out interpolation calculation on the target pixel point are in the acquired coordinates or not, if so, reading the data of each reference pixel point from the cache and carrying out interpolation calculation to obtain the data of each pixel point of the output image when the transformation unit in which each reference pixel point is positioned is effective. The device comprises a vertical scanning module, a judging module and the like, and is used for realizing the distortion correction image processing method. The invention can reduce the hardware cost of the image processing device and improve the image processing efficiency.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a method for processing an image and an apparatus for implementing the method.
Background
As the photographing technology has been developed, camera technology has also been developed, and some cameras now provide two lenses in order to calculate three-dimensional information of an object, and such cameras are called binocular cameras. The binocular camera can be used for calculating binocular depth information, the binocular depth information is extracted, the surrounding environment is sensed from two visual angles through the binocular camera, images under different visual angles are obtained, and the three-dimensional information of the object is calculated through a projection relation according to the matching relation of the object between pixels in the images at the two visual angles.
When the three-dimensional information of the object is calculated, a large amount of processing needs to be performed on the images, for example, epipolar geometric correction needs to be performed before two stereo images with different visual angles are matched, the images subjected to epipolar geometric correction are strictly aligned in a horizontal direction, and the images of the space point on the left image and the right image are always positioned in the same row, so that the search of stereo matching only needs to be performed in one row, and the calculation is simplified.
Typically, the process of epipolar geometry correction involves distortion correction of the image. For example, some existing lenses, such as fisheye lenses, generate serious distortion in images captured, and do not conform to the viewing angle of the images seen by human eyes, so the distorted images need to be corrected. Meanwhile, because of the difference in the degree of distortion of images photographed using different fisheye lenses, the order of fetching from a synchronous random access memory (SDRAM) is random and uncertain, and it is difficult to satisfy the requirement for high-speed correction of images in terms of bandwidth and performance.
Because the existing processing mode of the distorted image is mainly to geometrically correct the whole image in a line raster scanning mode, a line buffer (line buffer) is needed to load original image data for correction calculation, the method has poor general applicability, consumes large storage resources, is not beneficial to hardware implementation, needs a large amount of buffers, and causes huge hardware circuit area.
Disclosure of Invention
The invention mainly aims to provide a distortion correction image processing method which reduces hardware resource consumption and reduces the area of a circuit board.
Another object of the present invention is to provide a distortion correction image processing apparatus that increases the speed of image correction processing.
In order to achieve the above main object, the distortion correction image processing method provided by the present invention includes obtaining coordinates of a plurality of pixel points from a mapping relationship correspondence table, where the mapping relationship correspondence table is a mapping relationship correspondence table of pixel point coordinates in an output image and pixel point coordinates in an input image; the acquired coordinates are coordinates obtained by vertical scanning in a row unit; and judging whether the coordinates of a plurality of reference pixel points used for carrying out interpolation calculation on the target pixel point are in the acquired coordinates or not, if so, reading the data of each reference pixel point from the cache and carrying out interpolation calculation to obtain the data of each pixel point of the output image when the transformation unit in which each reference pixel point is positioned is effective.
According to the scheme, the acquisition of the pixel point coordinates is realized by adopting a vertical scanning geometric correction mode, so that the hardware realization mode is simple, a line buffer is not required, and the requirement on hardware resources is low. And, the occupied bandwidth resource is less.
In a preferred embodiment, when it is determined that the coordinates of at least one reference pixel are not in the acquired coordinates, the coordinates of the missing reference pixel are written into the missing pixel coordinate buffer and the image pixel coordinate buffer, and a data read request is sent to the sdram.
Therefore, pixel coordinates are cached by using the pixel coordinate cache lacking the pixel coordinate cache and the image pixel coordinate cache, on one hand, the delay of a synchronous random dynamic memory bus is eliminated, on the other hand, the depth of the two caches is configurable, and the image with larger distortion rate can be suitable for images with different distortion rates by adjusting the depth of the two caches.
The further scheme is that before sending a data reading request to the synchronous random dynamic memory, the following steps are executed: and determining that the missing pixel coordinate register is in a non-full state, the image pixel coordinate register is in a non-full state, and the count value of the counter of the transformation unit where the missing reference pixel is located is zero.
Therefore, the request is sent to the synchronous random dynamic memory only when the three conditions are met, on one hand, data overflow caused by data writing in the full data writing state of the two buffers can be avoided, on the other hand, new data can be prevented from being wrongly written in the transformation unit under the condition that useful data still exist, and normal data writing is ensured.
Further, each conversion unit is provided with a counter; when the coordinates are written into the image pixel coordinate buffer each time, the count value of the counter of the corresponding conversion unit is added once; each time coordinates are read from the image pixel coordinate buffer, the count value of the counter of the corresponding transformation unit is decremented by one.
Therefore, the counter is used for calculating the writing and reading conditions of the data of the conversion unit, the data in the conversion unit can be ensured to be used before new data is written into the conversion unit, and the data in the conversion unit can be prevented from being wrongly cleared.
Preferably, each transformation unit is provided with a corresponding readable mark; after the data of the conversion unit is written into the conversion unit, the readable mark of the conversion unit is set; when the data reading of the conversion unit is finished, the readable mark of the conversion unit is reset.
It can be seen that by setting the readable flag of the transformation unit to mark whether the data of the transformation unit has been written or read completely, it can be ensured that the data in the transformation unit has been used before the new data is written into the transformation unit, and the new data can be written into the transformation unit.
The further scheme is that the sending the data reading request to the synchronous random dynamic memory comprises the following steps: and sending the request information of the data reading request to a request buffer, and then sending the request information stored in the buffer to the synchronous random dynamic memory.
Therefore, the request information is cached by using the request buffer, the request information can be intensively sent to the synchronous random access memory, the request information can be cached in advance, and the data sending and reading efficiency is improved.
In order to achieve another object, the present invention provides an image processing apparatus for distortion correction, comprising: the vertical scanning module is used for acquiring the coordinates of a plurality of pixel points obtained by vertical scanning in a row unit from the mapping relation corresponding table, and the mapping relation corresponding table is a mapping relation corresponding table of the pixel point coordinates in the output image and the pixel point coordinates in the input image; the judging module is used for judging whether the coordinates of the reference pixel points used for carrying out interpolation calculation on the target pixel points are in the acquired coordinates or not; and the data reading module is used for determining that the coordinates of the plurality of reference pixels are in the acquired coordinates, and when the transformation unit where each reference pixel is located is effective, reading the data of each reference pixel from the cache and performing interpolation calculation to obtain the data of each pixel of the output image.
According to the scheme, the pixel point coordinates are obtained by adopting a vertical scanning geometric correction mode, compared with a line scanning mode, the hardware implementation mode is simple, a line buffer is not needed, and the requirement on hardware resources is low. And, the occupied bandwidth resource is less.
Preferably, the image processing device further comprises a missing pixel coordinate buffer and an image pixel coordinate buffer; the missing pixel coordinate buffer is used for storing the coordinates of the missing reference pixels output from the judging module when the coordinates of at least one reference pixel are not in the acquired coordinates; the image pixel coordinate buffer is used for storing the coordinates of the reference pixel points output from the judging module.
According to the scheme, the pixel coordinates are cached by using the pixel coordinate cache which is lack of the pixel coordinate cache and the image pixel coordinate cache, so that the delay of the synchronous random access memory bus can be eliminated, and the efficiency of reading data from the synchronous random access memory is improved.
Further, the lack of the pixel coordinate buffer and the storage depth of the image pixel coordinate buffer may be set according to the distortion rate of the input image.
It can be seen that the depths of the two buffers are configured according to the distortion rate of the input image, and the image with larger distortion rate can be applied to images with different distortion rates by adjusting the depths of the two buffers, so that the application range of the image processing device is wider.
In a further aspect, the image processing apparatus further includes a request buffer for storing request information of the data reading request output by the determining module, and the request information may be sent to the sdram.
Therefore, the request information is cached by using the request cache, and the request information can be cached in advance, so that the request information to be sent is prepared in advance before the request information is sent to the synchronous random dynamic memory, and the data sending and reading efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of an input image scan using an embodiment of the distortion correction image processing method of the present invention.
FIG. 2 is a first portion of a flow chart of an embodiment of the distortion corrected image processing method of the present invention.
Fig. 3 is a second portion of a flow chart of an embodiment of the distortion corrected image processing method of the present invention.
FIG. 4 is a block diagram showing the configuration of an embodiment of the image processing apparatus for distortion correction according to the present invention.
FIG. 5 is a diagram of a plurality of transform units in an embodiment of the distortion corrected image processing apparatus according to the present invention.
The invention is further described with reference to the following figures and examples.
Detailed Description
The distortion correction image processing method is applied to various shooting devices such as cameras and monitoring devices, and can also be applied to mobile devices with a shooting function such as electronic devices such as smart phones, tablet computers or automobile data recorders. The distortion correction image processing apparatus of the present invention can be applied to various photographing devices.
The distortion correction image processing method of the present invention is a distortion correction image processing method for geometric correction implemented by using a cache (cache), and a distortion correction image processing apparatus is a hardware architecture for implementing the above method. The main idea of the invention is to realize the mapping relation between the output pixel and the original input pixel by adopting reverse mapping, and for the pixel point of the output image, the reverse mapping appoints the pixel point of which position of the input image the output pixel point comes from, and the mapping relation is related to the camera characteristic and is a fixed mapping relation, so that after the lens position of the camera is determined, real-time calculation is not needed.
The embodiment of the distortion correction image processing method comprises the following steps:
the method of the embodiment is firstly used for calculating the mapping relationship offline through a computer program to obtain a reduced mapping relationship corresponding table (LUT) and storing the LUT, and when the hardware needs to process the image, the mapping relationship corresponding table is only required to be read and interpolated and amplified to obtain the one-to-one mapping relationship. Since the process of implementing the calculation by a computer program is an existing technique, it is not discussed herein. The minimum storage unit in the hardware structure applying the embodiment is a Transformation Unit (TU), one transformation unit can store 64 bytes of original image data, and the transformation unit forms a cache buffer.
The embodiment is realized based on the coordinates of the pixel points acquired by the mapping relationship correspondence table, and referring to fig. 1, the embodiment generates the output coordinates of the vertical scanning according to 16 lines, and finally generates the output coordinates of the vertical scanning according to the unit of the remainder of dividing the image height by 16 if the number of the lines is less than 16. Therefore, in this embodiment, it is first required to determine which row coordinates of each pixel point in the output image correspond to which pixels in the input image, and determine which column coordinates of each pixel point in the output image correspond to which pixels in the input image. After the correspondence between the pixels of the output image and the pixels of the input image is determined, a mapping correspondence table is formed, and then the image processing apparatus calculates and processes the data of the input image according to the data of the mapping correspondence table, for example, performs bilinear interpolation, thereby obtaining the output image.
Preferably, in this embodiment, the mapping relationship correspondence table is a mapping table that is reduced by a certain magnification compared with the original image size, specifically, the buffer used for buffering the mapping relationship correspondence table is lut _ sram, the buffer lut _ sram uses a single-port random access memory, and the mapping relationship correspondence table is decompressed by bilinear interpolation to obtain a one-to-one mapping relationship between each pixel point of the input image and the pixel point coordinate of the output image.
Referring to fig. 2, in this embodiment, step S1 is first executed to obtain coordinates of a plurality of pixel points from the mapping relationship correspondence table. Since the mapping relationship correspondence table is stored in the sdram, step S1 is to read the mapping relationship correspondence table from the sdram and obtain the coordinates of a plurality of pixels. Generally, the mapping table is a compressed table, and therefore, after the mapping table is read from the sdram, the read data needs to be decompressed, and the original image coordinates corresponding to the output coordinates are obtained through bilinear interpolation, and the obtained original image coordinates are input into a buffer for storing the image coordinates, where the buffer is named as src _ coord _ buf _ fifo. Preferably, the buffer src _ core _ buf _ FIFO is formed by a First-in First-out (FIFO) buffer, and the depth of the buffer is 16, that is, the buffer can store 16 data at most. When the buffer src _ coord _ buf _ fifo is full of data, bilinear interpolation will be stopped until the buffer is emptied and new coordinates are written again.
The image with geometric distortion is parabolic in the vertical direction, that is, the distortion of the image is that the two ends change fast and the middle changes slowly. By adopting a vertical scanning mode, most of the access situations are continuous when the periphery and the middle of the image meet the boundary of the transformation unit, so that the data reading request to the synchronous random dynamic memory is continuously sent, and the updating time of the synchronous random dynamic memory can be avoided as much as possible. The vertical scanning adopted by the embodiment is different from the raster scanning, and does not need to buffer a whole line of data each time when the data is replaced. When the geometric correction of raster scanning is adopted, at least two rows of data are needed during bilinear interpolation, so that at least two rows of original images need to be cached, and if only two rows of data are cached, the calculation efficiency around the images is low, and the strategy for ensuring the data updating of the calculation efficiency becomes complicated. Therefore, the vertical scanning mode can initiate continuous and dense requests for reading the data of the synchronous random dynamic memory when the boundary of the transformation unit is met around and in the middle of the image, and a line buffer is not needed to be used as a buffer, so that the area overhead of hardware is reduced, and the hardware is simple to realize.
And after the coordinates of the mapping relation corresponding table are obtained, executing a step S2, and judging whether the coordinates of the four reference pixel points of the target pixel point currently processed are all hit, if so, executing a step S3, otherwise, executing a step S11. Since bilinear interpolation calculation of a certain pixel needs to use data of four related pixels, namely upper and lower lines, as reference, the calculated pixel is called a target pixel, and the used pixel as reference is called a reference pixel.
In step S2, the coordinates of the pixel points read from the synchronous random access memory are determined, and it is determined whether the coordinates of the four reference pixel points of the currently calculated target pixel point are all hit, that is, whether the coordinates of the four reference pixel points are included in the coordinates of the pixel points read from the synchronous random access memory, and if hit, it indicates that the cache buffer stores the required data, and the data can be directly read; if there is no hit, a request to read data needs to be issued to the synchronous random access memory.
Since the data read from the synchronous random access memory is written into the transformation unit, the readable flag of the transformation unit is set if the data of the current transformation unit is written, and the readable flag of the transformation unit is reset after the data of the transformation unit is read. When the readable flag of one transformation unit is set, the data of the transformation unit can be read, namely the transformation unit is valid, and if the readable flag of the transformation unit is reset, the data of the transformation unit cannot be read, namely the transformation unit is invalid.
Specifically, when the buffer src _ coord _ buf _ fifo is in a non-empty state, that is, when data is stored in the buffer src _ coord _ buf _ fifo, coordinates of pixel points of an original image are read from the buffer src _ coord _ buf _ fifo, and comparison and determination are performed to determine whether coordinates of four reference pixel points required for geometric correction bilinear interpolation of a target pixel point are all hit. In this embodiment, the cache buffer is composed of 2 × 32 transform units, and two transform units in the horizontal direction are grouped into one group, which is 32 groups in total. The judgment method in step S2 is to analyze the coordinates of the pixel points of the original image after reading the coordinates of the pixel points of the original image from the buffer src _ coord _ buf _ fifo. Assuming that the vertical coordinate of the pixel is y _ color and the horizontal coordinate is x _ color, the y _ color is divided by 32 to obtain the row index tu _ ver _ idx of the transform unit, and the x _ color is divided by 64 to obtain the column index tu _ hor _ idx of the transform unit.
In addition, in this embodiment, an additional register is set for each transform unit, where the register is a 16-bit (bit) register, called tu _ coord _ reg, and when the coordinates of the reference pixel do not hit, the coordinate { y _ coord [10 ]. Therefore, the judgment in step S2 is to compare the coordinates { y _ coord [10 ], x _ coord [10 ].
Further, an additional counter register called tu _ cnt _ reg is provided for each transform unit, and when the coordinates { y _ coord [10 ], x _ coord [10 ] } and the value stored in the register tu _ coord _ reg are equal, and the count value of the corresponding counter tu _ cnt _ reg is not 0, it indicates that the transform unit is hit.
When the coordinates are read from the buffer src _ coord _ buf _ fifo and analyzed, since there is a coordinate point exceeding the image range after reverse mapping, it is necessary to determine whether the read coordinate values exceed the image size range, and the coordinates are directly cropped to the image width when the horizontal coordinates exceed the image width, and the coordinates are directly cropped to the image height when the vertical coordinates exceed the image height. Suppose that pic _ width represents the picture width, pic _ height represents the picture height, src _ x _ coord represents the x-direction coordinate read out from the buffer src _ coord _ buf _ fifo, and src _ y _ coord represents the y-direction coordinate read out from the buffer src _ coord _ buf _ fifo. The geometric correction needs value participation calculation of four reference pixel points which are adjacent up and down, so that the coordinates of the four reference pixel points which are needed can be expressed as:
x_coord00＝src_x_coord,y_coord00＝src_y_coord
x_coord01＝src_x_coord+1,y_coord01＝src_y_coord
x_coord10＝src_x_coord,y_coord10＝src_y_coord+1
x_coord11＝src_x_coord+1,y_coord11＝src_y_coord+1
if the horizontal coordinate src _ x _ coord exceeds the image width, the four horizontal coordinates are directly assigned as pic _ width, and if the vertical coordinate src _ x _ coord exceeds the image height, the four vertical coordinates are directly assigned as pic _ height. If the horizontal coordinate src _ x _ coord is exactly equal to the image width, x _ coord01 and x _ coord11 are assigned src _ x _ coord, and if the vertical coordinate src _ y _ coord is exactly equal to the image height, y _ coord10 and y _ coord11 are assigned image height. The other cases are represented by the above four points.
After the conversion, the index of the corresponding transformation unit can be obtained as follows:
tu_line_idx_00＝y_coord00[4:0]
tu_line_idx_01＝y_coord01[4:0]
tu_line_idx_10＝y_coord10[4:0]
tu_line_idx_11＝y_coord11[4:0]
tu_row_idx_00＝x_coord00[6]
tu_row_idx_01＝x_coord01[6]
tu_row_idx_10＝x_coord10[6]
tu_row_idx_11＝x_coord11[6]
where tu _ line _ idx denotes a row index of a transform unit, tu _ row _ idx denotes a column index of the transform unit, and the column index denotes which transform unit belongs to the horizontal direction. Since there may be coordinate points that just need to read data across two transform units in the horizontal direction during the mapping process, if this occurs, two consecutive transform unit data need to be read. To this end, the transform unit index can be fully expressed as:
tu_idx_00＝{tu_line_idx_00，tu_row_idx_00}
tu_idx_01＝{tu_line_idx_01，tu_row_idx_01}
tu_idx_10＝{tu_line_idx_10，tu_row_idx_00}
tu_idx_11＝{tu_line_idx_11，tu_row_idx_11}
after the index of the conversion unit is obtained, a comparison can be performed, the signals involved in the comparison are
camp_00＝{y_coord00[10:0],x_coord00[10:6]}
camp_01＝{y_coord01[10:0],x_coord01[10:6]}
camp_10＝{y_coord10[10:0],x_coord10[10:6]}
camp_11＝{y_coord11[10:0],x_coord11[10:6]}
Since each time the data of the sdram is read in 64 bytes as the minimum unit, the x _ coord/64 value is used for the comparison. Therefore, the correspondence of camp _00, camp _01, camp _10, camp _11 is compared with tu _ coord _ reg [ tu _ idx _00], tu _ coord _ reg [ tu _ idx _01], tu _ coord _ reg [ tu _ idx _10], tu _ coord _ reg [ tu _ idx _11], if all are equal, it means that the transform unit is hit, otherwise it is not hit.
In the present embodiment, the lack of the corresponding reference pixel is common, and as shown in fig. 5, the pixel with the hatched line in the figure is the lack of the reference pixel, and the lack of the reference pixel may occur at the non-boundary of the transformation unit, which is 3 cases, such as the lack of the data of two reference pixels in the previous row, as shown by reference numeral 32 in fig. 5, or the lack of the data of two reference pixels in the next row, as shown by reference numeral 33 in fig. 5, or the lack of the data of four reference pixels, as shown by reference numeral 31 in fig. 5. The lack of reference pixels may also occur at the boundary of the transform unit, which is 15 cases, such as lack of data of one or two reference pixels in the previous row, lack of data of one or two reference pixels in the next row, and so on, as shown by reference numerals 34 and 35 in fig. 5. The subdivision of the condition of lacking the reference pixel points is to read only the data of the transformation units of the lacking reference pixel points when the data of the reference pixel points is lacking, so that redundant data reading is not brought, the data volume of reading data from the synchronous random dynamic memory is reduced, and the occupied bandwidth can be reduced.
If the judgment result in the step S2 is yes, it indicates that the coordinates of the four reference pixels are hit, then step S3 is executed, the coordinates of the four reference pixels are written into the image pixel coordinate buffer coord _ buf _ fifo, and meanwhile, the count value of the counter tu _ cnt _ reg of the corresponding transformation unit is added by 1, that is, the count value is added by one time.
And then executing step S4 to judge whether the image pixel coordinate buffer is in a non-empty state, if so, executing step S6, otherwise, executing step S5 to wait for the state of the image pixel coordinate buffer to be changed into the non-empty state.
When step S6 is executed, the coordinates stored in the image pixel coordinate buffer coord _ buf _ fifo are read, and the read coordinates are analyzed, for example, to find out in which transformation unit the data of the pixel point of the corresponding coordinates is stored. Then, step S7 is executed to determine whether the transformation units corresponding to the four reference pixel points are all valid. Because the data of the four reference pixel points may be stored in different transformation units respectively, and the data of the four reference pixel points needs to be used when performing bilinear interpolation calculation on the target pixel point, it is first required to determine whether the transformation units corresponding to the four reference pixel points are all valid before performing bilinear interpolation calculation. In this embodiment, the validity of the transformation unit is whether the valid flag of the transformation unit is 1, if the valid flag is 1, the transformation unit is valid, and if the valid flag is 0, the transformation unit is invalid. Optionally, the transformation unit may set a readable flag, if the readable flag is set, it indicates that the transformation unit stores valid data, and the transformation unit may be read, and if the readable flag is reset, it indicates that the transformation unit does not store valid data, and the transformation unit may not be read.
If the judgment result in step S7 is no, which indicates that the transformation unit cannot be read, step S8 is executed, and step S9 is executed after the readable flags of the transformation units corresponding to the four reference pixel points are all set. Of course, if the determination result in step S7 is yes, it indicates that the conversion units corresponding to the four reference pixel points are all valid, then step S9 is directly executed, the data in the cache buffer of the cache is read, the data of the four reference pixel points are all read out, and bilinear interpolation calculation is performed, so as to obtain the output data of the target pixel point.
If the judgment result in the step S2 is negative, which indicates that at least one reference pixel point among the four reference pixel points is missing, the step S11 is executed, as shown in fig. 3, to determine which reference pixel point is missing, that is, to judge the condition of missing pixel point. For example, it is determined whether the pixel points at the upper left corner are missing or the pixel points at the lower right corner are missing. Then, step S12 is executed to write the coordinates of the missing reference pixel into the missing pixel coordinate buffer miss _ coord _ buf _ fifo, for example, if the coordinates of the missing reference pixel are { y _ coord [10 ]. The coordinates of the missing reference pixel are then written into the missing pixel coordinate buffer miss _ coord _ buf _ fifo.
Next, step S13 is executed to write the coordinates of the missing reference pixel into the image pixel coordinate buffer coord _ buf _ fifo, which is used for writing the original image data read from the sdram into the address of the cache buffer. Then, step S14 is executed to issue a data read request to the sdram.
In this embodiment, there is a handshake mechanism for initiating a data read request to the sdram in the determining module, that is, before sending the request, it needs to be confirmed that the following three conditions are satisfied at the same time: the missing pixel coordinate buffer miss _ color _ buf _ fifo is in a non-full state, the image pixel coordinate buffer color _ buf _ fifo is in a non-full state, and the count value of the counter of the conversion unit corresponding to the missing pixel is 0. Only when the three conditions are met simultaneously, the data reading request is sent, and if any one of the three conditions is not met, the data reading request is not sent.
If the counter value of the counter of the transformation unit corresponding to the missing reference pixel point is not 0 when the data reading request is sent, it indicates that the unused coordinates are still left in the image pixel coordinate buffer coord _ buf _ fifo, and if the data reading request is sent at this time, the situation that the original valid data in the transformation unit is covered by new data is caused, and just in order to avoid the phenomenon, it is set that the three situations need to be met before the data reading request can be sent.
In this embodiment, the data read request sent to the synchronous random access memory includes the following contents: the number of the requested reading pixels and the address of the requested reading pixels. The number of the requested reading pixels indicates how many 64-byte data need to be read, and since the least ideal case is that four reference pixels miss, four 64-byte data need to be read at this time. The address of the request-to-read pixel point indicates which line of the original image and which 64 bytes of data need to be read, wherein the position information of the 64 bytes of data is represented by x _ coord [10 ] = x _ coord [10 ]/64.
After sending the data read request, step S15 is executed to determine whether a data read request response is received, for example, determine whether a signal of mem _ rd _ req _ ack is received, and if the signal is received, indicate that the data of the sdram can be read. At this time, the count value of the counter tu _ cnt _ reg of the corresponding conversion unit is increased by 1 when the signal mem _ rd _ req _ ack is high. Then, step S17 is performed to read the data of the sdram and write the coordinates corresponding to the read data into the missing pixel coordinate buffer miss _ coord _ buf _ fifo and the image pixel coordinate buffer coord _ buf _ fifo.
If the determination result in step S15 is no, which indicates that the current condition does not satisfy the condition for sending the data reading request, that is, the missing pixel coordinate buffer mis _ coord _ buf _ fifo is not in the non-full state, the image pixel coordinate buffer coord _ buf _ fifo is not in the non-full state, or the count value of the counter of the conversion unit corresponding to the missing pixel point is not all 0, step S16 is executed to wait for the timing when the condition for sending the data reading request is satisfied, and after the condition is satisfied, the data reading request is sent.
Preferably, when it is determined that there are missing reference pixels and a data read request needs to be sent to the dram, the data read request is not issued immediately, but the requested information, including information about the number and location of pixels to be read, is buffered in a request buffer req _ fifo, which has a depth of 32, that is, the request buffer req _ fifo can store 32 data at maximum. After the data is written into the request buffer req _ fifo, the next hit or miss determination can be continued, so as to separate the hit determination from the data read request sending operation to improve the data reading efficiency of the sdram. As long as the request buffer req _ fifo is in a non-empty state, a request for data reading can be issued from the information of the read request in the request buffer req _ fifo.
In order to improve the efficiency of reading data from the synchronous random access memory, the data needs to be ready to be written into the address of the cache buffer before the data comes, so that the address data needs to be read from the missing pixel coordinate buffer miss _ core _ buf _ fifo in advance. In order to realize the address reading in advance, effective signals and data returned by the synchronous random dynamic memory are subjected to delay processing, so that when the data of the synchronous random dynamic memory arrive, the address information of the cache buffer of the cache is ready.
Since the data of the synchronous ram needs to be written into the cache buffer, since the cache buffer is composed of random access memory (sram), the requests to initiate the synchronous ram are sequential. And the missing pixel coordinate buffer miss _ coord _ buf _ fifo for storing the coordinates of the missing pixel point is composed of a first-in first-out memory, so that when the data of the synchronous random access memory is read back, the address used for writing the cache buffer of the cache is directly read from the missing pixel coordinate buffer miss _ coord _ buf _ fifo. And writing the data into a conversion unit corresponding to the cache buffer of the cache. Each conversion unit corresponds to 64 effective signals indicating that data is effective, and after a certain piece of data is written into the conversion unit, the effective signal indicating that the conversion unit is effective is set to be 1, namely set. When a request for reading the synchronous random dynamic memory is initiated to the transformation unit, which indicates that the data of the transformation unit can be updated, the valid signal indicating that the transformation unit is valid is set to 0.
Preferably, after the data writing of each transform unit is completed, a completion signal tu _ wr _ end is generated, which triggers the pulling up of the transform unit valid signal. When sending a data read request to the synchronous random access memory, the valid signal of the corresponding transformation unit is set to 0, namely reset. Therefore, when the data of the cache buffer of the cache is read each time, whether the signals of the corresponding four conversion units are all effective signals or not can be judged, and the correct data which is required currently is ensured when the data is fetched from the conversion units.
Therefore, after step S17 is executed, step S18 is executed to determine whether the data of the transform unit is completely written, that is, whether the completion signal tu _ wr _ end is generated, if yes, step S20 is executed to update the valid flag of the transform unit, otherwise, step S19 is executed to wait for the completion of the data writing of the transform unit, and step S20 is executed after the data writing is completed.
And after the data of the conversion units are written, executing the step S7, and performing bilinear interpolation calculation when the conversion units corresponding to the four reference pixel points are determined to be valid. The bilinear interpolation of this embodiment needs to read data of four adjacent original pixels in two rows at the top and bottom each time of correction, and valid data is already mapped into the transformation unit in the data buffering stage of the synchronous random dynamic memory, and the valid signal of the corresponding transformation unit is set to 1, that is, valid. Therefore, the interpolation calculation process is relatively simple, only the effective signals of the corresponding four transformation units need to be judged whether to be 1 before the cache buffer data of the cache is read, if all the effective signals are 1, the data can be read for calculation, and otherwise, the calculation process needs to wait until the effective signals of the four transformation units are all 1. And after the data of the four reference pixels corresponding to one target pixel point is calculated, the data of the image pixel coordinate buffer coord _ buf _ fifo can be continuously read, and the calculation of the next target pixel point is carried out until all the pixels of the image are calculated.
In this embodiment, the cache buffer is used as a way for data prefetching, and therefore, the problem of data update replacement is involved. The principle that the data of the current transform unit can be replaced is: all the valid data cached by the current transformation unit are used up. Therefore, each transform unit has a counter, and when the counter is 0, the current transform unit data is used up and can be replaced by new data. Each time one coordinate information is written into the image pixel coordinate buffer coord _ buf _ fifo, 1 is added to the count values of the counters of the corresponding four conversion units, and each time one coordinate information is read from the image pixel coordinate buffer coord _ buf _ fifo, 1 is subtracted from the count values of the counters of the corresponding four conversion units. Therefore, when there is no reference pixel, a data reading request is issued only when the counter value of the counter corresponding to the transformation unit is 0, otherwise, the old valid data may be overwritten by the new data.
Distortion correction image processing apparatus embodiment:
referring to fig. 4, the present embodiment has a vertical scanning module 11, a determining module 12, a request buffer 14, a missing image coordinate buffer 15, an image pixel coordinate buffer 16, a sdram 20, a write control module 21, a read control module 22, and a cache 23. The vertical scanning module 11 obtains the coordinates of the plurality of pixels obtained by vertical scanning in a row unit from the mapping relationship correspondence table, that is, the coordinates output by vertical scanning are provided for geometric correction.
The judgment module 12 serves as a main body part of the distortion correction image processing apparatus, and is configured to judge whether four reference pixels used for performing interpolation calculation on a target pixel are all hit, that is, judge whether coordinates of the four reference pixels have been read, and write the coordinates of the four reference pixels into the image pixel coordinate buffer 16 if the four reference pixels are all hit. If any one of the four reference pixel points is not hit, the condition of lacking pixel points is distinguished, and the coordinates of the lacking pixel points are written into the lacking pixel coordinate buffer 15.
Then, the determining module 12 further determines whether a condition for sending a data reading request to the sdram 20 is satisfied, and if the condition is satisfied, sends the data reading request to the sdram 20. Of course, the request is first written into the request buffer 14 and then sent from the request buffer 14 to the synchronous dynamic random access memory 20.
Preferably, in order to separate the determination of hit or miss from the data reading request sent to the sdram 20, the request buffer 14 can be flexibly configured according to the size of the original image distortion rate, for example, can be set to 32 bytes or other values. In addition, in order to eliminate the delay of bus read request and data return, the image pixel coordinate buffer 16 can be flexibly configured according to the size of bus delay.
The data of the synchronous random access memory 20 is written into the cache 23, the write control module 21 is used for controlling the data writing of the cache 23, and the read control module 22 is used for controlling the data reading of the cache 23. In this embodiment, the cache 23 includes a plurality of conversion units, each conversion unit has a counter and a register, and each conversion unit can set a readable flag and a valid flag, and the control of data writing and reading is realized by the count value of the counter and the register during the data writing and reading processes.
Because the invention adopts the vertical scanning mode to obtain the pixel points of the image, compared with the prior geometric correction processing technology aiming at the whole image and scanning according to the raster, the invention has the advantages of simple hardware implementation mode, no need of using a line buffer and lower requirement on hardware resources. On the other hand, the use of the request buffer eliminates the delay of the synchronous dynamic random access memory bus on the one hand, and on the other hand, the depth of the buffer is configurable, so that the image with larger distortion rate can be applied to images with different distortion rates by adjusting the depths of a plurality of buffers. In addition, the invention also subdivides the condition of the lacking reference pixel points, reduces the redundancy of distortion correction on the original image reading of the synchronous random dynamic memory, and achieves the purpose of reducing the bandwidth.
Of course, the above-mentioned solution is only a preferred embodiment of the present invention, and practical applications may also have more variations, for example, a request buffer is not provided, a data read request is directly sent to the sdram, or the depths of multiple buffers are fixed and are not configurable, and these variations do not affect the implementation of the present invention, and should also be included in the protection scope of the present invention.
Claims (7)
1. An image processing method for distortion correction, comprising:
acquiring coordinates of a plurality of pixel points from a mapping relation corresponding table, wherein the mapping relation corresponding table is a mapping relation corresponding table of pixel point coordinates in an output image and pixel point coordinates in an input image;
the method is characterized in that:
the acquired coordinates are coordinates obtained by vertical scanning in units of lines;
judging whether the coordinates of a plurality of reference pixels used for carrying out interpolation calculation on a target pixel are in the acquired coordinates or not, if so, writing the coordinates of the plurality of reference pixels into an image pixel coordinate buffer, judging whether the image pixel coordinate buffer is in a non-empty state or not, if so, reading the coordinates stored in the image pixel coordinate buffer, analyzing the read coordinates, and reading the data of each reference pixel from a high-speed buffer and carrying out interpolation calculation to obtain the data of each pixel of the output image when a transformation unit in which each reference pixel is positioned is effective;
when the coordinates of at least one reference pixel point are not in the acquired coordinates, the missing coordinates of the reference pixel points are written into a missing pixel coordinate buffer and an image pixel coordinate buffer, and a data reading request is sent to a synchronous random dynamic memory;
before sending a data reading request to the synchronous random dynamic memory, the following steps are also executed: and confirming that the missing pixel coordinate buffer is in a non-full state, the image pixel coordinate buffer is in a non-full state, and the count value of the counter of the transformation unit where the missing reference pixel point is located is zero.
2. The distortion-corrected image processing method according to claim 1, characterized in that:
each transformation unit is provided with one counter;
when the coordinates are written into the image pixel coordinate buffer, the count value of the counter of the corresponding conversion unit is added once; and each time coordinates are read from the image pixel coordinate buffer, the count value of the counter of the corresponding transformation unit is reduced by one time.
3. The distortion corrected image processing method according to claim 1 or 2, characterized in that:
each transformation unit is provided with a corresponding readable mark;
after the data of the transformation unit is written, the readable mark of the transformation unit is set; and when the data reading of the conversion unit is finished, the readable mark of the conversion unit is reset.
4. The distortion-corrected image processing method according to claim 1 or 2, characterized in that:
issuing a data read request to the synchronous random access memory comprises: and sending the request information of the data reading request to a request buffer, and then sending the request information stored in the request buffer to the synchronous random dynamic memory.
5. An image processing apparatus for correcting distortion, comprising:
the vertical scanning module is used for acquiring the coordinates of a plurality of pixel points obtained by vertical scanning in a row unit from a mapping relation corresponding table, wherein the mapping relation corresponding table is a mapping relation corresponding table of the pixel point coordinates in the output image and the pixel point coordinates in the input image;
the judging module is used for judging whether the coordinates of the reference pixel points used for carrying out interpolation calculation on the target pixel points are in the acquired coordinates or not;
the data reading module is used for writing the coordinates of the reference pixels into an image pixel coordinate buffer when the coordinates of the reference pixels are determined to be in the acquired coordinates, judging whether the image pixel coordinate buffer is in a non-empty state, if so, reading the coordinates stored in the image pixel coordinate buffer, analyzing the read coordinates, and reading the data of each reference pixel from a high-speed buffer and performing interpolation calculation when a transformation unit where each reference pixel is located is effective to obtain the data of each pixel of the output image;
the distortion correction image processing device also comprises a missing pixel coordinate buffer and an image pixel coordinate buffer;
when the coordinate of at least one reference pixel point is determined not to be in the acquired coordinate, the missing coordinate of the reference pixel point is written into the missing pixel coordinate buffer and the image pixel coordinate buffer, and a data reading request is sent to a synchronous random dynamic memory;
before sending a data reading request to the synchronous random dynamic memory, the following steps are also executed: and confirming that the missing pixel coordinate buffer is in a non-full state, the image pixel coordinate buffer is in a non-full state, and the count value of the counter of the transformation unit where the missing reference pixel point is located is zero.
6. The distortion corrected image processing apparatus according to claim 5, characterized in that:
the storage depth of the missing pixel coordinate buffer and the image pixel coordinate buffer may be set according to a distortion rate of the input image.
7. The distortion correction image processing apparatus according to claim 5 or 6, characterized by further comprising:
and the request buffer is used for storing the request information of the data reading request output by the judging module, and the request information can be sent to a synchronous random dynamic memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910040943.XA CN109785265B (en) | 2019-01-16 | 2019-01-16 | Distortion correction image processing method and image processing apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910040943.XA CN109785265B (en) | 2019-01-16 | 2019-01-16 | Distortion correction image processing method and image processing apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109785265A CN109785265A (en) | 2019-05-21 |
| CN109785265B true CN109785265B (en) | 2022-11-11 |
Family
ID=66500736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910040943.XA Active CN109785265B (en) | 2019-01-16 | 2019-01-16 | Distortion correction image processing method and image processing apparatus |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109785265B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112308787B (en) * | 2019-08-02 | 2024-05-07 | 北京地平线机器人技术研发有限公司 | Distortion correction method and device and electronic equipment |
| CN111984548B (en) * | 2020-07-22 | 2024-04-02 | 深圳云天励飞技术股份有限公司 | Neural network computing device |
| EP4139804A1 (en) * | 2020-08-25 | 2023-03-01 | Google LLC | Color management caching for display processing pipelines |
| CN114648449A (en) * | 2020-12-18 | 2022-06-21 | 华为技术有限公司 | Image remapping method and image processing device |
| CN117057982B (en) * | 2023-10-11 | 2024-01-26 | 北京地平线信息技术有限公司 | Integrated circuit for performing distortion transformation on image and electronic equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06268925A (en) * | 1993-03-17 | 1994-09-22 | Sharp Corp | Distortion correction device |
| CN102196221A (en) * | 2010-03-01 | 2011-09-21 | 精工爱普生株式会社 | Image processing device, projector and image processing method |
| CN102714696A (en) * | 2010-11-11 | 2012-10-03 | 松下电器产业株式会社 | Image processing device, image processing method, program, and imaging device |
| CN105354809A (en) * | 2015-12-05 | 2016-02-24 | 中国航空工业集团公司洛阳电光设备研究所 | Predistortion method and device based on pixel position index of output image |
| JP2017017672A (en) * | 2015-07-01 | 2017-01-19 | 株式会社リコー | Image processing device, image processing method and image processing program |
| CN106713755A (en) * | 2016-12-29 | 2017-05-24 | 北京疯景科技有限公司 | Method and apparatus for processing panoramic image |
| CN107004261A (en) * | 2015-09-15 | 2017-08-01 | 华为技术有限公司 | Image distortion correction method and device |
| CN107424126A (en) * | 2017-05-26 | 2017-12-01 | 广州视源电子科技股份有限公司 | Image correction method, device, equipment and system, image pickup equipment and display equipment |
| CN107945130A (en) * | 2017-11-28 | 2018-04-20 | 华南理工大学 | A kind of projection malformation correction method of photocuring 3D printer |
| CN108564551A (en) * | 2018-04-25 | 2018-09-21 | 珠海全志科技股份有限公司 | Fish eye images processing method and fish eye images processing unit |
-
2019
- 2019-01-16 CN CN201910040943.XA patent/CN109785265B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06268925A (en) * | 1993-03-17 | 1994-09-22 | Sharp Corp | Distortion correction device |
| CN102196221A (en) * | 2010-03-01 | 2011-09-21 | 精工爱普生株式会社 | Image processing device, projector and image processing method |
| CN102714696A (en) * | 2010-11-11 | 2012-10-03 | 松下电器产业株式会社 | Image processing device, image processing method, program, and imaging device |
| JP2017017672A (en) * | 2015-07-01 | 2017-01-19 | 株式会社リコー | Image processing device, image processing method and image processing program |
| CN107004261A (en) * | 2015-09-15 | 2017-08-01 | 华为技术有限公司 | Image distortion correction method and device |
| CN105354809A (en) * | 2015-12-05 | 2016-02-24 | 中国航空工业集团公司洛阳电光设备研究所 | Predistortion method and device based on pixel position index of output image |
| CN106713755A (en) * | 2016-12-29 | 2017-05-24 | 北京疯景科技有限公司 | Method and apparatus for processing panoramic image |
| CN107424126A (en) * | 2017-05-26 | 2017-12-01 | 广州视源电子科技股份有限公司 | Image correction method, device, equipment and system, image pickup equipment and display equipment |
| CN107945130A (en) * | 2017-11-28 | 2018-04-20 | 华南理工大学 | A kind of projection malformation correction method of photocuring 3D printer |
| CN108564551A (en) * | 2018-04-25 | 2018-09-21 | 珠海全志科技股份有限公司 | Fish eye images processing method and fish eye images processing unit |
Non-Patent Citations (4)
| Title |
|---|
| 双目数字图像实时预畸变校正技术研究;廖书红 等;《电光与控制》;20180331;第25卷(第05期);第109-114页 * |
| 基于FPGA的双目相机研制;李正阳;《中国优秀硕士学位论文全文数据库 信息科技辑》;20180315(第03期);第1-79页 * |
| 基于非线性方程的鱼眼图像畸变矫正方法研究;魏利胜 等;《电子测量与仪器学报》;20151130;第29卷(第11期);第1584-1593页 * |
| 面向视频流的畸变矫正算法的FPGA实现;徐晓 等;《电视技术》;20130702;第37卷(第13期);第53-55页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109785265A (en) | 2019-05-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109785265B (en) | Distortion correction image processing method and image processing apparatus | |
| US9262807B2 (en) | Method and system for correcting a distorted input image | |
| WO2022218253A1 (en) | Image block processing method and apparatus, and electronic device and readable storage medium | |
| JP2001101396A (en) | Processor and method for correcting image distortion and medium with program performing image distortion correction processing stored therein | |
| CN110958362A (en) | Image correction system and method based on block table look-up | |
| CN108616717B (en) | Real-time panoramic video splicing display device and method thereof | |
| JP6136190B2 (en) | Image processing apparatus and imaging apparatus | |
| EP1721298A2 (en) | Embedded system with 3d graphics core and local pixel buffer | |
| CN113643414A (en) | Three-dimensional image generation method and device, electronic equipment and storage medium | |
| WO2021179605A1 (en) | Gpu-based camera video projection method and apparatus, device, and storage medium | |
| US8462222B2 (en) | Image processing method and device, and imaging apparatus using the image processing device | |
| CN101196860A (en) | Optimized two-dimension DMA transmission method especially for access to image block | |
| CN109685715B (en) | Stepless rotation processing method for high-precision image in display system | |
| CN100568275C (en) | DSP is last to carry out the Cache bounding box accelerated method that refractive and reflective panorama launches with look-up table | |
| JP5569312B2 (en) | Memory access determination circuit, memory access determination method, cache control device, and electronic device | |
| JP2010176547A (en) | Controller included in image processor, control method and control processing program | |
| CN106780291B (en) | Real-time distortion image processing accelerating device | |
| JP4970378B2 (en) | Memory controller and image processing apparatus | |
| CN108629734B (en) | Image geometric transformation method and device and terminal | |
| US11508031B2 (en) | Warping data | |
| CN113592844A (en) | Image distortion correction method and device, terminal and storage medium | |
| CN115601223A (en) | Image preprocessing device, method and chip | |
| JP6762775B2 (en) | Image processing equipment, imaging equipment, control methods and programs | |
| TWI756771B (en) | Image transformation method | |
| CN117472801B (en) | Video memory bandwidth optimization method, device and system and BMC system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |