CN117274068A

CN117274068A - Image correction method, system, terminal and computer storage medium

Info

Publication number: CN117274068A
Application number: CN202210663213.7A
Authority: CN
Inventors: 张承果; 周阳; 李鹲翱; 谭其林; 唐林; 唐诗然; 段智涓; 鲁良; 郭怀成; 李燕华
Original assignee: Chengdu Pudu Robot Co ltd; Shenzhen Pudu Technology Co Ltd
Current assignee: Chengdu Pudu Robot Co ltd; Shenzhen Pudu Technology Co Ltd
Priority date: 2022-06-13
Filing date: 2022-06-13
Publication date: 2023-12-22

Abstract

The application relates to an image correction method, an image correction system, a terminal and a storage medium. The method comprises the following steps: acquiring a source image in real time through a camera; converting the source image according to the internal and external parameters of the camera to obtain a converted source image, and taking the converted source image as a new image; traversing the new image line by line, determining M rows of pixels to be corrected from the new image, and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image; and determining the pixels of the row corresponding to the M rows of pixels to be corrected from the source image according to the row-column offset of each pixel in the M rows of pixels to be corrected, and correcting the pixel values of the pixels of the row corresponding to the M rows of pixels to be corrected to obtain a corrected source image. By adopting the method, only partial source image data can be cached, the data of the whole image is not required to be cached, the occupation of camera correction logic resources is reduced, and the processing performance of camera correction is further improved.

Description

Image correction method, system, terminal and computer storage medium

Technical Field

The present disclosure relates to the field of camera distortion correction technologies, and in particular, to a method, a system, a terminal, and a storage medium for correcting an image.

Background

Because the image data is affected by a plurality of factors in the process of acquisition and transmission, slight deformation and distortion can be generated between an actual image and an ideal image, and only the image subjected to strict correction can be subjected to stereo matching to obtain scene depth information. Therefore, binocular correction is an essential component of stereo vision, and the accuracy and precision of binocular correction directly determine the effect of stereo matching.

However, the correction algorithm has a large calculation amount and high requirement on correction accuracy, and occupies a large amount of logic resources, which results in low processing performance of camera correction.

Disclosure of Invention

In view of the foregoing, it is desirable to provide an image correction method, system, terminal, computer-readable storage medium, and computer program product that can reduce the occupation of camera correction logic resources and improve the processing performance of camera correction.

In a first aspect, the present application provides a method of correcting an image. The method comprises the following steps:

acquiring a source image in real time through a camera;

converting the source image according to internal and external parameters of the camera to obtain a converted source image, and taking the converted source image as a new image, wherein each pixel of the new image has a one-to-one correspondence with a corresponding pixel in the source image;

Traversing the new image row by row, determining M rows of pixels to be corrected from the new image, and determining row-column offset of each pixel in the M rows of pixels to be corrected according to row-column serial numbers of each pixel in the M rows of pixels to be corrected in the new image;

determining pixels of a row corresponding to the M rows of pixels to be corrected from the source image according to the row-column offset of each pixel in the M rows of pixels to be corrected;

and correcting pixel values of pixels in a row of pixels corresponding to the M rows of pixels to be corrected in the source image to obtain a corrected source image.

In one embodiment, determining pixels of a row corresponding to the M rows of pixels to be corrected from the source image according to a row-column offset of each pixel of the M rows of pixels to be corrected includes:

caching pixels of a row corresponding to the M rows of pixels to be corrected in the source image into a cache region, wherein M is an integer greater than or equal to 1;

and determining an N line pixel to be corrected from the M line pixels to be corrected, and acquiring a target cache line corresponding to the N line pixel to be corrected from the cache region.

In one embodiment, correcting the pixel value of the pixel in the pixel of the row corresponding to the M rows of pixels to be corrected in the source image includes:

Determining a target pixel point corresponding to each pixel in the N line of pixels to be corrected from the target cache line according to the row-column offset of each pixel in the N line of pixels to be corrected and the storage address of each pixel in the target cache line; the storage address corresponding to the row-column offset of the corresponding pixel in the N-th row of pixels to be corrected is the same as the storage address corresponding to the target pixel point;

and correcting the pixel value of the target pixel point by using a bilinear difference method to obtain the corrected pixel value of the target pixel point.

In one embodiment, the determining, from the target cache line data, a target pixel point corresponding to each pixel in the nth row of pixels to be corrected according to the column-row offset of each pixel in the nth row of pixels to be corrected and the storage address of each pixel in the target cache line includes:

aligning the N-th line pixels to be corrected with the target cache according to the line offset of each pixel in the N-th line pixels to be corrected and the line serial number of each pixel in the target cache line to obtain a position mapping relation; the position mapping relation represents that each pixel in the N-th row of pixels to be corrected corresponds to a corresponding pixel in the target cache row one by one;

And determining a target pixel point corresponding to each pixel in the N line of pixels to be corrected from the target cache line according to the position mapping relation, the row-column offset of each pixel in the N line of pixels to be corrected and the storage address of each pixel in the target cache line.

In one embodiment, the method further comprises:

determining the total number of the line numbers of the pixels cached in the cache area;

and when the total number of the line numbers of the pixels cached in the cache region is smaller than a first threshold value, caching the uncached pixels in the source image to the cache region until the total number of the line numbers of the pixels cached in the cache region is equal to the first threshold value.

In one embodiment, determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image includes:

and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the conversion relation between the new image and the source image and the row-column serial numbers of each pixel in the M rows of pixels to be corrected in the new image and the source image, wherein the row-column offset of each pixel in the M rows of pixels to be corrected comprises a column offset and a row offset.

In one embodiment, the converting the source image according to the internal and external parameters of the camera to obtain the converted source image, and taking the converted source image as a new image includes:

converting the source image from an image pixel coordinate system to a camera coordinate system according to the conversion relation of the internal reference matrix to obtain an image coordinate corresponding to the pixel coordinate of the source image in the camera coordinate system;

and after rotating and de-distorting the image coordinates according to the internal and external parameters of the camera, mapping the processed image coordinates to the image pixel coordinate system to obtain a new image of the source image in the image pixel coordinate system.

In a second aspect, the present application also provides a correction system for an image. The system comprises:

the image acquisition module is used for acquiring a source image in real time through a camera;

the image processing module is used for converting the source image according to the internal and external parameters of the camera to obtain a converted source image, and taking the converted source image as a new image, wherein each pixel of the new image has a one-to-one correspondence with the corresponding pixel in the source image;

the image traversing module is used for traversing the new image line by line, determining M rows of pixels to be corrected from the new image, and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image;

The image correction module is used for determining pixels of a row corresponding to the M rows of pixels to be corrected from the source image according to the row-column offset of each pixel in the M rows of pixels to be corrected;

In a third aspect, the present application further provides a terminal. The terminal comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the following steps when executing the computer program:

acquiring a source image in real time through a camera;

converting the source image according to the internal and external parameters of the camera to obtain a converted source image, and taking the converted source image as a new image, wherein each pixel of the new image has a one-to-one correspondence with a corresponding pixel in the source image;

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

acquiring a source image in real time through a camera;

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

acquiring a source image in real time through a camera;

The image correction method, the system, the terminal, the storage medium and the computer program product are used for carrying out coordinate conversion on the source image according to the coordinate conversion relation between the image pixel coordinate system and the camera coordinate system by acquiring the source image to obtain a new image under the image pixel coordinate system; obtaining M rows of pixels to be corrected by traversing the new image row by row, storing source image data by adopting a row caching mechanism and determining target pixel points of each pixel of the M rows of pixels to be corrected by adopting a coordinate offset mechanism; and correcting the pixel value of the corresponding pixel according to the target pixel point to obtain a corrected source image. In the process of correcting the image, M rows of pixels to be corrected are determined by traversing the new image row by row, only part of source image data is required to be cached by adopting a row caching and offset accessing mechanism, and the data of the whole image is not required to be cached, so that occupation of camera correction logic resources is reduced, and further the processing performance of camera correction is improved.

Drawings

FIG. 1 is an application environment diagram of a method of correcting an image in one embodiment;

FIG. 2 is a flow chart of a method of correcting an image in one embodiment;

FIG. 3 is a flow chart of a method of determining pixel values of pixels to be corrected in one embodiment;

FIG. 4 is a diagram of a correction offset access mechanism according to one embodiment;

FIG. 5 is a flow chart of a method for correcting an image according to another embodiment;

FIG. 6 is a block diagram of the components of the correction platform for an image in one embodiment;

FIG. 7 is a timing diagram of the correction of a binocular camera of the correction platform of the image in one embodiment;

FIG. 8 is a block diagram of a system for correcting an image in one embodiment;

fig. 9 is an internal structural diagram of a terminal in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

The image correction method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 is communicatively connected to the camera 104. The camera is used for collecting a source image to be corrected of the terminal. The terminal acquires a source image in real time through a camera; converting the source image according to the internal and external parameters of the camera to obtain a converted source image, taking the converted source image as a new image, wherein each pixel of the new image has a one-to-one correspondence with a corresponding pixel in the source image; traversing the new image line by line, determining M rows of pixels to be corrected from the new image, and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image; determining pixels of a row corresponding to the M rows of pixels to be corrected from the source image according to the row-column offset of each pixel in the M rows of pixels to be corrected; and correcting the pixel values of the pixels in the pixels of the row corresponding to the M rows of pixels to be corrected in the source image to obtain a corrected source image. . The terminal 102 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and internet of things devices. The camera 104 may be, but is not limited to, various monocular, binocular, and multi-view cameras.

In one embodiment, as shown in fig. 2, there is provided a method for correcting an image, which is described by taking an example that the method is applied to the terminal in fig. 1, and includes the following steps:

step 202, acquiring a source image in real time through a camera.

The method comprises the steps that a camera calibration program is developed on the terminal based on OpenCV+FPGA, the calibration program runs in a Linux environment based on OpenCV, calibration parameters are loaded to an EEPROM through the terminal, the calibration parameters are automatically loaded when a sensor (can be an image acquisition sensor such as a camera) is started, the calibration parameters are quantized through the FPGA and then loaded to an embedded correction module (such as a binocular correction module) to finish correction of a camera image, and the corrected image is output and displayed on the terminal. FPGA (Field-Programmable Gate Array, field programmable gate array) is a product of further development on the basis of programmable devices such as PAL, GAL, CPLD.

The camera in the present embodiment may be, but not limited to, a monocular camera, a binocular camera, or a multi-view camera, and the present embodiment will be described by taking the camera as a binocular camera. The binocular camera comprises a left camera and a right camera, the source image acquired by the corresponding left camera is a first source image, and the image acquired by the right camera is a second source image. The terminal is connected with the camera through a communication line (the communication line can be an I2C bus), and the camera uploads the acquired source image to the terminal. And when the camera is started to collect images, calibration parameters are automatically loaded for calibration. It will be understood that the correction processing of the left camera and the right camera is the same, and the correction processing of any one of the left camera and the right camera will be described as an example.

And 204, converting the source image according to the internal and external parameters of the camera to obtain a converted source image, and taking the converted source image as a new image.

Wherein each pixel of the new image has a one-to-one correspondence with a corresponding pixel in the source image.

Specifically, an internal reference matrix is obtained, and a source image is converted from an image pixel coordinate system to a camera coordinate system according to the internal reference matrix, so that image coordinates corresponding to pixels in the source image in the camera coordinate system are obtained; and (3) carrying out rotation and de-distortion processing on the image coordinates according to the internal and external parameters of the camera, and mapping the processed image coordinates to an image pixel coordinate system to obtain a new image of the source image in the image pixel coordinate system.

Further, taking a source image of a left camera as an example to explain according to the internal reference matrix, converting the image pixel coordinates of the source image from an image pixel coordinate system to a camera coordinate system to obtain image coordinates corresponding to the pixel coordinates of the source image in the camera coordinate system; and (3) carrying out rotation and de-distortion processing on the image coordinates according to the internal and external parameters of the camera, and mapping the processed image coordinates to an image pixel coordinate system to obtain a new image of the source image in the image pixel coordinate system. The image pixel coordinate systems of the two images are respectively converted into a camera coordinate system through a common internal reference matrix, and the two camera coordinate systems are respectively rotated to obtain a new camera coordinate system. By multiplying the rotation matrices IR1 and IR2; and respectively carrying out de-distortion operation on the left camera and the right camera aiming at the new camera coordinates, and after the de-distortion operation is finished, respectively converting the left camera coordinate system and the right camera coordinate system into a left image pixel coordinate system and a right image pixel coordinate system by using internal reference matrixes of the left camera and the right camera.

The method comprises the steps that the internal and external parameters of a camera are acquired through the camera, image data under various angles are acquired through the camera, and calibration processing is carried out according to the image data, so that the internal and external parameters of the camera are obtained. When the camera is a binocular camera, the binocular camera is used for collecting binocular image data of a plurality of pairs of angles, and the calibration processing is carried out according to the binocular image data to obtain the internal and external parameters of the left and right cameras in the binocular camera. For example, acquisition of an internal reference matrix of a cameraConverting a source image from an image pixel coordinate system to a camera coordinate system according to an internal reference matrix of the camera to obtain image coordinates corresponding to the pixel coordinates of the source image in the camera coordinate system, namely the image coordinates can be expressed as: />Where (v, u) represents pixel coordinates and (x, y) represents image coordinates; rotating and de-distorting the left and right cameras according to the internal and external parameters of the cameras, wherein the rotation is performed by multiplying the rotation matrix by the leftIs realized as follows:

r＝x ² +y ²

x'＝x*(1+k1*r+k2*r ² +k3*r ³ )+2*p1*x*y+p2*(r+2*x ² )

y'＝y*(1+k1*r+k2*r ² +k3*r ³ )+2*p2*x*y+p1*(r+2*y ² )

wherein d1= [ k1 k2 k3 p1 p2] is a distortion coefficient, and the rotation and distortion correction of the binocular camera can be implemented by the existing method, which is not described herein.

Step 206, traversing the new image row by row, determining M rows of pixels to be corrected from the new image, and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected.

Where N is a positive integer and may be, but is not limited to, 1, 2, 5, etc. Each pixel of each row has a corresponding row-column serial number. The line offset refers to the deviation value of the current line count in the new image from the corresponding mapped pixel coordinates. The width of the new image is obtained, and the column offset of each pixel in the N-th row of pixels to be corrected is determined according to the column serial number y of each pixel in the N-th row of pixels to be corrected, the y coordinate value remapy in the mapping pixel coordinates of the corresponding pixel and the width img_w of the new image, wherein the column offset can represent (remapy-y) x img_w. And determining the row offset of each pixel in the N line of pixels to be corrected according to the row serial number x of each pixel in the N line of pixels to be corrected, the x coordinate value remapx in the mapping pixel coordinates of the corresponding pixel and the new image, wherein the row offset can represent remapx-x. The column offset δ for each pixel can be expressed as: δ= (remapy-y) img_w+remapx-x. The calculation method of the row-column offset delta of each pixel in the nth row pixels to be corrected in the new image corresponding to the left and right source images is the same, and will not be described in detail herein.

Specifically, a new image is traversed row by row, an nth row of pixels to be corrected is determined from M rows of pixels to be corrected, mapping pixel coordinates of each pixel in the nth row of pixels to be corrected are obtained, and row-column offset of each pixel in the nth row of pixels to be corrected is determined according to row-column serial numbers of each pixel in the nth row of pixels to be corrected and corresponding mapping pixel coordinates.

Traversing the new image line by line, determining M rows of pixels to be corrected from the new image, acquiring row and column numbers of each pixel in each row of pixels in the M rows of pixels to be corrected, corresponding mapping pixel coordinates, and determining row and column offset of each pixel in the M rows of pixels to be corrected according to the row and column numbers of each pixel in each row of pixels in the M rows of pixels to be corrected and the corresponding mapping pixel coordinates. Further, determining a column offset for each of the corrected M rows of pixels includes: determining the row offset of each pixel in each row of pixels in M rows of pixels to be corrected according to the row serial number of each row of pixels in the M rows of pixels to be corrected and the abscissa in the corresponding mapping pixel coordinates; and determining the column offset of each pixel in each column of the M rows of pixels to be corrected according to the vertical coordinate in the corresponding mapping pixel coordinates of the column serial number of each column of the M rows of pixels to be corrected. The column and row offset for each pixel is determined based on the row offset and the corresponding column offset for each pixel.

Optionally, in one embodiment, before traversing the new image row by row, determining M rows of pixels to be corrected from the new image, the method further comprises: splitting the new image to obtain a plurality of line images, and generating sequence identifiers of the line images; the new image is traversed row by row based on the sequence identification.

And step 208, correcting the pixel value of the corresponding pixel in the M rows of pixels to be corrected according to the row-column offset of each pixel in the M rows of pixels to be corrected, so as to obtain a corrected source image.

Specifically, according to the row-column offset of each pixel in the M rows of pixels to be corrected, pixels in a row corresponding to the M rows of pixels to be corrected are determined from the source image; and correcting the pixel values of the pixels in the pixels of the row corresponding to the M rows of pixels to be corrected in the source image to obtain a corrected source image. The correction is performed in a correction offset storage mechanism, a target pixel point corresponding to each pixel is read from a line buffer area according to the row-column offset of each pixel in M lines of pixels to be corrected, four pixel points adjacent in the width and height directions are determined according to the target pixel point, bilinear interpolation is performed on the pixel value of the corresponding pixel in the N lines of pixels to be corrected according to the pixel value of the four pixel points, and a corrected source image is obtained, wherein the corrected source image is an undistorted and line-parallel image.

The cache size of the line cache area can be set according to actual requirements, or can be a fixed value. The target cache line data cached in the line cache region is determined according to the offset of the mapped pixel coordinates. For example, the shift of the mapped pixel coordinates is maximally up and down row _ max, the buffer area needs to buffer the original image data of row row_max+1. Bilinear interpolation can be realized by the existing bilinear difference processing mode, and details are not repeated here.

In the image correction method, a new image under an image pixel coordinate system is obtained by acquiring a source image and carrying out coordinate conversion on the source image according to a coordinate conversion relation between the image pixel coordinate system and a camera coordinate system; obtaining M rows of pixels to be corrected by traversing the new image row by row, storing source image data by adopting a row caching mechanism and determining target pixel points of each pixel of the M rows of pixels to be corrected by adopting a coordinate shifting mechanism; and correcting the pixel value of the corresponding pixel according to the target pixel point to obtain a corrected source image. In the process of correcting the image, M rows of pixels to be corrected are determined by traversing the new image row by row, only part of source image data is required to be cached by adopting a row caching and offset accessing mechanism, and the data of the whole image is not required to be cached, so that occupation of camera correction logic resources is reduced, and further the processing performance of camera correction is improved.

Optionally, in order to further improve the processing performance of camera correction, determining an nth row of pixels to be corrected from M rows of pixels to be corrected, acquiring a target cache row corresponding to the nth row of pixels to be corrected from the cache region, and further correcting the pixel value of a corresponding pixel in the nth row of pixels to be corrected, including the following steps:

Step 302, a target cache line for the nth row pixels to be corrected is determined.

Specifically, each pixel of the nth row of pixels to be corrected is written into a storage area, the mapping pixel coordinate of each pixel in the nth row of pixels to be corrected is obtained, a row to be cached is determined from a source image according to the offset of the mapping pixel coordinate of each pixel relative to the pixel coordinate of the corresponding pixel of the source image, and data of the row to be cached is cached in a row cache area of a terminal, so that cache data of a target cache row of the nth row of pixels to be corrected is obtained. The cache data of the target cache line comprises pixel values of pixel points of each line and write addresses written into the cache region.

Step 304, determining a target pixel point corresponding to each pixel in the N line of pixels to be corrected from the target cache line according to the row-column offset of each pixel in the N line of pixels to be corrected and the storage address of each pixel in the target cache line.

The storage address of each pixel in the target cache line refers to the write address written into the cache region.

Specifically, corresponding target pixel points are determined from target cache line data according to a correction offset access mechanism and a line offset of each pixel in an N-th line of pixels to be corrected, namely, according to the line offset of each pixel in the N-th line of pixels to be corrected and a line serial number of each pixel in the target cache line, the N-th line of pixels to be corrected and the target cache are aligned to obtain a position mapping relation, wherein the position mapping relation represents that each pixel in the N-th line of pixels to be corrected corresponds to corresponding pixel in the target cache line one by one. After each pixel in the N-th line of pixels to be corrected is aligned with the target cache line data, determining a mapping address according to the row-column offset of each pixel in the N-th line of pixels to be corrected and the storage address of the corresponding row-column pixel in the target cache line, and reading the pixel point corresponding to the mapping address to obtain the target pixel point corresponding to each pixel in the N-th line of pixels to be corrected. The target pixel point is a virtual pixel point corresponding to the pixel point in the new image in the source image.

As shown in fig. 4, for the schematic diagram of the correction offset access mechanism, the target cache line data is determined, and the target cache line data is written into the write address (wr_addr) of the line cache region by enabling the write in the terminal line cache region (wr_en). According to the row-column offset delta of each pixel in the N-th row of pixels to be corrected, when a target pixel point corresponding to each pixel in the N-th row of pixels to be corrected is read, aligning each row of pixels in the row of data to be corrected with a target cache line to obtain a position mapping relation between each pixel in the N-th row of pixels to be corrected and the cache pixel in the target cache line data; according to the position mapping relation, reading data (rd_out) is carried out through reading enabling (rd_en) to obtain a mapping address in the buffer area, then the reading address (rd_addr) is determined, and further the corresponding target pixel point is read from the target buffer line.

And 306, correcting the pixel value of the target pixel point by using a bilinear difference method to obtain the corrected pixel value of the target pixel point.

Specifically, corresponding target pixel points are determined from target cache line data according to the mapping address, four adjacent pixel points in the width and height directions are determined according to the target pixel points, bilinear interpolation is carried out on pixel values of corresponding pixels in N-th line pixels to be corrected according to the determined four adjacent pixel points, the pixel values of the corresponding pixels in the N-th line pixels to be corrected in a new image are obtained, the pixel values of each pixel in each line of the new image values are sequentially calculated, and the corrected source image is obtained.

In the method for determining the pixel value of the pixel to be corrected, the target cache line data is determined according to the offset of the pixel in the nth line to be corrected, the corresponding virtual pixel point in the source image is determined from the cached target cache line data by adopting a line offset storage mechanism according to the line offset of each pixel in the nth line to be corrected, four adjacent pixel points in the width and height directions are determined according to the virtual pixel point, bilinear interpolation calculation is performed according to the four pixel points, and the pixel value in the target image is obtained, so that the corrected source image is obtained. By adopting a row-column offset storage mechanism, the efficient utilization of the on-chip memory in the terminal is realized, the use of an external memory is avoided, the time consumption of a camera correction algorithm is obviously improved, and the processing performance of the terminal on camera correction is improved.

In one embodiment, as shown in fig. 5, there is provided a method for correcting an image, which is applied to the terminal in fig. 1, and the camera is exemplified as a binocular camera, and includes the following steps:

step 502, acquiring a source image in real time through a camera.

Step 504, converting the source image according to the internal and external parameters of the camera to obtain a converted source image, and taking the converted source image as a new image.

Specifically, binocular image data under each angle of a plurality of pairs of checkerboards are collected through binocular cameras, binocular image data are subjected to double-target determination, inner and outer parameters of the left camera and the right camera are obtained, and an inner reference matrix is determined according to the inner and outer parameters. Acquiring a left source image and a right source image in real time through a camera, and converting the left source image and the right source image from an image pixel coordinate system to a camera coordinate system according to the conversion relation of an internal reference matrix to obtain image coordinates corresponding to the pixel coordinates of the left source image and the right source image in the camera coordinate system; and after rotating and de-distorting the image coordinates according to the internal and external parameters of the camera, mapping the processed image coordinates to an image pixel coordinate system to obtain new images corresponding to the left source image and the right source image in the image pixel coordinate system.

Step 506, traversing the new image line by line, determining M lines of pixels to be corrected from the new image, and buffering pixels of a line corresponding to the M lines of pixels to be corrected in the source image to the buffer area.

Wherein M is an integer greater than or equal to 1.

Specifically, new images corresponding to the left source image and the right source image in the image pixel coordinate system are traversed line by line respectively, M rows of pixels to be corrected are determined in the new images corresponding to the left source image and the right source image in the image pixel coordinate system, and pixels of a row corresponding to the M rows of pixels to be corrected are determined from the source image according to the offset of each pixel in the M rows of pixels to be corrected and are cached in a cache region.

Optionally, in one embodiment, after determining the pixels of the row corresponding to the M rows of pixels to be corrected from the source image, further determining the total number of rows of pixels buffered in the buffer; and when the total number of the line numbers of the pixels cached in the cache region is smaller than a first threshold value, caching the uncached pixels in the source image into the cache region until the total number of the line numbers of the pixels cached in the cache region is equal to the first threshold value. The first threshold is determined according to actual requirements such as actual processing performance.

In step 508, the nth row of pixels to be corrected is determined from the M rows of pixels to be corrected, and the target cache line corresponding to the nth row of pixels to be corrected is obtained from the cache region.

Specifically, the offset of the mapping pixel coordinates of each pixel in the nth row of pixels to be corrected in the new image corresponding to the left and right source images is obtained respectively, and the cache data of the target cache line is determined from the cache area of the pixels in the row corresponding to the M rows of pixels to be corrected in the source image according to the offset.

Step 510, determining the row-column offset of each pixel in the nth row of pixels to be corrected according to the row-column serial number of each pixel in the nth row of pixels to be corrected.

Specifically, determining a column offset of each pixel in the corresponding N-th row of pixels to be corrected according to a column sequence number of each pixel in the N-th row of pixels to be corrected in the new image corresponding to the left source image and the right source image in an image pixel coordinate system; and determining the row offset of each pixel in the N line pixels to be corrected in the new image corresponding to the left source image and the right source image according to the line serial numbers of the N line pixels to be corrected, and carrying out weighting processing on the column offset and the row offset to obtain the row-column offset delta of each pixel in the N line pixels to be corrected.

In step 512, a target pixel point corresponding to each pixel in the nth row of pixels to be corrected is determined from the target cache line according to the row-column offset of each pixel in the nth row of pixels to be corrected and the storage address of each pixel in the target cache line.

The storage address corresponding to the row-column offset of the corresponding pixel in the N-th row of pixels to be corrected is the same as the storage address corresponding to the target pixel point.

Step 514, correcting the pixel value of the target pixel by using the bilinear difference method to obtain the corrected pixel value of the target pixel.

Specifically, according to the row-column offset of each pixel in the N-th line of pixels to be corrected in the new image corresponding to the left and right source images, aligning each line of pixels in the line data to be corrected with the target cache line to obtain the position mapping relation between each pixel in the N-th line of pixels to be corrected and the cache pixel in the target cache line; and obtaining a mapping address in the buffer area according to the position mapping relation, and determining a corresponding target pixel point from the target buffer line data according to the mapping address. Further, each line of pixels in the line data to be corrected is aligned with the target cache line, a line cache area address corresponding to each pixel address in the N line of pixels to be corrected is determined in the line cache area, and when the line cache area address is larger than the line offset of the corresponding pixels, a difference value is processed through the line cache area address wr_addr and the line offset to obtain a mapping address. When the address of the line buffer is smaller than or equal to the row-column offset delta of the corresponding pixel, the line buffer address and the row-column offset delta are subjected to difference processing, and then summed with the size len of the line buffer to obtain the mapping address.

Wherein, the determination of the mapping address rd_addr in the line buffer in the buffer area can be expressed as:

determining four adjacent pixel points in the width and height directions according to the target pixel points, performing bilinear interpolation on pixel values of corresponding pixels in N rows of pixels to be corrected according to the determined four adjacent pixel points to obtain pixel values of corresponding pixels in N rows of pixels to be corrected in a new image, and sequentially calculating pixel values of each pixel in each row of the new image value to obtain new images corresponding to corrected left and right source images.

In the image correction method, a new image under an image pixel coordinate system is obtained by acquiring a source image and carrying out coordinate conversion on the source image according to a coordinate conversion relation between the image pixel coordinate system and a camera coordinate system; the new image is traversed line by line to obtain the N line pixels to be corrected, and a line caching mechanism is adopted to store the target line source image data without storing the whole image data; determining a target pixel point of each pixel of the N-th row of pixels to be corrected by adopting a coordinate offset machine; and correcting the pixel value of the corresponding pixel according to the target pixel point to obtain a corrected source image. In the process of correcting the image, the N line pixels to be corrected are determined by traversing the new image line by line, only part of source image data is required to be cached by adopting a line caching and offset accessing mechanism, the data of the whole image is not required to be cached, the problem of distortion and rotating cross-line access is solved by adopting a multi-line offset accessing mechanism, the occupation of camera correction logic resources is reduced, the processing performance of camera correction is further improved, and the smoother image data of the new image is obtained by adopting a bilinear interpolation algorithm.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides an image correction platform for realizing the image correction method. The implementation of the solution provided by the platform is similar to the implementation described in the above method, so the specific limitation in the embodiments of the correction platform for one or more images provided below may be referred to the limitation of the correction method for an image hereinabove, and will not be described herein.

In one embodiment, as shown in fig. 6, a block diagram of an image correction platform is provided, the camera correction platform comprising a host computer, an image acquisition sensor, and a terminal, the terminal comprising a memory and a processor. The image acquisition sensor can be a binocular camera, the platform develops a camera calibration program (for example, the binocular camera calibration program) based on OpenCV+FPGA, the binocular calibration program operates in a Linux environment based on OpenCV, calibration parameters are loaded into an EEPROM (Electrically Erasable Programmable Read-Only Memory) through an upper computer, the calibration parameters are automatically loaded when the image acquisition sensor is started, the calibration parameters are loaded into an embedded binocular correction module after being quantized by the FPGA, the correction of a binocular camera image is finally completed, and the corrected image is output and displayed by the upper computer.

The double-target programming is developed based on OpenCV, and internal and external parameters of the double-camera are generated by utilizing the multi-angle checkerboard image. The upper computer has functions of loading internal and external parameters of the camera, displaying a correction target image and communicating with the binocular correction processor, and the binocular camera correction processor based on the FPGA receives image data of the binocular sensor in real time, outputs the correction target image to the upper computer in real time, and initializes the EEPROM and the binocular image sensor.

Specifically, binocular image data under each angle of a plurality of pairs of checkerboards are collected through a binocular camera and transmitted to an upper computer through USB2.0, and the upper computer stores the binocular image data; the method comprises the steps of reading checkerboard images acquired by an upper computer, and outputting internal and external parameters of left and right cameras through OpenCV calibration; the FPGA-based binocular camera correction processor acquires internal and external parameters output by the upper computer through USB2.0, writes the internal and external parameters into the EEPROM in software driving, initializes an internal and external parameter register of the binocular image correction module when the FPGA-based binocular camera correction processor is started, and acquires a source image in real time through a camera; performing rotation and de-distortion processing on the source image according to the internal and external parameters of the camera to obtain a rotated and de-distorted source image, and taking the rotated and de-distorted source image as a new image; traversing the new image line by line, determining M rows of pixels to be corrected from the new image, and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image; determining pixels of a row corresponding to the M rows of pixels to be corrected from the source image according to the row-column offset of each pixel in the M rows of pixels to be corrected; and correcting the pixel values of the pixels in the pixels of the row corresponding to the M rows of pixels to be corrected in the source image to obtain a corrected source image. As shown in fig. 7, a timing diagram of the correction of the binocular camera of the correction stage of the image. The parameters marked with axi prefix represent the internal and external parameters marked by the binocular camera, are stored in a register form in the FPGA, img_frm_vld represents an image frame header, img_row_vld represents an image data effective signal, img_dat represents an image data signal, remap_frm_vld represents a corrected image frame header, remap_row_vld represents a corrected image data effective signal, and remap_img_dat represents a corrected image data signal. It can be seen that the FPGA-based binocular camera epipolar correction process is a pure pipelined process.

Based on the same inventive concept, the embodiments of the present application also provide an image correction system for implementing the above-mentioned image correction method. The implementation of the solution provided by the system is similar to the implementation described in the above method, so the specific limitation in the embodiments of the correction system for one or more images provided below may be referred to the limitation of the correction method for an image hereinabove, and will not be described herein.

In one embodiment, as shown in FIG. 8, there is provided a correction system for an image, comprising: an image acquisition module 802, an image processing module 804, an image traversal module 806, and an image correction module 808, wherein:

an image acquisition module 802 for acquiring a source image in real time by a camera;

the image processing module 804 is configured to convert the source image according to the internal and external parameters of the camera, obtain a converted source image, and use the converted source image as a new image, where each pixel of the new image has a one-to-one correspondence with a corresponding pixel in the source image.

The image traversing module 806 is configured to traverse the new image line by line, determine M rows of pixels to be corrected from the new image, and determine a row-column offset of each pixel in the M rows of pixels to be corrected according to a row-column serial number of each pixel in the M rows of pixels to be corrected in the new image.

An image correction module 808, configured to determine pixels of a row corresponding to the M rows of pixels to be corrected from the source image according to a row-column offset of each pixel in the M rows of pixels to be corrected; and correcting the pixel values of the pixels in the pixels of the row corresponding to the M rows of pixels to be corrected in the source image to obtain a corrected source image. .

According to the correction system of the image, the source image is obtained, and the source image is subjected to coordinate conversion according to the coordinate conversion relation between the image pixel coordinate system and the camera coordinate system, so that a new image under the image pixel coordinate system is obtained; obtaining M rows of pixels to be corrected by traversing the new image row by row, storing source image data by adopting a row caching mechanism and determining target pixel points of each pixel of the M rows of pixels to be corrected by adopting a coordinate shifting mechanism; and correcting the pixel value of the corresponding pixel according to the target pixel point to obtain a corrected source image. In the process of correcting the image, M rows of pixels to be corrected are determined by traversing the new image row by row, only part of source image data is required to be cached by adopting a row caching and offset accessing mechanism, and the data of the whole image is not required to be cached, so that occupation of camera correction logic resources is reduced, and further the processing performance of camera correction is improved. In another embodiment, a correction system for an image is provided, which includes, in addition to an image acquisition module 802, an image processing module 804, an image traversing module 806, and an image correction module 808: the system comprises a caching module, an alignment module, a target pixel point determining module and a row-column offset determining module, wherein:

The buffer module is used for buffering pixels of a row corresponding to M rows of pixels to be corrected in the source image to a buffer area, wherein M is an integer greater than or equal to 1;

the image obtaining module 802 is further configured to determine an nth row of pixels to be corrected from the M rows of pixels to be corrected, and obtain a target cache row corresponding to the nth row of pixels to be corrected from the cache region.

The image correction module 808 is further configured to determine a target pixel point corresponding to each pixel in the nth row of pixels to be corrected from the target cache line according to the row-column offset of each pixel in the nth row of pixels to be corrected and the storage address of each pixel in the target cache line; the storage address corresponding to the row-column offset of the corresponding pixel in the N-th row of pixels to be corrected is the same as the storage address corresponding to the target pixel point; and correcting the pixel value of the target pixel point by using a bilinear difference method to obtain the corrected pixel value of the target pixel point.

The alignment module is used for aligning the N-th line pixels to be corrected with the target cache according to the line offset of each pixel in the N-th line pixels to be corrected and the line serial number of each pixel in the target cache line to obtain a position mapping relation; the position mapping relation represents that each pixel in the N line of pixels to be corrected corresponds to a corresponding pixel in the target cache line one by one.

The target pixel point determining module is used for determining a target pixel point corresponding to each pixel in the N line of pixels to be corrected from the target cache line according to the position mapping relation, the row-column offset of each pixel in the N line of pixels to be corrected and the storage address of each pixel in the target cache line.

The row-column offset determining module is used for determining row-column offset of each pixel in the M rows of pixels to be corrected according to the conversion relation between the new image and the source image and the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image and the source image, wherein the row-column offset of each pixel in the M rows of pixels to be corrected comprises a row offset and a row offset.

The image conversion module is used for converting the source image from the image pixel coordinate system to the camera coordinate system according to the conversion relation of the internal reference matrix to obtain the image coordinate corresponding to the pixel coordinate of the source image in the camera coordinate system;

and (3) carrying out rotation and de-distortion processing on the image coordinates according to the internal and external parameters of the camera, and mapping the processed image coordinates to an image pixel coordinate system to obtain a new image of the source image in the image pixel coordinate system.

In one example, acquiring a source image in real time through a camera, and converting the source image from an image pixel coordinate system to a camera coordinate system according to the conversion relation of an internal reference matrix to obtain an image coordinate corresponding to the pixel coordinate of the source image in the camera coordinate system; and (3) carrying out rotation and de-distortion processing on the image coordinates according to the internal and external parameters of the camera, and mapping the processed image coordinates to an image pixel coordinate system to obtain a new image of the source image in the image pixel coordinate system.

Traversing the new image line by line, determining M rows of pixels to be corrected from the new image, and determining the row-column offset of each pixel in the M rows of pixels to be corrected according to the row-column serial number of each pixel in the M rows of pixels to be corrected in the new image. Caching pixels of a row corresponding to M rows of pixels to be corrected in a source image into a cache region, wherein M is an integer greater than or equal to 1; and determining an N line pixel to be corrected from M lines of pixels to be corrected, and acquiring a target cache line corresponding to the N line pixel to be corrected from the cache region. And determining a target pixel point corresponding to each pixel in the N line of pixels to be corrected from the target cache line according to the row-column offset of each pixel in the N line of pixels to be corrected and the storage address of each pixel in the target cache line. And correcting the pixel value of the target pixel point by using a bilinear difference method to obtain the corrected pixel value of the target pixel point.

The various modules in the correction system for images described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or independent of a processor in the terminal, or may be stored in software in a memory in the terminal, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a terminal is provided, which may be a terminal, and an internal structure diagram thereof may be as shown in fig. 9. The terminal comprises a processor, a memory, a communication interface, a display screen and an input system which are connected through a system bus. Wherein the processor of the terminal is adapted to provide computing and control capabilities. The memory of the terminal includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the terminal is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a method of correcting an image. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen, the input system of the terminal can be a touch layer covered on the display screen, can be a key, a track ball or a touch pad arranged on a terminal shell, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 9 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the terminal to which the present application is applied, and that a particular terminal may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an embodiment, there is also provided a terminal including a memory and a processor, the memory storing a computer program, the processor implementing the steps of the method embodiments described above when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

It should be noted that, user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples represent only a few embodiments of the present application, which are described in more detail and are not thereby to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method for correcting an image, applied to an electronic device, the method comprising:

acquiring a source image in real time through a camera;

2. The method of claim 1, wherein determining pixels of a row corresponding to the M rows of pixels to be corrected from the source image based on the row-column offset of each of the M rows of pixels to be corrected, comprises:

3. The method according to claim 2, wherein correcting pixel values of pixels in a row of pixels in the source image corresponding to the M rows of pixels to be corrected comprises:

4. A method according to claim 3, wherein said determining a target pixel point corresponding to each pixel in the nth row of pixels to be corrected from the target cache line data based on the column-row offset of each pixel in the nth row of pixels to be corrected and the storage address of each pixel in the target cache line comprises:

5. The method according to claim 2, wherein the method further comprises:

6. The method of claim 1, wherein determining a rank offset for each of the M rows of pixels to be corrected based on a rank order number of each of the M rows of pixels to be corrected in the new image, comprises:

7. The method according to claim 1, wherein converting the source image according to the internal and external parameters of the camera to obtain the converted source image, and taking the converted source image as a new image comprises:

8. A system for correcting an image, the system comprising:

9. A terminal comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.