CN114757985A - Binocular depth sensing device based on ZYNQ improved algorithm and image processing method - Google Patents

Binocular depth sensing device based on ZYNQ improved algorithm and image processing method Download PDF

Info

Publication number
CN114757985A
CN114757985A CN202210398448.8A CN202210398448A CN114757985A CN 114757985 A CN114757985 A CN 114757985A CN 202210398448 A CN202210398448 A CN 202210398448A CN 114757985 A CN114757985 A CN 114757985A
Authority
CN
China
Prior art keywords
image
zynq
parallax
cost
programmable logic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210398448.8A
Other languages
Chinese (zh)
Inventor
李婷
杨宇
刘海桥
刘海俊
徐涵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan Institute of Engineering
Original Assignee
Hunan Institute of Engineering
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan Institute of Engineering filed Critical Hunan Institute of Engineering
Priority to CN202210398448.8A priority Critical patent/CN114757985A/en
Publication of CN114757985A publication Critical patent/CN114757985A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

The invention provides an image processing method based on a ZYNQ improved algorithm, which comprises the steps of S1, configuring a ZYNQ platform; step S2, forming an image data window; step S3, calculating a cost value; step S4, obtaining floating point parallax with sub-pixel precision; step S5, post optimization and data output. The invention also provides a binocular depth sensing device based on the ZYNQ improved algorithm, which comprises a binocular camera and a ZYNQ platform, wherein the ZYNQ platform comprises a processing system PS end and a programmable logic PL end, the processing system PS end and the programmable logic PL end carry out data interaction through an AXI bus, the programmable logic PL end is connected with the binocular camera, and a ZYNQ platform memory program is used for realizing the image processing method based on the ZYNQ improved algorithm. The invention can realize high quality, high resolution and real-time high dynamic output of the depth map, and has better high dynamic adaptability.

Description

Binocular depth perception device based on ZYNQ improved algorithm and image processing method
Technical Field
The invention relates to the technical field of computer vision application, in particular to a binocular depth perception device based on a ZYNQ improved algorithm and an image processing method.
Background
With the continuous development of automatic driving and 3D environmental perception technologies, 3D visual perception devices are increasingly widely used. And many application scenarios have high requirements on the real-time performance and accuracy of the 3D visual perception device for processing data. Especially in outdoor applications, people demand higher real-time and accuracy of processing data by 3D visual perception devices. Meanwhile, the high-speed motion of the outdoor vehicle puts higher requirements on the dynamic adaptability of the 3D visual perception device.
However, existing 3D visual perception devices, such as TOF-based 3D perception devices and structured light-based 3D depth imaging devices, are difficult to be applied outdoors due to insufficient capability of resisting natural light interference, and the speed of generating depth images is in the order of seconds, so that the devices mounted on a vehicle with a certain movement speed have a problem of poor high dynamic adaptability. Specifically, the method comprises the following steps:
1) the 3D sensing device based on TOF has long measuring distance but low resolution ratio, and is not suitable for outdoor low-power devices. Because the low-power device is the equipment of polishing that output is limited mostly, its volume heat dissipation receives the restriction, and then leads to the light intensity of beating away not enough, easily receives external strong natural light interference, finally makes the real-time nature and the accuracy nature of 3D perception greatly influenced.
2) Structured light based 3D depth imaging devices employ a depth measurement scheme, i.e. projecting a coded image or speckle spot, calculating the difference after reception, and calculating the depth by quantifying the difference. It is also not good for outdoor applications because the projected encoded image or speckle spot is easily flooded by strong natural light outdoors.
In recent years, a 3D sensing device based on a binocular camera has appeared in the industry, has the advantages of low cost, suitability for indoor and outdoor scenes, high reliability and the like, and is widely applied to the fields of automatic driving, autonomous mobile robots, three-dimensional reconstruction and the like. However, for binocular stereo vision, the existing method for calculating depth of a binocular camera has a large amount of calculation, real-time calculation is difficult to realize only by a CPU, and acceleration by using a GPU causes a large burden on the aspects of equipment size, cost and power consumption, and directly inputting data of the binocular camera to an upper layer application still occupies a large amount of upper computer calculation resources. Although some depth perception systems based on FPGA or ZYNQ exist in the industry, the output resolution of such a perception system is low, and is only 640 × 480.
Disclosure of Invention
The invention aims to provide a binocular depth perception device based on a ZYNQ improved algorithm and an image processing method, which are used for solving the problems that the existing 3D perception device is difficult to apply outdoors, high dynamic adaptability is poor, output resolution is low and parallax calculation accuracy is low. The specific technical scheme is as follows:
first, a first object of the present invention is to provide an image processing method based on a ZYNQ improved algorithm, which includes the steps of:
step S1, configuring a ZYNQ platform: the ZYNQ platform comprises a processing system PS end and a programmable logic PL end, wherein the processing system PS end and the programmable logic PL end carry out data interaction through an AXI bus, the programmable logic PL end is connected with a binocular camera, and the binocular camera needs to calibrate parameters in advance;
step S2, forming an image data window: after parameters are calibrated in step S1, preprocessing, ping-pong buffering and line buffering are carried out on a left image and a right image synchronously output by a left camera and a right camera in the binocular camera by utilizing the programmable logic PL terminal, so as to form an image data window;
step S3, calculating a cost value: performing cost calculation by using the programmable logic PL terminal to execute an improved algorithm on the image data window formed in the step S2, determining a parallax range in the cost calculation, and calculating a cost value by using the improved algorithm; the improved algorithm comprises a gray scale cost algorithm and a census transformation algorithm which are used in combination; the gray scale cost algorithm comprises the absolute value sum of gray scale difference and the absolute value sum of gray scale gradient difference which are used in combination;
step S4, obtaining the floating point parallax of the sub-pixel precision: performing cost aggregation on the left image along the left-to-right direction and the right-to-left direction to obtain a final cost value of the pixel position in the left image in the parallax range, finding out the parallax D with the minimum cost value in the parallax range determined in the step S3 by adopting a WTA (WTA) rule, simultaneously recording the cost values of two parallaxes adjacent to the parallax D, and performing sub-pixel fitting by using the parallax D and the cost values corresponding to the two parallaxes adjacent to the parallax D to obtain the floating point type parallax with sub-pixel precision;
step S5, post optimization and data output: converting the floating point type parallax in the step S4 into integer data, uploading the integer data to a Processing System (PS) end through a Programmable Logic (PL) end, performing LRC consistency check and uniqueness constraint on the integer data at the PS end of the processing system, and returning an optimization result to the Programmable Logic (PL) end to obtain a depth map, wherein at the moment, median filtering is performed to remove noise in the depth map; and outputting the depth map obtained by median filtering to a display for displaying according to an HDMI time sequence, and transmitting the depth map to an upper computer through a USB interface and/or a GIGE interface.
Further, in step S3, the improvement algorithm includes the following steps:
step S3.1, using the parallel characteristic provided by the PL end of the programmable logic, performing sobel edge detection on the image data window formed in step S2 at the same time, calculating the sum of the absolute value of the gray level difference and the sum of the absolute value of the gray level gradient difference, and combining the two to calculate the gray level cost of the image, wherein the expression (1) of the sum of the absolute value of the gray level difference is as follows:
Figure BDA0003598488670000031
in expression (1), d is a parallax predetermined for the grayscale image processing window, and the range of d depends on the pitch of the cameras in the left and right cameras of the binocular camera; (x, y) is the coordinates of the left image pixel location or the coordinates of the right image pixel location; Ω is an image processing window, and has a size of (N +1) × (N + 1); r is the number of lines of the image processing window; c is the number of columns of the image processing window; cSAD(x, y, d) is the sum of the absolute values of the gray differences of the pixels with the parallax d at (x, y) in the image; i isl(r + x, c + y) is the gray scale value of the left image pixel at (r + x, c + y); i isr(r + x + d, c + y) is the gray scale value of the right image pixel at (r + x + d, c + y)
Expression (2) of the sum of absolute values of the gradation gradient differences is as follows:
Figure BDA0003598488670000032
in expression (2), CGSAD(x, y, d) is the sum of absolute values of the gray gradient differences of the pixels with parallax d at (x, y) in the image;
Figure BDA0003598488670000033
representing the gray scale gradient of the left image pixel at (r + x, c + y);
Figure BDA0003598488670000034
representing the gray scale gradient of the right image pixel at (r + x + d, c + y);
the expression (3) of the gray scale cost is as follows:
C′SAD(x,y,d)＝CSAD(x,y,d)+w·CGSAD(x,y,d) (3)
in expression (3), C'SAD(x, y, d) represents the total gray scale cost of the pixels with parallax d at (x, y) in the left image; w is a weight coefficient;
step S3.2, unifying the gray scale cost obtained in step S3.1 and the cost scale of Census transformation into a (0, 1) interval by using an exponential function to obtain a cost function, where the cost function is used to calculate a cost value C (x, y, d) of a pixel with parallax d at (x, y) in the image, and an expression (4) of the cost function is as follows:
C(x,y,d)＝ρ(Ccensus(x,y,d),λcensus)+ρ(CS′AD(x,y,d),λSAD) (4)
in expression (4), C (x, y, d) is a cost value of a pixel having a parallax of d at (x, y) in the image;
λcensusand λSADIs a regulatory factor;
Figure BDA0003598488670000041
representing the cost value of a pixel with the parallax d at (x, y) in the image obtained by census transformation;
Figure BDA0003598488670000042
further, the parallax d ranges from 0 to 64 pixels.
Further, in step S2, the preprocessing, ping-pong buffer and line buffer processing procedure is as follows:
the pretreatment process comprises the following steps: firstly, a first register inside a binocular camera is configured by utilizing the programmable logic PL end analog IIC time sequence to drive a left camera and a right camera to respectively and synchronously output a left image and a right image, and effective pixel values in the left image and the right image are subjected to gray scale conversion; secondly, caching the image data after the gray level conversion through FIFO; finally, remapping the image data cached by the FIFO to the same plane through polar line rectification operation to obtain a left image and a right image aligned in a coplanar line;
the ping-pong cache processing process comprises the following steps: the left camera controls a first SDRAM storage chip to transmit a left image through a read-write time sequence generated by a first DDR3 read-write control module; the right camera controls a second SDRAM storage chip to transmit right images through a read-write time sequence generated by a second DDR3 read-write control module; a ping-pong operation read-write arbitration module is arranged between the left camera and the first DDR3 read-write control module and between the right camera and the second DDR3 read-write control module, and output signals of the ping-pong operation read-write arbitration module realize read-write switching of different BANKs in the first SDRAM storage chip and the second SDRAM storage chip, so that respective ping-pong cache operations on the parallel coplanar left image and right image are completed;
the line cache processing process comprises the following steps: and outputting the ping-pong cached left image and right image in lines by using N second registers with the length of the left image width or the right image width, and inputting the image output by each line into N third registers connected in series to form (N +1) × (N +1) image data windows capable of temporarily storing.
Further, in step S2, the left camera and the right camera both use a camera with an output of 720P @60 FPS.
Further, the calibration parameters in step S1 include camera parameters, rotation matrix and translation vector of the left and right cameras.
Secondly, a second objective of the present invention is to provide a binocular depth perception device based on a ZYNQ improved algorithm, which includes a binocular camera and a ZYNQ platform, where the ZYNQ platform includes a processing system PS terminal and a programmable logic PL terminal, the processing system PS terminal and the programmable logic PL terminal perform data interaction through an AXI bus, and the programmable logic PL terminal is connected to the binocular camera, and the ZYNQ platform stores a program, and the image processing method based on the ZYNQ improved algorithm is implemented by executing the program through the processing system PS terminal and the programmable logic PL terminal.
The technical scheme of the invention at least has the following beneficial effects:
(1) according to the image processing method based on the ZYNQ improved algorithm, the gray gradient reflecting edge information can be integrated through the improved algorithm under the ZYNQ platform, the processing capability of edge details in an image and the matching accuracy of the edge of the image are improved, the influence of environmental factors such as illumination and noise is effectively eliminated, and the parallax calculation precision is finally improved; the output of USB and Ethernet GIGE interfaces is realized through hard wire resources in the ZYNQ platform, the problem of large occupation of computing resources in a CPU and a GPU under the condition of realizing stereo matching through an upper computer is solved, and finally high-quality, high-resolution and real-time high-dynamic output of a depth map is realized.
(2) According to the binocular depth perception device based on the ZYNQ improved algorithm, the ZYNQ platform executes the image processing method based on the ZYNQ improved algorithm, so that the hardware compatibility of the binocular depth perception device can be improved, and the characteristic of mass data transmission can be met. And the output pixels of the binocular depth perception device can reach the higher resolution of 720P, the output frame rate can reach 60FPS, the higher updating frequency is realized while the precision is higher, the binocular depth perception device can be used for environment perception of an outdoor carrier, has better high dynamic adaptability, and can be used for obstacle recognition of an intelligent vehicle or an autonomous mobile robot with a certain movement speed.
In addition to the objects, features and advantages described above, other objects, features and advantages of the present invention are also provided. The present invention will be described in further detail below with reference to the drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. In the drawings:
FIG. 1 is a schematic diagram of the processing procedure before line caching of an image processing method based on ZYNQ improved algorithm in embodiment 1 of the present invention (not shown in the processing system PS end and the parameter calibration procedure in step S1);
FIG. 2 is a schematic diagram of the processing procedure after line caching of an image processing method based on the ZYNQ improved algorithm in embodiment 1 of the present invention;
FIG. 3 is a comparison graph of a depth map obtained by the image processing method based on the ZYNQ improved algorithm in the embodiment 1 of the present invention and a depth map obtained by the conventional AD _ Census algorithm.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
Example 1:
referring to fig. 1-2, an image processing method based on ZYNQ improved algorithm includes the following steps:
step S1, configuring a ZYNQ platform: the ZYNQ platform comprises a processing system PS end and a programmable logic PL end, wherein the processing system PS end and the programmable logic PL end carry out data interaction through an AXI bus, the programmable logic PL end is connected with a binocular camera, and the binocular camera needs to calibrate parameters in advance;
step S2, forming an image data window: after parameters are calibrated in step S1, the programmable logic PL end is utilized to carry out preprocessing, ping-pong buffer and line buffer on a left image and a right image which are respectively and synchronously output by a left camera and a right camera in the binocular camera, so as to form an image data window;
step S3, calculating a cost value: performing cost calculation by using the programmable logic PL terminal to execute an improved algorithm on the image data window formed in the step S2, determining a parallax range in the cost calculation, and calculating a cost value by using the improved algorithm; the improved algorithm comprises a gray scale cost algorithm and a census transformation algorithm which are used in combination; the gray scale cost algorithm comprises the absolute value sum of gray scale difference and the absolute value sum of gray scale gradient difference which are used in combination;
step S4, obtaining a floating point disparity of sub-pixel precision: performing cost aggregation on the left image along the left-to-right direction and the right-to-left direction to obtain a final cost value of the pixel position in the left image in the parallax range, finding out the parallax D with the minimum cost value in the parallax range determined in the step S3 by adopting a WTA (WTA) rule, simultaneously recording the cost values of two parallaxes adjacent to the parallax D, and performing sub-pixel fitting (specifically, quadratic curve fitting) by using the parallax D and the cost values corresponding to the two parallaxes adjacent to the parallax D to obtain the floating-point type parallax with sub-pixel precision;
step S5, post optimization and data output: converting the floating point parallax in the step S4 into integer data, uploading the integer data from the programmable logic PL end to the processing system PS end through dma, performing LRC consistency check and uniqueness constraint on the processing system PS end to optimize the integer data, and returning an optimization result to the programmable logic PL end to obtain a depth map, at this time, performing median filtering to remove noise in the depth map; and outputting the depth map obtained by median filtering to a display for displaying according to an HDMI time sequence, and transmitting the depth map to an upper computer through a USB interface and/or a GIGE interface.
In step S3, the improvement algorithm includes the steps of:
step S3.1, using the parallel characteristic provided by the PL end of the programmable logic, performing sobel edge detection on the image data window formed in step S2 at the same time, and taking an edge grayscale image processing window of 3 × 3 size, calculating the sum of the absolute value of the grayscale difference and the sum of the absolute value of the grayscale gradient difference on the edge grayscale image processing window, and combining the two to calculate the grayscale cost of the image, wherein the expression (1) of the sum of the absolute values of the grayscale difference is as follows:
Figure BDA0003598488670000071
in expression (1), d is a parallax predetermined for the grayscale image processing window, the range of d depending on the pitch of the cameras in the left and right cameras of the binocular camera, and the range of the parallax d is 0 to 64 pixels in the present embodiment; (x, y) is the coordinates of the left image pixel location or the coordinates of the right image pixel location; Ω is the image processing window; r is the number of rows of the image processing window, and r is taken to be [ -3, 3 [)]Any integer within; c is the number of columns of the image processing window, and c is [ -3, 3 [ ]]Any integer within; cSAD(x, y, d) is the sum of absolute values of the gray differences of the pixels located at the coordinates (x, y) and having the parallax d in the image; i isl(r + x, c + y) is the gray scale value of the left image pixel located at the coordinates (r + x, c + y) in the middle; i isr(r + x + d, c + y) is the gray scale value of the right image pixel located at the coordinate (r + x + d, c + y);
expression (2) of the sum of absolute values of the gradation gradient differences is as follows:
Figure BDA0003598488670000072
in expression (2), CGSAD(x, y, d) is the sum of absolute values of gray gradient differences of pixels located at coordinates (x, y) and having a parallax d in the image;
Figure BDA0003598488670000073
representing the gray scale gradient of the left image pixel at coordinates (r + x, c + y);
Figure BDA0003598488670000074
representing the gray scale gradient of the right image pixel at coordinates (r + x + d, c + y);
the expression (3) of the gray scale cost is as follows:
C′SAD(x,y,d)＝CSAD(x,y,d)+w·CGSAD(x,y,d) (3)
in expression (3), C'SAD(x, y, d) represents the total gray cost of pixels located at coordinates (x, y) and having a parallax of d in the left image; w is a weight coefficient, and is 0.8-1;
step S3.2, unifying the gray scale cost obtained in step S3.1 and the cost scale of Census transformation to (0, 1) interval by using an exponential function to obtain a cost function, where the cost function is used to calculate a cost value C (x, y, d), and an expression (4) thereof is as follows:
C(x,y,d)＝ρ(Ccensus(x,y,d),λcensus)+ρ(C′SAD(x,y,d),λSAD) (4)
in expression (4), C (x, y, d) is a cost value of a pixel located at coordinates (x, y) in an image and having a parallax of d; lambda [ alpha ]censusTaking 40 as a regulating factor; lambda [ alpha ]SADTaking 3, 4 or 5 as a regulating factor;
Figure BDA0003598488670000081
representing the cost value of a pixel with parallax d at the coordinate (x, y) in the image obtained by census transformation;
Figure BDA0003598488670000082
in the invention, if the existing simple SAD algorithm is adopted, namely the absolute values of the corresponding pixel differences of the corresponding pixel blocks of the left and right view images after line alignment are summed, the difference response of the left and right view images on the brightness details is large, and the matching distortion can occur under the condition of inconsistent brightness of the left and right images. If the existing pure census transformation algorithm is not sensitive to the overall brightness change of the gray-scale image, a better matching effect can be obtained even if the brightness of the left image and the brightness of the right image are inconsistent because of a comparative relative gray-scale relation. If a simple SAD algorithm and a simple census transform algorithm are simply combined for use, although the overall and detailed expressions are considered, the processing capability of edge details in an image is still insufficient, so that the result is that many mismatches exist at the edge, and the parallax calculation accuracy cannot be improved. The improved algorithm is adopted, so that the gray gradient reflecting the edge information can be blended, the processing capability of edge details in the image and the matching accuracy of the edge of the image are improved, the influence of environmental factors such as illumination, noise and the like is effectively eliminated, and the parallax calculation precision is finally improved.
In step S5, the floating-point disparity is converted into 16-bit integer data.
In step S2, the preprocessing, ping-pong cache, and line cache processes are as follows:
the pretreatment process comprises the following steps: firstly, a first register inside a binocular camera is configured by utilizing the programmable logic PL end analog IIC time sequence to drive a left camera and a right camera to respectively and synchronously output a left image and a right image, and effective pixel values in the left image and the right image are subjected to gray scale conversion; secondly, caching the image data after the gray level conversion through FIFO (first in first out) to solve the problem that the frequency of inputting the image data into the programmable logic PL end by a left camera and a right camera is inconsistent with the working frequency of the programmable logic PL end, so that the stable connection of the image data in the ZYNQ platform is realized; finally, polar line rectification operation is achieved by using the calibration parameters in the step S1 and floating point operation resources inside the ZYNQ platform, image data cached in FIFO are remapped to the same plane through the polar line rectification operation, and a left image and a right image aligned in a coplanar line are obtained, so that the calculation amount of later-stage cost aggregation and the realization difficulty of hardware are reduced conveniently;
the ping-pong cache processing process comprises the following steps: the left camera controls a first SDRAM storage chip to transmit a left image through a read-write time sequence generated by a first DDR3 read-write control module; the right camera controls a second SDRAM storage chip to transmit right images through a read-write time sequence generated by a second DDR3 read-write control module; a ping-pong operation read-write arbitration module is arranged between the left camera and the first DDR3 read-write control module and between the right camera and the second DDR3 read-write control module, and output signals of the ping-pong operation read-write arbitration module realize read-write switching of different BANKs in the first SDRAM storage chip and the second SDRAM storage chip, and complete respective ping-pong cache operations on the parallel coplanar left image and right image, so that stable storage and output of an epipolar line image pair of the left image and an epipolar line image pair of the right image are realized;
the line cache processing process comprises the following steps: and outputting the ping-pong cached left image and right image in lines by using N second registers with the length of the left image width or the right image width, inputting the image output by each line into N third registers connected in series, and forming (N +1) × (N +1) image data windows capable of temporarily storing, wherein N is 6.
In step S2, the left and right cameras each employ a cmos camera outputting 720P @60 FPS.
The calibration parameters in step S1 include camera parameters, rotation matrices, and translation vectors of the left and right cameras.
A binocular depth perception device based on a ZYNQ improved algorithm, a binocular camera and a ZYNQ platform are disclosed, the ZYNQ platform comprises a processing system PS end and a programmable logic PL end, the processing system PS end and the programmable logic PL end carry out data interaction through an AXI bus, the programmable logic PL end is connected with the binocular camera, a memory program of the ZYNQ platform is stored, and the image processing method based on the ZYNQ improved algorithm is realized through the processing system PS end and the programmable logic PL end execution program.
The invention adopts an improved algorithm, and theoretical analysis is as follows: the basic idea of SAD is to sum the absolute values of the corresponding pixel differences of the corresponding pixel blocks of the left and right view images after line alignment, and to respond to the difference in detail greatly, but the simple SAD will generate matching distortion when the left and right images have inconsistent brightness, and Census transformation is not sensitive to the overall brightness change, and because of the comparative relative gray relationship, a good matching effect can be obtained even if the left and right images have inconsistent brightness. The combination of the former two has taken into account the performance of whole and detail, but the throughput of edge detail in the image is still not enough, and the result that leads to is exactly that there are many mismatches in the edge, for further promotion algorithm interference killing feature, can merge into the grey scale gradient of reaction edge information through the mode of weighing, promotes the matching accuracy of image edge.
According to the image processing method based on the ZYNQ improved algorithm, the parallax calculation precision is improved through the improved algorithm under the ZYNQ platform, specifically, compared with a depth map (see a right image in a figure 3) obtained through the existing AD _ Census algorithm, the depth map (see a left image in the figure 3) obtained through the method reduces an invalid cavity area (see a black area m in a square frame in the left image and the right image in the figure 3), edge information is kept to the maximum, and the whole noise point is obviously reduced; the USB and Ethernet GIGE interface output is realized through hard wire resources in the ZYNQ platform, the problem of large occupation of computing resources in a CPU and a GPU under the condition of realizing stereo matching through an upper computer is solved, and finally high-quality, high-resolution and real-time high-dynamic output of a depth map is realized.
According to the binocular depth perception device based on the ZYNQ improved algorithm, the ZYNQ platform executes the image processing method based on the ZYNQ improved algorithm, so that the hardware compatibility of the binocular depth perception device can be improved, and the characteristic of mass data transmission can be met. And the output pixels of the binocular depth sensing device can reach the high resolution of 720P, the output frame rate can reach 60FPS, the high updating frequency is realized while the precision is high, the binocular depth sensing device can be used for environment sensing of outdoor carriers, has high dynamic adaptability, and can be used for obstacle recognition of intelligent vehicles or autonomous mobile robots with certain movement speed.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. An image processing method based on a ZYNQ improved algorithm is characterized by comprising the following steps:
step S1, configuring a ZYNQ platform: the ZYNQ platform comprises a processing system PS end and a programmable logic PL end, wherein the processing system PS end and the programmable logic PL end carry out data interaction through an AXI bus, the programmable logic PL end is connected with a binocular camera, and the binocular camera needs to calibrate parameters in advance;
step S2, forming an image data window: after parameters are calibrated in step S1, preprocessing, ping-pong buffering and line buffering are carried out on a left image and a right image synchronously output by a left camera and a right camera in the binocular camera by utilizing the programmable logic PL terminal, so as to form an image data window;
step S3, calculating a cost value: performing cost calculation by using the programmable logic PL terminal to execute an improved algorithm on the image data window formed in the step S2, determining a parallax range in the cost calculation, and calculating a cost value by using the improved algorithm; the improved algorithm comprises a gray scale cost algorithm and a census transformation algorithm which are used in combination; the gray scale cost algorithm comprises a sum of absolute values of gray scale differences and a sum of absolute values of gray scale gradient differences which are used in combination;
step S4, obtaining a floating point disparity of sub-pixel precision: performing cost aggregation on the left image along the left-to-right direction and the right-to-left direction to obtain a final cost value of the pixel position in the left image in the parallax range, finding out the parallax D with the minimum cost value in the parallax range determined in the step S3 by adopting a WTA (WTA) rule, simultaneously recording the cost values of two parallaxes adjacent to the parallax D, and performing sub-pixel fitting by using the parallax D and the cost values corresponding to the two parallaxes adjacent to the parallax D to obtain the floating point type parallax with sub-pixel precision;
step S5, post optimization and data output: converting the floating point parallax in the step S4 into integer data, uploading the integer data to a processing system PS end from the programmable logic PL end, performing LRC consistency check and uniqueness constraint on the processing system PS end to optimize the integer data, and returning an optimization result to the programmable logic PL end to obtain a depth map, at this time, performing median filtering to remove noise in the depth map; and outputting the depth map obtained by median filtering to a display for displaying according to an HDMI time sequence, and transmitting the depth map to an upper computer through a USB interface and/or a GIGE interface.
2. The image processing method based on ZYNQ improvement algorithm as claimed in claim 1, wherein in step S3, said improvement algorithm comprises the steps of:
step S3.1, using the parallel characteristic provided by the PL end of the programmable logic, performing sobel edge detection on the image data window formed in step S2 at the same time, calculating the sum of the absolute value of the gray level difference and the sum of the absolute value of the gray level gradient difference, and combining the two to calculate the gray level cost of the image, wherein the expression (1) of the sum of the absolute value of the gray level difference is as follows:
Figure FDA0003598488660000011
in expression (1), d is a parallax predetermined for the grayscale image processing window, and the range of d depends on the pitch of the cameras in the left and right cameras of the binocular camera; (x, y) is the coordinates of the left image pixel location or the coordinates of the right image pixel location; Ω is an image processing window, and the size is (N +1) × (N + 1); r is the number of lines of the image processing window; c is the number of columns of the image processing window; cSAD(x, y, d) is in the imageThe sum of absolute values of the gray differences of the pixels with the parallax d at (x, y); i isl(r + x, c + y) is the gray scale value of the left image pixel at (r + x, c + y);
Ir(r + x + d, c + y) is the gray scale value of the right image pixel at (r + x + d, c + y)
Expression (2) of the sum of absolute values of the gradation gradient differences is as follows:
Figure FDA0003598488660000021
in expression (2), CGSAD(x, y, d) is the sum of absolute values of gray gradient differences of pixels with parallax d at (x, y) in the image;
Figure FDA0003598488660000022
representing the gray scale gradient of the left image pixel at (r + x, c + y);
Figure FDA0003598488660000023
representing the gray scale gradient of the right image pixel at (r + x + d, c + y);
the expression (3) of the gray scale cost is as follows:
C′SAD(x,y,d)＝CSAD(x,y,d)+w·CGSAD(x,y,d) (3)
in expression (3), C'SAD(x, y, d) represents the total gray scale cost of the pixels with parallax d at (x, y) in the left image; w is a weight coefficient;
step S3.2, unifying the gray scale cost obtained in step S3.1 and the cost scale of Census transformation to the (0, 1) interval by using an exponential function to obtain a cost function, where the cost function is used to calculate a cost value C (x, y, d) of a pixel with a parallax d at (x, y) in the image, and an expression (4) of the cost function is as follows:
C(x,y,d)＝ρ(Ccensus(x,y,d),λcensus)+ρ(C′SAD(x,y,d),λSAD) (4)
in expression (4), C (x, y, d) is a cost value of a pixel having a parallax of d at (x, y) in the image;
λcensusand λSADIs a regulatory factor;
Figure FDA0003598488660000031
Ccensus(x, y, d) represents the cost value of the pixel with the parallax of d in the image obtained by census transformation, wherein the parallax of the pixel is d;
Figure FDA0003598488660000032
3. the image processing method based on the ZYNQ improvement algorithm as claimed in claim 2, wherein the parallax d is in the range of 0-64 pixels.
4. The image processing method based on the ZYNQ improved algorithm according to any one of claims 1 to 3, wherein in step S2, the processing procedures of the preprocessing, the ping-pong buffer and the line buffer are as follows:
the pretreatment process comprises the following steps: firstly, a first register inside a binocular camera is configured by utilizing the programmable logic PL end analog IIC time sequence to drive a left camera and a right camera to respectively and synchronously output a left image and a right image, and effective pixel values in the left image and the right image are subjected to gray scale conversion; secondly, caching the image data after the gray level conversion through FIFO; finally, remapping the image data cached by the FIFO to the same plane through polar line rectification operation to obtain a left image and a right image aligned in a coplanar line;
the ping-pong cache processing process comprises the following steps: the left camera controls a first SDRAM storage chip to transmit a left image through a read-write time sequence generated by a first DDR3 read-write control module; the right camera controls a second SDRAM storage chip to transmit right images through a read-write time sequence generated by a second DDR3 read-write control module; a ping-pong operation read-write arbitration module is arranged between the left camera and the first DDR3 read-write control module and between the right camera and the second DDR3 read-write control module, and output signals of the ping-pong operation read-write arbitration module realize read-write switching of different BANKs in the first SDRAM storage chip and the second SDRAM storage chip, so that respective ping-pong cache operations on the parallel coplanar left image and right image are completed;
the line cache processing process comprises the following steps: and outputting the ping-pong cached left image and right image in lines by using N second registers with the length of the left image width or the right image width, and inputting the image output by each line into N third registers connected in series to form (N +1) × (N +1) image data windows capable of temporarily storing.
5. The image processing method based on ZYNQ improvement algorithm of claim 4, wherein in step S2, said left camera and said right camera both use a camera whose output is 720P @60 FPS.
6. The image processing method based on ZYNQ improved algorithm as claimed in claim 5, wherein the calibration parameters in step S1 include camera parameters, rotation matrix and translation vector of the left and right cameras.
7. A binocular depth sensing device based on a ZYNQ improved algorithm is characterized by comprising a binocular camera and a ZYNQ platform, wherein the ZYNQ platform comprises a processing system PS end and a programmable logic PL end, the processing system PS end and the programmable logic PL end carry out data interaction through an AXI bus, the programmable logic PL end is connected with the binocular camera, and the ZYNQ platform is internally provided with a program to implement the image processing method based on the ZYNQ improved algorithm, which is disclosed by claim 6, through the processing system PS end and the programmable logic PL end execution program.
CN202210398448.8A 2022-04-15 2022-04-15 Binocular depth sensing device based on ZYNQ improved algorithm and image processing method Pending CN114757985A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210398448.8A CN114757985A (en) 2022-04-15 2022-04-15 Binocular depth sensing device based on ZYNQ improved algorithm and image processing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210398448.8A CN114757985A (en) 2022-04-15 2022-04-15 Binocular depth sensing device based on ZYNQ improved algorithm and image processing method

Publications (1)

Publication Number Publication Date
CN114757985A true CN114757985A (en) 2022-07-15

Family

ID=82330628

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210398448.8A Pending CN114757985A (en) 2022-04-15 2022-04-15 Binocular depth sensing device based on ZYNQ improved algorithm and image processing method

Country Status (1)

Country Link
CN (1) CN114757985A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117635849A (en) * 2024-01-26 2024-03-01 成都万联传感网络技术有限公司 Dynamic real-time high-precision three-dimensional imaging system

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150288948A1 (en) * 2014-04-08 2015-10-08 Tk Holdings Inc. System and method for night vision object detection and driver assistance
CN108629812A (en) * 2018-04-11 2018-10-09 深圳市逗映科技有限公司 A kind of distance measuring method based on binocular camera
CN109360235A (en) * 2018-09-29 2019-02-19 中国航空工业集团公司上海航空测控技术研究所 A kind of interacting depth estimation method based on light field data
CN110473217A (en) * 2019-07-25 2019-11-19 沈阳工业大学 A kind of binocular solid matching process based on Census transformation
CN111415305A (en) * 2020-03-10 2020-07-14 桂林电子科技大学 Method for recovering three-dimensional scene, computer-readable storage medium and unmanned aerial vehicle
CN111445380A (en) * 2020-03-19 2020-07-24 熵智科技(深圳)有限公司 Method and device for realizing real-time binocular stereo matching based on ZYNQ
CN113329219A (en) * 2021-05-07 2021-08-31 华南理工大学 Multi-output parameter dynamically configurable depth camera

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150288948A1 (en) * 2014-04-08 2015-10-08 Tk Holdings Inc. System and method for night vision object detection and driver assistance
CN108629812A (en) * 2018-04-11 2018-10-09 深圳市逗映科技有限公司 A kind of distance measuring method based on binocular camera
CN109360235A (en) * 2018-09-29 2019-02-19 中国航空工业集团公司上海航空测控技术研究所 A kind of interacting depth estimation method based on light field data
CN110473217A (en) * 2019-07-25 2019-11-19 沈阳工业大学 A kind of binocular solid matching process based on Census transformation
CN111415305A (en) * 2020-03-10 2020-07-14 桂林电子科技大学 Method for recovering three-dimensional scene, computer-readable storage medium and unmanned aerial vehicle
CN111445380A (en) * 2020-03-19 2020-07-24 熵智科技(深圳)有限公司 Method and device for realizing real-time binocular stereo matching based on ZYNQ
CN113329219A (en) * 2021-05-07 2021-08-31 华南理工大学 Multi-output parameter dynamically configurable depth camera

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117635849A (en) * 2024-01-26 2024-03-01 成都万联传感网络技术有限公司 Dynamic real-time high-precision three-dimensional imaging system
CN117635849B (en) * 2024-01-26 2024-04-09 成都万联传感网络技术有限公司 Dynamic real-time high-precision three-dimensional imaging system

Similar Documents

Publication Publication Date Title
CN109949899B (en) Image three-dimensional measurement method, electronic device, storage medium, and program product
Humenberger et al. A fast stereo matching algorithm suitable for embedded real-time systems
US20210110599A1 (en) Depth camera-based three-dimensional reconstruction method and apparatus, device, and storage medium
US11941831B2 (en) Depth estimation
Zhu et al. Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps
CN111833393A (en) Binocular stereo matching method based on edge information
WO2022057598A1 (en) Image rendering method and device
US8634637B2 (en) Method and apparatus for reducing the memory requirement for determining disparity values for at least two stereoscopically recorded images
EP3135033B1 (en) Structured stereo
CN110637461B (en) Compact optical flow handling in computer vision systems
US11417065B2 (en) Methods and systems for reprojection in augmented-reality displays
WO2020119467A1 (en) High-precision dense depth image generation method and device
CN113643414B (en) Three-dimensional image generation method and device, electronic equipment and storage medium
Zhao et al. Real-time stereo on GPGPU using progressive multi-resolution adaptive windows
US20220230327A1 (en) Graphics processing systems
CN116129037B (en) Visual touch sensor, three-dimensional reconstruction method, system, equipment and storage medium thereof
WO2022165722A1 (en) Monocular depth estimation method, apparatus and device
CN115714855A (en) Three-dimensional visual perception method and system based on stereoscopic vision and TOF fusion
CN111161398A (en) Image generation method, device, equipment and storage medium
CN112907557A (en) Road detection method, road detection device, computing equipment and storage medium
CN114757985A (en) Binocular depth sensing device based on ZYNQ improved algorithm and image processing method
US20130187918A1 (en) Method for modelling a 3d scene and corresponding device
Dong et al. A 4.29 nJ/pixel stereo depth coprocessor with pixel level pipeline and region optimized semi-global matching for IoT application
WO2020092051A1 (en) Rolling shutter rectification in images/videos using convolutional neural networks with applications to sfm/slam with rolling shutter images/videos
US10540809B2 (en) Methods and apparatus for tracking a light source in an environment surrounding a device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination