CN111815693B - Depth image generation method and device - Google Patents
Depth image generation method and device Download PDFInfo
- Publication number
- CN111815693B CN111815693B CN202010923720.0A CN202010923720A CN111815693B CN 111815693 B CN111815693 B CN 111815693B CN 202010923720 A CN202010923720 A CN 202010923720A CN 111815693 B CN111815693 B CN 111815693B
- Authority
- CN
- China
- Prior art keywords
- scene
- pixel point
- pixel
- value
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Measurement Of Optical Distance (AREA)
Abstract
The invention discloses a method and a device for generating a depth image, which comprise the following steps: and calculating the gray gradient of each scene pixel point in the scene image block, and taking a plurality of scene pixel points exceeding a gradient threshold value as effective scene pixel points. And acquiring initial coordinates and gray values of corresponding sub-pixel points of the same name in the reference image of the effective scene pixel points, and expressing the parallax of the effective scene pixel points as linear polynomial functions of the pixel point coordinates. And acquiring a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the homonymous sub-pixel point. And acquiring the corresponding target polynomial coefficient when the normalized cross-correlation function value is maximum. And acquiring the target parallax of the scene pixel point in the center of the image block. And calculating the depth value corresponding to the scene pixel point by the parallax to generate a depth image corresponding to the scene image, and improving the accuracy of the depth estimation result of each pixel point in the scene image.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a depth image generation method and device.
Background
The monocular speckle structured light system is a depth perception device and is applied to the fields of intelligent security, mobile robots, augmented reality and the like. The system utilizes a speckle projection device to project a fine speckle pattern onto the surface of a scene, and simultaneously, a camera acquires an image of the scene and matches the image with a pre-stored reference image, so that a depth map of the scene is obtained. In the process, image matching is an important step for generating a depth map, and the accuracy of the depth result is directly influenced by the quality of a matching result.
The existing matching methods are mainly divided into two categories, namely a grayscale-based method and a binary-based method, and the two categories of methods generally adopt an image block matching idea to calculate the matching cost. Image block matching is based on the assumption that the disparity values of all pixels in the same image block are substantially equal. For a relatively complex scene image, the parallax of each pixel in a local area may have a large variation range, resulting in low accuracy of the depth estimation result of each pixel in the scene image.
Disclosure of Invention
The invention aims to provide a depth image generation method. And for the region with larger local depth change, the accuracy of the depth estimation result of each pixel point in the scene image is improved.
In order to achieve the purpose, the adopted technical scheme comprises the following steps:
s110: the method comprises the steps of obtaining a scene image and a pre-stored reference image, and calculating a sub-pixel parallax value of each scene pixel point in the scene image according to the coordinates and the gray scale of the scene pixel point in the scene image and the coordinates and the gray scale of the corresponding reference pixel point in the reference image.
S120: and (3) constructing a scene image block E with the size of M multiplied by N scene pixel points by taking a single scene pixel point in the scene image as a center. And setting the sub-pixel parallax value of each scene pixel point in the scene image block to meet the formula (1).
di＝A0Xi+B0Yi+C0Formula (1)
Wherein d isiIs the sub-pixel disparity value, x, of the ith scene pixel pointiIs the x component, y, of the coordinates of the ith scene pixel pointiIs the y component of the ith scene pixel point coordinate, A0、B0、C0Is a polynomial coefficient.
And generating a transfinite equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the transfinite equation set.
S130: and calculating the gray gradient of each scene pixel point in the scene image block, and taking a plurality of scene pixel points exceeding a gradient threshold value as effective scene pixel points.
S140: and acquiring coordinates of the same-name sub-pixel points in the reference image block corresponding to the effective scene pixel points according to the coordinates and the polynomial coefficients of the effective scene pixel points, and acquiring gray values of the same-name sub-pixel points according to the abscissa of the same-name sub-pixel points and the gray values of two adjacent pixel points at the left and right of the same-name sub-pixel points.
S150: and acquiring a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point according to the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point.
S160: and acquiring a corresponding target polynomial coefficient when the normalized cross-correlation function value is maximum according to the relation between the normalized cross-correlation function value and the polynomial coefficient.
S170: and acquiring the target parallax of the scene pixel points according to the formula (1), the target polynomial coefficients and the coordinates of the single scene pixel points.
S180: and acquiring the depth value corresponding to the scene pixel point according to the target parallax of the scene pixel point, the depth of the reference image, the length of the system baseline and the focal length of the camera.
S190: repeating S120 to S180 for all scene pixel points in the scene image, and obtaining the corresponding depth value of each scene pixel point to generate the depth image corresponding to the scene image.
The present invention also provides a depth image generating apparatus, including:
the sub-pixel parallax value obtaining module is configured to obtain a scene image and a pre-stored reference image, and calculate a sub-pixel parallax value of each scene pixel point in the scene image according to the coordinates and the gray scale of the scene pixel point in the scene image and the coordinates and the gray scale of the corresponding reference pixel point in the reference image.
The system comprises a polynomial coefficient acquisition module, a scene image processing module and a scene image processing module, wherein the polynomial coefficient acquisition module is configured to construct a scene image block with the size of M multiplied by N scene pixels by taking a single scene pixel in a scene image as a center. And setting the sub-pixel parallax value of each scene pixel point in the scene image block E to meet the formula (1).
di＝A0Xi+B0Yi+C0Formula (1)
Wherein d isiIs the sub-pixel disparity value, x, of the ith scene pixel pointiIs the x component, y, of the coordinates of the ith scene pixel pointiIs the y component of the ith scene pixel point coordinate, A0、B0、C0Is a polynomial coefficient.
And generating a transfinite equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the transfinite equation set.
And the effective scene pixel point acquisition module is configured to calculate the gray gradient of each scene pixel point in the scene image block, and take a plurality of scene pixel points exceeding a gradient threshold value as effective scene pixel points.
And the gray value acquisition module is configured to acquire coordinates of the same-name sub-pixel points in the reference image block corresponding to the effective scene pixel points according to the coordinates and the polynomial coefficients of the effective scene pixel points, and acquire gray values of the same-name sub-pixel points according to the abscissa of the same-name sub-pixel points and gray values of two adjacent whole pixel points on the left and right of the same-name sub-pixel points.
And the function value acquisition module is configured to acquire a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point according to the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point.
And a target polynomial coefficient acquisition module configured to acquire a corresponding target polynomial coefficient when the normalized cross-correlation function value is maximum, according to a relationship between the normalized cross-correlation function value and the polynomial coefficient.
And the target parallax obtaining module is configured to obtain the target parallax of the scene pixel according to the formula (1), the target polynomial coefficient and the coordinate of the single scene pixel.
And the depth value acquisition module is configured to acquire a depth value corresponding to the scene pixel point according to the target parallax of the scene pixel point, the depth of the reference image, the system baseline length and the camera focal length.
And the depth image generation module is configured to acquire a depth value corresponding to each scene pixel point to generate a depth image corresponding to the scene image.
Compared with the prior art, the invention has the technical effects that: generating a transfinite equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the transfinite equation set.
The key point of the method is that a polynomial coefficient is optimized through steps S130 to S160 to obtain a target polynomial coefficient, and the target parallax of a scene pixel point is obtained according to the optimized target polynomial coefficient, a formula (1) and the coordinate of a single scene pixel point. So that the target disparity is closer to the actual disparity value. The depth image corresponding to the scene image is generated through S180 and S190, so that the accuracy of the depth estimation result of each pixel point in the scene image can be improved for the area with large local depth change.
Drawings
Fig. 1 is a schematic flow chart of a depth image generation method according to the present invention.
Fig. 2 is a schematic diagram of mapping of an effective scene pixel point corresponding to a homonymous point in the present invention.
Fig. 3 is a schematic diagram of obtaining a sub-pixel parallax value of a scene pixel point according to the present invention.
Fig. 4 is a schematic flow chart of obtaining a sub-pixel parallax value of a scene pixel point in the present invention.
Fig. 5 is a schematic structural diagram of the depth image generating apparatus according to the present invention.
Detailed Description
The following describes embodiments of the present invention with reference to the drawings.
As shown in fig. 1, an embodiment of the present invention is a depth image generating method, including the following steps:
s110: the method comprises the steps of obtaining a scene image and a pre-stored reference image, and calculating a sub-pixel parallax value of each scene pixel point in the scene image according to the coordinates and the gray scale of the scene pixel point in the scene image and the coordinates and the gray scale of the corresponding reference pixel point in the reference image.
Fig. 2 shows a scene image 1 and a pre-stored reference image 2. The scene image 1 is an image of a scene actually captured by the camera. Reference image 2 is an image that is projected onto the scene surface with a speckle projection device to form a fine speckle pattern, which is then captured by a camera.
S120: and (3) constructing a scene image block with the size of M multiplied by N scene pixel points by taking a single scene pixel point in the scene image as a center. Setting the sub-pixel parallax value of each scene pixel point in the scene image block to satisfy the formula (1) di＝A0Xi+B0Yi+C0。
And generating a transfinite equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the transfinite equation set.
How to obtain the polynomial coefficients corresponding to the scene image block E is specifically described below with reference to fig. 2.
And (3) constructing a scene image block E with the size of M multiplied by N scene pixel points by taking a single scene pixel point p in the scene image 1 as a center. Setting the sub-pixel parallax value of each scene pixel point in the scene image block E to satisfy the formula (1) di＝A0Xi+B0Yi+C0。
Wherein d isiIs the sub-pixel disparity value, x, of the ith scene pixel pointiIs the x component, y, of the coordinates of the ith scene pixel pointiIs the y component of the ith scene pixel point coordinate, A0、B0、C0Is a polynomial coefficient.
And generating a transfinite equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the transfinite equation set.
Specifically, the x component and the y component of the coordinates of each scene pixel point in the scene image block E, and the sub-pixel parallax value calculated in S110 are substituted into di＝Axi+Byi+ C, generating an over-limit equation set containing M × N equations, wherein the over-limit equation set is as follows:
a, B and C are unknown numbers, the over-limit equation set is solved by a least square method to obtain initial values A of A, B and C corresponding to the scene image block E0，B0，C0，A0、B0And C0I.e. polynomial coefficients.
The present invention obtains a target polynomial coefficient for an optimized polynomial coefficient through the following steps S130 to S160, and steps S130 to S160 are key steps of the present invention. And acquiring the target parallax of the scene pixel points according to the optimized target polynomial coefficients, the formula (1) and the coordinates of the single scene pixel points. The target parallax thus acquired is closer to the actual parallax value.
The following describes the steps for optimizing the polynomial coefficients.
S130: and calculating the gray gradient of each scene pixel point in the scene image block, and taking a plurality of scene pixel points exceeding a gradient threshold value as effective scene pixel points.
That is, for the scene image block E, the gray gradient of each scene pixel point in the scene image block E may be calculated by the sobel operator, for example, if the scene image block E includes i scene pixel points, the scene pixel point P is calculated1To PiThe gray scale gradient of (a).
And setting a gradient threshold value, and taking a plurality of scene pixel points exceeding the gradient threshold value as effective scene pixel points. In other words, the scene pixel point P is eliminated1To PiAnd the quality of the pixel points with the middle gray gradient lower than the gradient threshold is improved in the following steps.
The Sobel operator is an important processing method in the field of computer vision. Mainly for obtaining the first order gradient of a digital image. The Sobel operator is used for obtaining the gray gradient of each scene pixel point by adding the weight difference of the gray values of the four upper, lower, left and right neighborhoods of each scene pixel point in the image.
S140: and acquiring coordinates of the same-name sub-pixel points in the reference image block corresponding to the effective scene pixel points according to the coordinates and the polynomial coefficients of the effective scene pixel points, and acquiring the gray value of the same-name sub-pixel points according to the abscissa of the same-name sub-pixel points and the gray values of two adjacent whole pixel points at the left and right of the same-name sub-pixel points.
In FIG. 2, a pixel point P is shown according to the effective sceneiObtaining effective scene pixel point P by the coordinate and polynomial coefficientiCorresponding to the same-name sub-pixel point q in the reference image blockiAccording to the co-name sub-pixel point qiAbscissa and synonym ofPixel point qiLeft and right adjacent two whole pixel points qi-1And q isi+1Obtaining the same-name sub-pixel point qiThe gray value of (a).
Effective scene pixel point P in scene image 1iThe mapping of the same name point of (2) in the reference image. And a set of corresponding sub-pixel points with the same name in each scene pixel point in the scene image block forms a mapping area F of the scene image block E. Namely, the scene image blocks E in the scene image 1 are mapped into the mapping area F in the reference image 2.
The homonymous point is an image point of the same point on different images of the actual object. For example, a scene pixel p and a corresponding sub-pixel q are formed on the scene image and the reference image respectively at the same point on the ground.
Specifically, in S140, obtaining coordinates of a corresponding sub-pixel point of the same name in the reference image block of the effective scene pixel point according to the coordinates and the polynomial coefficient of the effective scene pixel point includes:
valid scene pixel piIs set as pi(xi,yi) Effective scene pixel point piThe coordinate of the sub-pixel point with the same name in the corresponding reference image block is qi(xi-di，yi) Wherein d isiObtained by calculation of formula (1).
Specifically, in S140, obtaining the gray value of the corresponding sub-pixel point according to the abscissa of the corresponding sub-pixel point and the gray values of two adjacent pixel points of the corresponding sub-pixel point includes:
homonymous sub-pixel point qiGray value of g'iObtained by the formula (3).
Where round () represents the rounding function, g'iLAnd g'iRRespectively being homonymous sub-pixel points qiThe gray value of the left pixel point and the gray value of the right pixel point.
S150: and acquiring a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point according to the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point.
Specifically, the normalized cross-correlation function value is z, and z is set to 1-c, where c is calculated by formula (4).
Wherein, giIs the ith effective scene pixel point piGray scale of g'iIs piIs the same as the name point qiGray scale of gVIs n effective scene pixel points piGrayscale mean value of (i-1 … n), g'VIs n points of identity qi(i-1 … n).
S160: and acquiring a corresponding target polynomial coefficient when the normalized cross-correlation function value is maximum according to the relation between the normalized cross-correlation function value and the polynomial coefficient.
Specifically, with A0，B0，C0And calculating a partial derivative matrix and a Hessian matrix of the C pairs A, B and C by an incremental method as a starting point.
Partial derivative matrix of
Hessian matrix is
And (4) gradually calculating the increments delta A, delta B and delta C of the A, B and C by a Newton iteration method until the increments are smaller than a set increment threshold value, and obtaining the solution of target polynomial coefficients A, B and C.
S170: and acquiring the target parallax of the scene pixel points according to the formula (1), the target polynomial coefficients and the coordinates of the single scene pixel points.
Specifically, the solution formula d of the target polynomial coefficients A, B and C is taken to be Ax + By + C, and the target parallax d of the single scene pixel point p in the center of the scene image block E is obtained. Wherein, x and y are respectively an x component and a y component of a p coordinate of a single scene pixel point.
S180: and acquiring the depth value corresponding to the scene pixel point according to the target parallax of the scene pixel point, the depth of the reference image, the length of the system baseline and the focal length of the camera.
Specifically, the depth Z corresponding to the scene pixel point p is calculated by formula (5).
Where d is the target parallax, Z0And b is the depth corresponding to the reference image, b is the length of the system baseline, and f is the focal length of the camera.
S190: repeating S120 to S180 for all scene pixel points in the scene image, and obtaining the corresponding depth value of each scene pixel point to generate the depth image corresponding to the scene image.
In the invention, the polynomial coefficient is obtained through S110 and S120, the parallax of scene pixel points in the scene image is not calculated by using the polynomial coefficient, but the polynomial coefficient is optimized through S130 to S160 to obtain the target polynomial coefficient after the polynomial coefficient is optimized, and the target parallax of the scene pixel points is calculated through the formula (1), the target polynomial coefficient and the coordinates of single scene pixel points, so that for the inclined plane and other areas with large local depth change in the scene to be measured, a more accurate depth estimation result of the scene pixel points can be obtained, and meanwhile, the calculation speed is higher.
As shown in fig. 3 and 4, based on the above embodiments of the present invention, S110 in the present invention specifically includes the following steps.
S111: based on an initial scene pixel point in a scene image, an initial scene image block with a set size is obtained by taking the initial scene pixel point as a center.
Specifically, for any initial scene pixel point p (x, y) in the scene image 1, an M × N initial scene image block of a set size is selected with the initial scene pixel point p as a center.
S112: and acquiring a reference pixel point with the same position as the scene pixel point in the reference image. And acquiring an initial reference image block with a set size in the reference image by taking the reference pixel point as the center.
In fig. 3, a reference pixel point p in the same position as the scene pixel point p in the reference image 2 is obtained1'. In reference image 2 with reference pixel point p1' obtaining an initial reference image block B of a set size for the center1'。
A reference pixel point p' (x, y) having the same position as p (x, y) is acquired in the reference image 2. Selecting a search region G [ x-Delta 1-x + Delta 2, y ] near the reference pixel point p' (x, y)]For each pixel point p 'in region A'iSelecting image block B 'of scene pixel points with size of M multiplied by N by taking the pixel point as center'i. Respectively calculate B'i(i ═ 1, … r, where r is the number of pixels in the search area G) and the scene image block E, i.e. cost, the cost value is preferably calculated by using the following formula:
s113: and setting a search area based on the positions of reference pixel points of the reference image, and selecting a plurality of reference image blocks by taking each pixel point in the search area as a center.
S114: calculating the difference value between the initial scene image block and each reference image block, taking the reference image block with the minimum difference value as a matching reference image block, and taking a reference pixel point in the center of the matching image block as a matching pixel point.
For example, the difference values between the initial scene image block E and the reference image blocks B1', B2' and B3 'in fig. 3 are the smallest reference image block B1' as the matching reference image block.
S115: and taking the horizontal coordinate difference value of the matching pixel point and the scene pixel point as the initial parallax value of the scene pixel point.
S116: and acquiring a difference value corresponding to two reference pixel points adjacent to the left and right of the matching pixel point.
By formula (2)
The corresponding sub-pixel disparity values.
And d is the sub-pixel parallax corresponding to the initial pixel point. The initial disparity value corresponding to the initial pixel point is dI. The difference value of the corresponding matched pixel points of the initial pixel points in the reference image is ca. The difference value of the left adjacent point of the matched pixel point is c1. The difference value of the adjacent points on the right side of the matched pixel point is c2。
In the invention, the sub-pixel parallax value of each scene pixel point in the scene image is obtained by a block matching method, so that the accuracy of the sub-pixel parallax value of each scene pixel point is improved.
Referring to fig. 5, another embodiment of the present invention further provides a depth image generating apparatus, including:
the sub-pixel disparity value obtaining module 202 is configured to obtain a scene image and a pre-stored reference image, and calculate a sub-pixel disparity value of each scene pixel point in the scene image according to the coordinates and the gray scale of the scene pixel point in the scene image and the coordinates and the gray scale of the corresponding reference pixel point in the reference image.
A polynomial coefficient obtaining module 204 configured to construct a scene image block E with a size of M × N scene pixels centered on a single scene pixel in the scene image. And setting the sub-pixel parallax value of each scene pixel point in the scene image block E to meet the formula (1).
di＝A0Xi+B0Yi+C0Formula (1)
Wherein d isiIs the sub-pixel disparity value, x, of the ith scene pixel pointiIs the x component, y, of the coordinates of the ith scene pixel pointiIs the y component of the ith scene pixel point coordinate, A0、B0、C0Is a polynomial coefficient.
And generating a transfinite equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the transfinite equation set.
An effective scene pixel point obtaining module 206, configured to calculate a gray gradient of each scene pixel point in the scene image block, and take a plurality of scene pixel points exceeding a gradient threshold as effective scene pixel points.
The gray value obtaining module 208 is configured to obtain coordinates of the same-name sub-pixel points in the reference image block corresponding to the effective scene pixel points according to the coordinates and the polynomial coefficients of the effective scene pixel points, and obtain gray values of the same-name sub-pixel points according to the abscissa of the same-name sub-pixel points and gray values of two whole pixels adjacent to the same-name sub-pixel points from left to right.
The function value obtaining module 210 is configured to obtain a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point according to the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point.
And a target polynomial coefficient obtaining module 212 configured to obtain a corresponding target polynomial coefficient when the normalized cross-correlation function value is maximum according to the relationship between the normalized cross-correlation function value and the polynomial coefficient.
A target disparity obtaining module 214 configured to obtain a target disparity of a scene pixel according to formula (1), a target polynomial coefficient, and a coordinate of a single scene pixel.
A depth value obtaining module 216 configured to obtain a depth value corresponding to a scene pixel point according to the target parallax of the scene pixel point, the depth of the reference image, the system baseline length, and the camera focal length.
A depth image generation module 218 configured to obtain a depth value corresponding to each scene pixel point to generate a depth image corresponding to the scene image.
The depth image generation apparatus of this embodiment and the depth image generation method are the same inventive concept, and refer to the specific description of the depth image generation method, which is not described herein again.
Claims (9)
1. A depth image generation method, comprising:
s110: acquiring a scene image and a pre-stored reference image, and calculating a sub-pixel parallax value of each scene pixel point in the scene image according to the coordinate and the gray level of the scene pixel point in the scene image and the coordinate and the gray level of a corresponding reference pixel point in the reference image;
s120: constructing a scene image block with the size of M multiplied by N scene pixels by taking a single scene pixel in the scene image as a center; setting the sub-pixel parallax value of each scene pixel point in the scene image block to meet a formula (1);
di＝A0Xi+B0Yi+C0formula (1)
Wherein d isiIs the sub-pixel disparity value, x, of the ith scene pixel pointiIs the x component, y, of the coordinates of the ith scene pixel pointiIs the y component of the ith scene pixel point coordinate, A0、B0、C0Is a polynomial coefficient;
generating an over-limit equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block, and acquiring a polynomial coefficient corresponding to the scene image block according to the over-limit equation set;
s130: calculating the gray gradient of each scene pixel in the scene image block, and taking a plurality of scene pixels exceeding a gradient threshold as effective scene pixels;
s140: acquiring coordinates of the effective scene pixel points corresponding to the same-name sub-pixel points in the reference image block according to the coordinates and polynomial coefficients of the effective scene pixel points, and acquiring gray values of the same-name sub-pixel points according to the abscissa of the same-name sub-pixel points and the gray values of two adjacent whole pixel points on the left and right of the same-name sub-pixel points;
s150: acquiring a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point according to the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point;
s160: acquiring a corresponding target polynomial coefficient when the normalized cross-correlation function value is maximum according to the relation between the normalized cross-correlation function value and the polynomial coefficient;
s170: acquiring the target parallax of the scene pixel points according to the formula (1), the target polynomial coefficient and the coordinates of the single scene pixel points;
s180: acquiring a depth value corresponding to the scene pixel point according to the target parallax of the scene pixel point, the depth of the reference image, the length of a system base line and the focal length of a camera;
s190: repeating S120 to S180 for all scene pixel points in the scene image, and obtaining the corresponding depth value of each scene pixel point to generate the depth image corresponding to the scene image.
2. The depth image generation method according to claim 1, wherein the S110 includes:
s111: based on initial scene pixel points in the scene image, acquiring an initial scene image block with a set size by taking the initial scene pixel points as a center;
s112: acquiring reference pixel points with the same positions as the scene pixel points in the reference image; acquiring an initial reference image block with a set size by taking a reference pixel point as a center in a reference image;
s113: setting a search area based on the positions of reference pixel points of the reference image, and selecting a plurality of reference image blocks by taking each pixel point in the search area as a center;
s114: calculating difference values of the initial scene image blocks and each reference image block, taking the reference image block with the minimum difference value as a matching reference image block, and taking a reference pixel point in the center of the matching image block as a matching pixel point;
s115: taking the horizontal coordinate difference value of the matching pixel point and the scene pixel point as the initial parallax value of the scene pixel point;
s116: acquiring a difference value corresponding to two reference pixel points adjacent to the left and right of the matching pixel point;
calculating a sub-pixel parallax value corresponding to an initial scene pixel point through a formula (2);
wherein d is the sub-pixel parallax corresponding to the initial pixel point; the initial parallax value corresponding to the initial pixel point is dI(ii) a The difference value of the corresponding matched pixel points of the initial pixel points in the reference image is ca(ii) a The difference value of the left adjacent point of the matched pixel point is c1(ii) a The difference value of the adjacent points on the right side of the matched pixel point is c2。
3. The depth image generation method according to claim 1, wherein the S130 includes:
and calculating the gray gradient of each scene pixel point in the scene image block through a Sobel operator.
4. The method of claim 1, wherein the obtaining, in S140, coordinates of a pixel point of the effective scene corresponding to a same-name sub-pixel point in the reference image block according to the coordinates and polynomial coefficients of the pixel point of the effective scene comprises:
setting the coordinate of the effective scene pixel point as pi(xi,yi) The effective scene pixel point piThe coordinate of the sub-pixel point corresponding to the same name in the reference image block is qi(xi-di，yi) Wherein d isiObtained by calculation of formula (1);
in the S140, obtaining the gray value of the corresponding sub-pixel according to the abscissa of the corresponding sub-pixel and the gray values of the two adjacent pixels on the left and right of the corresponding sub-pixel, includes:
the homonymous sub-pixel point qiGray value of g'iObtained by formula (3);
g′i＝(round(xi-di)+1-(xi-di))g′iL+((xi-di)-round(xi-di))g′iRformula (3)
Where round () represents the rounding function, g'iLAnd g'iRRespectively being homonymous sub-pixel points qiThe gray value of the left pixel point and the gray value of the right pixel point.
5. The depth image generation method according to claim 1, wherein the S150 includes:
the normalized cross-correlation function value is z, and z is set to be 1-c, wherein c is obtained by calculation of formula (4);
wherein, giIs the ith effective scene pixel point piGray scale of g'iIs piIs the same as the name point qiGray scale of gVIs n effective scene pixel points piGrayscale mean value of (i-1 … n), g'VIs n points of identity qi(i-1 … n).
6. The depth image generation method according to claim 5, wherein the S160 includes:
with A0、B0And C0Calculating partial derivative matrixes and Hessian matrixes of the pairs of C A, B and C by an incremental method as a starting point;
the partial derivative matrix is
The Hessian matrix is
Gradually calculating the increments delta A, delta B and delta C of A, B and C by a Newton iteration method until the increments are smaller than a set increment threshold value, and obtaining the solution of target polynomial coefficients A, B and C; the target polynomial is: d is Ax + By + C.
7. The depth image generation method according to claim 6, wherein the S170 includes:
bringing the solution d of the target polynomial coefficient A, B and C to Ax + By + C to obtain the target parallax d of a single scene pixel point p in the center of the image block; wherein x and y are the x component and y component of the image coordinate of a single scene pixel point p, respectively.
8. The depth image generation method according to claim 7, wherein the depth Z corresponding to the scene pixel point is calculated by formula (5);
wherein Z is0And b is the depth corresponding to the reference image, b is the length of the system baseline, and f is the focal length of the camera.
9. A depth image generation apparatus, characterized by comprising:
the sub-pixel parallax value acquisition module is configured to acquire a scene image and a pre-stored reference image, and calculate the sub-pixel parallax value of each scene pixel point in the scene image according to the coordinates and the gray scale of the scene pixel point in the scene image and the coordinates and the gray scale of the corresponding reference pixel point in the reference image;
a polynomial coefficient obtaining module configured to construct a scene image block E of size M × N scene pixels centered on a single scene pixel in the scene image; setting the sub-pixel parallax value of each scene pixel point in the scene image block E to meet a formula (1);
di＝A0Xi+B0Yi+C0formula (1)
Wherein d isiIs the sub-pixel disparity value, x, of the ith scene pixel pointiIs the x component, y, of the coordinates of the ith scene pixel pointiIs the y component of the ith scene pixel point coordinate, A0、B0、C0Is a polynomial coefficient;
generating an over-limit equation set containing M multiplied by N equations according to the formula (1) and the sub-pixel parallax value and the coordinates of each scene pixel point in the scene image block E, and acquiring a polynomial coefficient corresponding to the scene image block according to the over-limit equation set;
an effective scene pixel point obtaining module configured to calculate a gray gradient of each scene pixel point in the scene image block, and take a plurality of scene pixel points exceeding a gradient threshold as effective scene pixel points;
the gray value acquisition module is configured to acquire coordinates of the pixel points of the effective scene corresponding to the same-name sub-pixel points in the reference image block according to the coordinates and polynomial coefficients of the pixel points of the effective scene, and acquire gray values of the same-name sub-pixel points according to the abscissa of the same-name sub-pixel points and gray values of two whole pixels adjacent to the left and right of the same-name sub-pixel points;
a function value obtaining module configured to obtain a normalized cross-correlation function value between the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point according to the gray value of the effective scene pixel point and the gray value of the same-name sub-pixel point;
a target polynomial coefficient obtaining module configured to obtain a target polynomial coefficient corresponding to the maximum normalized cross-correlation function value according to a relationship between the normalized cross-correlation function value and the polynomial coefficient;
a target parallax obtaining module configured to obtain a target parallax of the scene pixel according to the formula (1), a target polynomial coefficient and a coordinate of the single scene pixel;
a depth value obtaining module configured to obtain a depth value corresponding to the scene pixel point according to a target parallax of the scene pixel point, a depth of a reference image, a system baseline length, and a camera focal length;
a depth image generation module configured to obtain a depth value corresponding to each scene pixel point to generate a depth image corresponding to the scene image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010923720.0A CN111815693B (en) | 2020-09-04 | 2020-09-04 | Depth image generation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010923720.0A CN111815693B (en) | 2020-09-04 | 2020-09-04 | Depth image generation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111815693A CN111815693A (en) | 2020-10-23 |
| CN111815693B true CN111815693B (en) | 2021-01-12 |
Family
ID=72859928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010923720.0A Active CN111815693B (en) | 2020-09-04 | 2020-09-04 | Depth image generation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111815693B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114627134B (en) * | 2022-05-18 | 2022-08-09 | 深圳元象信息科技有限公司 | Scene image generation method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012247597A (en) * | 2011-05-27 | 2012-12-13 | Seiko Epson Corp | Image processing method, image processing device, electro-optic device, and electronic equipment |
| CN103810708B (en) * | 2014-02-13 | 2016-11-02 | 西安交通大学 | A kind of laser speckle image depth perception method and device |
| CN104331890B (en) * | 2014-10-30 | 2017-06-16 | 北京大学深圳研究生院 | A kind of global disparity method of estimation and system |
| CN108109130A (en) * | 2016-11-24 | 2018-06-01 | 广州映博智能科技有限公司 | Self-adapting window stereo vision matching method based on laser facula |
| CN106651794B (en) * | 2016-12-01 | 2019-12-03 | 北京航空航天大学 | A kind of projection speckle bearing calibration based on virtual camera |
| CN107945222A (en) * | 2017-12-15 | 2018-04-20 | 东南大学 | A kind of new Stereo matching cost calculates and parallax post-processing approach |
-
2020
- 2020-09-04 CN CN202010923720.0A patent/CN111815693B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111815693A (en) | 2020-10-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI729995B (en) | Generating a merged, fused three-dimensional point cloud based on captured images of a scene | |
| CN113129430B (en) | Underwater three-dimensional reconstruction method based on binocular structured light | |
| KR100681320B1 (en) | Method for modelling three dimensional shape of objects using level set solutions on partial difference equation derived from helmholtz reciprocity condition | |
| CN107990846B (en) | Active and passive combination depth information acquisition method based on single-frame structured light | |
| CN112184811B (en) | Monocular space structured light system structure calibration method and device | |
| CN106068646A (en) | Degree of depth drawing generating method, device and non-transience computer-readable medium | |
| CN109724537B (en) | Binocular three-dimensional imaging method and system | |
| CN111709985A (en) | Underwater target ranging method based on binocular vision | |
| CN116188558B (en) | Stereo photogrammetry method based on binocular vision | |
| CN111815693B (en) | Depth image generation method and device | |
| CN114821497A (en) | Method, device and equipment for determining position of target object and storage medium | |
| CN106323190A (en) | Depth measurement range-customizable depth measurement method and system for obtaining depth image | |
| CN110487254B (en) | Rapid underwater target size measuring method for ROV | |
| CN112734862A (en) | Depth image processing method and device, computer readable medium and equipment | |
| CN113808070B (en) | Binocular digital speckle image related parallax measurement method | |
| CN108242044A (en) | Image processing apparatus and image processing method | |
| JP2018081378A (en) | Image processing apparatus, imaging device, image processing method, and image processing program | |
| KR102107465B1 (en) | System and method for generating epipolar images by using direction cosine | |
| CN110068308B (en) | Distance measurement method and distance measurement system based on multi-view camera | |
| CN112215879A (en) | Depth extraction method of light field polar plane image | |
| CN111508071B (en) | Binocular camera-based 3D modeling method and shooting terminal | |
| CN114299129B (en) | Depth recovery method, electronic device, and computer-readable storage medium | |
| KATO et al. | Deformation Measurement Using Digital Image Correlation: Interpolation Method for Sub-Pixel Accuracy (Optical Method 3) | |
| Pedersini et al. | Calibration and self-calibration of multi-ocular camera systems | |
| CN113379663B (en) | Space positioning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |