CN105374039A - Monocular image depth information estimation method based on contour acuity - Google Patents

Abstract

The invention provides a monocular image depth information estimation method based on contour acuity. The method uses edge contour acuity as a fuzzy information estimation characteristic, and through low-level cue information, depth information extraction is carried out. Edge detection is firstly carried out on the image; edge energy and contour acuity are calculated for the edge in the image, the edge energy and the contour acuity serve as outer energy of the contour, inner energy, contour line characteristic energy and contour line distance energy, of the contour is combined to build a contour tracking model, the minimal value of an energy function is solved, and an image contour is searched; then, depth gradient hypothesis serves as a priori assumed gradient model, depth value filling is carried out on an area with different contour lines, and depth distribution is obtained through calculation; and finally, the original image information and the obtained depth image information are used for optimization on the obtained depth image, and a final disparity map is obtained. As is proved by an experimental result, the depth estimation algorithm is simple, and the depth image for the monocular image can be quickly and accurately estimated.

Description

Monocular image depth information estimation method based on contour sharpness

Technical Field

The invention relates to a method for estimating depth information of a monocular image, which can obtain the depth information of an object shown in the monocular image through low-level clues of the image.

Background

Depth information perception is a precondition for generating stereoscopic vision, and the depth information-based practical application plays an important role in the fields of scene re-solution, three-dimensional reconstruction, mode recognition, target tracking and the like. In practical application, a monocular camera or a depth camera can be adopted for extracting the depth information. The monocular camera-based depth information estimation method has certain advantages due to the fact that the method is simple in actual operation and low in hardware cost, and depth information can be directly extracted from existing monocular image materials.

Monocular image depth information estimation extracts two-dimensional and three-dimensional geometric information such as color, shape, coplanarity and the like of a target through a single image, and therefore space three-dimensional information of the target is obtained by using a small amount of known conditions. Most algorithms at present are realized by adopting high-level and middle-level clues of images. For example, the semantic label of the known image is obtained by learning the reference image, and the semantic label is defined for the target image by using the semantic label, so that the relative depth information is obtained. The method comprises the steps of segmenting a target image by utilizing image structure information obtained by training in a real parallax image, such as color, texture, shape and the like, and deriving depth information by applying a Markov random field with distinguishing training. And the depth information can be recovered from the image by directly applying the local information without analyzing the content of the image by using the low-level clue information, and the algorithm is relatively simple.

The fuzzy information can be used as an important characteristic of depth estimation in the process of estimating the depth information of the monocular image. The blurring is mostly caused by out-of-focus or different depth of objects in the imaging area during the imaging process. According to the characteristic, monocular image depth information estimation determines the foreground and the background of an image according to the blurring condition of the image by defocusing the image so as to estimate the depth of a scene. Relative depth extraction is achieved by applying vanishing light and markov random fields to spread the blur across the entire image, as initiated by the blur values at the edge locations. Or the modeling of the defocusing fuzzy process is realized by utilizing the thermal diffusion process, and the scene depth is recovered by utilizing the uneven inverse thermal diffusion equation to estimate the fuzzy amount of the edge position. And the fuzzy diffusion method is complex, low in efficiency and poor in practicability.

Therefore, the method for estimating the monocular image depth information is simplified by utilizing the low-level clues and the new fuzzy information characteristics of the image.

Disclosure of Invention

The invention provides a monocular image depth information estimation method based on contour sharpness.

The purpose of the invention is realized by the following technical scheme:

the monocular image depth information estimation method based on the contour sharpness is characterized in that the gradient amplitude and contour sharpness information are used for reflecting the characteristics of object contours with different depths in an image, and when depth estimation is carried out, the gradient amplitude of edges is not only considered, and meanwhile, the spatial information of the edges is added, so that the fuzzy change trend of the edges of the objects is more fully reflected. The method comprises the following steps:

(1) performing edge detection on an input image to obtain edge points P ═ P of an object in the image₁,p₂,…p_n}；

(2) According to the prior depth gradient model, defining an initial contour line, and defining a group of contour lines which are parallel to each other and have equal intervals as the initial contour line V ═ { V ═ V }₀,v₁…v_m}; wherein v(s) ═ x(s), y(s)]X(s), y(s) are coordinates of the point s in the initial contour line;

the total energy of the contour is defined as:

E_total＝w₁E_edge+w₂E_sharp+w₃E_s+w₄E_d

wherein E_edgeAs a function of the contour edge energy, E_sharpEnergy function of contour sharpness, E_sAs a function of the characteristic energy of the contour line, E_dAs a function of contour distance energy, w₁、w₂、w₃、w₄The parameters for each function weight control can be assigned according to specific image characteristics.

Contour edge energy function E_edgeGradient of representing contourAmplitude, defined as the magnitude of the gradient amplitude of the image I (x, y) along the gradient direction, is expressed as:

$E_{edge}=\exp (-\frac{{\left|g\left(I\right(x,y\left)\right)\right|}^2}{a})$

wherein, $g\left(I\right(x,y\left)\right)=\sqrt{{\left({\∂}_{x}I\right(x,y\left)\right)}^2+{\left({\∂}_{y}I\right(x,y\left)\right)}^2}$ solving gradient amplitude of I (x, y);

contour sharpness energy function E_sharpThe fuzzy degree of the contour is represented and obtained by solving the sharpness of the gradient contour. Is defined as:

$E_{sharp}=\exp (-\frac{\mathrm{\σ}{\left(p\right(q_0\left)\right)}^2}{b});$

wherein the gradient profile sharpness σ (p (q)₀) Is the root mean square of the variance of the gradient profile variables. Where the gradient profile is an edge pixel q in the image₀(x₀,y₀) As a starting point, tracing to the boundary of the edge along the gradient direction until the gradient amplitude of the edge does not change any more, and obtaining a one-dimensional path p (q)₀) The formed gradient amplitude curve. The sharpness of the gradient profile is defined as

$\mathrm{\σ}\left(p\right(q_0\left)\right)=\sqrt{\underset{x\∈p\left(q_0\right)}{\Σ}\frac{g\left(q\right)}{G\left(q_0\right)}{d_c}^2(q,q_0)};G\left(q_0\right)=\underset{s\∈p\left(q_0\right)}{\Σ}g\left(s\right)$

Wherein d is_c(q,q₀) For the points q and q of the gradient profile₀Length of the curve in between, G (q) is the gradient amplitude at point q, G (q)₀) Is the sum of gradient amplitudes of all points in the gradient profile, s is any point in the gradient profile, and b is weight controlAnd (4) parameters.

Contour line characteristic energy function E_sSmoothness for constraining the contour;

contour distance energy function E_dThe method is used for controlling the contour tracing curve not to exceed the search area;

(3) updating the position of each contour point from the bottom to the top of each initial contour line in the image from the starting point on the left side of the image, and calculating contour tracking energy for each contour point according to the total energy definition of the contour lines in the step (2);

(4) solving the minimum value of the energy function; searching pixel points in a pixel column adjacent to the current contour point, and searching for an edge point P which meets the definition of an energy function, wherein the edge point P is equal to { P }₁,p₂,…p_nSelecting the position of the pixel point with the minimum energy as a new contour point; repeatedly searching the minimum value from the left side to the right side of the image to obtain a final contour search result, namely obtaining a target contour line V;

(5) and filling depth values in the areas with different contour lines by taking the depth gradient hypothesis as a priori hypothesis gradient model, and calculating to obtain the depth distribution.

Contour line V ═ V in contour line set₀,v₁,…,v_mAnd the corresponding allocated Depth value Depth is:

$Depth\left(v_i\right)=255\×\frac{m-i}{m},i=0...m$

(6) and optimizing the obtained depth image by using the gray information of the original image and the obtained depth image information:

${{\mathrm{Depth}}_f}^{bf}\left(x_i\right)=\frac{1}{W\left(x_i\right)}\underset{x_j\∈\mathrm{\Ω}\left(x_i\right)}{\Σ}\exp \left(G_{{\mathrm{\σ}}_s}\right(\left|\right|x_i-x_j\left|\right|)G_{{\mathrm{\σ}}_r}\left(I\right(x_i)-I(x_j\left)\right))Depth\left(x_j\right)$

$W\left(x_i\right)=\underset{q\∈S}{\Σ}\exp \left(G_{{\mathrm{\σ}}_s}\right(\left|\right|x_i-x_j\left|\right|\left)G_{{\mathrm{\σ}}_r}\right(I\left(x_i\right)-I\left(x_j\right)))$

wherein Depth (x)_i) To input the depth image, Ω (x)_i) Is a pixel x_iA central neighborhood, I (x)_i) Is a pixel x_iLuminance I component, x of_jIs a pixel x_iIn the neighborhood of Ω (x)_i) Of the neighborhood of pixels, W (x)_i) Is a normalization factor of the filter parameters; | x_i-x_j| | is the spatial Euclidean distance of two pixels; i (x)_i)-I(x_j) Representing the similarity of luminance of two pixelsPixel x_iAnd x_jRespectively have a spatial coordinate of (x)_ix,x_iy) And (x)_jx,x_jy). Spatial weight coefficientAnd chroma weight coefficientIs defined as:

$G_{{\mathrm{\σ}}_s}\left(\right||x_i-x_j|\left|\right)=\frac{{\[\sqrt{{(x_{jx}-x_{ix})}^2+{(x_{jy}-x_{iy})}^2}\]}^2}{2{{\mathrm{\σ}}_s}^2}$

$G_{{\mathrm{\σ}}_r}\left(I\right(x_i)-I(x_j\left)\right)=\frac{{|I\left(x_j\right)-I\left(x_i\right)|}^2}{2{{\mathrm{\σ}}_r}^2};$

wherein sigma_sIs the variance, σ, of the spatial weight_rIs the variance of the chroma weights.

(7) A disparity map obtained by estimating depth information of the input monocular image is obtained.

The method has the advantage that the monocular image depth information estimation method based on the outline sharpness is provided. The traditional monocular image depth information estimation method needs to perform learning training, image understanding and other steps by utilizing high-level and middle-level clues of images, and the algorithm is complex. The method utilizes low-level clues of the images and has simple calculation. Different from the traditional method for distinguishing the object depth by using the fuzzy information, the method disclosed by the invention effectively distinguishes the object contour by using the contour sharpness information when calculating the fuzzy information, so that the object contour with different depths is obtained, the steps of fuzzy diffusion and the like are avoided, and the experimentality of the method is improved. The method reflects the characteristics of object contours with different depths in an image by utilizing the information of the gradient amplitude and the contour sharpness, not only considers the gradient amplitude of the edge when carrying out depth estimation, but also adds the spatial information of the edge, and more fully reflects the fuzzy change trend of the edge of the object.

Drawings

FIG. 1 is a flow chart of the method.

Fig. 2 shows the definition of contour sharpness.

FIG. 3 is a diagram illustrating the relative relationship of depth estimation assigned by contour lines.

Detailed Description

The following detailed description of the embodiments of the invention is provided in connection with the accompanying drawings.

(1) For an input image, edge detection is carried out by adopting a Canny edge detection algorithm, and the edge point P of an object in the image is obtained as { P ═ P }₁,p₂,…p_n}；

(2) According to the prior depth gradient model, defining an initial contour line, and defining a group of contour lines which are parallel to each other and have equal intervals as the initial contour line V ═ { V ═ V }₀,v₁,...,v_m}. Wherein v(s) ═ x(s), y(s)]X(s), y(s) are coordinates of the point s in the initial contour line;

the total energy of the contour line is defined as

E_total＝w₁E_edge+w₂E_sharp+w₃E_s+w₄E_d

Wherein E_edgeAs a function of the contour edge energy, E_sharpEnergy function of contour sharpness, E_sAs a function of the characteristic energy of the contour line, E_dAs a function of contour distance energy, w₁、w₂、w₃、w₄For each function weight control parameter, can be set to w for the generic image₁＝0.25、w₂＝0.5、w₃＝0.125、w₄＝0.125。

(3) Contour edge energy function E_edgeThe magnitude of the gradient amplitude of the image I (x, y) along the gradient direction is defined as

$E_{edge}=\exp (-\frac{{\left|g\left(I\right(x,y\left)\right)\right|}^2}{a})$

Wherein, $g\left(I\right(x,y\left)\right)=\sqrt{{\left({\∂}_{x}I\right(x,y\left)\right)}^2+{\left({\∂}_{y}I\right(x,y\left)\right)}^2}$ solving gradient amplitude of I (x, y);

wherein the parameter a is a weight control parameter.

(4) In the method, the gradient contour sharpness of the edge is used for representing the blurring degree of the contour, so that a contour sharpness energy function E obtained by solving the gradient contour sharpness is defined_sharpAs a characteristic value of the degree of contour blur. As shown in fig. 2, with an edge pixel q in the figure₀(x₀,y₀) As a starting point, tracing to both sides of the edge along the gradient direction until the gradient magnitude of the edge no longer changes, thus obtaining a path p (q)₀). Along a one-dimensional path p (q)₀) The resulting gradient magnitude profile is referred to as the gradient profile. Using variance of gradient profile variablesRoot mean square defines the sharpness of the profile, expressed as:

$\mathrm{\σ}\left(p\right(q_0\left)\right)=\sqrt{\underset{x\∈p\left(q_0\right)}{\Σ}\frac{g\left(q\right)}{G\left(q_0\right)}{d_c}^2(q,q_0)}$

$G\left(q_0\right)=\underset{s\∈p\left(q_0\right)}{\Σ}g\left(s\right)$

where d is_c(q,q₀) For the points q and q of the gradient profile₀Length of the curve in between, G (q) is the gradient amplitude at point q, G (q)₀) Is the sum of the gradient amplitudes of all points in the gradient profile, and s is any point in the gradient profile.

Sharpness energy function E_sharpIs defined as:

$E_{sharp}=\exp (-\frac{\mathrm{\σ}{\left(p\right(q_0\left)\right)}^2}{b})$

wherein the parameter b is a weight control parameter.

(5) Contour line characteristic energy function E_sAnd controlling contour tracking as a smooth term constraint to ensure that a curve generated by tracking is smooth and prevent the curve from falling into local extreme values during solving.

Defining outline points in the outline as N ═ N₀,n₁,...,n_nIn which n is₀For the starting point of the contour, the characteristic energy function of the contour is defined as

$E_s\left(n_i\right)=\frac{d_s(n_i,n_{i-1})}{c}=\frac{\left|n_{iy}-n_{i-1y}\right|}{c},i=0,1,...n$

Wherein d is_s(n_i,n_i-1) Representing point n_iAnd point n_i-1The length of the curve between the two parameters, and the parameter c is a weight control parameter.

(6) Contour distance energy function E_dThe elastic constraint term is used for constraining the distance of each contour point in the contour line tracking process, so that the contour line tracking curve does not exceed the search area, and the contour lines are ensured not to be crossed.

$E_d\left(n_i\right)=\frac{d_e(n_i,n_0)}{d}=\frac{\left|n_{iy}-n_{0y}\right|}{d},i=0,1,...n$

Wherein d is_e(n_i,n₀) Representing parametric representation points n_iAnd point n₀D is a weight control parameter.

(7) Updating the position of each contour point from the bottom to the top of each initial contour line in the image from the starting point on the left side of the image, and calculating contour tracking energy for each contour point according to the total energy calculation method of the contour lines in the steps (2-6);

(8) the minimum of the energy function is solved. Searching pixel points in a pixel column adjacent to the current contour point, and searching for an edge point P which meets the definition of an energy function, wherein the edge point P is equal to { P }₁,p₂,…p_nThe point with the smallest energy value in the pixel is selected as the new pixel point position with the smallest energyThe contour points of (1); and repeatedly searching the minimum value from the left side to the right side of the image to obtain a final contour search result, namely obtaining the target contour line V.

(9) And filling depth values of areas with different contour lines according to the graph 3 by taking a depth gradient hypothesis which is gradually deepened from bottom to top as a priori hypothesis gradient model, and calculating to obtain depth distribution.

Contour line V ═ V in contour line set₀,v₁,...,v_mAnd h, correspondingly allocating the depth value as.

$Depth\left(v_i\right)=255\×\frac{m-i}{m},i=0...m$

(10) And optimizing the obtained depth image by using the original image information and the obtained depth image information:

${{\mathrm{Depth}}_f}^{bf}\left(x_i\right)=\frac{1}{W\left(x_i\right)}\underset{x_j\∈\mathrm{\Ω}\left(x_i\right)}{\Σ}\exp \left(G_{{\mathrm{\σ}}_s}\right(\left|\right|x_i-x_j\left|\right|\left)G_{{\mathrm{\σ}}_r}\right(I\left(x_i\right)-I\left(x_j\right)\left)\right)Depth\left(x_j\right)$

$W\left(x_i\right)=\underset{q\∈S}{\Σ}\exp \left(G_{{\mathrm{\σ}}_s}\right(\left|\right|x_i-x_j\left|\right|\left)G_{{\mathrm{\σ}}_r}\right(I\left(x_i\right)-I\left(x_j\right)\left)\right)$

wherein Depth (x)_i) To input the depth image, Ω (x)_i) Is a pixel x_iA central neighborhood, I (x)_i) Is a pixel x_iLuminance I component, pixel x_jIs a pixel x_iIn the neighborhood of Ω (x)_i) Of the neighborhood of pixels, W (x)_i) Is a normalization factor of the filter parameters; | x_i-x_j| | is the spatial Euclidean distance of two pixels; i (x)_i)-I(x_j) Representing the similarity of the luminance of two pixels, pixel x_iAnd x_jRespectively have a spatial coordinate of (x)_ix,x_iy) And (x)_jx,x_jy). Spatial weight coefficientAnd chroma weight coefficientIs defined as:

$G_{{\mathrm{\σ}}_s}\left(\right||x_i-x_j|\left|\right)=\frac{{\[\sqrt{{(x_{jx}-x_{ix})}^2+{(x_{jy}-x_{iy})}^2}\]}^2}{2{{\mathrm{\σ}}_s}^2}$

$G_{{\mathrm{\σ}}_r}\left(I\right(x_i)-I(x_j\left)\right)=\frac{{|I\left(x_j\right)-I\left(x_i\right)|}^2}{2{{\mathrm{\σ}}_r}^2}$

wherein sigma_sIs the variance, σ, of the spatial weight_rIs the variance of the chroma weights.

(11) A disparity map obtained by estimating depth information of the input monocular image is obtained.

Claims (3)

1. The monocular image depth information estimation method based on the contour sharpness is characterized by comprising the following steps of:

(1) performing edge detection on an input image to obtain edge points P ═ P of an object in the image₁,p₂,…p_n}；

(2) According to the prior depth gradient model, defining an initial contour line, and defining a group of contour lines which are parallel to each other and have equal intervals as the initial contour line V ═ { V ═ V }₀,v₁...v_m}; wherein v(s) ═ x(s), y(s)]X(s), y(s) are the initial contour linesThe coordinates of point s in (1);

the total energy of the contour line is defined as

E_total＝w₁E_edge+w₂E_sharp+w₃E_s+w₄E_d

Wherein E_edgeRepresenting the gradient amplitude of the contour as a function of the contour edge energy;

E_sharpa contour sharpness energy function representing a degree of blurring of the contour;

E_sis a contour characteristic energy function used for restraining the smoothness of the contour;

E_dthe contour line distance energy function is used for controlling the contour tracing curve not to exceed the search area;

w₁、w₂、w₃、w₄controlling parameters for each function weight;

(3) updating the position of each contour point from the bottom to the top of each initial contour line in the image from the starting point on the left side of the image, and calculating contour tracking energy for each contour point according to the total energy definition of the contour lines in the step (2);

(4) solving the minimum value of the energy function; searching pixel points in a pixel column adjacent to the current contour point, and searching for an edge point P which meets the definition of an energy function, wherein the edge point P is equal to { P }₁,p₂,…p_nSelecting the position of the pixel point with the minimum energy as a new contour point; repeatedly searching the minimum value from the left side to the right side of the image to obtain a final contour search result, namely obtaining a target contour line V;

(5) filling depth values of areas with different contour lines by taking a depth gradient hypothesis as a priori hypothesis gradient model, and calculating to obtain depth distribution;

contour line V ═ V in contour line set₀,v₁,...,v_mFor an assigned Depth value Depth corresponding to i-0 … m, the Depth value Depth is:

$Depth\left(v_i\right)=255\×\frac{m-i}{m}$

(6) and optimizing the obtained depth image by using the gray information of the original image and the obtained depth image information:

${{\mathrm{Depth}}_f}^{bf}\left(x_i\right)=\frac{1}{W\left(x_i\right)}\underset{x_j\∈\mathrm{\Ω}\left(x_i\right)}{\Σ}\exp \left(G_{{\mathrm{\σ}}_s}\right(\left|\right|x_i-x_j\left|\right|\left)G_{{\mathrm{\σ}}_r}\right(I\left(x_i\right)-I\left(x_j\right)\left)\right)Depth\left(x_j\right)$

$W\left(x_i\right)=\underset{q\∈S}{\Σ}\exp \left(G_{{\mathrm{\σ}}_s}\right(\left|\right|x_i-x_j\left|\right|\left)G_{{\mathrm{\σ}}_r}\right(I\left(x_i\right)-I\left(x_j\right)\left)\right)$

wherein Depth (x)_i) To input the depth image, Ω (x)_i) Is a pixel x_iA central neighborhood, I (x)_i) Is a pixel x_iLuminance I component, x of_jIs a pixel x_iIn the neighborhood of Ω (x)_i) Of the neighborhood of pixels, W (x)_i) Is a normalization factor of the filter parameters; | x_i-x_j| | is the spatial Euclidean distance of two pixels; i (x)_i)-I(x_j) Representing the similarity of the luminance of two pixels, pixel x_iAnd x_jRespectively have a spatial coordinate of (x)_ix,x_iy) And (x)_jx,x_jy) (ii) a Spatial weight coefficientAnd chroma weight coefficientIs defined as:

$G_{{\mathrm{\σ}}_s}\left(\right||x_i-x_j|\left|\right)=\frac{{\[\sqrt{{(x_{jx}-x_{ix})}^2+{(x_{jy}-x_{iy})}^2}\]}^2}{2{{\mathrm{\σ}}_s}^2}$

$G_{{\mathrm{\σ}}_r}\left(I\right(x_i)-I(x_j\left)\right)=\frac{|I\left(x_j\right)-I\left(x_i\right){|}^2}{2{{\mathrm{\σ}}_r}^2};$

wherein sigma_sIs the variance, σ, of the spatial weight_rIs the variance of the chroma weights.

(7) A disparity map obtained by estimating depth information of the input monocular image is obtained.

2. The method of claim 1, wherein the contour edge energy function E is_edgeThe magnitude of the gradient amplitude of the image I (x, y) along the gradient direction is defined as:

$E_{edge}=\exp (-\frac{\left|g\left(I\right(x,y)\right){|}^2}{a})$

wherein, $g\left(I\right(x,y\left)\right)=\sqrt{{\left({\∂}_{x}I\right(x,y\left)\right)}^2+{\left({\∂}_{y}I\right(x,y\left)\right)}^2}$ solving gradient amplitude of I (x, y);

wherein the parameter a is a weight control parameter.

3. The method of claim 1, wherein the contour sharpness energy function E is_sharpSolved for gradient profile sharpness, defined as:

$E_{sharp}=\exp (-\frac{\mathrm{\σ}{\left(p\right(q_0\left)\right)}^2}{b});$

wherein the gradient profile sharpness σ (p (q)₀) Is the root mean square of the variance of the gradient profile variables. Where the gradient profile is an edge pixel q in the image₀(x₀,y₀) As a starting point, tracing to the boundary of the edge along the gradient direction until the gradient amplitude of the edge does not change any more, and obtaining a one-dimensional path p (q)₀) The formed gradient amplitude curve; the sharpness of the gradient profile is defined as

$\mathrm{\σ}\left(p\right(q_0\left)\right)=\sqrt{\underset{x\∈p\left(q_0\right)}{\Σ}\frac{g\left(q\right)}{G\left(q_0\right)}{d_c}^2(q,q_0)};G\left(q_0\right)=\underset{s\∈p\left(q_0\right)}{\Σ}g\left(s\right)$

Wherein d is_c(q,q₀) For the points q and q of the gradient profile₀Length of the curve in between, G (q) is the gradient amplitude at point q, G (q)₀) Is the sum of the gradient amplitudes of all points in the gradient profile, s is any point in the gradient profile, and b is a weight control parameter.