CN115115689A - Depth estimation method of multiband spectrum - Google Patents

Abstract

The present invention provides a multi-band spectrum depth estimation method, comprising the following steps: Step 1: Use a camera to capture original images P ₀ (x, y), P ₁ (x, y), ···, P of different wavelength bands _i (x,y), where i represents different bands; Step 2: Take the original image of a single band as the input, take h(x,y) as the convolution kernel, according to the image convolution formula P'(x,y)=P (x,y)*h(x,y), get the blurred image P'(x,y) after convolution; Step 3: Calculate the original image P(x,y) and the blurred image P'(x,y respectively ), calculate the original edge gradient P _{E_G} (x, y) and the blurred edge gradient P' _{E_G} (x, y) according to the obtained edge area; Step 4: Divide the original image edge P _{E_G} (x, y) by the blurred edge Gradient P' _{E_G} (x, y), get the blur ratio σ(x, y) of the original image and the edge image; Step 5: According to the obtained blur ratio σ(x, y), according to the lens imaging formula

Calculate the sparse depth; Step 6: Repeat the above steps 2-5 for the subsequent original images of the bands, and combine the focus position of each band to realize the real structure position constraint in the image.

Description

A Depth Estimation Method for Multiband Spectra

technical field

The invention relates to a depth estimation method for multi-band spectrum.

Background technique

Image-based depth estimation refers to restoring the 3D topography information of the measured sample surface from a single or multiple 2D map. It has important research significance and application value, and is an important research problem in the field of computer vision and graphics. Traditional imaging equipment uses the image information under natural light illumination to restore. This method is limited by the non-deterministic constraint relationship in the image under the monocular camera, and it is difficult to restore the depth information of the image with a single image. In the multi-camera imaging system, although the constraint relationship between images is increased, the redundant information of the restored image increases, and the image information obtained from different angles has a certain collection dead angle, so it is difficult to restore the complete depth information on the image. Although the deep learning restoration method can solve the depth of a monocular single frame image to a certain extent through training, it requires a large amount of data sets and lacks a certain generalization ability. Therefore, it is necessary to find a method that can solve the above problems.

SUMMARY OF THE INVENTION

The main technical problem to be solved by the present invention is to provide a multi-band spectrum depth estimation method.

In order to solve the above-mentioned technical problems, the present invention provides a depth estimation method for multi-band spectrum, comprising the following steps:

Step 1: Use the camera to capture original images of different bands P ₀ (x,y), P ₁ (x,y),..., P _i (x,y), where i represents different bands; i≥2;

Step 2: Take the original image of a single band as the input, take h(x,y) as the convolution kernel, and according to the image convolution formula P'(x,y)=P(x,y)*h(x,y), Obtain the convoluted blurred image P'(x,y);

Step 3: Calculate the edges of the original image P(x,y) and the blurred image P'(x,y) respectively, and calculate the original edge gradient P _{E_G} (x, y) and the blurred edge gradient P' _{E_G} (x according to the obtained edge area ,y);

Step 4: Divide the original image edge P _{E_G} (x, y) by the blurred edge gradient P' _{E_G} (x, y) to obtain the blur ratio σ(x, y) of the original image and the edge image;

计算稀疏深度，其中k为常数，D为光学系统通光孔径；s为像面与光学系统距离；d_f为物方焦距；d为物体深度。Step 5: According to the obtained blur ratio σ(x,y) according to the lens imaging formula

Calculate the sparse depth, where k is a constant, D is the clear aperture of the optical system; s is the distance between the image plane and the optical system; d _f is the focal length of the object; d is the depth of the object.

Step 6: Repeat the above steps 2-5 for the subsequent original images of the bands, and finally combine the focus position of each band to realize the real structure position constraint in the image.

In a preferred embodiment: in the step 1, the original images of different wavebands are captured by adding filters of different wavebands to the monocular camera, or using a multispectral camera capable of collecting images of multiple wavebands in one imaging , or use a multi-eye camera to configure filters to obtain images of different spectral bands.

In a preferred embodiment: the convolution kernel in the step 2 includes: any approximation of the point spread function of the imaging system.

In a preferred embodiment: the point spread function is a Gaussian kernel, a Cauchy kernel or a Gauss-Cauchy kernel.

In a preferred embodiment: the edge gradient in step 3 includes:

Image edge gradient extraction can first use image edge detection operators such as canny to extract image edges, and then use edge gradient operators such as sobel, robert or threshold extraction methods to calculate image edge gradient changes; or directly use edge gradient operators or threshold extraction. The edge method calculates the gradient change of the image edge.

In a preferred embodiment: the step 4 includes the following sub-steps:

(1) Assuming that the absolutely clear image is P ₀ (x, y), the original image P (x, y) collected can be regarded as the absolutely clear image P ₀ (x, y). Product h(x,y,σ), and the blurred image is the original image P(x,y) to do a convolution operation with a known standard deviation value h(x,y,σ ₁ ), that is, the blurred image P' (x,y) can be regarded as an absolutely clear image and do two convolution operations P'(x,y)=P(x,y)*h(x,y,σ)*h(x,y,σ ₁ ) ;

(2) The blur ratio σ(x, y) of the original image and the edge image can be expressed as:

finally

In a preferred embodiment: the step 6 includes: using each spectral band to have its own focus position, by obtaining the absolute distance relationship of the focal plane of the image relative to each individual band, combined with the obtained relative position of the band itself. The absolute relationship between the depths of the structures in the image can be obtained.

Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

According to a multi-band spectrum depth estimation method involved in the present invention, firstly, because the multi-band spectrum is used to collect image information, multiple spectral images with different depth information can be collected in a monocular scene, and the characteristics of a single image can be solved. The problem of insufficient constraints; secondly, the relative depth relationship on each spectral image is obtained by using the image reblurring theory, and the relative depth solution between the structural information in the image that requires multi-view localization is solved; then, the original image gradient and The ratio method of blurred image gradients eliminates the multiplicative error caused by imaging equipment in the process of image acquisition to a certain extent; finally, combined with the focal depth relationship of different spectral bands and the relative depth of single-band image solution, the absolute image is realized. Deep restoration.

Therefore, the multi-band spectrum depth estimation method involved in the present invention is less affected by the external environment during the implementation process, and reduces the interference factors caused by imaging equipment and other multiplicative errors. Secondly, in the implementation steps of depth estimation, the basic operations between images are used, which greatly reduces the time of depth information recovery and the time complexity of the algorithm. In addition, the estimation model is based on the most basic lens imaging. Starting from the principle, it satisfies the basic models of most of the current measurement fields, so that the monocular depth estimation method proposed by the present invention has certain universality. .

Description of drawings

FIG. 1 is a schematic diagram of out-of-focus blur imaging in a preferred embodiment of the present invention;

FIG. 2 is a flowchart of a preferred embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

Referring to FIG. 1 to FIG. 2 , this embodiment provides a method for depth estimation of multi-band spectrum, including the following steps:

Step 1: Use the camera to capture original images of different bands P ₀ (x,y), P ₁ (x,y),..., P _i (x,y), where i represents different bands; i≥2;

Step 2: Take the original image of a single band as the input, take h(x,y) as the convolution kernel, and according to the image convolution formula P'(x,y)=P(x,y)*h(x,y), Obtain the convoluted blurred image P'(x,y);

Step 3: Calculate the edges of the original image P(x,y) and the blurred image P'(x,y) respectively, and calculate the original edge gradient P _{E_G} (x, y) and the blurred edge gradient P' _{E_G} (x according to the obtained edge area ,y);

Step 4: Divide the original image edge P _{E_G} (x, y) by the blurred edge gradient P' _{E_G} (x, y) to obtain the blur ratio σ(x, y) of the original image and the edge image;

计算稀疏深度，其中k为常数，D为光学系统通光孔径；s为像面与光学系统距离；d_f为物方焦距；d为物体深度。Step 5: According to the obtained blur ratio σ(x,y) according to the lens imaging formula

Calculate the sparse depth, where k is a constant, D is the clear aperture of the optical system; s is the distance between the image plane and the optical system; d _f is the focal length of the object; d is the depth of the object.

Step 6: Repeat the above steps 2-5 for the subsequent original images of the bands, and finally combine the focus position of each band to realize the real structure position constraint in the image.

The original images of different wavebands in the step 1 are obtained by adding filters of different wavebands to the monocular camera, or using a multispectral camera capable of collecting images of multiple wavebands in one imaging, or using a multispectral camera to configure filters. The light sheet obtains pictures of different spectral bands.

The convolution kernel in the step 2 includes: arbitrarily approximate the point spread function of the imaging system. The point spread function is a Gaussian kernel, a Cauchy kernel, a Gauss-Cauchy kernel or other kernel functions that approximate the point spread function of an imaging system.

The edge gradient in step 3 includes:

Image edge gradient extraction can first use canny and other image edge detection operators to extract image edges, and then use sobel, robert and other edge gradient operators or threshold extraction methods to calculate image edge gradient changes; or directly use edge gradient operators or threshold extraction. The edge method calculates the gradient change of the image edge.

The step 4 includes the following sub-steps:

(1) Assuming that the absolutely clear image is P ₀ (x, y), the original image P (x, y) collected can be regarded as the absolutely clear image P ₀ (x, y). Product h(x,y,σ), and the blurred image performs a convolution operation with a known standard deviation value h(x,y,σ ₁ ) for the original image P(x,y), that is, the blurred image P' (x,y) can be regarded as an absolutely clear image and do two convolution operations P'(x,y)=P(x,y)*h(x,y,σ)*h(x,y,σ ₁ ) ;

(2) The blur ratio σ(x, y) of the original image and the edge image can be expressed as:

finally

The step 6 includes: using each spectral band to have its own focus position, by obtaining the absolute distance relationship of the focal plane of the image relative to each individual band, and combining the obtained relative position relationship of the band itself, the image can be obtained. Depth absolute relationship between structures.

The above description is only a preferred embodiment of the present invention, but the design concept of the present invention is not limited to this. Insubstantial changes are acts that infringe the protection scope of the present invention.

Claims (7)

1. A method of depth estimation of a multi-band spectrum, comprising the steps of:

step 1: shooting original images P of different wave bands by a camera ₀ (x,y)、P ₁ (x,y)、···、P _i (x, y), wherein i represents different bands; i is more than or equal to 2;

step 2: taking a single-waveband original image as input, taking h (x, y) as a convolution kernel, and obtaining a convolved blurred image P '(x, y) according to an image convolution formula P' (x, y) × P (x, y) × h (x, y);

and step 3: the edges of the original image P (x, y) and the blurred image P' (x, y) are calculated respectively according to the edgeComputing a raw edge gradient P to an edge region _{E_G} (x, y) and blurred edge gradient P' _{E_G} (x,y)；

And 4, step 4: edge P of original image _{E_G} (x, y) divided by blurred edge gradient P' _{E_G} (x, y) obtaining the fuzzy ratio sigma (x, y) of the original image and the edge image;

and 5: according to the obtained fuzzy ratio sigma (x, y) and lens imaging formula

Calculating the sparse depth, wherein k is a constant and D is the clear aperture of the optical system; s is the distance between the image plane and the optical system; d _f Is the object space focal length; d is the object depth.

Step 6: and repeating the steps 2-5 on the subsequent wave band original images respectively, and finally combining the focusing position of each wave band to realize the real structure position constraint in the images.

2. The method of claim 1 wherein the method of estimating the depth of a multiband spectrum comprises: the original images shot in the step 1 are obtained by additionally arranging optical filters with different wave bands on a monocular camera, or by adopting a multispectral camera which can collect images with a plurality of wave bands through once imaging, or by using the optical filters configured by the monocular camera to obtain images with different spectral wave bands.

3. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the convolution kernel in step 2 includes: the point spread function of the imaging system is arbitrarily approximated.

4. The method of claim 3 wherein the method of estimating the depth of a multiband spectrum comprises: the point spread function is a gaussian kernel, cauchy kernel, or gaussian-cauchy kernel.

5. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the edge gradient in step 3 comprises:

the image edge gradient extraction can be realized by firstly adopting an image edge detection operator such as canny and the like to extract an image edge, and then adopting an edge gradient operator such as sobel, robert and the like or a threshold value to extract an edge to calculate the change of the image edge gradient; or directly utilizing an edge gradient operator or a threshold value to extract an edge to calculate the gradient change of the image edge.

6. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the step 4 comprises the following substeps:

(1) suppose an absolute sharp image is P ₀ (x, y) the acquired original image P (x, y) can be regarded as an absolutely sharp image P ₀ (x, y) is convolved with h (x, y, sigma) with unknown standard deviation but very small value, and the blurred image is convolved with h (x, y, sigma) with known standard deviation value being very large for the original image P (x, y) ₁ ) The convolution operation of (a), that is, the blurred image P '(x, y) can be regarded as an absolute sharp image, and two times of convolution operations P' (x, y) ═ P (x, y) × (x, y, σ) are performed on the absolute sharp image ₁ )；

(2) The blur ratio σ (x, y) of the original image and the edge image can be expressed as:

finally, the product is processed

7. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the step 6 comprises the following steps: by utilizing the fact that each spectral waveband has an independent focusing position, the absolute distance relation of the image relative to the focusing surface of each independent waveband is obtained, and the obtained relative position relation of the waveband is combined, so that the depth absolute relation among structures in the image can be obtained.

Images