CN115115689A - Depth estimation method of multiband spectrum - Google Patents
Depth estimation method of multiband spectrum Download PDFInfo
- Publication number
- CN115115689A CN115115689A CN202210640198.4A CN202210640198A CN115115689A CN 115115689 A CN115115689 A CN 115115689A CN 202210640198 A CN202210640198 A CN 202210640198A CN 115115689 A CN115115689 A CN 115115689A
- Authority
- CN
- China
- Prior art keywords
- image
- edge
- original
- depth
- blurred
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000001228 spectrum Methods 0.000 title claims abstract description 18
- 238000003384 imaging method Methods 0.000 claims abstract description 17
- 230000003287 optical effect Effects 0.000 claims description 12
- 230000003595 spectral effect Effects 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 5
- 238000003708 edge detection Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 6
- 238000011084 recovery Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012876 topography Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10036—Multispectral image; Hyperspectral image
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Measurement Of Optical Distance (AREA)
Abstract
The invention provides a depth estimation method of a multiband spectrum, which comprises the following steps: step 1: shooting original images P of different wave bands by a camera 0 (x,y)、P 1 (x,y)、···、P i (x, y), wherein i represents different bands; step 2: taking a single-waveband original image as input, taking h (x, y) as a convolution kernel, and obtaining a convolved blurred image P '(x, y) according to an image convolution formula P' (x, y) × P (x, y) × h (x, y); and step 3: respectively calculating the edges of the original image P (x, y) and the blurred image P' (x, y), and calculating the original edge gradient P according to the obtained edge regions E_G (x, y) and blurred edge gradient P' E_G (x, y); and 4, step 4: edge P of original image E_G (x, y) divided by blurred edge gradient P' E_G (x, y) obtaining the fuzzy ratio sigma (x, y) of the original image and the edge image; and 5: according to the obtained fuzzy ratio sigma (x, y) and the lens imaging formula
Calculating the sparse depth; step 6: and repeating the steps 2-5 for the subsequent wave band original images respectively, and combining the focusing position of each wave band to realize the real structure position constraint in the images.
Description
Technical Field
The invention relates to a depth estimation method of a multiband spectrum.
Background
The depth estimation based on the image refers to the recovery of the three-dimensional topography information of the surface of a measured sample from a single or multi-view two-dimensional image, the estimated depth image can be applied to the fields of unmanned driving, micro-nano structural part topography detection and the like, has important research significance and application value, and is an important research problem in the fields of computer vision and graphics. The traditional imaging equipment adopts image information under natural light illumination for restoration, and the method is limited by the nondeterministic constraint relation in the image under a monocular camera, so that the depth information of the image is difficult to restore by using a single image. Under the multi-view camera imaging system, although the constraint relation among the images is increased, redundant information of the restored images is increased, and the image information acquired at different angles has certain acquisition dead angles, so that complete depth information on the images is difficult to restore. Although the deep learning restoration method can solve the depth of a monocular single-frame image to a certain extent through training, a large amount of data sets are required, and certain generalization capability is lacked. Therefore, there is a need to find a solution to the above problems.
Disclosure of Invention
The invention aims to provide a depth estimation method of a multiband spectrum.
In order to solve the above technical problem, the present invention provides a method for estimating the depth of a multiband spectrum, comprising the following steps:
step 1: shooting original images P of different wave bands by a camera 0 (x,y)、P 1 (x,y)、···、P i (x, y), wherein i represents different bands; i is more than or equal to 2;
and 2, step: taking a single-waveband original image as input, taking h (x, y) as a convolution kernel, and obtaining a convolved blurred image P '(x, y) according to an image convolution formula P' (x, y) × P (x, y) × h (x, y);
and step 3: respectively calculating the edges of the original image P (x, y) and the blurred image P' (x, y), and calculating the original edge gradient P according to the obtained edge regions E_G (x, y) and blurred edge gradient P' E_G (x,y)；
And 4, step 4: edge P of original image E_G (x, y) divided by blurred edge gradient P' E_G (x, y) obtaining the fuzzy ratio sigma (x, y) of the original image and the edge image;
and 5: according to the obtained fuzzy ratio sigma (x, y) and lens imaging formula
Calculating the sparse depth, wherein k is a constant and D is the clear aperture of the optical system; s is the distance between the image plane and the optical system; d f Is the object space focal length; d is the object depth.
Step 6: and repeating the steps 2-5 on the subsequent wave band original images respectively, and finally combining the focusing position of each wave band to realize the real structure position constraint in the images.
In a preferred embodiment: the original images shot in the step 1 are obtained by additionally arranging optical filters with different wave bands on a monocular camera, or by adopting a multispectral camera which can collect images with a plurality of wave bands through once imaging, or by using the optical filters configured by the monocular camera to obtain images with different spectral wave bands.
In a preferred embodiment: the convolution kernel in step 2 includes: the point spread function of the imaging system is arbitrarily approximated.
In a preferred embodiment: the point spread function is a gaussian kernel, cauchy kernel, or gaussian-cauchy kernel.
In a preferred embodiment: the edge gradient in step 3 comprises:
the image edge gradient extraction can be realized by firstly adopting an image edge detection operator such as canny and the like to extract an image edge and then adopting an edge gradient operator such as sobel, robert and the like or a threshold value to extract an edge to calculate the image edge gradient change; or directly calculating the gradient change of the image edge by using an edge gradient operator or a threshold value edge extraction method.
In a preferred embodiment: the step 4 comprises the following substeps:
(1) suppose an absolute sharp image is P 0 (x, y) the acquired original image P (x, y) can be regarded as an absolutely sharp image P 0 (x, y) is convolved with h (x, y, sigma) with unknown standard deviation but very small value, and the blurred image is convolved with h (x, y, sigma) with known standard deviation value being very large for the original image P (x, y) 1 ) The convolution operation of (a), i.e. the blurred image P '(x, y) can be regarded as an absolute clear image, and two convolution operations P' (x, y) ═ P (x, y) × (x, y, σ) can be performed on the absolute clear image 1 )；
(2) The blur ratio σ (x, y) of the original image and the edge image can be expressed as:
finally, the product is processed
In a preferred embodiment: the step 6 comprises the following steps: by utilizing the fact that each spectral waveband has an independent focusing position, the absolute distance relation of the image relative to the focusing surface of each independent waveband is obtained, and the obtained relative position relation of the waveband is combined, so that the depth absolute relation among structures in the image can be obtained.
Compared with the prior art, the technical scheme of the invention has the following beneficial effects:
according to the depth estimation method of the multiband spectrum, firstly, because the multiband spectrum is adopted to collect image information, a plurality of spectrum images with different depth information can be collected in a monocular scene, and the problem of insufficient characteristic constraint of a single image is solved; secondly, acquiring a relative depth relation on each spectral image by using an image re-blurring theory, and solving the problem that the relative depth between structural information in a multi-view positioning image needs to be solved; then, a ratio method of the original image gradient and the blurred image gradient is adopted, and multiplicative errors caused by imaging equipment in the image acquisition process are eliminated to a certain extent; and finally, combining the focusing depth relation of different spectral wave bands and the relative depth solved by the single wave band image to realize the absolute depth restoration of the image.
Therefore, the depth estimation method of the multiband spectrum is less influenced by the external environment in the implementation process, and interference factors caused by imaging equipment and other multiplicative errors are reduced. Secondly, in the depth estimation implementation step, basic operation among images is adopted, so that the depth information recovery time is greatly reduced, and the time complexity of the algorithm is reduced; in addition, the estimation model meets basic models in most measurement fields at present from the most basic lens imaging principle, so that the monocular depth estimation method provided by the invention has certain universality. .
Drawings
FIG. 1 is a schematic diagram of out-of-focus blur imaging in a preferred embodiment of the present invention;
fig. 2 is a flow chart of a preferred embodiment of the present invention.
Detailed Description
The technical solution of the present invention is further explained with reference to the accompanying drawings and the detailed description.
Referring to fig. 1-2, the present embodiment provides a method for estimating the depth of a multiband spectrum, comprising the following steps:
step 1: shooting original images P of different wave bands by a camera 0 (x,y)、P 1 (x,y)、···、P i (x, y), wherein i represents different wavebands; i is more than or equal to 2;
step 2: taking a single-waveband original image as input, taking h (x, y) as a convolution kernel, and obtaining a convolved blurred image P '(x, y) according to an image convolution formula P' (x, y) × P (x, y) × h (x, y);
and step 3: respectively calculating the edges of the original image P (x, y) and the blurred image P' (x, y), and calculating the original edge according to the obtained edge regionGradient P E_G (x, y) and blurred edge gradient P' E_G (x,y)；
And 4, step 4: edge P of original image E_G (x, y) divided by blurred edge gradient P' E_G (x, y) obtaining the fuzzy ratio sigma (x, y) of the original image and the edge image;
and 5: according to the obtained fuzzy ratio sigma (x, y) and lens imaging formula
Calculating the sparse depth, wherein k is a constant, and D is the clear aperture of the optical system; s is the distance between the image plane and the optical system; d is a radical of f Is the object space focal length; d is the object depth.
Step 6: and repeating the steps 2-5 on the subsequent wave band original images respectively, and finally combining the focusing position of each wave band to realize the real structure position constraint in the images.
The original images shot in the step 1 are obtained by additionally arranging optical filters with different wave bands on a monocular camera, or by adopting a multispectral camera which can collect images with a plurality of wave bands through once imaging, or by using the optical filters configured by the monocular camera to obtain images with different spectral wave bands.
The convolution kernel in step 2 includes: the point spread function of the imaging system is arbitrarily approximated. The point spread function is a kernel function of a gaussian kernel, cauchy kernel, gaussian-cauchy kernel or other approximate imaging system point spread functions.
The edge gradient in step 3 comprises:
the image edge gradient extraction can be realized by firstly adopting an image edge detection operator such as canny and the like to extract an image edge, and then adopting an edge gradient operator such as sobel, robert and the like or a threshold value to extract an edge to calculate the change of the image edge gradient; or directly calculating the gradient change of the image edge by using an edge gradient operator or a threshold value edge extraction method.
The step 4 comprises the following substeps:
(1) suppose an absolute sharp image is P 0 (x, y) the acquired original image P (x, y) can be regarded as an absolutely sharp image P 0 (x, y) one standard deviationThe known but extremely small convolution h (x, y, σ), and the blurred image is the original image P (x, y) with a known large standard deviation value h (x, y, σ) 1 ) The convolution operation of (a), that is, the blurred image P '(x, y) can be regarded as an absolute sharp image, and two times of convolution operations P' (x, y) ═ P (x, y) × (x, y, σ) are performed on the absolute sharp image 1 )；
(2) The blur ratio σ (x, y) of the original image and the edge image can be expressed as:
finally, the product is processed
The step 6 comprises the following steps: by utilizing the fact that each spectral waveband has an independent focusing position, the absolute distance relation of the image relative to the focusing surface of each independent waveband is obtained, and the obtained relative position relation of the waveband is combined, so that the depth absolute relation among structures in the image can be obtained.
The above description is only a preferred embodiment of the present invention, but the design concept of the present invention is not limited thereto, and any person skilled in the art can make insubstantial changes to the present invention within the technical scope of the present invention, and all actions infringing the protection scope of the present invention.
Claims (7)
1. A method of depth estimation of a multi-band spectrum, comprising the steps of:
step 1: shooting original images P of different wave bands by a camera 0 (x,y)、P 1 (x,y)、···、P i (x, y), wherein i represents different bands; i is more than or equal to 2;
step 2: taking a single-waveband original image as input, taking h (x, y) as a convolution kernel, and obtaining a convolved blurred image P '(x, y) according to an image convolution formula P' (x, y) × P (x, y) × h (x, y);
and step 3: the edges of the original image P (x, y) and the blurred image P' (x, y) are calculated respectively according to the edgeComputing a raw edge gradient P to an edge region E_G (x, y) and blurred edge gradient P' E_G (x,y)；
And 4, step 4: edge P of original image E_G (x, y) divided by blurred edge gradient P' E_G (x, y) obtaining the fuzzy ratio sigma (x, y) of the original image and the edge image;
and 5: according to the obtained fuzzy ratio sigma (x, y) and lens imaging formula
Calculating the sparse depth, wherein k is a constant and D is the clear aperture of the optical system; s is the distance between the image plane and the optical system; d f Is the object space focal length; d is the object depth.
Step 6: and repeating the steps 2-5 on the subsequent wave band original images respectively, and finally combining the focusing position of each wave band to realize the real structure position constraint in the images.
2. The method of claim 1 wherein the method of estimating the depth of a multiband spectrum comprises: the original images shot in the step 1 are obtained by additionally arranging optical filters with different wave bands on a monocular camera, or by adopting a multispectral camera which can collect images with a plurality of wave bands through once imaging, or by using the optical filters configured by the monocular camera to obtain images with different spectral wave bands.
3. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the convolution kernel in step 2 includes: the point spread function of the imaging system is arbitrarily approximated.
4. The method of claim 3 wherein the method of estimating the depth of a multiband spectrum comprises: the point spread function is a gaussian kernel, cauchy kernel, or gaussian-cauchy kernel.
5. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the edge gradient in step 3 comprises:
the image edge gradient extraction can be realized by firstly adopting an image edge detection operator such as canny and the like to extract an image edge, and then adopting an edge gradient operator such as sobel, robert and the like or a threshold value to extract an edge to calculate the change of the image edge gradient; or directly utilizing an edge gradient operator or a threshold value to extract an edge to calculate the gradient change of the image edge.
6. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the step 4 comprises the following substeps:
(1) suppose an absolute sharp image is P 0 (x, y) the acquired original image P (x, y) can be regarded as an absolutely sharp image P 0 (x, y) is convolved with h (x, y, sigma) with unknown standard deviation but very small value, and the blurred image is convolved with h (x, y, sigma) with known standard deviation value being very large for the original image P (x, y) 1 ) The convolution operation of (a), that is, the blurred image P '(x, y) can be regarded as an absolute sharp image, and two times of convolution operations P' (x, y) ═ P (x, y) × (x, y, σ) are performed on the absolute sharp image 1 )；
(2) The blur ratio σ (x, y) of the original image and the edge image can be expressed as:
finally, the product is processed
7. The method of claim 1 for depth estimation of a multiband spectrum, characterized in that: the step 6 comprises the following steps: by utilizing the fact that each spectral waveband has an independent focusing position, the absolute distance relation of the image relative to the focusing surface of each independent waveband is obtained, and the obtained relative position relation of the waveband is combined, so that the depth absolute relation among structures in the image can be obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210640198.4A CN115115689B (en) | 2022-06-08 | 2022-06-08 | Depth estimation method of multiband spectrum |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210640198.4A CN115115689B (en) | 2022-06-08 | 2022-06-08 | Depth estimation method of multiband spectrum |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115115689A true CN115115689A (en) | 2022-09-27 |
| CN115115689B CN115115689B (en) | 2024-07-26 |
Family
ID=83326782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210640198.4A Active CN115115689B (en) | 2022-06-08 | 2022-06-08 | Depth estimation method of multiband spectrum |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115115689B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049906A (en) * | 2012-12-07 | 2013-04-17 | 清华大学深圳研究生院 | Image depth extraction method |
| CN103473743A (en) * | 2013-09-12 | 2013-12-25 | 清华大学深圳研究生院 | Method for obtaining image depth information |
| CN107767332A (en) * | 2017-10-23 | 2018-03-06 | 合肥师范学院 | A kind of single image depth recovery method and system in real time |
| US20180146847A1 (en) * | 2015-07-16 | 2018-05-31 | Olympus Corporation | Image processing device, imaging system, image processing method, and computer-readable recording medium |
| CN109584210A (en) * | 2018-10-30 | 2019-04-05 | 南京理工大学 | Multispectral three-dimensional vein imaging system |
| CN110942480A (en) * | 2019-11-19 | 2020-03-31 | 宁波五维检测科技有限公司 | Monocular single-frame multispectral three-dimensional imaging method |
| CN111192238A (en) * | 2019-12-17 | 2020-05-22 | 南京理工大学 | Nondestructive blood vessel three-dimensional measurement method based on self-supervision depth network |
| CN114463206A (en) * | 2022-01-24 | 2022-05-10 | 武汉理工大学 | Multispectral image quality improvement method based on global iterative fusion |
-
2022
- 2022-06-08 CN CN202210640198.4A patent/CN115115689B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049906A (en) * | 2012-12-07 | 2013-04-17 | 清华大学深圳研究生院 | Image depth extraction method |
| CN103473743A (en) * | 2013-09-12 | 2013-12-25 | 清华大学深圳研究生院 | Method for obtaining image depth information |
| US20180146847A1 (en) * | 2015-07-16 | 2018-05-31 | Olympus Corporation | Image processing device, imaging system, image processing method, and computer-readable recording medium |
| CN107767332A (en) * | 2017-10-23 | 2018-03-06 | 合肥师范学院 | A kind of single image depth recovery method and system in real time |
| CN109584210A (en) * | 2018-10-30 | 2019-04-05 | 南京理工大学 | Multispectral three-dimensional vein imaging system |
| CN110942480A (en) * | 2019-11-19 | 2020-03-31 | 宁波五维检测科技有限公司 | Monocular single-frame multispectral three-dimensional imaging method |
| CN111192238A (en) * | 2019-12-17 | 2020-05-22 | 南京理工大学 | Nondestructive blood vessel three-dimensional measurement method based on self-supervision depth network |
| CN114463206A (en) * | 2022-01-24 | 2022-05-10 | 武汉理工大学 | Multispectral image quality improvement method based on global iterative fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115115689B (en) | 2024-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111340844B (en) | Multi-scale characteristic optical flow learning calculation method based on self-attention mechanism | |
| Petrovai et al. | Exploiting pseudo labels in a self-supervised learning framework for improved monocular depth estimation | |
| Zhuo et al. | Defocus map estimation from a single image | |
| CN113450410B (en) | Monocular depth and pose joint estimation method based on epipolar geometry | |
| CN103020965B (en) | A kind of foreground segmentation method based on significance detection | |
| CN103093479B (en) | A kind of object localization method based on binocular vision | |
| Dong et al. | Infrared image colorization using a s-shape network | |
| CN111080709A (en) | Multispectral stereo camera self-calibration algorithm based on track feature registration | |
| CN110969667A (en) | Multi-spectrum camera external parameter self-correction algorithm based on edge features | |
| CN112016478B (en) | Complex scene recognition method and system based on multispectral image fusion | |
| CN112200854B (en) | Leaf vegetable three-dimensional phenotype measuring method based on video image | |
| CN103268482B (en) | A kind of gesture of low complex degree is extracted and gesture degree of depth acquisition methods | |
| CN109977834B (en) | Method and device for segmenting human hand and interactive object from depth image | |
| Bailey et al. | Fast depth from defocus from focal stacks | |
| CN103841298A (en) | Video image stabilization method based on color constant and geometry invariant features | |
| CN105654493A (en) | Improved method for optimizing optical affine-invariant binocular stereo matching cost and parallax | |
| CN109285183B (en) | Multimode video image registration method based on motion region image definition | |
| Moghimi et al. | A joint adaptive evolutionary model towards optical image contrast enhancement and geometrical reconstruction approach in underwater remote sensing | |
| CN111951339A (en) | Image processing method for performing parallax calculation by using heterogeneous binocular cameras | |
| CN105719251A (en) | Compression and quality reduction image restoration method used for large image motion linear fuzziness | |
| CN113538545B (en) | Monocular depth estimation method based on electro-hydraulic adjustable-focus lens and corresponding camera and storage medium | |
| CN107945119B (en) | Method for estimating correlated noise in image based on Bayer pattern | |
| CN117315138A (en) | Three-dimensional reconstruction method and system based on multi-eye vision | |
| CN108805937B (en) | Single-camera polarization information prediction method | |
| CN110910457B (en) | Multispectral three-dimensional camera external parameter calculation method based on angular point characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |