CN108288286A - A kind of half global solid matching method preferential based on surface orientation - Google Patents
A kind of half global solid matching method preferential based on surface orientation Download PDFInfo
- Publication number
- CN108288286A CN108288286A CN201810046411.2A CN201810046411A CN108288286A CN 108288286 A CN108288286 A CN 108288286A CN 201810046411 A CN201810046411 A CN 201810046411A CN 108288286 A CN108288286 A CN 108288286A
- Authority
- CN
- China
- Prior art keywords
- parallax
- pixel
- priori
- plane
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/536—Depth or shape recovery from perspective effects, e.g. by using vanishing points
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of half global solid matching methods preferential based on surface orientation.Which solve, for the problem that the matching effect of the indoor scene of large-scale not texture is undesirable, include the following steps present in existing Stereo Matching Technology：S1, SGM algorithm；S2, two-dimensional directional priori；S3, three-dimensional priori；S4, surface normal priori.Method disclosed by the invention is the priori in simple surfaces, can also be significantly improved in the matching of the complex scene containing inclined-plane and weaker texture.
Description
Technical field
The present invention relates to computer vision techniques, and in particular to a kind of half global Stereo matching preferential based on surface orientation
Method.
Background technology
Stereo matching as the technologies such as three-dimensional reconstruction, stereo navigation, non-contact distance-measurement committed step by matching two width
Or multiple image obtains depth information.And it is widely used in, it is industrial production automation, Pipeline control, unmanned
Automobile (ranging, navigation), safety monitoring, remote Sensing Image Analysis, intelligent robot control etc..Although Stereo matching application is wide
General but also many still unsolved problems therefore the technology become the difficulty of computer vision field extensive concern in recent years
Point and hot spot.With the scientific and technological progress of society, the development of Stereo Matching Technology is maked rapid progress, with matching algorithm precision and speed
Raising, application scenarios further expand.In this context, research Stereo matching becomes to mean a great.
Stereo Matching Technology is crucial in computer vision one of studies a question.It can be generally divided into partial approach and complete
Office's method.Both methods has all carried out smooth hypothesis to the world observed；The former passes through the polymerization in local window
The cost matched, and surface before is explicitly imposed in the world by the latter by a smooth item.It is most simple and most common
Smooth hypothesis be single order, and point out two adjacent pixel most probable depth having the same.The smooth hypothesis of single order introduces
Parallel biasing before one.When there is enough textures in scene, this is not problem, but in the inclined of not texture
Surface will produce mistake, this is typically problematic in scene indoors.
Therefore for the matching for the indoor scene of large-scale not texture present in existing Stereo Matching Technology
The undesirable problem of effect, the present invention propose a kind of half global solid matching method preferential based on surface orientation, can be with
It is effective to improve the above problem.
Invention content
The technical problem to be solved by the present invention is to propose a kind of half global Stereo matching side preferential based on surface orientation
Method, the matching effect that can solve the indoor scene present in existing matching technique for large-scale not texture are undesirable
The problem of.
Scheme is used by the present invention solves above-mentioned technical problem：
A kind of half global solid matching method preferential based on surface orientation, includes the following steps：
S1, half global registration (SGM) algorithm；
S2, two-dimensional directional priori；
S3, three-dimensional priori；
S4, surface normal priori.
As advanced optimizing, half Global Algorithm in S1 its between local algorithm and Global Algorithm, so-called half is complete
Office refers to that algorithm both without the regional area of only consideration pixel, did not accounted for all pixels yet；
Half global registration (SGM) is a kind of widely used Stereo Matching Technology, is proposed by Hirschmuller, it will
The efficiency of partial approach is combined with the precision of global approach, by being similar to two-dimentional MRF optimizations and several one-dimensional scan lines
Optimization problem can obtain effective solution by Dynamic Programming.Studies have shown that SGM is a kind of special message transmission calculation
Method.
Although this method is suitable for Aerial Images and textured Outdoor Scene, for the interior of large-scale not texture
The effect of scene is not so good.The reason is that the algorithm uses, a kind of simple single order is smooth it is assumed that this smooth hypothesis has
Conducive to preceding parallel surface.Therefore, it tends to generation and preceding parallel patch on the inclined surface of weak texture, this is on the surface of reconstruction
On be failure, especially when matching high-definition picture, will produce in larger " hole ".
As advanced optimizing, content is specific as follows in S2：
Assuming that the parallax face S of a real value, on any given pixel, we want to encourage all possible difference；
It allows r to become current " the comprehensive direction " of SGM, gives the successor p'=p+r of a pixel p and it, rasterizing
Surface S is to the inconsistent of integer；
Then discrete parallax step (or jump) is calculated
With a new function VsOriginal smooth punishment V is substituted, and it contains parallax jump:
VS(dp,d′p)=V (dp+jp,d′p) (3)
At pixel, the value of j is non-zero, and VsThen tend to take this species diversity step, and difference face is encouraged to be protected with S
Maintain an equal level row.We can be stored in migrated image by the j that will jump effectively to calculate Vs.Since jump depends on direction r,
So needing four different migrated images, two opposite directions of a correspondence.
But we need not precalculate and store all four migrated images.On the contrary, we only need to store
Before the direction operation scan line optimization opposite to every a pair, we generate relevant deflection graph using formula (7) and (8)
Picture.When reverse directions, we understand the symbol of roll over offset amount.
As advanced optimizing, content is specific as follows in S3：
Loosen the requirement to single direction before each pixel, and allows the surface of multiple overlappings of different depth false
If.Before, each surface should first be used as a direction, but should influence the point (S3 in Fig. 2) near all.
Assuming that there is K inconsistent surfaces in pixel pFor given parallax d, immediate table can be found
Face.
Then, we are its rasterizing to the inconsistent of integer
The discrete parallax between adjacent pixel is calculated again.However, this time, they depend on d.
New smooth penalty term have now becomes
VS(dp,d′p)=V (dp+jp(dp),d′p) (7)
In order to effectively calculate VS, we are by the pre-calculated values of all p and dIt is stored in an auxiliary quantity.This can be with
It is counted as the Voronoi diagram of an one-dimensional discrete, each pixel is on the asymmetric axis of volume.In the value of each row
In, it can effectively be calculated by scanning forward and scanning backward, all parallax faces are then presented on this pixel
In row.
As advanced optimizing, content is specific as follows in S4：
We had been contemplated that the priori for using parallax curved surface as orientation.Focusing on be to obtain a surface normal
Textures, such as obtained by photometric stereo figure or Manhattan-world's priori.Such normal map can be in our SGM-
It is used in P algorithms.First, we cannot direct use direction because SGM-P need a practical table that can be rasterized
Face.Therefore, normal mapping must be integrated.Secondly, it would be desirable to distinguish scene space (in Euclidean world coordinates) and regard
Difference space.Although surface normal textures be considered it is two-dimensional because it carries out the single surface direction of each pixel
Coding, this direction provides in scene space.
Derive the relationship between the surface normal under scene coordinate and the direction in parallax face.Given surface normal vectorThe equation of the tangent plane of p (x, y, z) plane is under scene space coordinate：
Here, hpThe unknown depth of plane is encoded.In the case of perspective projection, there are the image of x=uz/f and y=vz/f
These values that the focal length f of coordinate (u, v) and the camera of at the origin is obtained instead of us.
For three-dimensional right, it is baseline and parallax to have z=bf/d, b and d respectively.Z is updated to formula (14), has obtained this
A disparity plane equation.
The direction of disparity plane depends on hp, it encodes the depth of tangent plane in scene space.Therefore, one
The scene plane for having fixed-direction but unknown depth, will produce one group of inconsistent plane, and direction depends on relevant parallax.
In order to use the normal priori in surface in sgm-p, it is necessary first to be integrated into a surface.In scene space
It is integrated in scene projection model using least square method.The result is that the faces a z scene space coordinate, is to appoint at the beginning
The depth of meaning.We scale this surface with scale factor appropriate, are then converted into d, reach the substantially uniform surfaces d and cover
Cover full disparity range.Finally, we construct an offset from this family, different as described in previous section, not
Before same parallax face, the direction of 3D is produced.
The beneficial effects of the invention are as follows：The present invention is the priori in simple surfaces, can also containing inclined-plane and compared with
It is significantly improved in the matching of the complex scene of weak texture.
Description of the drawings
Fig. 1 is inventive algorithm flow chart.
Fig. 2 is the diagram of the smooth items of SGM-P.
Fig. 3 is that surface normal is converted into parallax directions priori figure.
Specific implementation method
To make the object of the invention, technical solution and advantage be more clearly understood, with reference to embodiment, to the present invention make into
The detailed description of one step, exemplary embodiment of the invention and its explanation are only used for explaining the present invention, are not intended as to this hair
Bright restriction.
Referring to Fig. 1-Fig. 3, a kind of half global solid matching method preferential based on surface orientation includes the following steps：
S1, half global registration (SGM) algorithm
Before the embodiment of the present invention is described in detail, first SGM algorithms are simply introduced.
Half global registration (SGM) algorithm is a kind of effective method, the two-dimentional MRF minimized for approximate energy.
Cp(d) it is a metadata item, indicates in d ∈ D={ dmin,…,dmaxWhen matched pixel P cost, V (d, d ')
It is a pair of smooth item, its parallax between adjacent pixel is corrected.Specifically, V realizes the smooth vacation of single order
If.
Rather than 2D MRF are minimized, this is NP-hard problems, and SGM is effectively by the one-dimensional version of formula (1) minimum
Change, by Dynamic Programming, advances along 8 directions.The matching cost L of a polymerization is calculated for each direction r, SGMr(p,
D) it is recursively defined from image boundary：
8 polymerization costs are summarized on each pixel, to generate the cost amount of a polymerization.
The minimum value of each pixel is selected as the gap won.
The summation of the minimum value for 8 Lr (p, d) that Drory et al. is observed is one and lower is integrated into totle drilling cost most
Small value S (p, d) defines a uncertainty as difference between the two on each pixel.
It is intuitively the U on the position of 8 least-cost pathspTo be 0, for example, in incorrect unbalanced region
In, there is very high cost C in incorrect regionp.However, in not textured region, multiple inconsistent places have phase
As unitary cost, and have 8 LrMinimum value, it may occur however that in different difference, especially on inclined surface.At this
In invention, U is usedpProbabilistic disparity error is drawn, and selects the matching of high confidence level.
The priori of S2, two-dimensional directional.
Assuming that we obtain the parallax face S of a real value, r is allowed to become in current " the comprehensive direction " of SGM.Give a picture
The successor p'=p+r of plain p and it, our rasterizing surface S are to the inconsistent of integer.
Then discrete parallax step (or jump) is calculated
We are with a new function VsOriginal smooth punishment V is substituted, and it contains parallax jump：
VS(dp,d′p)=V (dp+jp,d′p) (9)
At pixel, the value of j is non-zero, and VsThen tend to take this species diversity step, and difference face is encouraged to be protected with S
Maintain an equal level row.We can be stored in by the j that will jump and effectively calculate V in migrated image (S2 in Fig. 2)s.Due to jump according to
Rely in direction r, so needing four different migrated images, two opposite directions of a correspondence.
But we need not precalculate and store all four migrated images.On the contrary, we only need to store
Before the direction operation scan line optimization opposite to every a pair, we generate relevant deflection graph using formula (7) and (8)
Picture.When reverse directions, the symbol of meeting roll over offset amount.
The priori of S3, three-dimensional.
Assuming that there is K inconsistent surfaces in pixel pFor given parallax d, can find most
Close surface.
Then, we are its rasterizing to the inconsistent of integer
The discrete parallax between adjacent pixel is calculated again.However, this time, they depend on d：
New smooth penalty term have now becomes：
VS(dp,d′p)=V (dp+jp(dp),d′p) (13)
S2 illustrates the direction that this difference relies in Fig. 2.In order to effectively calculate VS, we are pre- by all p's and d
Calculated valueIt is stored in an auxiliary quantity.This can be counted as the Voronoi diagram of an one-dimensional discrete, each pixel
It is on the asymmetric axis of volume.In the value of each row, can effectively it be calculated by scanning forward and scanning backward,
Then all parallax faces are presented in the row of this pixel.
S4, surface normal priori.
Normal map can use in sgm-p, cannot direct use direction because sgm-p need one can be by light
The actual surface of gated.Therefore, normal mapping must be integrated.Secondly, it is also necessary to distinguish scene space and (be sat in the Euclidean world
In mark) and disparity space.When being transformed into disparity space under a perspective projection model, surface direction can become depth according to
Rely, as a result will produce three-dimensional parallax, need the expression of an offset.
Referring to Fig. 3, the relationship between the surface normal under scene coordinate and the direction in parallax face is derived.Given surface method
VectorThe equation of the tangent plane of p (x, y, z) plane is under scene space coordinate：
Herein, hpThe unknown depth of plane is encoded.In the case of perspective projection, we have x=uz/f
These values obtained instead of us with the focal length f of the camera of the image coordinate (u, v) and at the origin of y=vz/f.
For three-dimensional right, it is baseline and parallax to have z=bf/d, b and d respectively.Z is updated to formula (15), obtains this
Disparity plane equation.
The direction of disparity plane depends on hp, it encodes the depth of tangent plane in scene space.Therefore, one
The scene plane for having fixed-direction but unknown depth, will produce one group of inconsistent plane, and direction depends on relevant parallax.
In order to use the normal priori in surface in sgm-p, it is necessary first to be integrated into a surface.In scene space
It is integrated in scene projection model using least square method.The result is that the faces a z scene space coordinate, is to appoint at the beginning
The depth of meaning.This surface is scaled with scale factor appropriate (in Fig. 3 at S2), is then converted into d, is reached substantially uniform
The surfaces d cover full disparity range (in Fig. 3 at S3).Finally, an offset is constructed from this family, as previous section is retouched
The difference stated produces the direction of 3D before different parallax faces (in Fig. 3 at S4).
Claims (5)
1. a kind of half global solid matching method preferential based on surface orientation, which is characterized in that including with step：
S1, SGM algorithm；
S2, two-dimensional directional priori；
S3, three-dimensional priori；
S4, surface normal priori.
2. being existed according to a kind of method of the half global Stereo matching preferential based on surface orientation, feature described in claim 1 book
In the step of SGM algorithms is as follows in S 1：
S11, SGM algorithm are a kind of effective methods, the two-dimentional MRF minimized for approximate energy：
Cp(d) it is a metadata item, indicates in d ∈ D={ dmin,…,dmaxWhen matched pixel P cost, V (d, d ') is a pair of
Smooth item, its parallax between adjacent pixel are corrected；
V realizes the smooth hypothesis of single order：
Rather than 2D MRF are minimized, this is NP-hard problems, and SGM effectively minimizes the one-dimensional version of formula (1), leads to
Dynamic Programming is crossed, is advanced along 8 directions, 8 polymerization costs are obtained；
S12, it is each direction r, SGM calculates the matching cost L of a polymerizationr(p, d) is recursively defined from image boundary：
8 polymerization costs are summarized on each pixel, to generate the cost amount of a polymerization：
The minimum value of each pixel is selected as the gap won：
8 LrThe summation of the minimum value of (p, d) is a lower minimum value S (p, d) for being integrated into totle drilling cost in each pixel
On, and a uncertainty is defined as difference between the two：
It is intuitively the U on the position of 8 least-cost pathspTo be 0.
3. a kind of half global solid matching method preferential based on surface orientation according to claim 1, feature exist
In S2, two-dimensional directional priori include the following steps：
S21, hypothesis obtain the parallax face S of a real value, and r is allowed to become current " the comprehensive direction " of SGM, give a pixel p
With its successor p'=p+r, rasterizing surface S to the inconsistent of integer；
Then discrete parallax step is calculated：
S22, with a new function VsOriginal smooth punishment V is replaced, it contains parallax jump：
VS(dp,d′p)=V (dp+jp,d′p) (9)
At pixel, the value of j is non-zero, and VsThen tend to take this species diversity step, and difference face is encouraged to keep flat with S
Row, can be stored in migrated image by the j that will jump effectively to calculate Vs；
Since jump depends on direction r, so needing four different migrated images, two opposite directions of a correspondence；
All four migrated images need not be precalculated and store, on the contrary, it is only necessary to storeIn the side opposite to every a pair
To before operation scan line optimization, relevant migrated image is generated using formula (7) and (8), when reverse directions, can be overturn partially
The symbol of shifting amount.
4. a kind of half global solid matching method preferential based on surface orientation according to claim 1, feature exist
In S3, three-dimensional priori include the following steps：
S31, hypothesis have K inconsistent surfaces in pixel pK=1...K can find and most connect for given parallax d
Close surface：
Then, we are its rasterizing to the inconsistent of integer：
S32, discrete parallax between adjacent pixel is calculated again, however, this time, they depend on d：
New smooth penalty term have now becomes：
VS(dp,d′p)=V (dp+jp(dp),d′p) (13)
In order to effectively calculate VS, by the pre-calculated values of all p and dIt is stored in an auxiliary quantity；This is regarded as one
The Voronoi diagram of a one-dimensional discrete, each pixel be on the asymmetric axis of volume, in the value of each row, Ke Yitong
It crosses scanning forward and scans backward effectively to calculate, then all parallax faces are presented in the row of this pixel.
5. a kind of half global solid matching method preferential based on surface orientation according to claim 1, feature exist
In S4, surface normal direction priori include the following steps：
S41, relationship between the surface normal under scene coordinate and the direction in parallax face, given surface normal vector are derivedThe equation of the tangent plane of p (x, y, z) plane is under scene space coordinate：
Hp encodes the unknown depth of plane, and in the case of perspective projection, we have x=uz/f's and y=vz/f
These values that the focal length f of the camera of image coordinate (u, v) and at the origin is obtained instead of us；
S42, for three-dimensional right, it is baseline and parallax to have z=bf/d, b and d respectively, and z is updated to formula (15), has obtained this
A disparity plane equation：
The direction of disparity plane depends on hp, it encodes the depth of tangent plane in scene space, and therefore, one has admittedly
The scene plane for determining direction but unknown depth, will produce one group of inconsistent plane, and direction depends on relevant parallax.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810046411.2A CN108288286A (en) | 2018-01-17 | 2018-01-17 | A kind of half global solid matching method preferential based on surface orientation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810046411.2A CN108288286A (en) | 2018-01-17 | 2018-01-17 | A kind of half global solid matching method preferential based on surface orientation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108288286A true CN108288286A (en) | 2018-07-17 |
Family
ID=62835292
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810046411.2A Pending CN108288286A (en) | 2018-01-17 | 2018-01-17 | A kind of half global solid matching method preferential based on surface orientation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108288286A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111080689A (en) * | 2018-10-22 | 2020-04-28 | 杭州海康威视数字技术股份有限公司 | Method and device for determining face depth map |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761739A (en) * | 2014-01-23 | 2014-04-30 | 武汉大学 | Image registration method based on half energy optimization |
| CN106530337A (en) * | 2016-10-31 | 2017-03-22 | 武汉市工程科学技术研究院 | Non local stereopair dense matching method based on image gray scale guiding |
-
2018
- 2018-01-17 CN CN201810046411.2A patent/CN108288286A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103761739A (en) * | 2014-01-23 | 2014-04-30 | 武汉大学 | Image registration method based on half energy optimization |
| CN106530337A (en) * | 2016-10-31 | 2017-03-22 | 武汉市工程科学技术研究院 | Non local stereopair dense matching method based on image gray scale guiding |
Non-Patent Citations (1)
| Title |
|---|
| DANIEL SCHARSTEIN等: ""Semi-Global Stereo Matching with Surface Orientation Priors"", 《ARXIV.ORG》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111080689A (en) * | 2018-10-22 | 2020-04-28 | 杭州海康威视数字技术股份有限公司 | Method and device for determining face depth map |
| CN111080689B (en) * | 2018-10-22 | 2023-04-14 | 杭州海康威视数字技术股份有限公司 | Method and device for determining face depth map |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Concha et al. | Using superpixels in monocular SLAM | |
| KR101928575B1 (en) | Piecewise planar reconstruction of three-dimensional scenes | |
| EP2064675B1 (en) | Method for determining a depth map from images, device for determining a depth map | |
| Zhu et al. | Generalized parallel-perspective stereo mosaics from airborne video | |
| US8791941B2 (en) | Systems and methods for 2-D to 3-D image conversion using mask to model, or model to mask, conversion | |
| US9613420B2 (en) | Method for locating a camera and for 3D reconstruction in a partially known environment | |
| Mordohai et al. | Stereo using monocular cues within the tensor voting framework | |
| CN106408596B (en) | Sectional perspective matching process based on edge | |
| WO2008112790A2 (en) | System and method for using feature tracking techniques for the generation of masks in the conversion of two-dimensional images to three-dimensional images | |
| WO2008112766A1 (en) | Systems and methods for allowing a user to dynamically manipulate stereoscopic parameters | |
| Gao et al. | Complete scene reconstruction by merging images and laser scans | |
| CN104537668B (en) | A kind of quick parallax image computational methods and device | |
| JP4996922B2 (en) | 3D visualization | |
| Shivakumar et al. | Real time dense depth estimation by fusing stereo with sparse depth measurements | |
| EP1997072B1 (en) | Method for determining a depth map from images, device for determining a depth map | |
| Román et al. | Automatic Multiperspective Images. | |
| CN108288286A (en) | A kind of half global solid matching method preferential based on surface orientation | |
| CN111179327B (en) | Depth map calculation method | |
| Lhuillier | Toward flexible 3d modeling using a catadioptric camera | |
| CN111951310A (en) | Binocular stereo matching method, disparity map acquisition device and computer storage medium | |
| CN106156471B (en) | A kind of multi-level flow field visualization method | |
| Andreasson et al. | Vision based interpolation of 3D laser scans | |
| Yoon et al. | Stereo matching with symmetric cost functions | |
| Andreasson et al. | Non-iterative vision-based interpolation of 3D laser scans | |
| Raviya et al. | Depth and Disparity Extraction Structure for Multi View Images-Video Frame-A Review |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180717 |
|
| RJ01 | Rejection of invention patent application after publication |