CN107945222A - A kind of new Stereo matching cost calculates and parallax post-processing approach - Google Patents
A kind of new Stereo matching cost calculates and parallax post-processing approach Download PDFInfo
- Publication number
- CN107945222A CN107945222A CN201711348132.3A CN201711348132A CN107945222A CN 107945222 A CN107945222 A CN 107945222A CN 201711348132 A CN201711348132 A CN 201711348132A CN 107945222 A CN107945222 A CN 107945222A
- Authority
- CN
- China
- Prior art keywords
- mrow
- parallax
- pixel
- msub
- mtd
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012805 post-processing Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims description 24
- 238000001514 detection method Methods 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000012417 linear regression Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 230000008439 repair process Effects 0.000 claims description 5
- 238000007667 floating Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000010146 3D printing Methods 0.000 description 1
- 206010019133 Hangover Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000000205 computational method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Abstract
Calculated the invention discloses a kind of new Stereo matching cost and parallax post-processing approach is to improve texture-free region, texture is similar but depth different zones, and the disparity estimation precision of oblique flat areas.The present invention collectively forms joint Stereo matching cost using the gray scale of description texture-rich area information and the Census features of Gradient Features, embodiment large area non-textured area domain structure and nuance, and disparity computation is carried out with this；Then, on the basis of original disparity map, introduce secondary parallax reparation and improve that algorithm is similar in texture but the disparity estimation precision of depth different zones；Become more meticulous by the parallax based on plane fitting, improve parallax precision of the algorithm in oblique flat areas.
Description
Technical field
The present invention provides a kind of new Stereo matching cost calculating and parallax post-processing approach, belongs to computer binocular
Stereoscopic vision field.
Background technology
Stereo matching is a most difficult and most important step during technique of binocular stereoscopic vision is realized.Stereo matching is calculated
Method includes global Stereo Matching Algorithm, half global Stereo Matching Algorithm and local three kinds of Stereo Matching Algorithm, wherein sectional perspective
Matching algorithm with its high accuracy, low consumption when obtain most commonly used application.The calculation procedure of sectional perspective matching algorithm is usual
For：1) Stereo matching cost calculates, 2) Matching power flow polymerize, 3) disparity map calculates, 4) parallax post-processes.
Due to the limitation of the sectional perspective matching algorithm Matching power flow zone of convergency, matching process is larger by noise jamming,
At present by the disparity map that sectional perspective matching algorithm obtains in large area texture-free region, texture is similar but depth different zones
And the parallax quality of the matching such as oblique flat areas difficult region is remarkably decreased.
The content of the invention
For solve traditional sectional perspective matching algorithm in large area texture-free region, texture is similar but depth different zones
With the parallax precision of oblique flat areas it is relatively low the problem of, the present invention propose a kind of new Stereo matching cost calculate and
Parallax post-processing approach, this method are suitable for all sectional perspective matching algorithms, can be obviously improved algorithm and be matched more than and are stranded
The parallax precision in difficult region.
In order to solve the above technical problems, the present invention provides a kind of new Stereo matching cost calculating, for based on gray scale
Difference, gradient difference and the joint Matching power flow corresponding to the Hamming distance of Census features calculate.
Further, the Hamming distance calculating of Census features is corresponded to during Matching power flow calculates to be included：
A rectangular window W (p) is chosen centered on pixel p, by the gray value of each pixel q in window respectively with
The gray value of imago vegetarian refreshments p is compared, if the gray value of q is bigger than the gray value of central point p, is encoded to 1 to q, is otherwise compiled
Code is 0；
Census conversion is defined as follows：
Wherein, I (p) represents the corresponding gray value of pixel p and q with I (q) respectively, and ξ (q, p) represents the volume at pixel q
Code value；
The corresponding encoded radio step-by-step connection of all pixels point in window, so as to form the Hamming of window center pixel p
Coding, is represented with following formula：
Wherein,Operator represents step-by-step connection, and H (p) represents the hamming code at p；
The calculation formula of Hamming distance is as follows：
Wherein, CH(p, d) represents pixel p corresponding Hamming distance when parallax is d, HL(p) and HR(p-d) represent left respectively
Figure pixel p and the hamming code of right figure pixel p-d, operatorRepresent xor operation, | | | |1For in statistics binary string 1
Number, dmaxIt is the maximum disparity value in scene.
Further, the gray scale difference is defined as follows：
CI(p, d)=| IL(p)-IR(p-d)|d∈[0,dmax-1] (4)
Wherein, IL(p) and IR(p-d) gray value of left figure pixel p and right figure pixel p-d is represented respectively；
The gradient difference is defined as follows
CG(p, d)=| GL(p)-GR(p-d)|d∈[0,dmax-1] (5)
Wherein, GL(p) and GR(p-d) left figure pixel p and the Grad of right figure pixel p-d are represented respectively.
Further, exponential function control gray scale difference, gradient difference and Hamming are introduced during joint Matching power flow is calculated
The scope of distance, when parallax is d, the corresponding joint Matching power flow of image p points is：
E (p, d)=ρ (CI(p,d),λI)+ρ(CG(p,d),λG)+ρ(CH(p,d),λH) (6)
Wherein, CI(p, d) represents gray scale difference, CG(p, d) represents gradient difference, CH(p, d) represents Hamming distance, E (p, d) table
Show joint Matching power flow, ρ (c, λ) is variable CI,CG,CHRobustness function, the every difference range of exponential function control is public
Formula is as follows：
The invention also discloses a kind of parallax post-processing approach, comprise the following steps：
S1, use secondary parallax reparation to original disparity map；
S2, carry out the disparity map after reparation the parallax process of refinement based on plane fitting.
Further, secondary parallax reparation is used to original disparity map in parallax post-processing, specifically includes following steps：
S1-1：Left and right consistency detection is made to horizontal parallax figure, detection principle is：
Wherein, pixel p coordinates are (x, y), dLRAnd dRLP point left figures and the corresponding parallax of right figure are represented respectively, work as left and right
The difference that parallax is corresponded in figure is less than Tc, p points are set to 1 by left and right consistency detection, Mask (x, y)；Conversely, p points do not pass through
Left and right consistency detection, the point that it is zero that Mask (x, y), which is set in 0, Mask, are the unreliable point of parallax；
S1-2：Parallax based on transversal scanning is tentatively repaired；Repairing principle is：Using the unreliable point p of parallax as starting point, to
It is left, scanned in certain contiguous range to the right, find first reliable pixel of parallax respectively, if be respectively present in field to
The reliable point of two parallaxes of from left to right, the unreliable parallax of p points is repaired with both medium and small parallax values, and sets the parallax being repaired not
The Mask values reliably put are 1；If only existing the leftward or rightward reliable point of a parallax in field, p is repaired not with this parallax value
Reliable parallax, and the Mask values for being repaired the unreliable point of parallax are set as 1；If it can not be repaiied without the reliable point of other parallaxes in field
The unreliable parallax of multiple p, maintains its Mask value constant；
S1-3：Second of reparation based on weight-parallax histogram；Repairing principle is：Centered on the unreliable point p of parallax
Delimit window, any pixel q votes its own parallax value in window, in window current ballot pixel with center parallax not
The distance reliably put is nearer, or color is closer, then the ballot weight of the ballot pixel is bigger, with the parallax of ballot value maximum
Value repairs the parallax value of window center, the i.e. unreliable point of parallax, and sets the Mask values for being repaired the unreliable point of parallax as 1, votes
Weight equation is as follows
Wherein, | p-q | represent the space length between pixel p and q, Ic represents the color value of one of R, G, B triple channels,
σdAnd σcIt is constant, for control weight size；
The ballot value calculation formula that q points correspond to parallax d is as follows
Histogram (d)=Histogram (d)+weight (q, p) d ∈ [0, dmax-1] (10)
Wherein, the corresponding disparity estimation value of current ballot pixel is d.
Further, the specific implementation step of step S2 is as follows：
S2-1：Image is split：Some subimage blocks are divided the image into using Mean-shift algorithms, it is assumed that each subgraph
As the pixel of block is in same plane；
S2-2：Disparity plane parameter fitting：If plane d=ax+by+c, wherein (x, y) is the coordinate of image plane vegetarian refreshments, d
For the parallax value of the pixel, using the reliable pixel of parallax in plane as fitting foundation, iterative linear regression analysis side is used
Method is fitted disparity plane, and the error vector by introducing last round of iteration redefines the error function of epicycle,
In the hope of obtaining more accurate plane fitting parameters；Assuming that the fitting parameter of disparity plane Ω is denoted as A=[a, b, c]T, in plane
The reliable pixel quantity of parallax is N, and coordinate set is expressed as X={ [x1,y1,1]；…；[xN,yN, 1] }, parallax collection is combined into D
=[d1,d1,…,dN]T, then error function be defined as follows：
Ek=| | D-XAk-E′k-1||2 (12)
Wherein, EkAnd AkThe error of fitting and fit Plane parameter of kth wheel iteration respectively, Ek-1＇ represents last round of iteration
Error vector, is defined as follows：
E′k-1=g (Ek-1)=g (D-XAk-1) (13)
Wherein, g () is a threshold function table, and g () is defined as follows：
Wherein, TgIt is threshold value；
S2-3：Parallax becomes more meticulous：The disparity plane being fitted using each subimage block, is recalculated in subimage block and owned
The parallax of pixel, is preserved with floating point type, and calculation formula is as follows：
dΩ(x, y)=aK·x+bK·y+cK (15)
Wherein, dΩThe parallax value of (x, y) place pixel, [a after (x, y) expression parallax becomes more meticulousK,bK,cK] it is disparity plane
The fitting parameter of Ω.
Further, the iterative linear regression analysis method in step S2-2 includes：
If the ratio of the reliable pixel quantity N of the parallax and pixel total quantity M in disparity plane Ω is less than 0.1, then
The plane is not fitted, algorithm terminates；Conversely, algorithm continues to execute；Assuming that maximum iteration is K；
As k=1, if E0＇ is null vector, and fit Plane parameter A is solved using least square method to formula 121, then use
Formula 13 calculates the error vector E that next round introduces in formula 121＇；
As 2≤k≤K-1, the error of fitting vector E that kth -1 is taken turns is introduced in formula 12k-1＇, redefines the mistake of kth wheel
Difference function simultaneously equally solves fitting parameter A using least square methodk, finally calculate Ek’；
As k=K, final fitting parameter A is solvedk, algorithm terminates.
Beneficial effect：Compared with prior art, the present invention the present invention has the following advantages：The present invention is rich using description texture
The gray scale and Gradient Features of rich area information, embody large area non-textured area domain structure and the Census features of nuance are total to
With joint Stereo matching cost is formed, disparity computation is carried out with this；Then, on the basis of original disparity map, secondary parallax is introduced
Reparation improves that algorithm is similar in texture but the disparity estimation precision of depth different zones；Pass through the parallax essence based on plane fitting
Refinement, improves parallax precision of the algorithm in oblique flat areas, the Stereo matching cost in the present invention calculate and parallax after
Processing method is suitable for any sectional perspective matching algorithm using this two step, can be obviously improved texture-free region, texture phase
Like but the different region of depth, and the disparity estimation precision of oblique flat areas.
Brief description of the drawings
Fig. 1 is the sectional perspective matching algorithm flow chart of the present invention；
Fig. 2 is using disparity map contrast schematic diagram obtained by different Stereo matching cost computational methods：
Wherein, it is artwork to scheme (a), and figure (b) is the experimental result based on pixel difference and gradient difference, and figure (c) is using joint
The parallax that Matching power flow obtains；
Fig. 3 is that front and rear disparity map contrast schematic diagram is repaired using the first time based on transversal scanning：
Wherein, it is artwork to scheme (a), and figure (b) is original disparity map, and figure (c) is the disparity map after first time parallax is repaired；
Fig. 4 is disparity map after being repaired using second based on weight-parallax histogram；
Fig. 5 is the disparity map and Error Graph contrast schematic diagram before and after being become more meticulous using the parallax based on plane fitting：
Wherein, Fig. 5 (a) and Fig. 5 (b) represents the disparity map and Error Graph, Fig. 5 (c) and Fig. 5 before plane fitting respectively
(d) respectively represent plane fitting after disparity map and Error Graph.
Embodiment
Below in conjunction with drawings and examples, the present invention will be further described.
The present invention is applied to the flow chart of sectional perspective matching algorithm as shown in Figure 1, wherein step 1, step 2 and step 5
For novel solid Matching power flow proposed by the present invention calculating and parallax post-processing approach related content, detailed solution is made below
Release.
The Stereo matching cost of the present embodiment calculates, for based on gray scale difference, gradient difference and corresponding to Census features
The joint Matching power flow of Hamming distance calculates, and specifically includes：
1st, left and right figure is calculated corresponding to the Hamming distance of Census features：
Census conversion is a kind of non-parametric transformations, is mainly used to the partial structurtes feature of phenogram picture, uses Census
The process that conversion obtains the hamming code of pixel p is：A rectangular window W (p) is chosen first centered on pixel p, so
Afterwards by the gray value of each pixel q in window respectively compared with the gray value of central pixel point p：If in the gray value ratio of q
The gray value of heart point p is big, then 1 is encoded to q, is otherwise encoded to 0；
Census conversion is defined as follows：
Wherein, I (p) represents the corresponding gray value of pixel p and q with I (q) respectively, and ξ (q, p) represents the volume at pixel q
Code value.
Finally the corresponding encoded radio step-by-step of all pixels point in window is connected, so that the hamming code of pixel p is formed,
Hamming code can be represented with following formula：
Wherein,Operator represents step-by-step connection, and H (p) represents the hamming code at p.
Hamming code can characterize the spatial structural form of pixel, thus the difference of hamming code can represent pixel it
Between similarity, this species diversity defines with Hamming distance, and the calculation formula of Hamming distance is as follows：
Wherein, CH(p, d) represents pixel p corresponding Hamming distance when parallax is d, HL(p) and HR(p-d) represent left respectively
Figure pixel p and the hamming code of right figure pixel p-d, operatorRepresent xor operation, | | | |1For in statistics binary string 1
Number, dmaxIt is the maximum disparity value in scene.
2nd, the gray scale difference of left and right figure calculates：Gray scale difference is defined as follows：
CI(p, d)=| IL(p)-IR(p-d)| d∈[0,dmax-1] (4)
Wherein, IL(p) and IR(p-d) gray value of left figure pixel p and right figure pixel p-d is represented respectively.
3rd, the gradient difference of left and right figure calculates：Gradient difference is defined as follows
CG(p, d)=| GL(p)-GR(p-d)| d∈[0,dmax-1] (5)
Wherein, GL(p) and GR(p-d) left figure pixel p and the Grad of right figure pixel p-d are represented respectively.
4th, the joint Matching power flow based on Hamming distance, gray scale difference and gradient difference calculates：
The scope of exponential function control gray scale difference, gradient difference and Hamming distance is introduced in calculating process：
E (p, d)=ρ (CI(p,d),λI)+ρ(CG(p,d),λG)+ρ(CH(p,d),λH) (6)
Wherein, E (p, d) represents joint Matching power flow, and ρ (c, λ) is the robustness function of variable c：
Various differences are converted to Matching power flow using ρ (c, λ) mainly two purposes：
1) scope of Matching power flow is controlled, significantly changing will not occur because of the exception of certain work factor in formula 6；
2) Different matching work factor is controlled to be supported for combining the weight of Matching power flow by λ, the λ in this algorithmI=
0.5,λG=0.25, λH=0.25；
3) consider the faint effective information to disparity computation of weak texture region, reach and screen regional disparity change
Purpose.
Stereo matching experimental result based on Different matching cost as shown in Figure 2：It is artwork to scheme (a), and figure (b) is to be based on picture
The experimental result of plain difference and gradient difference, its bad point percentage are 23.4%, and figure (c) is regarded using what joint Matching power flow obtained
Difference, its bad point percentage are 2.1%, it is seen that by introducing Hamming distance, error of the algorithm in texture-free region is effectively subtracted
It is few.
The parallax post processing of the present embodiment is divided into two steps：Step S1：Secondary parallax reparation and step S2：Based on plane fitting
Parallax become more meticulous.
S1, secondary parallax reparation：Including S1-1：Left and right consistency detection, S1-2：Parallax based on transversal scanning is tentatively repaiied
Multiple and S1-3：Second of reparation based on weight-parallax histogram.
S1-1：Left and right consistency detection：
Can the principle of the detection scheme be that parallax value reliability is judged by left and right consistency detection according to pixel, left
The mathematical definition of right uniformity detection is as follows：
Wherein, image p point coordinates is (x, y), dLRAnd dRLP point left figures and the corresponding parallax of right figure are represented respectively；Work as left and right
The difference that parallax is corresponded in figure is less than Tc, p points are set to 1 by left and right consistency detection, Mask (x, y)；Conversely, p points cannot lead to
Left and right consistency detection is crossed, Mask (x, y) is set to 0, the T of the present embodimentcIt is that parallax is unreliable to be set in 1, Mask the point for being zero
Point.
S1-2：Parallax based on transversal scanning is tentatively repaired：
The present embodiment finds reliable parallax in neighborhood using the mode of transversal scanning (mask figure Mask (x, y) value is 1)
And replace the parallax that Mask (x, y) value is 0 node failure；Wherein P represents the insecure pixel of parallax, PLAnd PRRepresent respectively
Using P as starting point to the left, scan find first reliable pixel of parallax to the right.
Specific execution flow is as follows：
Centered on pixel P respectively to the left, to the right each find reliable first pixel of parallax, be denoted as PLAnd PR；
If PLAnd PRAll exist, then the parallax of P points is repaired using wherein less parallax, corresponding setting Mask values are 1, and algorithm redirects
To a last step；If only PL(or PR) exist, then use PL(or PR) corresponding parallax repairs the parallaxes of P points, it is corresponding to set
Mask values are 1, and algorithm jumps to final step；If PLAnd PRAll it is not present, then transversal scanning can not repair the parallax of P points, calculate
Method jumps to final step；Next unreliable pixel of parallax to be repaired is found, algorithm jumps to the first step.
From the figure 3, it may be seen that the parallax based on transversal scanning is tentatively repaired as a result, accurate compared to original disparity value parallax profile
Degree is obviously improved, but is laterally repaired and introduced laterally " hangover " effect, such as the region that rectangle frame delimited, is needed second
Repair and solve.
S1-3：Second of reparation based on weight-parallax histogram, to solve tailing problem, but is not limited only to trail
Problem.Calculation process is as follows：
Centered on the insecure pixel p of parallax, the local window that size is N × N delimited, the value of N cannot be excessive
Can not be too small, N crosses senior general and increases time complexity, and N is too small cannot to obtain enough reliable parallax use in local window
In the parallax of renewal central pixel point, the N values of the present embodiment are 15；
Any one pixel q is to the ballot weight of its own parallax value, i.e. weight in calculation window.Ballot weight is based on
Color and range information, if current ballot pixel and the distance of the unreliable pixel of center parallax are nearer, or color is closer,
Then ballot weight is bigger, ballot weight definition：
Wherein, | p-q | represent the space length of pixel, Ic represents the color value of one of R, G, B triple channels, σdAnd σcIt is
Constant, for control weight size, the σ of the present embodimentdAnd σcIt is set to 17 and 25；
The ballot value calculation formula that q points correspond to parallax d is as follows：
Histogram (d)=Histogram (d)+weight (q, p) d ∈ [0, dmax-1] (10)
In window in weight-parallax statistics with histogram result, window center pixel is repaired with the parallax value of ballot value maximum
The unreliable parallax value of point, if the Mask values for being repaired the unreliable pixel of parallax are 1.Fig. 4 is second of reparation as a result, can
See that tailing problem has solved, and further improve the accuracy of parallax profile, so far, the secondary parallax reparation side of the present embodiment
Case is completed.
S2, the parallax based on plane fitting become more meticulous, and can effectively solve the problems, such as the disparity computation of clinoplain, be divided into three
Step：S2-1：Image is split；S2-2：Disparity plane parameter fitting；S2-3：Parallax becomes more meticulous.
S2-1：Image is split:
This algorithm carries out image segmentation using Mean-Shift, each segmentation sub-block represents a plane Ωi。
S2-2：Disparity plane parameter fitting:
After image is split, the parameter for the disparity plane that each segmentation sub-block represents is fitted, in a disparity plane
In, parallax can be represented with the coordinate and plane parameter of pixel, as follows：
D=ax+by+c (11)
Wherein, (a, b, c) represents the parameter of disparity plane, is denoted as A=[a, b, c], and X=[x, y, 1] represents certain in image
A bit, d represents the corresponding parallax of pixel (x, y).The parameter of disparity plane is solved first, is then calculated by formula 11 in plane
The parallax of each pixel.
The fitting of disparity plane parameter can be attributed to linear regression problem, the use of the reliable pixel of parallax in plane be plan
Close foundation.This algorithm uses iterative linear regression analysis method, by mistake of the error vector to epicycle for introducing last round of iteration
Difference function is redefined, in the hope of obtaining more accurate plane fitting parameters.
Assuming that the fitting parameter of disparity plane Ω is denoted as A=[a, b, c]T, the reliable pixel quantity of parallax is in plane
N, pixel coordinate set are expressed as X={ [x1,y1,1]；…；[xN,yN, 1] }, parallax collection is combined into D=[d1,d1,…,dN]T, then
Error function is defined as follows：
Ek=| | D-XAk-E′k-1||2 (12)
Wherein, EkAnd AkThe error of fitting and fit Plane parameter of kth wheel iteration respectively, Ek-1＇ represents last round of iteration
Error vector, is defined as follows：
E′k-1=g (Ek-1)=g (D-XAk-1) (13)
Wherein, g () is a threshold function table, and g () is defined as follows：
Wherein, TgIt is threshold value, the T of the present embodimentgFor 0.5, the purpose for carrying out threshold process to error vector is to pass through control
Error size enables iteration to restrain as early as possible.Note iterations is K, below carries out the flow analyzed iterative linear regression
Describe in detail：
If the ratio of the reliable pixel quantity N of the parallax and pixel total quantity M in disparity plane Ω is less than empirical value
0.1, then the plane is not fitted, algorithm terminates；Conversely, algorithm continues to execute；
As k=1, if E0＇ is null vector, and fit Plane parameter A is solved using least square method to formula 121, then use
Formula 13 calculates the error vector E that next round introduces in formula 121＇；
As 2≤k≤K-1, the error of fitting vector E that kth -1 is taken turns is introduced in formula 12k-1＇, redefines the mistake of kth wheel
Difference function simultaneously equally solves fitting parameter A using least square methodk, finally calculate Ek＇；
As k=K, final fitting parameter A is solvedk, algorithm terminates.
S2-3：Parallax becomes more meticulous：
Remember Ak=[aK,bK,cK] be disparity plane Ω fitting parameter, then all parallaxes in the plane can lead to
Cross formula 15 to recalculate, since plane parameter is represented with floating point type, so the parallax obtained by formula 15 can also save as
Floating point type.
dΩ(x, y)=aK·x+bK·y+cK (15)
Fig. 5 comparative analyses forward and backward experimental result of the parallax process of refinement based on plane fitting, wherein estimation error
The parallax actual value of Middleburry data sets offer is provided.Fig. 5 (a) and Fig. 5 (b) is represented before plane fitting respectively
Disparity map and Error Graph, Fig. 5 (c) and Fig. 5 (d) represent the disparity map and Error Graph after plane fitting respectively.Regarded in the scene
The region that poor precision is obviously improved is concentrated mainly on background wall, piano and stool, this is because these regions can pass through
Mean-Shift algorithms are partitioned into the plane domain of large area, they include more reliable parallax, can fit high accuracy
Disparity plane, so as to be obviously improved the income effect after plane fitting.
So far, the step of before, completes the calculating of Stereo matching cost and parallax post processing, can by the two steps
To be obviously improved algorithm in large area texture-free region, texture is similar but the parallax of depth different zones and oblique flat areas
Precision.
In conclusion the present invention is similar in large area texture-free region, texture for current sectional perspective matching algorithm
But depth different zones, and the relatively low problem of parallax precision of oblique flat areas, it is proposed that new Matching power flow calculate with
And parallax post-processing approach.This method is suitable for all sectional perspective matching algorithms.The innovative point of the present invention is to introduce
Census features are calculated for Matching power flow, introduce secondary parallax reparation and the parallax based on plane fitting becomes more meticulous for parallax
Post processing.It is an advantage of the invention that the parallax precision of above matching difficult region can be obviously improved, and this method is suitable for owning
Sectional perspective matching algorithm.The present invention is suitable for the application such as virtual reality, 3D printing.
Claims (8)
1. a kind of new Stereo matching cost calculates, it is characterised in that：Based on gray scale difference, gradient difference and to correspond to
The joint Matching power flow of the Hamming distance of Census features calculates.
2. a kind of new Stereo matching cost according to claim 1 calculates, it is characterised in that：The Matching power flow meter
Hamming distance in calculation corresponding to Census features, which calculates, to be included：
Centered on pixel p choose a rectangular window W (p), by the gray value of each pixel q in window respectively with middle imago
The gray value of vegetarian refreshments p is compared, if the gray value of q is bigger than the gray value of central point p, is encoded to 1 to q, is otherwise encoded to
0；
Census conversion is defined as follows：
<mrow>
<mi>&xi;</mi>
<mrow>
<mo>(</mo>
<mi>q</mi>
<mo>,</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mtable>
<mtr>
<mtd>
<mfenced open = "{" close = "">
<mtable>
<mtr>
<mtd>
<mn>1</mn>
</mtd>
<mtd>
<mrow>
<mi>I</mi>
<mrow>
<mo>(</mo>
<mi>q</mi>
<mo>)</mo>
</mrow>
<mo>></mo>
<mi>I</mi>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<mi>I</mi>
<mrow>
<mo>(</mo>
<mi>q</mi>
<mo>)</mo>
</mrow>
<mo>&le;</mo>
<mi>I</mi>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
</mtd>
<mtd>
<mrow>
<mi>q</mi>
<mo>&Element;</mo>
<mi>W</mi>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
</mtr>
</mtable>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>1</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein, I (p) represents the corresponding gray value of pixel p and q with I (q) respectively, and ξ (q, p) represents the coding at pixel q
Value；
The corresponding encoded radio step-by-step connection of all pixels point in window, so that the hamming code of window center pixel p is formed,
Represented with following formula：
<mrow>
<mi>H</mi>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
<mo>=</mo>
<munder>
<mrow>
<mo>&CircleTimes;</mo>
<mi></mi>
</mrow>
<mrow>
<mi>q</mi>
<mo>&Element;</mo>
<mi>W</mi>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
</mrow>
</munder>
<mi>&xi;</mi>
<mrow>
<mo>(</mo>
<mi>q</mi>
<mo>,</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>2</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein,Operator represents step-by-step connection, and H (p) represents the hamming code at p；
The calculation formula of Hamming distance is as follows：
<mrow>
<mtable>
<mtr>
<mtd>
<mrow>
<msub>
<mi>C</mi>
<mi>H</mi>
</msub>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>,</mo>
<mi>d</mi>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mo>|</mo>
<mo>|</mo>
<msub>
<mi>H</mi>
<mi>L</mi>
</msub>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
<mo>&CirclePlus;</mo>
<msub>
<mi>H</mi>
<mi>R</mi>
</msub>
<mrow>
<mo>(</mo>
<mi>p</mi>
<mo>-</mo>
<mi>d</mi>
<mo>)</mo>
</mrow>
<mo>|</mo>
<msub>
<mo>|</mo>
<mn>1</mn>
</msub>
</mrow>
</mtd>
<mtd>
<mrow>
<mi>d</mi>
<mo>&Element;</mo>
<mo>&lsqb;</mo>
<mn>0</mn>
<mo>,</mo>
<msub>
<mi>d</mi>
<mrow>
<mi>m</mi>
<mi>a</mi>
<mi>x</mi>
</mrow>
</msub>
<mo>-</mo>
<mn>1</mn>
<mo>&rsqb;</mo>
</mrow>
</mtd>
</mtr>
</mtable>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>3</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein, CH(p, d) represents pixel p corresponding Hamming distance when parallax is d, HL(p) and HR(p-d) left image is represented respectively
The hamming code of plain p and right figure pixel p-d, operatorRepresent xor operation, | | | |1For in statistics binary string 1
Number, dmaxIt is the maximum disparity value in scene.
3. a kind of new Stereo matching cost according to claim 2 calculates, it is characterised in that：
The gray scale difference is defined as follows：
CI(p, d)=| IL(p)-IR(p-d)|d∈[0,dmax-1] (4)
Wherein, IL(p) and IR(p-d) gray value of left figure pixel p and right figure pixel p-d is represented respectively；
The gradient difference is defined as follows
CG(p, d)=| GL(p)-GR(p-d)|d∈[0,dmax-1] (5)
Wherein, GL(p) and GR(p-d) left figure pixel p and the Grad of right figure pixel p-d are represented respectively.
4. a kind of new Stereo matching cost according to claim 3 calculates, it is characterised in that：Calculating joint matching
The scope of exponential function control gray scale difference, gradient difference and Hamming distance is introduced during cost, when parallax is d, image p points correspond to
Joint Matching power flow be：
E (p, d)=ρ (CI(p,d),λI)+ρ(CG(p,d),λG)+ρ(CH(p,d),λH) (6)
Wherein, CI(p, d) represents gray scale difference, CG(p, d) represents gradient difference, CH(p, d) represents Hamming distance, and E (p, d) represents connection
Matching power flow is closed, ρ (c, λ) is variable CI,CG,CHRobustness function, the every difference range of exponential function control, formula is such as
Under：
<mrow>
<mtable>
<mtr>
<mtd>
<mrow>
<mi>&rho;</mi>
<mrow>
<mo>(</mo>
<mi>c</mi>
<mo>,</mo>
<mi>&lambda;</mi>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mn>1</mn>
<mo>-</mo>
<mi>exp</mi>
<mrow>
<mo>(</mo>
<mo>-</mo>
<mfrac>
<mi>c</mi>
<mi>&lambda;</mi>
</mfrac>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
<mtd>
<mrow>
<mi>c</mi>
<mo>&Element;</mo>
<mo>{</mo>
<msub>
<mi>C</mi>
<mi>i</mi>
</msub>
<mo>,</mo>
<msub>
<mi>C</mi>
<mi>G</mi>
</msub>
<mo>,</mo>
<msub>
<mi>C</mi>
<mi>H</mi>
</msub>
<mo>}</mo>
</mrow>
</mtd>
</mtr>
</mtable>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>7</mn>
<mo>)</mo>
</mrow>
</mrow>
5. the parallax post processing side that a kind of new Stereo matching cost described in based on claim 1-4 any one calculates
Method, it is characterised in that：The parallax post-processing, comprises the following steps：
S1, use secondary parallax reparation to original disparity map；
S2, carry out the disparity map after reparation the parallax process of refinement based on plane fitting.
6. parallax post-processing approach according to claim 5, it is characterised in that：
Secondary parallax reparation is used to original disparity map in the parallax post-processing, specifically includes following steps：
S1-1：Left and right consistency detection is made to horizontal parallax figure, detection principle is：
Wherein, pixel p coordinates are (x, y), dLRAnd dRLP point left figures and the corresponding parallax of right figure are represented respectively, when in the figure of left and right
The difference of corresponding parallax is less than Tc, p points are set to 1 by left and right consistency detection, Mask (x, y)；Conversely, p points do not pass through left and right
Consistency detection, the point that it is zero that Mask (x, y), which is set in 0, Mask, are the unreliable point of parallax；
S1-2：Parallax based on transversal scanning is tentatively repaired；Repairing principle is：Using the unreliable point p of parallax as starting point, to the left, to
The right side is scanned in certain contiguous range, first reliable pixel of parallax is found respectively, if being respectively present in field to left-hand
The reliable point of right two parallaxes, the unreliable parallax of p points is repaired with both medium and small parallax values, and it is unreliable to set the parallax being repaired
The Mask values of point are 1；If only existing the leftward or rightward reliable point of a parallax in field, the unreliable of p is repaired with this parallax value
Parallax, and the Mask values for being repaired the unreliable point of parallax are set as 1；If it can not repair p's without the reliable point of other parallaxes in field
Unreliable parallax, maintains its Mask value constant；
S1-3：Second of reparation based on weight-parallax histogram；Repairing principle is：Delimited centered on the unreliable point p of parallax
Window, any pixel q votes its own parallax value in window, and current ballot pixel and center parallax are unreliable in window
The distance of point is nearer, or color is closer, then the ballot weight of the ballot pixel is bigger, is repaiied with the parallax value of ballot value maximum
The parallax value of multiple window center, the i.e. unreliable point of parallax, and the Mask values for being repaired the unreliable point of parallax are set as 1, ballot weight
Formula is as follows
<mrow>
<mi>w</mi>
<mi>e</mi>
<mi>i</mi>
<mi>g</mi>
<mi>h</mi>
<mi>t</mi>
<mrow>
<mo>(</mo>
<mi>q</mi>
<mo>,</mo>
<mi>p</mi>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mi>exp</mi>
<mrow>
<mo>(</mo>
<mo>-</mo>
<mfrac>
<mrow>
<mo>|</mo>
<mi>p</mi>
<mo>-</mo>
<mi>q</mi>
<mo>|</mo>
</mrow>
<msub>
<mi>&sigma;</mi>
<mi>d</mi>
</msub>
</mfrac>
<mo>-</mo>
<mfrac>
<msqrt>
<mrow>
<munder>
<mo>&Sigma;</mo>
<mrow>
<mi>c</mi>
<mo>&Element;</mo>
<mo>{</mo>
<mi>R</mi>
<mo>,</mo>
<mi>G</mi>
<mo>,</mo>
<mi>B</mi>
<mo>}</mo>
</mrow>
</munder>
<msup>
<mrow>
<mo>(</mo>
<msub>
<mi>I</mi>
<mi>c</mi>
</msub>
<mo>(</mo>
<mi>p</mi>
<mo>)</mo>
<mo>-</mo>
<msub>
<mi>I</mi>
<mi>c</mi>
</msub>
<mo>(</mo>
<mi>q</mi>
<mo>)</mo>
<mo>)</mo>
</mrow>
<mn>2</mn>
</msup>
</mrow>
</msqrt>
<msub>
<mi>&sigma;</mi>
<mi>c</mi>
</msub>
</mfrac>
<mo>)</mo>
</mrow>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>9</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein, | p-q | represent the space length between pixel p and q, Ic represents the color value of one of R, G, B triple channels, σdWith
σcIt is constant, for control weight size；
The ballot value calculation formula that q points correspond to parallax d is as follows
Histogram (d)=Histogram (d)+weight (q, p) d ∈ [0, dmax-1] (10)
Wherein, the corresponding disparity estimation value of current ballot pixel is d.
7. the parallax post-processing approach according to claim 5 or 6, it is characterised in that：The specific implementation step of the step S2
It is rapid as follows：
S2-1：Image is split：Some subimage blocks are divided the image into using Mean-shift algorithms, it is assumed that each subimage block
Pixel in same plane；
S2-2：Disparity plane parameter fitting：If plane d=ax+by+c, wherein (x, y) is the coordinate of image plane vegetarian refreshments, d is should
The parallax value of pixel, using the reliable pixel of parallax in plane as fitting foundation, is intended using iterative linear regression analysis method
Disparity plane is closed, and the error vector by introducing last round of iteration redefines the error function of epicycle, in the hope of
Obtain more accurate plane fitting parameters；Assuming that the fitting parameter of disparity plane Ω is denoted as A=[a, b, c]T, parallax in plane
Reliable pixel quantity is N, and coordinate set is expressed as X={ [x1,y1,1]；…；[xN,yN, 1] }, parallax collection is combined into D=[d1,
d1,…,dN]T, then error function be defined as follows：
Ek=| | D-XAk-E′k-1||2 (12)
Wherein, EkAnd AkThe error of fitting and fit Plane parameter of kth wheel iteration respectively, Ek-1＇ represents the error of last round of iteration
Vector, is defined as follows：
E′k-1=g (Ek-1)=g (D-XAk-1) (13)
Wherein, g () is a threshold function table, and g () is defined as follows：
<mrow>
<mi>g</mi>
<mrow>
<mo>(</mo>
<mi>x</mi>
<mo>)</mo>
</mrow>
<mo>=</mo>
<mfenced open = "{" close = "">
<mtable>
<mtr>
<mtd>
<mrow>
<mi>x</mi>
<mo>-</mo>
<msub>
<mi>T</mi>
<mi>g</mi>
</msub>
</mrow>
</mtd>
<mtd>
<mrow>
<mo>|</mo>
<mi>x</mi>
<mo>|</mo>
<mo>></mo>
<msub>
<mi>T</mi>
<mi>g</mi>
</msub>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<mo>|</mo>
<mi>x</mi>
<mo>|</mo>
<mo>&le;</mo>
<msub>
<mi>T</mi>
<mi>g</mi>
</msub>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>14</mn>
<mo>)</mo>
</mrow>
</mrow>
Wherein, TgIt is threshold value；
S2-3：Parallax becomes more meticulous：The disparity plane being fitted using each subimage block, recalculates all pixels in subimage block
Parallax, preserved with floating point type, calculation formula is as follows：
dΩ(x, y)=aK·x+bK·y+cK (15)
Wherein, dΩThe parallax value of (x, y) place pixel, [a after (x, y) expression parallax becomes more meticulousK,bK,cK] for disparity plane Ω's
Fitting parameter.
8. parallax post-processing approach according to claim 7, it is characterised in that：Iterative linear in the step S2-2 returns
Analysis method is returned to include：
If the ratio of the reliable pixel quantity N of the parallax and pixel total quantity M in disparity plane Ω is less than 0.1, then not right
The plane is fitted, and algorithm terminates；Conversely, algorithm continues to execute；Assuming that maximum iteration is K；
As k=1, if E0＇ is null vector, and fit Plane parameter A is solved using least square method to formula 121, then using formula 13
Calculate the error vector E that next round introduces in formula 121＇；
As 2≤k≤K-1, the error of fitting vector E that kth -1 is taken turns is introduced in formula 12k-1＇, redefines the error letter of kth wheel
Number simultaneously equally solves fitting parameter A using least square methodk, finally calculate Ek’；
As k=K, final fitting parameter A is solvedk, algorithm terminates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711348132.3A CN107945222A (en) | 2017-12-15 | 2017-12-15 | A kind of new Stereo matching cost calculates and parallax post-processing approach |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711348132.3A CN107945222A (en) | 2017-12-15 | 2017-12-15 | A kind of new Stereo matching cost calculates and parallax post-processing approach |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107945222A true CN107945222A (en) | 2018-04-20 |
Family
ID=61944257
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711348132.3A Pending CN107945222A (en) | 2017-12-15 | 2017-12-15 | A kind of new Stereo matching cost calculates and parallax post-processing approach |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107945222A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110176009A (en) * | 2019-05-23 | 2019-08-27 | 复旦大学 | A kind of segmentation of lung images and tracking and system |
| CN110533701A (en) * | 2018-05-25 | 2019-12-03 | 杭州海康威视数字技术股份有限公司 | A kind of image parallactic determines method, device and equipment |
| CN110533663A (en) * | 2018-05-25 | 2019-12-03 | 杭州海康威视数字技术股份有限公司 | A kind of image parallactic determines method, apparatus, equipment and system |
| CN111815693A (en) * | 2020-09-04 | 2020-10-23 | 北京清微智能科技有限公司 | Depth image generation method and device |
| CN112907714A (en) * | 2021-03-05 | 2021-06-04 | 兰州大学 | Mixed matching binocular vision system based on Census transformation and gray absolute difference |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170301108A1 (en) * | 2015-08-26 | 2017-10-19 | Digitalglobe, Inc. | Broad area geospatial object detection using autogenerated deep learning models |
| CN107392950A (en) * | 2017-07-28 | 2017-11-24 | 哈尔滨理工大学 | A kind of across yardstick cost polymerization solid matching method based on weak skin texture detection |
-
2017
- 2017-12-15 CN CN201711348132.3A patent/CN107945222A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170301108A1 (en) * | 2015-08-26 | 2017-10-19 | Digitalglobe, Inc. | Broad area geospatial object detection using autogenerated deep learning models |
| CN107392950A (en) * | 2017-07-28 | 2017-11-24 | 哈尔滨理工大学 | A kind of across yardstick cost polymerization solid matching method based on weak skin texture detection |
Non-Patent Citations (1)
| Title |
|---|
| 刘灿: "基于立体匹配的深度提取及动态场景虚拟视角合成技术的算法研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑（月刊）》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110533701A (en) * | 2018-05-25 | 2019-12-03 | 杭州海康威视数字技术股份有限公司 | A kind of image parallactic determines method, device and equipment |
| CN110533663A (en) * | 2018-05-25 | 2019-12-03 | 杭州海康威视数字技术股份有限公司 | A kind of image parallactic determines method, apparatus, equipment and system |
| CN110533663B (en) * | 2018-05-25 | 2022-03-04 | 杭州海康威视数字技术股份有限公司 | Image parallax determining method, device, equipment and system |
| CN110176009A (en) * | 2019-05-23 | 2019-08-27 | 复旦大学 | A kind of segmentation of lung images and tracking and system |
| CN110176009B (en) * | 2019-05-23 | 2023-03-07 | 复旦大学 | Lung image segmentation and tracking method and system |
| CN111815693A (en) * | 2020-09-04 | 2020-10-23 | 北京清微智能科技有限公司 | Depth image generation method and device |
| CN112907714A (en) * | 2021-03-05 | 2021-06-04 | 兰州大学 | Mixed matching binocular vision system based on Census transformation and gray absolute difference |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107945222A (en) | A kind of new Stereo matching cost calculates and parallax post-processing approach | |
| CN107481279B (en) | Monocular video depth map calculation method | |
| Pham et al. | Domain transformation-based efficient cost aggregation for local stereo matching | |
| US7876954B2 (en) | Method and device for generating a disparity map from stereo images and stereo matching method and device therefor | |
| CN104867135B (en) | A kind of High Precision Stereo matching process guided based on guide image | |
| Cigla et al. | Information permeability for stereo matching | |
| Saito et al. | Application of genetic algorithms to stereo matching of images | |
| CN114782691A (en) | Robot target identification and motion detection method based on deep learning, storage medium and equipment | |
| CN101765019B (en) | Stereo matching algorithm for motion blur and illumination change image | |
| EP3465611A1 (en) | Apparatus and method for performing 3d estimation based on locally determined 3d information hypotheses | |
| CN104270624B (en) | A kind of subregional 3D video mapping method | |
| CN111914913B (en) | Novel stereo matching optimization method | |
| CN109887008A (en) | Based on front and back to smooth and O (1) complexity parallax solid matching method, device and equipment | |
| Gupta et al. | A correlation-based approach for real-time stereo matching | |
| CN112991421A (en) | Robot vision stereo matching method | |
| CN112084855B (en) | Outlier elimination method for video stream based on improved RANSAC method | |
| CN110147809B (en) | Image processing method and device, storage medium and image equipment | |
| CN113920270B (en) | Layout reconstruction method and system based on multi-view panorama | |
| CN105809717A (en) | Depth estimation method, system and electronic equipment | |
| CN113344989B (en) | NCC and Census minimum spanning tree aerial image binocular stereo matching method | |
| CN110490877B (en) | Target segmentation method for binocular stereo image based on Graph Cuts | |
| Chao et al. | Occcasnet: occlusion-aware cascade cost volume for light field depth estimation | |
| Yuan et al. | An Improved Method of 3D Scene Reconstruction Based on SfM | |
| CN102413347B (en) | Method for correcting matching error based on depth-image-based rendering (DIBR) | |
| CN117078982B (en) | Deep learning-based large-dip-angle stereoscopic image alignment dense feature matching method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180420 |