CN107465911B - A kind of extraction of depth information method and device - Google Patents
A kind of extraction of depth information method and device Download PDFInfo
- Publication number
- CN107465911B CN107465911B CN201610382517.0A CN201610382517A CN107465911B CN 107465911 B CN107465911 B CN 107465911B CN 201610382517 A CN201610382517 A CN 201610382517A CN 107465911 B CN107465911 B CN 107465911B
- Authority
- CN
- China
- Prior art keywords
- information
- pixel
- optical flow
- frame image
- depth information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/004—Diagnosis, testing or measuring for television systems or their details for digital television systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/02—Diagnosis, testing or measuring for television systems or their details for colour television signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/513—Sparse representations
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention discloses a kind of extraction of depth information methods, comprising: obtains video frame images information, the video frame images information includes first frame image information and the second frame image information；The edge contour information in the first frame image is extracted, and first frame image information is matched to obtain sparse optical flow with second frame image information；According to the edge contour information and the sparse optical flow, the dense optical flow between first frame image information and second frame image information is calculated；Initial depth information is converted by the dense optical flow, and ultimate depth information is obtained according to the initial depth information and the video frame images information processing.The embodiment of the invention also discloses a kind of extraction of depth information devices.Using the embodiment of the present invention, the accuracy of the extraction of depth information of video image is improved.
Description
Technical field
The present invention relates to electronic technology field more particularly to a kind of extraction of depth information method and devices.
Background technique
Depth information refers to digit used in each pixel of storage, is also used for the color-resolution of measurement image.Image is deep
The presumable number of colours of each pixel for determining color image is spent, or determines the presumable gray scale of each pixel of gray level image
Series, it determines the maximum tonal gradation in the MaxColors or gray level image that may occur in which in color image.A such as width
Monochrome image, if each pixel has 8,8 powers that maximum gray scale number is 2, i.e., 256.One width color image RGB3 point
The pixel digit of amount is respectively 4,4,2, then the 4+4+2 power that maximum color number is 2, i.e., and 1024, that is the depth of pixel
It is 10, each pixel can be one of 1024 kinds of colors.
In the prior art scheme, a kind of method for extracting depth information is disclosed, comprising: from MP4/H.264 video pressure
Motion vector information is extracted in contracting information, converts depth information for motion vector information, extracts Color Image Edge information, knot
Conjunction image edge information carries out summation to intramarginal depth information and is averaged to obtain final depth information.But MP4/
H.264 the motion vector of video all corresponds to the motion vector of the sizes block of pixels such as 4X4,4X8,8X8,16X16 in video frame,
It is very poor that depth information back edge information is converted by the corresponding motion vector of block of pixels, also, includes I in MP4/H.264 video
(key frame), P (search frame forward), B (bidirectional research frame) do not include motion vector in I frame, P frame include front close on it is several
The motion vector of a I frame and P frame, B frame include the motion vector of the several frames in present frame front and back, causes can only to take and close on
The motion vector that P frame and B frame acquire to carry out I frame approximate valuation, and the motion vector of P frame, B frame and previous frame data is also very
Hardly possible calculates accurate.
Summary of the invention
The embodiment of the present invention provides a kind of extraction of depth information method and device.The depth information of video image can be improved
The accuracy of extraction.
First aspect present invention provides a kind of extraction of depth information method, comprising:
Video frame images information is obtained, the video frame images information includes first frame image information and the second frame image letter
Breath；
Extract the edge contour information in the first frame image, and by first frame image information and second frame
Image information is matched to obtain sparse optical flow；
According to the edge contour information and the sparse optical flow, first frame image information and described the is calculated
Dense optical flow between two frame image informations；
Initial depth information is converted by the dense optical flow, and according to the initial depth information and the video frame figure
As information processing obtains ultimate depth information.
Correspondingly, second aspect of the present invention provides a kind of extraction of depth information device, comprising:
Data obtaining module, for obtaining video frame images information, the video frame images information includes first frame image
Information and the second frame image information；
Information matches module, for extracting the edge contour information in the first frame image, and by the first frame figure
As information is matched to obtain sparse optical flow with second frame image information；
Information computational module, for being calculated described first according to the edge contour information and the sparse optical flow
Dense optical flow between frame image information and second frame image information；
Message processing module, for converting initial depth information for the dense optical flow, and according to the initial depth
Information and the video frame images information processing obtain ultimate depth information.
Correspondingly, third aspect present invention provides a kind of extraction of depth information device, described device include interface circuit,
Memory and processor, wherein batch processing code is stored in memory, and processor is used to call to store in memory
Program code, for performing the following operations:
Video frame images information is obtained, the video frame images information includes first frame image information and the second frame image letter
Breath；
Extract the edge contour information in the first frame image, and by first frame image information and second frame
Image information is matched to obtain sparse optical flow；
According to the edge contour information and the sparse optical flow, first frame image information and described the is calculated
Dense optical flow between two frame image informations；
Initial depth information is converted by the dense optical flow, and according to the initial depth information and the video frame figure
As information processing obtains ultimate depth information.
Implement the embodiment of the present invention, first acquisition video frame images information, then extracts the edge wheel in first frame image
Wide information, and the first frame image information is matched to obtain sparse optical flow with the second frame image information；Secondly according to edge wheel
The dense optical flow between the first frame image information and the second frame image information is calculated in wide information and sparse optical flow；Finally will
Dense optical flow is converted into initial depth information, and obtains ultimate depth according to initial depth information and video frame images information processing
Information, to improve the accuracy of extraction of depth information.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, required use in being described below to embodiment
Attached drawing be briefly described, it should be apparent that, drawings in the following description are some embodiments of the invention, for this field
For those of ordinary skill, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow chart of first embodiment of extraction of depth information method proposed by the present invention；
Fig. 2 is a kind of schematic diagram of the calculation method of dense optical flow provided in an embodiment of the present invention；
Fig. 3 is a kind of flow chart of the second embodiment of extraction of depth information method proposed by the present invention；
Fig. 4 is a kind of structural schematic diagram for extraction of depth information device that the embodiment of the present invention proposes；
Fig. 5 is the structural schematic diagram of information computational module in device provided in an embodiment of the present invention；
Fig. 6 is the structural schematic diagram of message processing module in device provided in an embodiment of the present invention；
Fig. 7 is the structural schematic diagram for another extraction of depth information device that the embodiment of the present invention proposes.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
Referring to FIG. 1, Fig. 1 is a kind of flow chart of first embodiment of extraction of depth information method proposed by the present invention.Such as
Shown in figure, the method in the embodiment of the present invention includes:
S101 obtains video frame images information, and the video frame images information includes first frame image information and the second frame
Image information.
It obtains in the specific implementation, can be shot by single camera based on the video frame images information for closing on frame, wherein
First frame image information is current frame image information, and the second frame image information is the next frame image information closed on present frame；
The video frame images information based on the synchronous viewpoint in left and right can be obtained by the camera shooting of left and right two, wherein described first
Frame image information is left view point frame image information, and the second frame image information is right viewpoint frame image information.
S102, extracts the edge contour information in the first frame image, and by first frame image information with it is described
Second frame image information is matched to obtain sparse optical flow.
In the specific implementation, can be mentioned using based on forest SED algorithm, GPB algorithm, candy algorithm or image gradient algorithm
Take the edge contour information in the first frame image, wherein edge contour information is moving object in video frame images information
Moving boundaries, the same motion layer for partly belonging to the same object within edge contour, the part except edge contour
Other motion layers for belonging to other objects, the cost between the pixel of the same same motion layer of object within the edge contour
Cost distance apart from small, and between other pixels except edge contour (except moving boundaries) is big.
Furthermore it is possible to first using depth matching algorithm, KLT matching algorithm, SIFT matching algorithm or HOG matching algorithm come
Matched pixel is obtained to M={ (pm,p'm), wherein (pm,p'm) respectively represent pixel p in current frame image informationmWith it is current
Pixel p ' in next frame image information that frame closes onmBetween light stream, or represent the pixel p in left view point frame image informationm
With the pixel p ' in right viewpoint synchronization frame imagemBetween light stream, then using the light stream of multiple matched pixels pair as sparse
The Optic flow information of pixel in light stream.
S103, according to the edge contour information and the sparse optical flow, be calculated first frame image information with
Dense optical flow between second frame image information.
In the specific implementation, can be according to the edge contour information, the determining target pixel points with the dense optical flow
Between N number of pixel in the minimum the multiple pixel of cost distance, N is the integer greater than 1；According to N number of pixel
In each pixel Optic flow information and preset each pixel apart from weight, calculate the target pixel points
Optic flow information.Further, the Optic flow information of each pixel weight at a distance from each pixel can be calculated
The sum of products and each pixel apart from weights sum；By the sum of products divided by it is described apart from weight it
With Optic flow information of the ratio as the target pixel points is calculated.
For example, for the point p in dense optical flow, N number of customized cost is found around p point first apart from the smallest sparse
The pixel p of Optic flow information has been acquired in light streami, calculate Optic flow information of N number of customized cost apart from the smallest pixel
Multiplied by weightAfter obtaining the sum of products, it is averaged the light stream as p point, i.e.,
Wherein,For based on customized
The weight of cost distance, Nk(p) light stream letter has been acquired in the smallest sparse optical flow for N number of customized cost around p point
The pixel p of breathi。
It should be noted that customized cost has acquired N number of pixel p of Optic flow information in the smallest sparse optical flowi
It is respectively positioned within edge contour belonging to p point, pixel p, piBetween cost distance can be expressed as
C(ps) represent passing point psCost, cost small distance is close between the pixel for belonging to the same motion layer in edge contour,
And the big distance of cost between other pixels outside edge contour is remote.
Optionally, in the dense optical flow being calculated between first frame image information and second frame image information
Later, energy can be carried out to the dense optical flow being calculated to minimize to obtain the better dense optical flow of marginal information, energy
It is E=E that amount, which minimizes formula,data+αEsmooth, wherein EdataFor color gradient consistency constraint, EsmoothFor smoothness constraint, U, v is respectively pixel
Component of the motion vector of point on the direction x, y, I1For current frame image information or left view point frame image information, I2For present frame
Next frame image information or right viewpoint frame image information, α be proportionality constant,
Optionally, the video frame images information can be acquired respectively under a variety of pixel resolutions；According to various described
The edge contour information and the sparse optical flow under pixel resolution, are calculated multiple dense optical flow components；To described
Multiple dense optical flow components are merged to obtain the dense optical flow.
As shown in Fig. 2, Fig. 2 is a kind of schematic diagram of the calculation method of dense optical flow provided in an embodiment of the present invention.Due to
It calculates under single pixel resolution ratio and inevitably malfunctions in flat and repetition texture region dense optical flow, it can be in acquisition video frame
It is more to frame image information/left and right viewpoint frame image information progress is closed on using pyramid frequency reducing rate sample mode when image information
Then secondary sampling calculates corresponding dense optical flow component to the dense optical flow matched under each pixel resolution, and to each
Calculated dense optical flow component is merged to obtain dense optical flow under pixel resolution.
The dense optical flow is converted initial depth information by S104, and according to the initial depth information and the view
Frequency frame image information handles to obtain ultimate depth information.
In the specific implementation, the motion vector of the target pixel points in the dense optical flow and described can be calculated first
The largest motion vector sum minimum movement vector of pixel in video frame images information；It is sweared according to the movement of the target pixel points
Minimum movement vector described in amount, the largest motion vector sum, calculates the parallax information of the target pixel points, and take the view
The inverse of poor information obtains the initial depth information；Then the hole region in the initial depth information is repaired；
According to the video frame images information, the initial depth information after the repairing hole region is filtered；To process
The initial depth information after filtering processing carries out edge compensation and obtains the ultimate depth information.
For example, the motion vector v in the x and y direction of the pixel (i, j) in dense optical flowxAnd vyQuadratic sum rooting obtains
To the motion vector v of the pixel, the i.e. motion vector of the pixelIf depth information uses 8bit's
Grayscale image indicates, then can pass through formulaBy the motion vectors mapping of the pixel to [0,255]
Between, and then the parallax information of the pixel is obtained, and take the inverse of parallax information as depth information.
Further, small hole region that can be few for pixel carries out empty repairing using median filtering, to pixel
The more macroscopic-void region of point carries out empty repairing using image repair algorithm.
Further, current pixel point and the current pixel point support window in the available video frame images information
The physical space range information and color space range information between pixel in mouthful；According to the physical space distance letter
The initial depth information after breath, the color space range information and the repairing hole region, determines by filtering
The initial depth information for the current pixel point that treated.
For example, can the initial depth information described in the guidance two-sided filter using Weighted Coefficients after hole region into
Row filtering, the guidance two-sided filter of Weighted Coefficients can effectively reduce the noise in depth information and avoid causing object edge
Part is fuzzy on a large scale.The guidance two-sided filter of Weighted Coefficients utilizes video frame images information and the depth after cavity is repaired
Spend depth information of the information as input, after output fining.The calculation formula of the guidance two-sided filter of Weighted Coefficients:
P is current pixel point
Coordinate, s are the coordinate of the pixel in p current pixel point support window, and I is video frame images information, DsFor the depth of input
Information, DpFor the depth information of output, N is the size of support window, and w (p, s) is current pixel point and the current pixel point
Physical space range information between pixel in support window, c (Ip,Is) it is current pixel point and the current pixel point
Color space range information between pixel in support window, Rs are weight.
In addition, guiding bilateral filtering if the Rs value in above-mentioned calculation formula takes 1, calculation formula is
S is current pixel point coordinate, and q is the pixel in s support window
Coordinate,The depth difference of s, q are represented, Ms is the mask of speckle regions, and speckle regions threshold value is 0, non-speckle regions threshold value
It is 1, the pixel color and depth information difference in speckle regions expression region are small, but speckle regions very little, typically represent
Noise, therefore the Rs value of speckle regions should be set to 0, that is, the influence of noise is removed, when the pixel and current picture in support window
Vegetarian refreshments is more similar in an object area (color difference and depth difference very little), then Rs value is bigger, and vice versa.
Further, the initial depth information and process of the available current pixel point after filtering processing
The initial depth information of the pixel in the current pixel point support window before filtering processing；According to after filtering processing
The current pixel point initial depth information and by filtering processing before the current pixel point support window in
The initial depth information of pixel carries out edge compensation to the initial depth information of the current pixel point.
For example, although the guidance bilateral filtering of Weighted Coefficients can carry out smoothly flat site while retaining marginal information
Excessively, edge blurry can be caused when the color difference but in foreground object and background object is small or depth difference is big, therefore is needed
Edge compensation is carried out to the initial depth information of the current pixel point, formula is as follows:
Wherein, p is current pixel point coordinate in edge blurry region, and s is the pixel coordinate within the scope of p support window,
After the depth information of the current pixel p in edge blurry region is compensated the guidance bilateral filtering for using Weighted Coefficients for current pixel
The difference between the depth information before the guidance bilateral filtering of Weighted Coefficients is used within the scope of obtained depth information and support window
The depth value for square being minimized corresponding position v.Such as: being converted into depth information local value by light stream motion vector isIt is obtained after the guidance bilateral filtering of Weighted Coefficients guarantor side is smoothWherein, 1 in the 2nd matrix
(surrounding) includes 0,0,3 three pixel in the 1st matrix within the scope of support window, and 1 and the difference of numerical value 0 square are 1,
1 and the difference of numerical value 3 square are 4, therefore are squared and compensate for 0 in 1 corresponding 1st matrix as depth information, are pressed
Method successively finally obtains the preferable step local value of marginal information and is like this
In embodiments of the present invention, video frame images information is obtained first, then extracts the edge wheel in first frame image
Wide information, and the first frame image information is matched to obtain sparse optical flow with the second frame image information；Secondly according to edge wheel
The dense optical flow between the first frame image information and the second frame image information is calculated in wide information and sparse optical flow；Finally will
Dense optical flow is converted into initial depth information, and obtains ultimate depth according to initial depth information and video frame images information processing
Information, to improve the accuracy of extraction of depth information.
Referring to FIG. 3, Fig. 3 is a kind of flow chart of the second embodiment of extraction of depth information method proposed by the present invention.Such as
Shown in figure, the method in the embodiment of the present invention includes:
S301 obtains video frame images information, and the video frame images information includes first frame image information and the second frame
Image information.
It obtains in the specific implementation, can be shot by single camera based on the video frame images information for closing on frame, wherein
First frame image information is current frame image information, and the second frame image information is the next frame image information closed on present frame；
The video frame images information based on the synchronous viewpoint in left and right can be obtained by the camera shooting of left and right two, wherein described first
Frame image information is left view point frame image information, and the second frame image information is right viewpoint frame image information.
S302, extracts the edge contour information in the first frame image, and by first frame image information with it is described
Second frame image information is matched to obtain sparse optical flow.
In the specific implementation, can be mentioned using based on forest SED algorithm, GPB algorithm, candy algorithm or image gradient algorithm
Take the edge contour information in the first frame image, wherein edge contour information is moving object in video frame images information
Moving boundaries, the same motion layer for partly belonging to the same object within edge contour, the part except edge contour
Other motion layers for belonging to other objects, the cost between the pixel of the same same motion layer of object within the edge contour
Cost distance apart from small, and between other pixels except edge contour (except moving boundaries) is big.
Furthermore it is possible to first using depth matching algorithm, KLT matching algorithm, SIFT matching algorithm or HOG matching algorithm come
Matched pixel is obtained to M={ (pm,p'm), wherein (pm,p'm) respectively represent pixel p in current frame image informationmWith it is current
Pixel p ' in next frame image information that frame closes onmBetween light stream, or represent the pixel p in left view point frame image informationm
With the pixel p ' in right viewpoint synchronization frame imagemBetween light stream, then using the light stream of multiple matched pixels pair as sparse
The Optic flow information of pixel in light stream.
S303, according to the edge contour information and the sparse optical flow, be calculated first frame image information with
Dense optical flow between second frame image information.
In the specific implementation, can be according to the edge contour information, the determining target pixel points with the dense optical flow
Between N number of pixel in the minimum the multiple pixel of cost distance, N is the integer greater than 1；According to N number of pixel
In each pixel Optic flow information and preset each pixel apart from weight, calculate the target pixel points
Optic flow information.Further, the Optic flow information of each pixel weight at a distance from each pixel can be calculated
The sum of products and each pixel apart from weights sum；By the sum of products divided by it is described apart from weight it
With Optic flow information of the ratio as the target pixel points is calculated.
For example, for the point p in dense optical flow, N number of customized cost is found around p point first apart from the smallest sparse light
The pixel p of Optic flow information has been acquired in streami, calculate the Optic flow information of N number of customized cost apart from the smallest pixel multiplied by
WeightAfter obtaining the sum of products, it is averaged the light stream as p point, i.e.,
Wherein,For the weight based on customized cost distance, NkIt (p) is N number of customized around p point
Cost the pixel p of Optic flow information has been acquired in the smallest sparse optical flowi。
It should be noted that customized cost has acquired N number of pixel p of Optic flow information in the smallest sparse optical flowi
It is respectively positioned within edge contour belonging to p point, pixel p, piBetween cost distance can be expressed as
C(ps) represent passing point psCost, cost small distance is close between the pixel for belonging to the same motion layer in edge contour,
And the big distance of cost between other pixels outside edge contour is remote.
Optionally, in the dense optical flow being calculated between first frame image information and second frame image information
Later, energy can be carried out to the dense optical flow being calculated to minimize to obtain the better dense optical flow of marginal information, energy
It is E=E that amount, which minimizes formula,data+αEsmooth, wherein EdataFor color gradient consistency constraint, EsmoothFor smoothness constraint, U, v is respectively pixel
Component of the motion vector on the direction x, y, I1For current frame image information or left view point frame image information, I2For present frame
Next frame image information or right viewpoint frame image information, α are proportionality constant,
Optionally, the video frame images information can be acquired respectively under a variety of pixel resolutions；According to various described
The edge contour information and the sparse optical flow under pixel resolution, are calculated multiple dense optical flow components；To described
Multiple dense optical flow components are merged to obtain the dense optical flow.
As shown in Fig. 2, Fig. 2 is a kind of schematic diagram of the calculation method of dense optical flow provided in an embodiment of the present invention.Due to
It calculates under single pixel resolution ratio and inevitably malfunctions in flat and repetition texture region dense optical flow, it can be in acquisition video frame
It is more to frame image information/left and right viewpoint frame image information progress is closed on using pyramid frequency reducing rate sample mode when image information
Then secondary sampling calculates corresponding dense optical flow component to the dense optical flow matched under each pixel resolution, and to each
Calculated dense optical flow component is merged to obtain dense optical flow under pixel resolution.
The dense optical flow is converted initial depth information by S304.
In the specific implementation, the motion vector of the target pixel points in the dense optical flow and described can be calculated first
The largest motion vector sum minimum movement vector of pixel in video frame images information；It is sweared according to the movement of the target pixel points
Minimum movement vector described in amount, the largest motion vector sum, calculates the parallax information of the target pixel points, and take the view
The inverse of poor information obtains the initial depth information.
For example, the motion vector v in the x and y direction of the pixel (i, j) in dense optical flowxAnd vyQuadratic sum rooting obtains
To the motion vector v of the pixel, the i.e. motion vector of the pixelIf depth information uses the ash of 8bit
Degree figure indicates, then can pass through formulaBy the motion vectors mapping of the pixel to [0,255] it
Between, and then the parallax information of the pixel is obtained, and take the inverse of parallax information as depth information.
S305 repairs the hole region in the initial depth information.
In the specific implementation, small hole region that can be few for pixel carries out empty repairing using median filtering, to picture
Macroscopic-void region more than vegetarian refreshments carries out empty repairing using image repair algorithm.
S306, according to the video frame images information, to repair the initial depth information after the hole region into
Row filtering.
In the specific implementation, current pixel point and the current pixel point are supported in the available video frame images information
Physical space range information and color space range information between pixel in window；According to the physical space distance letter
The initial depth information after breath, the color space range information and the repairing hole region, determines by filtering
The initial depth information for the current pixel point that treated.
For example, can the initial depth information described in the guidance two-sided filter using Weighted Coefficients after hole region into
Row filtering, the guidance two-sided filter of Weighted Coefficients can effectively reduce the noise in depth information and avoid causing object edge
Part is fuzzy on a large scale.The guidance two-sided filter of Weighted Coefficients utilizes video frame images information and the depth after cavity is repaired
Spend depth information of the information as input, after output fining.The calculation formula of the guidance two-sided filter of Weighted Coefficients:
P is current pixel point
Coordinate, s are the coordinate of the pixel in p current pixel point support window, and I is video frame images information, DsFor the depth of input
Information, DpFor the depth information of output, N is the size of support window, and w (p, s) is current pixel point and the current pixel point
Physical space range information between pixel in support window, c (Ip,Is) it is current pixel point and the current pixel point
Color space range information between pixel in support window, Rs are weight.
In addition, guiding bilateral filtering if the Rs value in above-mentioned calculation formula takes 1, calculation formula is
S is current pixel point coordinate, and q is the pixel in s support window
Coordinate,The depth difference of s, q are represented, Ms is the mask of speckle regions, and speckle regions threshold value is 0, non-speckle regions threshold value
It is 1, the pixel color and depth information difference in speckle regions expression region are small, but speckle regions very little, typically represent
Noise, therefore the Rs value of speckle regions should be set to 0, that is, the influence of noise is removed, when the pixel and current picture in support window
Vegetarian refreshments is more similar in an object area (color difference and depth difference very little), then Rs value is bigger, and vice versa.
S307 carries out edge compensation to the initial depth information after filtering processing and obtains the ultimate depth letter
Breath.
In the specific implementation, the initial depth information and warp of the available current pixel point after filtering processing
The initial depth information of the pixel in the current pixel point support window before crossing filtering processing；According to by being filtered
In the initial depth information of the current pixel point afterwards and the current pixel point support window before process filtering processing
Pixel initial depth information, edge compensation is carried out to the initial depth information of the current pixel point.
For example, although the guidance bilateral filtering of Weighted Coefficients can carry out smoothly flat site while retaining marginal information
Excessively, edge blurry can be caused when the color difference but in foreground object and background object is small or depth difference is big, therefore is needed
Edge compensation is carried out to the initial depth information of the current pixel point, formula is as follows:
Wherein, p is current pixel point coordinate in edge blurry region, and s is the pixel coordinate within the scope of p support window,
After the depth information of the current pixel p in edge blurry region is compensated the guidance bilateral filtering for using Weighted Coefficients for current pixel
The difference between the depth information before the guidance bilateral filtering of Weighted Coefficients is used within the scope of obtained depth information and support window
The depth value for square being minimized corresponding position v.Such as: being converted into depth information local value by light stream motion vector isIt is obtained after the guidance bilateral filtering of Weighted Coefficients guarantor side is smoothWherein, 1 in the 2nd matrix
(surrounding) includes 0,0,3 three pixel in the 1st matrix within the scope of support window, and 1 and the difference of numerical value 0 square are 1,
1 and the difference of numerical value 3 square are 4, therefore are squared and compensate for 0 in 1 corresponding 1st matrix as depth information, are pressed
Method successively finally obtains the preferable step local value of marginal information and is like this
In embodiments of the present invention, video frame images information is obtained first, then extracts the edge wheel in first frame image
Wide information, and the first frame image information is matched to obtain sparse optical flow with the second frame image information；Secondly according to edge wheel
The dense optical flow between the first frame image information and the second frame image information is calculated in wide information and sparse optical flow；Finally will
Dense optical flow is converted into initial depth information, and obtains ultimate depth according to initial depth information and video frame images information processing
Information, to improve the accuracy of extraction of depth information.
Referring to FIG. 4, Fig. 4 is a kind of structural schematic diagram for extraction of depth information device that the embodiment of the present invention proposes.Such as
Shown in figure, the device in the embodiment of the present invention includes:
Data obtaining module 401, for obtaining video frame images information, the video frame images information includes first frame figure
As information and the second frame image information.
It obtains in the specific implementation, can be shot by single camera based on the video frame images information for closing on frame, wherein
First frame image information is current frame image information, and the second frame image information is the next frame image information closed on present frame；
The video frame images information based on the synchronous viewpoint in left and right can be obtained by the camera shooting of left and right two, wherein described first
Frame image information is left view point frame image information, and the second frame image information is right viewpoint frame image information.
Information matches module 402, for extracting the edge contour information in the first frame image, and by the first frame
Image information is matched to obtain sparse optical flow with second frame image information.
In the specific implementation, can be mentioned using based on forest SED algorithm, GPB algorithm, candy algorithm or image gradient algorithm
Take the edge contour information in the first frame image, wherein edge contour information is moving object in video frame images information
Moving boundaries, the same motion layer for partly belonging to the same object within edge contour, the part except edge contour
Other motion layers for belonging to other objects, the generation between the pixel of the same same motion layer of object within the edge contour
Valence is apart from small, and the cost distance between other pixels except edge contour (except moving boundaries) is big.
Furthermore it is possible to first using depth matching algorithm, KLT matching algorithm, SIFT matching algorithm or HOG matching algorithm come
Matched pixel is obtained to M={ (pm,p'm), wherein (pm,p'm) respectively represent pixel p in current frame image informationmWith it is current
Pixel p ' in next frame image information that frame closes onmBetween light stream, or represent the pixel p in left view point frame image informationm
With the pixel p ' in right viewpoint synchronization frame imagemBetween light stream, then using the light stream of multiple matched pixels pair as sparse
The Optic flow information of pixel in light stream.
Information computational module 403, for according to the edge contour information and the sparse optical flow, being calculated described the
Dense optical flow between one frame image information and second frame image information.
In the specific implementation, as shown in figure 5, information computational module 403 may further include:
Pixel determination unit 501, for according to the edge contour information, the determining target with the dense optical flow
N number of pixel between pixel in the minimum the multiple pixel of cost distance, N are the integer greater than 1.
Information calculating unit 502, for according to the Optic flow information of each pixel in N number of pixel and preset
Each pixel apart from weight, calculate the Optic flow information of the target pixel points.
In the specific implementation, the Optic flow information of each pixel weight at a distance from each pixel can be calculated
The sum of products and each pixel apart from weights sum；By the sum of products divided by it is described apart from weight it
With Optic flow information of the ratio as the target pixel points is calculated.
For example, for the point p in dense optical flow, N number of customized cost is found around p point first apart from the smallest sparse
The pixel p of Optic flow information has been acquired in light streami, calculate Optic flow information of N number of customized cost apart from the smallest pixel
Multiplied by weightAfter obtaining the sum of products, it is averaged the light stream as p point, i.e.,
Wherein,For based on customized
The weight of cost distance, Nk(p) light stream letter has been acquired in the smallest sparse optical flow for N number of customized cost around p point
The pixel p of breathi。
It should be noted that customized cost has acquired N number of pixel of Optic flow information in the smallest sparse optical flow
piIt is respectively positioned within edge contour belonging to p point, pixel p, piBetween cost distance can be expressed as
C(ps) represent passing point psCost, cost small distance is close between the pixel for belonging to the same motion layer in edge contour,
And the big distance of cost between other pixels outside edge contour is remote.
Optionally, in the dense optical flow being calculated between first frame image information and second frame image information
Later, energy can be carried out to the dense optical flow being calculated to minimize to obtain the better dense optical flow of marginal information, energy
It is E=E that amount, which minimizes formula,data+αEsmooth, wherein EdataFor color gradient consistency constraint, EsmoothFor smoothness constraint, U, v is respectively pixel
Component of the motion vector on the direction x, y, I1For current frame image information or left view point frame image information, I2For under present frame
One frame image information or right viewpoint frame image information, α are proportionality constant,
Optionally, the video frame images information can be acquired respectively under a variety of pixel resolutions；According to various described
The edge contour information and the sparse optical flow under pixel resolution, are calculated multiple dense optical flow components；To described
Multiple dense optical flow components are merged to obtain the dense optical flow.
As shown in Fig. 2, Fig. 2 is a kind of schematic diagram of the calculation method of dense optical flow provided in an embodiment of the present invention.Due to
It calculates under single pixel resolution ratio and inevitably malfunctions in flat and repetition texture region dense optical flow, it can be in acquisition video frame
It is more to frame image information/left and right viewpoint frame image information progress is closed on using pyramid frequency reducing rate sample mode when image information
Then secondary sampling calculates corresponding dense optical flow component to the dense optical flow matched under each pixel resolution, and to each
Calculated dense optical flow component is merged to obtain dense optical flow under pixel resolution.
Message processing module 404, for converting initial depth information for the dense optical flow, and according to the initial depth
Degree information and the video frame images information processing obtain ultimate depth information.
In the specific implementation, the motion vector of the target pixel points in the dense optical flow and described can be calculated first
The largest motion vector sum minimum movement vector of pixel in video frame images information；It is sweared according to the movement of the target pixel points
Minimum movement vector described in amount, the largest motion vector sum, calculates the parallax information of the target pixel points, and take the view
The inverse of poor information obtains the initial depth information.
For example, the motion vector v in the x and y direction of the pixel (i, j) in dense optical flowxAnd vyQuadratic sum rooting obtains
To the motion vector v of the pixel, the i.e. motion vector of the pixelIf depth information uses the ash of 8bit
Degree figure indicates, then can pass through formulaBy the motion vectors mapping of the pixel to [0,255] it
Between, and then the parallax information of the pixel is obtained, and take the inverse of parallax information as depth information.
In addition, as shown in fig. 6, message processing module 404 may further include:
Cavity repairing unit 601, for being repaired to the hole region in the initial depth information；
In the specific implementation, small hole region that can be few for pixel carries out empty repairing using median filtering, to picture
Macroscopic-void region more than vegetarian refreshments carries out empty repairing using image repair algorithm.
Information filter unit 602 is used for according to the video frame images information, described in after the repairing hole region
Initial depth information is filtered；
In the specific implementation, current pixel point and the current pixel point are supported in the available video frame images information
Physical space range information and color space range information between pixel in window；According to the physical space distance letter
The initial depth information after breath, the color space range information and the repairing hole region, determines by filtering
The initial depth information for the current pixel point that treated.
For example, can the initial depth information described in the guidance two-sided filter using Weighted Coefficients after hole region into
Row filtering, the guidance two-sided filter of Weighted Coefficients can effectively reduce the noise in depth information and avoid causing object edge
Part is fuzzy on a large scale.The guidance two-sided filter of Weighted Coefficients utilizes video frame images information and the depth after cavity is repaired
Spend depth information of the information as input, after output fining.The calculation formula of the guidance two-sided filter of Weighted Coefficients:
P is current pixel point
Coordinate, s are the coordinate of the pixel in p current pixel point support window, and I is video frame images information, DsFor the depth of input
Information, DpFor the depth information of output, N is the size of support window, and w (p, s) is current pixel point and the current pixel point
Physical space range information between pixel in support window, c (Ip,Is) it is current pixel point and the current pixel point
Color space range information between pixel in support window, Rs are weight.
In addition, guiding bilateral filtering if the Rs value in above-mentioned calculation formula takes 1, calculation formula is
S is current pixel point coordinate, and q is the pixel in s support window
Coordinate,The depth difference of s, q are represented, Ms is the mask of speckle regions, and speckle regions threshold value is 0, non-speckle regions threshold value
It is 1, the pixel color and depth information difference in speckle regions expression region are small, but speckle regions very little, typically represent
Noise, therefore the Rs value of speckle regions should be set to 0, that is, the influence of noise is removed, when the pixel and current picture in support window
Vegetarian refreshments is more similar in an object area (color difference and depth difference very little), then Rs value is bigger, and vice versa.
Edge compensation unit 603 is obtained for carrying out edge compensation to the initial depth information after filtering processing
To the ultimate depth information.
In the specific implementation, the initial depth information and warp of the available current pixel point after filtering processing
The initial depth information of the pixel in the current pixel point support window before crossing filtering processing；According to by being filtered
In the initial depth information of the current pixel point afterwards and the current pixel point support window before process filtering processing
Pixel initial depth information, edge compensation is carried out to the initial depth information of the current pixel point.
For example, although the guidance bilateral filtering of Weighted Coefficients can carry out smoothly flat site while retaining marginal information
Excessively, edge blurry can be caused when the color difference but in foreground object and background object is small or depth difference is big, therefore is needed
Edge compensation is carried out to the initial depth information of the current pixel point, formula is as follows:
Wherein, p is current pixel point coordinate in edge blurry region, and s is that the pixel within the scope of p support window is sat
Mark compensates the depth information of the current pixel p in edge blurry region bilateral using the guidance of Weighted Coefficients for current pixel
Using between the depth information before the guidance bilateral filtering of Weighted Coefficients within the scope of the depth information and support window obtained after filtering
Difference the depth value for square being minimized corresponding position v.Such as: depth information local value is converted by light stream motion vector
ForIt is obtained after the guidance bilateral filtering of Weighted Coefficients guarantor side is smoothWherein, in the 2nd matrix
1 support window within the scope of (surrounding) include 0,0,3 three pixel in the 1st matrix, 1 with square of the difference of numerical value 0
Be 1,1 with the difference of numerical value 3 square it is 4, therefore is squared and is mended for 0 in 1 corresponding 1st matrix as depth information
It repays, successively finally obtaining the preferable step local value of marginal information in this way is
In embodiments of the present invention, video frame images information is obtained first, then extracts the edge wheel in first frame image
Wide information, and the first frame image information is matched to obtain sparse optical flow with the second frame image information；Secondly according to edge wheel
The dense optical flow between the first frame image information and the second frame image information is calculated in wide information and sparse optical flow；Finally will
Dense optical flow is converted into initial depth information, and obtains ultimate depth according to initial depth information and video frame images information processing
Information, to improve the accuracy of extraction of depth information.
Referring to FIG. 7, Fig. 7 is the structural schematic diagram for another extraction of depth information device that the embodiment of the present invention proposes.
As shown in fig. 7, the device includes processor 701 and interface circuit 702, memory 703 and bus 704 are given in figure, it should
Processor 701, interface circuit 702 and memory 703 connect by bus 704 and complete mutual communication.
Wherein, processor 701 is for performing the following operations step:
Video frame images information is obtained, the video frame images information includes first frame image information and the second frame image letter
Breath；
Extract the edge contour information in the first frame image, and by first frame image information and second frame
Image information is matched to obtain sparse optical flow；
According to the edge contour information and the sparse optical flow, first frame image information and described the is calculated
Dense optical flow between two frame image informations；
Initial depth information is converted by the dense optical flow, and according to the initial depth information and the video frame figure
As information processing obtains ultimate depth information.
Wherein, processor 701 is for performing the following operations step:
According to the edge contour information, cost distance is minimum between the determining target pixel points with the dense optical flow
N number of pixel in the multiple pixel, N are the integer greater than 1；
According to the distance of the Optic flow information of each pixel in N number of pixel and preset each pixel
Weight calculates the Optic flow information of the target pixel points.
Wherein, processor 701 is for performing the following operations step:
Calculate the Optic flow information of each pixel at a distance from each pixel the sum of products of weight and
Each pixel apart from weights sum；
Ratio is calculated as the light of the target pixel points apart from weights sum divided by described in the sum of products
Stream information.
Wherein, processor 701 is for performing the following operations step:
Acquire the video frame images information respectively under a variety of pixel resolutions；
According under the various pixel resolutions the edge contour information and the sparse optical flow, be calculated multiple
Dense optical flow component；
The multiple dense optical flow component is merged to obtain the dense optical flow.
Wherein, processor 701 is for performing the following operations step:
Calculate pixel in the motion vector and the video frame images information of the target pixel points in the dense optical flow
The largest motion vector sum minimum movement vector of point；
According to minimum movement vector described in the motion vector of the target pixel points, the largest motion vector sum, calculate
The parallax information of the target pixel points, and the inverse of the parallax information is taken to obtain the initial depth information.
Wherein, processor 701 is for performing the following operations step:
Hole region in the initial depth information is repaired；
According to the video frame images information, the initial depth information after the repairing hole region is filtered
Wave；
Edge compensation is carried out to the initial depth information after filtering processing and obtains the ultimate depth information.
Wherein, processor 701 is for performing the following operations step:
Obtain the pixel in the video frame images information in current pixel point and the current pixel point support window
Between physical space range information and color space range information；
After the physical space range information, the color space range information and the repairing hole region
The initial depth information determines the initial depth information of the current pixel point after filtering processing.
Wherein, processor 701 is for performing the following operations step:
Before obtaining the initial depth information of the current pixel point after filtering processing and passing through filtering processing
The initial depth information of pixel in the current pixel point support window；
Before according to the initial depth information of the current pixel point after filtering processing and by filtering processing
The initial depth information of pixel in the current pixel point support window, to the initial depth of the current pixel point
Information carries out edge compensation.
It should be noted that processor 701 here can be a processing element, it is also possible to multiple processing elements
It is referred to as.For example, the processing element can be central processing unit (Central Processing Unit, CPU), it is also possible to spy
Determine integrated circuit (Application Specific Integrated Circuit, ASIC),
The device can also include input/output unit, be connected to bus 704, to wait it by bus and processor 701
Its part connects.The input/output unit can provide an input interface for operator, so that operator passes through the input
Interface selects item of deploying to ensure effective monitoring and control of illegal activities, and can also be other interfaces, can pass through the external other equipment of the interface.
Or be arranged to implement one or more integrated circuits of the embodiment of the present invention, such as: it is one or more micro-
Processor (digital singnal processor, DSP), or, one or more field programmable gate array (Field
Programmable Gate Array, FPGA).
Memory 703 can be a storage device, be also possible to the general designation of multiple memory elements, and for storing and can hold
Parameter, data required for line program code or application program running gear are run etc..And memory 703 may include random storage
Device (RAM) also may include nonvolatile memory (non-volatile memory), such as magnetic disk storage, flash memory
(Flash) etc..
It is total that bus 704 can be industry standard architecture (Industry Standard Architecture, ISA)
Line, external equipment interconnection (Peripheral Component, PCI) bus or extended industry-standard architecture (Extended
Industry Standard Architecture, EISA) bus etc..It is total that the bus 704 can be divided into address bus, data
Line, control bus etc..Only to be indicated with a thick line in Fig. 7, it is not intended that an only bus or a type convenient for indicating
The bus of type.
It should be noted that for simple description, therefore, it is stated as a systems for each embodiment of the method above-mentioned
The combination of actions of column, but those skilled in the art should understand that, the present invention is not limited by the sequence of acts described, because
For according to the present invention, certain some step can be performed in other orders or simultaneously.Secondly, those skilled in the art also should
Know, the embodiments described in the specification are all preferred embodiments, related actions and modules not necessarily this hair
Necessary to bright.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, is not described in some embodiment
Part, reference can be made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of above-described embodiment is can
It is completed with instructing relevant hardware by program, which can be stored in a computer readable storage medium, storage
Medium may include: flash disk, read-only memory (English: Read-Only Memory, abbreviation: ROM), random access device (English
Text: Random Access Memory, referred to as: RAM), disk or CD etc..
It is provided for the embodiments of the invention content download method above and relevant device, system are described in detail,
Used herein a specific example illustrates the principle and implementation of the invention, and the explanation of above embodiments is only used
In facilitating the understanding of the method and its core concept of the invention；At the same time, for those skilled in the art, according to the present invention
Thought, there will be changes in the specific implementation manner and application range, in conclusion the content of the present specification should not be construed as
Limitation of the present invention.
Claims (13)
1. a kind of extraction of depth information method, which is characterized in that the described method includes:
Video frame images information is obtained, the video frame images information includes first frame image information and the second frame image information；
Extract the edge contour information in the first frame image, and by first frame image information and the second frame image
Information is matched to obtain sparse optical flow；
According to the edge contour information and the sparse optical flow, first frame image information and second frame is calculated
Dense optical flow between image information；
Calculate pixel in the motion vector and the video frame images information of the target pixel points in the dense optical flow
Largest motion vector sum minimum movement vector；According to the motion vector of the target pixel points, the largest motion vector sum institute
Minimum movement vector is stated, the parallax information of the target pixel points is calculated, and the inverse of the parallax information is taken to obtain initial depth
Spend information, wherein the parallax informationV (i, j) is the motion vector of the target pixel points,
vmaxFor the largest motion vector, vminFor the minimum movement vector, N is positive integer；
Hole region in the initial depth information is repaired；According to the video frame images information, described in repairing
The initial depth information after hole region is filtered；Side is carried out to the initial depth information after filtering processing
Edge compensates to obtain ultimate depth information.
2. the method as described in claim 1, which is characterized in that the sparse optical flow includes the Optic flow information of multiple pixels,
It is described according to the edge contour information and the sparse optical flow, first frame image information and second frame is calculated
Dense optical flow between image information includes:
According to the edge contour information, between the determining target pixel points with the dense optical flow described in cost distance minimum
N number of pixel in multiple pixels, N are the integer greater than 1；Wherein, the edge contour information is the video frame images
The edge contour of moving object in information, the same motion layer for partly belonging to the same object within the edge contour,
Other motion layers for partly belonging to other objects except the edge contour, the same object is same within the edge contour
Cost is apart from small between the pixel of one motion layer, and the cost between other pixels except the edge contour away from
From big；
It is weighed according to the distance of the Optic flow information of each pixel in N number of pixel and preset each pixel
Value, calculates the Optic flow information of the target pixel points.
3. method according to claim 2, which is characterized in that the light according to each pixel in N number of pixel
Stream information and preset each pixel apart from weight, the Optic flow information for calculating the target pixel points includes:
Calculate the Optic flow information of each pixel sum of products of weight and described at a distance from each pixel
Each pixel apart from weights sum；
Ratio is calculated as the light stream of target pixel points letter apart from weights sum divided by described in the sum of products
Breath.
4. the method as described in claim 1, which is characterized in that the acquisition video frame images information includes:
Acquire the video frame images information respectively under a variety of pixel resolutions；
It is described according to the edge contour information and the sparse optical flow, first frame image information and described the is calculated
Dense optical flow between two frame image informations includes:
According under the various pixel resolutions the edge contour information and the sparse optical flow, be calculated multiple dense
Optical flow components；
The multiple dense optical flow component is merged to obtain the dense optical flow.
5. the method as described in claim 1, which is characterized in that it is described according to the video frame images information, described in repairing
The initial depth information after hole region, which is filtered, includes:
It obtains between the pixel in the video frame images information in current pixel point and the current pixel point support window
Physical space range information and color space range information；
According to after the physical space range information, the color space range information and the repairing hole region
Initial depth information determines the initial depth information of the current pixel point after filtering processing.
6. method as claimed in claim 5, which is characterized in that the described pair of initial depth information after filtering processing
Progress edge compensation obtains the ultimate depth information and includes:
Described in obtaining before the initial depth information and process filtering processing of the current pixel point after filtering processing
The initial depth information of pixel in current pixel point support window；
According to before the initial depth information of the current pixel point after filtering processing and process filtering processing
The initial depth information of pixel in current pixel point support window, to the initial depth information of the current pixel point
Carry out edge compensation.
7. a kind of extraction of depth information device, which is characterized in that described device includes:
Data obtaining module, for obtaining video frame images information, the video frame images information includes first frame image information
With the second frame image information；
Information matches module is believed for extracting the edge contour information in the first frame image, and by the first frame image
Breath is matched to obtain sparse optical flow with second frame image information；
Information computational module, for the first frame figure to be calculated according to the edge contour information and the sparse optical flow
As the dense optical flow between information and second frame image information；
Message processing module, for calculate the target pixel points in the dense optical flow motion vector and the video frame
The largest motion vector sum minimum movement vector of pixel in image information；According to the motion vector of the target pixel points, institute
Minimum movement vector described in largest motion vector sum is stated, the parallax information of the target pixel points is calculated, and the parallax is taken to believe
The inverse of breath obtains initial depth information；Wherein, the parallax informationV (i, j) is the target
The motion vector of pixel, vmaxFor the largest motion vector, vminFor the minimum movement vector, N is positive integer；
The message processing module is also used to repair the hole region in the initial depth information；According to the view
Frequency frame image information is filtered the initial depth information after the repairing hole region；To after filtering processing
The initial depth information carry out edge compensation obtain ultimate depth information.
8. device as claimed in claim 7, which is characterized in that the sparse optical flow includes the Optic flow information of multiple pixels,
The information computational module includes:
Pixel determination unit, for according to the edge contour information, the determining target pixel points with the dense optical flow
Between N number of pixel in the minimum the multiple pixel of cost distance, N is the integer greater than 1；Wherein, the edge contour
Information is the edge contour of moving object in the video frame images information, and partly belonging within the edge contour is same
The same motion layer of object, other motion layers for partly belonging to other objects except the edge contour, at the edge
Within profile between the pixel of the same same motion layer of object cost apart from small, and with its except the edge contour
Cost distance between his pixel is big；
Information calculating unit, for according to the Optic flow information of each pixel in N number of pixel and preset described every
A pixel apart from weight, calculate the Optic flow information of the target pixel points.
9. device as claimed in claim 8, which is characterized in that the information calculating unit is specifically used for:
Calculate the Optic flow information of each pixel sum of products of weight and described at a distance from each pixel
Each pixel apart from weights sum；
Ratio is calculated as the light stream of target pixel points letter apart from weights sum divided by described in the sum of products
Breath.
10. device as claimed in claim 7, which is characterized in that
The data obtaining module is also used to acquire the video frame images information respectively under a variety of pixel resolutions；
The information computational module is also used to according to the edge contour information under the various pixel resolutions and described dilute
Light stream is dredged, multiple dense optical flow components are calculated；The multiple dense optical flow component is merged to obtain the dense light
Stream.
11. device as claimed in claim 7, which is characterized in that
The message processing module is also used to obtain current pixel point and the current pixel point in the video frame images information
Physical space range information and color space range information between pixel in support window；According to the physical space away from
From the initial depth information after information, the color space range information and the repairing hole region, determines and pass through
The initial depth information of the current pixel point after filtering processing.
12. device as claimed in claim 11, which is characterized in that
The message processing module, be also used to obtain by filtering processing after the current pixel point initial depth information with
And the initial depth information by the pixel in the current pixel point support window before filtering processing；According to by filtering
The initial depth information for the current pixel point that treated and the current pixel point passed through before filtering processing support window
The initial depth information of pixel in mouthful carries out edge compensation to the initial depth information of the current pixel point.
13. a kind of extraction of depth information device, which is characterized in that described device includes interface circuit, memory and processor,
Wherein, batch processing code is stored in memory, and processor is used to call the program code stored in memory, for executing
It operates below:
Video frame images information is obtained, the video frame images information includes first frame image information and the second frame image information；
Extract the edge contour information in the first frame image, and by first frame image information and the second frame image
Information is matched to obtain sparse optical flow；
According to the edge contour information and the sparse optical flow, first frame image information and second frame is calculated
Dense optical flow between image information；
Calculate pixel in the motion vector and the video frame images information of the target pixel points in the dense optical flow
Largest motion vector sum minimum movement vector；According to the motion vector of the target pixel points, the largest motion vector sum institute
Minimum movement vector is stated, the parallax information of the target pixel points is calculated, and the inverse of the parallax information is taken to obtain initial depth
Spend information；Wherein, the parallax informationV (i, j) is the motion vector of the target pixel points,
vmaxFor the largest motion vector, vminFor the minimum movement vector, N is positive integer；
Hole region in the initial depth information is repaired；According to the video frame images information, described in repairing
The initial depth information after hole region is filtered；Side is carried out to the initial depth information after filtering processing
Edge compensates to obtain ultimate depth information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610382517.0A CN107465911B (en) | 2016-06-01 | 2016-06-01 | A kind of extraction of depth information method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610382517.0A CN107465911B (en) | 2016-06-01 | 2016-06-01 | A kind of extraction of depth information method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107465911A CN107465911A (en) | 2017-12-12 |
| CN107465911B true CN107465911B (en) | 2019-03-15 |
Family
ID=60545294
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610382517.0A Active CN107465911B (en) | 2016-06-01 | 2016-06-01 | A kind of extraction of depth information method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107465911B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108234988A (en) * | 2017-12-28 | 2018-06-29 | 努比亚技术有限公司 | Parallax drawing generating method, device and computer readable storage medium |
| CN110751672B (en) * | 2018-07-24 | 2022-06-21 | 深圳地平线机器人科技有限公司 | Method and apparatus for implementing multi-scale optical flow pixel transform using dilution convolution |
| CN109242811B (en) * | 2018-08-16 | 2021-09-17 | 广州视源电子科技股份有限公司 | Image alignment method and device, computer readable storage medium and computer equipment |
| CN110264455B (en) * | 2019-06-19 | 2021-07-23 | 北京市商汤科技开发有限公司 | Image processing method, neural network training method, device and storage medium |
| TWI788758B (en) * | 2021-01-15 | 2023-01-01 | 國立臺北科技大學 | Target image tracking system and method |
| CN113033552B (en) * | 2021-03-19 | 2024-02-02 | 北京字跳网络技术有限公司 | Text recognition method and device and electronic equipment |
| CN113793272B (en) * | 2021-08-11 | 2024-01-26 | 东软医疗系统股份有限公司 | Image noise reduction method and device, storage medium and terminal |
| CN114089330B (en) * | 2022-01-18 | 2022-05-20 | 北京航空航天大学 | Indoor mobile robot glass detection and map updating method based on depth image restoration |
| CN115190286B (en) * | 2022-07-06 | 2024-02-27 | 敏捷医疗科技(苏州)有限公司 | 2D image conversion method and device |
| CN116228834B (en) * | 2022-12-20 | 2023-11-03 | 阿波罗智联(北京)科技有限公司 | Image depth acquisition method and device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101699512A (en) * | 2009-10-30 | 2010-04-28 | 无锡景象数字技术有限公司 | Depth generating method based on background difference sectional drawing and sparse optical flow method |
| CN101710426A (en) * | 2009-10-30 | 2010-05-19 | 无锡景象数字技术有限公司 | Method for tracking depth image |
| CN102542601A (en) * | 2010-12-10 | 2012-07-04 | 三星电子株式会社 | Equipment and method for modeling three-dimensional (3D) object |
| CN104318569A (en) * | 2014-10-27 | 2015-01-28 | 北京工业大学 | Space salient region extraction method based on depth variation model |
| CN104766275A (en) * | 2014-01-02 | 2015-07-08 | 株式会社理光 | Method and device for making sparse disparity map dense |
| CN103049906B (en) * | 2012-12-07 | 2015-09-30 | 清华大学深圳研究生院 | A kind of image depth extracting method |
| CN105590327A (en) * | 2014-10-24 | 2016-05-18 | 华为技术有限公司 | Motion estimation method and apparatus |
| CN106256124A (en) * | 2014-04-24 | 2016-12-21 | 英特尔公司 | Structuring is three-dimensional |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9153027B2 (en) * | 2013-11-01 | 2015-10-06 | Nvidia Corporation | System, method, and computer program product for performing fast, non-rigid registration for high dynamic range image stacks |
-
2016
- 2016-06-01 CN CN201610382517.0A patent/CN107465911B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101699512A (en) * | 2009-10-30 | 2010-04-28 | 无锡景象数字技术有限公司 | Depth generating method based on background difference sectional drawing and sparse optical flow method |
| CN101710426A (en) * | 2009-10-30 | 2010-05-19 | 无锡景象数字技术有限公司 | Method for tracking depth image |
| CN102542601A (en) * | 2010-12-10 | 2012-07-04 | 三星电子株式会社 | Equipment and method for modeling three-dimensional (3D) object |
| CN103049906B (en) * | 2012-12-07 | 2015-09-30 | 清华大学深圳研究生院 | A kind of image depth extracting method |
| CN104766275A (en) * | 2014-01-02 | 2015-07-08 | 株式会社理光 | Method and device for making sparse disparity map dense |
| CN106256124A (en) * | 2014-04-24 | 2016-12-21 | 英特尔公司 | Structuring is three-dimensional |
| CN105590327A (en) * | 2014-10-24 | 2016-05-18 | 华为技术有限公司 | Motion estimation method and apparatus |
| CN104318569A (en) * | 2014-10-27 | 2015-01-28 | 北京工业大学 | Space salient region extraction method based on depth variation model |
Non-Patent Citations (1)
| Title |
|---|
| Dense depth map generation using sparse depth data from normal flow;Tak-Wai Hui; King Ngi Ngan;《2014 IEEE International Conference on Image Processing (ICIP)》;20150129;第3837-3841页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107465911A (en) | 2017-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107465911B (en) | A kind of extraction of depth information method and device | |
| US10846836B2 (en) | View synthesis using deep convolutional neural networks | |
| CN111598998B (en) | Three-dimensional virtual model reconstruction method, three-dimensional virtual model reconstruction device, computer equipment and storage medium | |
| CN110287846A (en) | A kind of face critical point detection method based on attention mechanism | |
| CN104008538B (en) | Based on single image super-resolution method | |
| CN107292804B (en) | Direct more exposure fusion parallel acceleration methods based on OpenCL | |
| CN107316286B (en) | Method and device for synchronously synthesizing and removing rain and fog in image | |
| CN113674421B (en) | 3D target detection method, model training method, related device and electronic equipment | |
| CN112581370A (en) | Training and reconstruction method of super-resolution reconstruction model of face image | |
| CN109255758A (en) | Image enchancing method based on full 1*1 convolutional neural networks | |
| CN107506792B (en) | Semi-supervised salient object detection method | |
| CN111179196B (en) | Multi-resolution depth network image highlight removing method based on divide-and-conquer | |
| CN110942484B (en) | Camera self-motion estimation method based on occlusion perception and feature pyramid matching | |
| CN112699885A (en) | Semantic segmentation training data augmentation method and system based on antagonism generation network GAN | |
| CN107766803B (en) | Video character decorating method and device based on scene segmentation and computing equipment | |
| CN112528782A (en) | Underwater fish target detection method and device | |
| CN114170290A (en) | Image processing method and related equipment | |
| CN109255773A (en) | Different resolution ratio based on full variation is infrared with visible light image fusion method and system | |
| CN109978897A (en) | A kind of multiple dimensioned heterologous remote sensing image registration method and device for generating confrontation network | |
| JP2023003763A (en) | Learning apparatus, image processing apparatus, learning processing method, and program | |
| CN112489144A (en) | Image processing method, image processing apparatus, terminal device, and storage medium | |
| CN116912645A (en) | Three-dimensional target detection method and device integrating texture and geometric features | |
| Liu et al. | Non-homogeneous haze data synthesis based real-world image dehazing with enhancement-and-restoration fused CNNs | |
| CN110717879A (en) | Multi-focus image processing method and device, storage medium and electronic equipment | |
| CN110766631A (en) | Face image modification method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |