CN103024420A - 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration - Google Patents
2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration Download PDFInfo
- Publication number
- CN103024420A CN103024420A CN2013100163339A CN201310016333A CN103024420A CN 103024420 A CN103024420 A CN 103024420A CN 2013100163339 A CN2013100163339 A CN 2013100163339A CN 201310016333 A CN201310016333 A CN 201310016333A CN 103024420 A CN103024420 A CN 103024420A
- Authority
- CN
- China
- Prior art keywords
- depth
- image
- original
- images
- rgbd
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention relates to a 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration. The method includes: retrieving K neighbor images in original images and corresponding depths from an RGBD data on internet, and calculating matching cost of the images to be matched; establishing denseness correspondence between the original images and the neighbor images by image matching; migrating the corresponding depths of the K neighbor images in the original 2D images to an original 2D image plane according to the denseness correspondence; selecting an optimal depth from the K migrating depth images as an initial estimated depth; post-processing the fused depth images to further improve estimated quality of the depth images and suppress noise; and subjecting the depth images to nonlinear processing according to visual significance of the original 2D images, and synthetizing 3D images. The method has the advantages high scenario adaptability, high depth estimation quality and simplicity and quickness in calculation.
Description
Technical field
The present invention relates to 2D and turn the 3D technology, relate in particular to a kind of single image 2D and turn the 3D method, the single image 2D that refers in particular to a kind of RGBD data depth migration turns the 3D method.
Background technology
The 3D video is the expansion of traditional 2D video, by increasing depth information, makes the user experience third dimension and the telepresenc of video content.The 3D video relates to the technology such as 3D content production, transmission, storage, broadcast and display device as the developing direction of video of future generation, has important practical significance for developing national economy.3D is devoid of matter to be one of principal element of the current 3D video development of restriction, and wanting to create a collection of higher quality that has still has larger difficulty with the 3D program that meets 3D displaying medium demand.But turning the 3D method, single image 2D can will have the 3D video that magnanimity 2D video resource is converted to stereo display now with lower cost within a short period of time, can alleviate current 3D situation devoid of matter, can solve again and utilize the 3D capture apparatus directly to make the expensive of 3D content and operation inconvenience.
The core missions that 2D turns the 3D method are to extract the depth information of scenery from the 2D image.Be subjected to the inspiration of human vision monocular depth reasoning, the researcher has proposed multiple theory and the method for utilizing visual cues to carry out the monocular image estimation of Depth, such as perspective, texture, shading value, degree of focus etc.Monocular image estimation of Depth principle based on perspective is that parallel lines will meet at a bit behind perspective imaging, be called vanishing point (vanishing point), corresponding line is called the line that goes out (vanishing line), according to the direction that the line that goes out intersects entire image is carried out depth assignment.The move the camera to follow the subject's movement position relationship of camera and grain surface of the geometric shape of texture primitive is relevant on the image, thereby also can have an X-rayed convergent-divergent (relevant with surface direction) and variable density effect (with the distance dependent of observer and texture primitive) is carried out estimation of Depth according to texture.Under the supposed premise of Lambert surface, the body surface gray scale thereby can the shading value by the object analysis surface change the degree of depth that estimate object along with surface normal variation under light illumination.The depth of focus is extracted and to be considered only to be in from the camera lens specific location in the low depth scene and just can be focused, and other positions all can produce in various degree fuzzy and fog-level and residing distance dependent.The defective of said method is that versatility is poor, just may obtain reliable depth estimation result when only having scene to comprise corresponding Depth cue; And these methods can only obtain relative depth, are difficult to obtain accurate depth information.Thereby for promoting the development of 3D video, turn the 3D method in the urgent need to designing a kind of high-quality 2D with scene universality.
Along with the popular of Kinect and popularization, magnanimity RGBD data have been assembled on the Internet.On the other hand, Content-based image retrieval is ripe.Experience is told us, and similar scene has the similar degree of depth.Thereby according to scene matching, the degree of depth of inquiry input scene becomes possibility from magnanimity RGBD database.
Summary of the invention
Technical problem to be solved by this invention is the present situation for prior art, provides the scene applicability strong, and the estimation of Depth quality is high, and the single image 2D that calculates a kind of RGBD data depth migration of Simple fast turns the 3D method.
The present invention solves the problems of the technologies described above the technical scheme that adopts:
A kind of single image 2D of RGBD data depth migration turns the 3D method, comprises following:
The calculating of step 1, global description's symbol: by calculating global description's symbol of image;
Step 2, retrieval neighbour image: in the Internet RGBD database, retrieve candidate image according to described global description symbol, and calculate its coupling cost, obtain from small to large K neighbour's image and the corresponding degree of depth thereof of original 2D image according to the coupling cost that calculates, wherein K is positive integer;
Step 3, set up dense corresponding relation: set up dense corresponding relation between original 2D image and K the neighbour's image by the sheet coupling;
Step 4, depth migration: according to dense corresponding relation with the corresponding depth migration of K neighbour's image of original 2D image to the original 2D plane of delineation;
Step 5, depth map merge: for each pixel of original 2D image, choose the degree of depth an of the best as its initial estimation degree of depth from the depth map of K migration;
Step 6, depth map reprocessing: for further improving the depth map estimated quality and suppressing noise, the depth map after merging is carried out reprocessing;
Synthesizing of step 7,3D rendering: the visual saliency according to original 2D image carries out Nonlinear Processing to depth map, synthetic 3D rendering.
The technical measures of optimizing also comprise:
The GIST feature is adopted in the calculating of above-mentioned global description's symbol.
The computing formula of above-mentioned coupling cost is:
Wherein Q is original 2D image, the to be matched image of C for retrieving from the RGBD database, the GIST characteristic vector of G (x) presentation video.
The computing formula of the coupling cost of above-mentioned sheet coupling is:
Above-mentioned sheet coupling adopts random search and neighborhood transmission to improve computational speed.
The formula of above-mentioned depth migration is:
Wherein
f(x) be pixel x among the Q (
i,
j) correspondence position in image C to be matched;
d C Be depth map corresponding to image C to be matched.
The formula that above-mentioned depth map merges is:
Wherein fusion function adopts medium filtering or weighted average filtering.
The formula of above-mentioned depth map reprocessing is:
Wherein
w x ,
w c With
w d The weight that represents respectively position, color and three components of the degree of depth.
The formula of above-mentioned Nonlinear Processing is:
Wherein d represents through the depth map behind the nonlinear adjustment;
s xRepresent original 2D image Q at the visual saliency coefficient at position x place, its value is positioned at interval [0 1], and N (x) is the neighborhood of x place pixel.
The synthetic employing DIBR technology of above-mentioned 3D rendering.
Compared with prior art, the single image 2D of a kind of RGBD data depth migration of the present invention turns the 3D method, by from the RGBD database of the Internet, retrieves K neighbour's image and the corresponding degree of depth thereof of original image, and calculates the coupling cost of image to be matched; Mate the dense corresponding relation of setting up between original image and the neighbour's image by sheet again; Again according to dense corresponding relation with the corresponding depth migration of K neighbour's image of original 2D image to the original 2D plane of delineation; From the depth map of K migration, choose the degree of depth an of the best as its initial estimation degree of depth; For further improving the depth map estimated quality and suppressing noise, the depth map after merging is carried out reprocessing; Visual saliency according to original 2D image carries out Nonlinear Processing to depth map at last, synthetic 3D rendering.Nowadays magnanimity RGBD database existing and that continue to increase has guaranteed for any original 2D image on the Internet, can retrieve on the internet neighbour's image as image to be matched, thereby guaranteed versatility of the present invention, be applicable to all kinds of complicated shooting environmental and scene; The present invention utilizes the RGBD data that depth transducer gathers on the Internet, can obtain the accurate depth information of image, turns the 3D method unlike traditional 2D and only obtains relative depth information; Because the dense matching that the present invention adopts random search and neighborhood transmission to set up between original 2D image and the image to be matched concerns, has significantly reduced the nearest neighbor search space, thereby has improved computational speed; On the other hand, Local Structure of Image and scenery composition relation have been considered in the neighborhood transmission, and the alignment of target has improved the depth migration quality between the easier assurance scene; According to visual saliency depth map is carried out nonlinear adjustment when using the synthetic 3D figure of DIBR technology, guaranteed the three-dimensional telepresenc of visual salient region.
Description of drawings
Fig. 1 is flow chart of the present invention.
Embodiment
Embodiment is described in further detail the present invention below in conjunction with accompanying drawing.
Be illustrated in figure 1 as flow chart of the present invention,
The calculating of step 1, global description's symbol: the global description's symbol that adopts the original 2D image of GIST feature calculation, its process is: at first, the Gabor filter that creates different directions and yardstick carries out filtering to image and obtains several characteristic patterns, and each characteristic pattern is divided into 4 * 4 piece; Secondly, the mean value of each piece in the calculated characteristics figure; At last, the mean value of piece in all characteristic patterns is arranged in order GIST feature as original 2D image, global description's symbol that this GIST feature is original 2D image.
Step 2, retrieval neighbour image: retrieve candidate image the magnanimity RGBD database of global description's symbol from the Internet according to original 2D image, utilize following formula to calculate the coupling cost of original 2D image and image to be matched;
Wherein Q is original 2D image, the to be matched image of C for retrieving from the RGBD database, G (x) expression GIST characteristic vector; Then, will mate cost sorts from small to large; At last, therefrom choose from small to large K as the candidate matches scene, its corresponding degree of depth is as candidate's estimating depth of Q; Wherein K is positive integer.
Step 3, set up dense corresponding relation: set up dense corresponding relation between original 2D image and K the neighbour's image by the sheet coupling.Detailed process is: the first step, each pixel among Q and the C is considered as the upper left corner coordinate of a W * W sheet, and to each such sheet among the Q, specify the sheet of its correspondence in C with the method for random pair, the pairing cost of two sheets is calculated with following formula:
Second step will be worked as the displacement of anter coupling to the neighborhood transmission, if the pairing cost of this transmission less than the neighborhood sheet, then keeps and adjust the coupling of neighborhood sheet when the anter coupling, work as anter otherwise the coupling of neighborhood sheet is delivered to; In the 3rd step, take current matching as starting point, in a large window, make minimum coupling as the optimum Match when anter with the index step length searching; In the 4th step, do not return second step if iteration reaches predetermined number of times, otherwise withdraw from.
Step 4, depth migration: the dense corresponding relation of setting up according to sheet coupling with K neighbour's image correspondence depth migration of original 2D image to original 2D image.The depth migration formula is:
Pixel x among the Q (
i,
j) correspondence position is in matching image C
f(x).Adopt following formula with matching image
CCorresponding depth map
d C Move on the original 2D plane of delineation, as its candidate's estimation of Depth.
The 5th step, depth map merge: for each pixel of original 2D image, choose the degree of depth an of the best as its initial estimation degree of depth from the depth map of K migration.Detailed process is shown below:
Adopt simple medium filtering or weighted average filtering in the fusion function.
The 6th step, depth map reprocessing: for further improving the depth map estimated quality and suppressing noise, adopt following formula to carry out the depth map reprocessing:
Wherein
w x ,
w c With
w d The weight that represents respectively position, color and three components of the degree of depth.
Synthesizing of the 7th step, 3D rendering: original 2D image is considered as left view, according to the degree of depth of estimating original 2D image conversion is arrived new position as right view; In order to guarantee the third dimension of visual salient region, before conversion, by following formula the degree of depth is carried out nonlinear adjustment:
Wherein
dExpression is through the depth map behind the nonlinear adjustment;
s xRepresent original 2D image
QVisual saliency coefficient at position x place, its value are positioned at interval [0 1], and be more more remarkable near 1 expression; N (x) is the neighborhood of x place pixel; Obtain synthesizing 3D rendering by the DIBR technology behind the left and right view.
Claims (10)
1. the single image 2D of a RGBD data depth migration turns the 3D method, it is characterized in that: may further comprise the steps:
The calculating of step 1, global description's symbol: by calculating global description's symbol of image;
Step 2, retrieval neighbour image: in the Internet RGBD database, retrieve candidate image according to described global description symbol, and calculate its coupling cost, obtain from small to large K neighbour's image and the corresponding degree of depth thereof of original 2D image according to the coupling cost that calculates, wherein K is positive integer;
Step 3, set up dense corresponding relation: set up dense corresponding relation between original 2D image and K the neighbour's image by the sheet coupling;
Step 4, depth migration: according to described dense corresponding relation with the corresponding depth migration of K neighbour's image of original 2D image to the original 2D plane of delineation;
Step 5, depth map merge: for each pixel of original 2D image, choose the degree of depth an of the best as its initial estimation degree of depth from the depth map of K migration;
Step 6, depth map reprocessing: for further improving the depth map estimated quality and suppressing noise, the depth map after merging is carried out reprocessing;
Synthesizing of step 7,3D rendering: the visual saliency according to original 2D image carries out Nonlinear Processing to depth map, synthetic 3D rendering.
2. the single image 2D of a kind of RGBD data depth migration according to claim 1 turns the 3D method, it is characterized in that: the GIST feature is adopted in the calculating of described global description symbol.
3. the single image 2D of a kind of RGBD data depth migration according to claim 2 turns the 3D method, and it is characterized in that: the computing formula of described coupling cost is:
Wherein Q is original 2D image, the to be matched image of C for retrieving from the RGBD database, the GIST characteristic vector of G (x) presentation video.
5. the single image 2D of a kind of RGBD data depth migration according to claim 4 turns the 3D method, it is characterized in that: described coupling adopts random search and neighborhood transmission to improve computational speed.
6. the single image 2D of a kind of RGBD data depth migration according to claim 5 turns the 3D method, and it is characterized in that: the formula of described depth migration is:
Wherein
f(x) be pixel x among the Q (
i,
j) correspondence position in image C to be matched;
d C Be depth map corresponding to image C to be matched.
8. the single image 2D of a kind of RGBD data depth migration according to claim 7 turns the 3D method, and it is characterized in that: the formula of described depth map reprocessing is:
Wherein
w x ,
w c With
w d The weight that represents respectively position, color and three components of the degree of depth.
9. the single image 2D of a kind of RGBD data depth migration according to claim 8 turns the 3D method, and it is characterized in that: the formula of described Nonlinear Processing is:
Wherein d represents through the depth map behind the nonlinear adjustment;
s xRepresent original 2D image Q at the visual saliency coefficient at position x place, its value is positioned at interval [0 1], and N (x) is the neighborhood of x place pixel.
10. the single image 2D of a kind of RGBD data depth migration according to claim 9 turns the 3D method, it is characterized in that: the synthetic employing DIBR technology of described 3D rendering.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201310016333.9A CN103024420B (en) | 2013-01-17 | 2013-01-17 | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201310016333.9A CN103024420B (en) | 2013-01-17 | 2013-01-17 | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration |
Publications (2)
Publication Number | Publication Date |
---|---|
CN103024420A true CN103024420A (en) | 2013-04-03 |
CN103024420B CN103024420B (en) | 2014-12-10 |
Family
ID=47972485
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201310016333.9A Expired - Fee Related CN103024420B (en) | 2013-01-17 | 2013-01-17 | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN103024420B (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106548494A (en) * | 2016-09-26 | 2017-03-29 | 浙江工商大学 | A kind of video image depth extraction method based on scene Sample Storehouse |
CN109191506A (en) * | 2018-08-06 | 2019-01-11 | 深圳看到科技有限公司 | Processing method, system and the computer readable storage medium of depth map |
CN110866882A (en) * | 2019-11-21 | 2020-03-06 | 湖南工程学院 | Layered joint bilateral filtering depth map restoration algorithm based on depth confidence |
CN113034385A (en) * | 2021-03-01 | 2021-06-25 | 嘉兴丰鸟科技有限公司 | Grid generating and rendering method based on blocks |
CN115170739A (en) * | 2022-07-11 | 2022-10-11 | 杭州时戳信息科技有限公司 | Vehicle three-dimensional design device based on artificial intelligence |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120127267A1 (en) * | 2010-11-23 | 2012-05-24 | Qualcomm Incorporated | Depth estimation based on global motion |
CN102695070A (en) * | 2012-06-12 | 2012-09-26 | 浙江大学 | Depth consistency fusion processing method for stereo image |
-
2013
- 2013-01-17 CN CN201310016333.9A patent/CN103024420B/en not_active Expired - Fee Related
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120127267A1 (en) * | 2010-11-23 | 2012-05-24 | Qualcomm Incorporated | Depth estimation based on global motion |
CN102695070A (en) * | 2012-06-12 | 2012-09-26 | 浙江大学 | Depth consistency fusion processing method for stereo image |
Non-Patent Citations (3)
Title |
---|
BRANDON M. SMITH,ET AL: "Stereo Matching with Nonparametric Smoothness Priors in Feature Space", 《IEEE XPLORE DIGITAL LIBRARY》, 31 December 2009 (2009-12-31), pages 485 - 492 * |
CONNELLY BARNES,ET AL: "The PatchMatch randomized matching algorithm for image manipulation", 《COMMUNICATIONS OF THE ACM》, vol. 54, no. 11, 30 November 2011 (2011-11-30), pages 103 - 110 * |
J. KONRAD,ET AL: "Automatic 2D-to-3D image conversion using 3D examples from the internet", 《PROC. OF SPIE-IS&T ELECTRONIC IMAGING》, vol. 8288, 31 December 2012 (2012-12-31), pages 1 - 12 * |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106548494A (en) * | 2016-09-26 | 2017-03-29 | 浙江工商大学 | A kind of video image depth extraction method based on scene Sample Storehouse |
WO2018053952A1 (en) * | 2016-09-26 | 2018-03-29 | 浙江工商大学 | Video image depth extraction method based on scene sample library |
CN109191506A (en) * | 2018-08-06 | 2019-01-11 | 深圳看到科技有限公司 | Processing method, system and the computer readable storage medium of depth map |
CN109191506B (en) * | 2018-08-06 | 2021-01-29 | 深圳看到科技有限公司 | Depth map processing method, system and computer readable storage medium |
CN110866882A (en) * | 2019-11-21 | 2020-03-06 | 湖南工程学院 | Layered joint bilateral filtering depth map restoration algorithm based on depth confidence |
CN113034385A (en) * | 2021-03-01 | 2021-06-25 | 嘉兴丰鸟科技有限公司 | Grid generating and rendering method based on blocks |
CN113034385B (en) * | 2021-03-01 | 2023-03-28 | 嘉兴丰鸟科技有限公司 | Grid generating and rendering method based on blocks |
CN115170739A (en) * | 2022-07-11 | 2022-10-11 | 杭州时戳信息科技有限公司 | Vehicle three-dimensional design device based on artificial intelligence |
CN115170739B (en) * | 2022-07-11 | 2023-09-01 | 杭州时戳信息科技有限公司 | Vehicle three-dimensional design device based on artificial intelligence |
Also Published As
Publication number | Publication date |
---|---|
CN103024420B (en) | 2014-12-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN101287143B (en) | Method for converting flat video to tridimensional video based on real-time dialog between human and machine | |
CN103810708B (en) | A kind of laser speckle image depth perception method and device | |
CN103024420B (en) | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration | |
CN102665086B (en) | Method for obtaining parallax by using region-based local stereo matching | |
CN105374039B (en) | Monocular image depth information method of estimation based on contour acuity | |
CN109064505B (en) | Depth estimation method based on sliding window tensor extraction | |
CN115205489A (en) | Three-dimensional reconstruction method, system and device in large scene | |
CN108596975A (en) | A kind of Stereo Matching Algorithm for weak texture region | |
CN106056622B (en) | A kind of multi-view depth video restored method based on Kinect cameras | |
Wu et al. | A novel method for semi-automatic 2D to 3D video conversion | |
CN102496183A (en) | Multi-view stereo reconstruction method based on Internet photo gallery | |
CN104159098B (en) | The translucent edge extracting method of time domain consistence of a kind of video | |
CN104778673B (en) | A kind of improved gauss hybrid models depth image enhancement method | |
CN115601406A (en) | Local stereo matching method based on fusion cost calculation and weighted guide filtering | |
CN103679699B (en) | A kind of based on notable figure translation and the solid matching method of combined measure | |
Fan et al. | Vivid-DIBR based 2D–3D image conversion system for 3D display | |
CN112270701A (en) | Packet distance network-based parallax prediction method, system and storage medium | |
Yang et al. | Depth map generation using local depth hypothesis for 2D-to-3D conversion | |
Sisi et al. | The overview of 2D to 3D conversion system | |
CN110610503A (en) | Three-dimensional information recovery method for power disconnecting link based on stereo matching | |
CN103400380A (en) | Single camera underwater target three-dimensional trace stimulation method merged with image matrix offset | |
CN103559702B (en) | Based on the two-dimensional single-view image depth estimation method of wavelet coefficient entropy | |
CN203027382U (en) | RGBD (Red/Green/Blue Data) depth migration equipment for converting single 2D (Two-Dimensional) images into 3D (Three-Dimensional) images | |
CN102595167B (en) | Depth uniformization method and device for 2D/3D video conversion | |
Akimov et al. | Single-image depth map estimation using blur information |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C53 | Correction of patent of invention or patent application | ||
CB02 | Change of applicant information |
Address after: 315211 Zhejiang Province, Ningbo Jiangbei District Fenghua Road No. 201 Applicant after: Ningbo University of Technology Address before: 315016 Zhejiang Province, Ningbo city Haishu District Cuibai Road No. 89 Ningbo University of Technology College of electronic and Information Engineering Applicant before: Ningbo University of Technology |
|
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20141210 Termination date: 20160117 |
|
EXPY | Termination of patent right or utility model |