CN103024420B - 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration - Google Patents
2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration Download PDFInfo
- Publication number
- CN103024420B CN103024420B CN201310016333.9A CN201310016333A CN103024420B CN 103024420 B CN103024420 B CN 103024420B CN 201310016333 A CN201310016333 A CN 201310016333A CN 103024420 B CN103024420 B CN 103024420B
- Authority
- CN
- China
- Prior art keywords
- image
- depth
- original
- images
- rgbd
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention relates to a 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration. The method includes: retrieving K neighbor images in original images and corresponding depths from an RGBD data on internet, and calculating matching cost of the images to be matched; establishing denseness correspondence between the original images and the neighbor images by image matching; migrating the corresponding depths of the K neighbor images in the original 2D images to an original 2D image plane according to the denseness correspondence; selecting an optimal depth from the K migrating depth images as an initial estimated depth; post-processing the fused depth images to further improve estimated quality of the depth images and suppress noise; and subjecting the depth images to nonlinear processing according to visual significance of the original 2D images, and synthetizing 3D images. The method has the advantages high scenario adaptability, high depth estimation quality and simplicity and quickness in calculation.
Description
Technical field
The present invention relates to 2D and turn 3D technology, relate in particular to a kind of single image 2D and turn 3D method, the single image 2D that refers in particular to a kind of RGBD data depth migration turns 3D method.
Background technology
3D video is the expansion of traditional 2D video, by increasing depth information, makes user experience third dimension and the telepresenc of video content.3D video, as the developing direction of video of future generation, relates to the technology such as 3D content production, transmission, storage, broadcasting and display device, for developing national economy, has important practical significance.3D is devoid of matter is one of principal element of the current 3D video development of restriction, and wanting to create a collection of higher quality that has still has larger difficulty with the 3D program that meets 3D displaying medium demand.Single image 2D turn 3D method can within a short period of time with lower cost by existing magnanimity 2D video resource be converted to can stereo display 3D video, can alleviate current 3D situation devoid of matter, can solve again and utilize 3D capture apparatus directly to make the expensive of 3D content and operation inconvenience.
The core missions that 2D turns 3D method are from 2D image, to extract the depth information of scenery.Be subject to the inspiration of human vision monocular depth reasoning, researcher has proposed multiple theory and the method for utilizing visual cues to carry out monocular image estimation of Depth, such as perspective, texture, shading value, degree of focus etc.Monocular image estimation of Depth principle based on perspective is that parallel lines will meet at a bit after perspective imaging, be called vanishing point (vanishing point), corresponding line is called the line that goes out (vanishing line), according to the direction that the line that goes out is crossing, entire image is carried out to depth assignment.On image, the move the camera to follow the subject's movement position relationship of camera and grain surface of the geometric shape of texture primitive is relevant, thereby also can have an X-rayed convergent-divergent (relevant with surface direction) and variable density effect (with the distance dependent of observer and texture primitive) is carried out estimation of Depth according to texture.Under the supposed premise of Lambert surface, under light illumination, body surface gray scale, along with surface normal variation, thereby can the shading value by object analysis surface change the degree of depth that estimate object.The depth of focus is extracted and to be considered in low depth scene and be only in from camera lens specific location and just can be focused, and other positions all can produce in various degree fuzzy and fog-level and residing distance dependent.The defect of said method is that versatility is poor, while only having scene to comprise corresponding Depth cue, just may obtain reliable depth estimation result; And these methods can only obtain relative depth, be difficult to obtain accurate depth information.Thereby for promoting the development of 3D video, in the urgent need to designing a kind of high-quality 2D with scene universality, turn 3D method.
Along with the popular of Kinect and popularization, magnanimity RGBD data on the Internet, have been assembled.On the other hand, CBIR technology is ripe.Experience is told us, and similar scene has the similar degree of depth.Thereby according to scene matching, from magnanimity RGBD database, the degree of depth of inquiry input scene becomes possibility.
Summary of the invention
Technical problem to be solved by this invention is the present situation for prior art, provides scene applicability strong, and estimation of Depth quality is high, and the single image 2D that calculates simple and quick a kind of RGBD data depth migration turns 3D method.
The present invention solves the problems of the technologies described above adopted technical scheme:
The single image 2D of RGBD data depth migration turns a 3D method, comprises following:
The calculating of step 1, global description's symbol: accord with by calculating the global description of image;
Step 2, retrieval neighbour image: according to described global description's symbol, in the Internet RGBD database, retrieve candidate image, and calculate its coupling cost, K the neighbour's image and the corresponding degree of depth thereof that according to the coupling cost calculating, obtain from small to large original 2D image, wherein K is positive integer;
Step 3, set up dense corresponding relation: by sheet, mate the dense corresponding relation of setting up between original 2D image and K neighbour's image;
Step 4, depth migration: according to dense corresponding relation by the corresponding depth migration of the K of original 2D image neighbour's image to the original 2D plane of delineation;
Step 5, depth map merge: for each pixel of original 2D image, from the depth map of K migration, choose a best degree of depth as its initial estimation degree of depth;
Step 6, depth map reprocessing: for further improving depth map estimated quality and suppressing noise, the depth map after merging is carried out to reprocessing;
Synthesizing of step 7,3D rendering: according to the visual saliency of original 2D image, depth map is carried out to Nonlinear Processing, synthetic 3D rendering.
The technical measures of optimizing also comprise:
The calculating of above-mentioned global description's symbol adopts GIST feature.
The computing formula of above-mentioned coupling cost is:
Wherein Q is original 2D image, the to be matched image of C for retrieving from RGBD database, the GIST characteristic vector of G (x) presentation video.
The computing formula of the coupling cost of above-mentioned sheet coupling is:
Above-mentioned sheet coupling adopts random search and neighborhood transmission to improve computational speed.
The formula of above-mentioned depth migration is:
Wherein
f(x) be pixel x in Q (
i,
j) correspondence position in image C to be matched;
d c for depth map corresponding to image C to be matched.
The formula that above-mentioned depth map merges is:
Wherein fusion function adopts medium filtering or weighted average filtering.
The formula of above-mentioned depth map reprocessing is:
Wherein
w x ,
w c with
w d the weight that represents respectively position, color and three components of the degree of depth.
The formula of above-mentioned Nonlinear Processing is:
Wherein d represents the depth map after nonlinear adjustment;
s xrepresent that original 2D image Q is at the visual saliency coefficient at x place, position, its value is positioned at interval [0 1], the neighborhood that N (x) is x place pixel.
The synthetic employing DIBR technology of above-mentioned 3D rendering.
Compared with prior art, the single image 2D of a kind of RGBD data depth migration of the present invention turns 3D method, by the RGBD database from the Internet, retrieves K neighbour's image and the corresponding degree of depth thereof of original image, and calculates the coupling cost of image to be matched; By sheet, mate the dense corresponding relation of setting up between original image and neighbour's image again; Again according to dense corresponding relation by the corresponding depth migration of the K of original 2D image neighbour's image to the original 2D plane of delineation; From the depth map of K migration, choose a best degree of depth as its initial estimation degree of depth; For further improving depth map estimated quality and suppressing noise, the depth map after merging is carried out to reprocessing; Finally according to the visual saliency of original 2D image, depth map is carried out to Nonlinear Processing, synthetic 3D rendering.Nowadays on the Internet, magnanimity RGBD database existing and that continue to increase has guaranteed for any original 2D image, can retrieve on the internet neighbour's image as image to be matched, thereby guaranteed versatility of the present invention, be applicable to all kinds of complicated shooting environmental and scene; The present invention utilizes the RGBD data that on the Internet, depth transducer gathers, and can obtain the accurate depth information of image, turns 3D method only obtain relative depth information unlike traditional 2D; Because the present invention adopts random search and neighborhood transmission, set up the dense matching relation between original 2D image and image to be matched, significantly reduced nearest neighbor search space, thereby improved computational speed; On the other hand, Local Structure of Image and scenery composition relation have been considered in neighborhood transmission, more easily guarantee the alignment of target between scene, have improved depth migration quality; While using the synthetic 3D figure of DIBR technology, according to visual saliency, depth map is carried out to nonlinear adjustment, guaranteed the three-dimensional telepresenc of visual salient region.
Accompanying drawing explanation
Fig. 1 is flow chart of the present invention.
Embodiment
Below in conjunction with accompanying drawing, embodiment is described in further detail the present invention.
Be illustrated in figure 1 flow chart of the present invention,
The calculating of step 1, global description's symbol: the global description's symbol that adopts the original 2D image of GIST feature calculation, its process is: first, the Gabor filter that creates different directions and yardstick carries out filtering to image and obtains several characteristic patterns, and each characteristic pattern is divided into 4 * 4 piece; Secondly, the mean value of each piece in calculated characteristics figure; Finally, the mean value of piece in all characteristic patterns is arranged in order to the GIST feature as original 2D image, global description's symbol that this GIST feature is original 2D image.
Step 2, retrieval neighbour image: in the magnanimity RGBD database according to global description's symbol of original 2D image from the Internet, retrieve candidate image, utilize following formula to calculate the cost of mating of original 2D image and image to be matched;
Wherein Q is original 2D image, the to be matched image of C for retrieving from RGBD database, and G (x) represents GIST characteristic vector; Then, coupling cost is sorted from small to large; Finally, therefrom choose from small to large K as candidate matches scene, its corresponding degree of depth is as candidate's estimating depth of Q; Wherein K is positive integer.
Step 3, set up dense corresponding relation: by sheet, mate the dense corresponding relation of setting up between original 2D image and K neighbour's image.Detailed process is: the first step, each pixel in Q and C is considered as to the upper left corner coordinate of W * W sheet, and to each such sheet in Q, by the method for random pair, specify its corresponding sheet in C, the pairing cost of two sheets is calculated with following formula:
Second step, to neighborhood transmission, if the pairing cost of this transmission is less than neighborhood sheet, when anter coupling retains and adjust the coupling of neighborhood sheet, otherwise is delivered to the displacement when anter coupling to work as anter by the coupling of neighborhood sheet; The 3rd step, take current matching as starting point, and the index step length searching of usining in a large window makes minimum coupling as the optimum Match when anter; The 4th step, does not return to second step if iteration reaches predetermined number of times, otherwise exits.
Step 4, depth migration: the dense corresponding relation of setting up according to sheet coupling by the K of original 2D image neighbour's image correspondence depth migration to original 2D image.Depth migration formula is:
Pixel x in Q (
i,
j) in matching image C, correspondence position is
f(x).Adopt above formula by matching image
ccorresponding depth map
d c move on the original 2D plane of delineation, as its candidate's estimation of Depth.
The 5th step, depth map merge: for each pixel of original 2D image, from the depth map of K migration, choose a best degree of depth as its initial estimation degree of depth.Detailed process is shown below:
In fusion function, adopt simple medium filtering or weighted average filtering.
The 6th step, depth map reprocessing: for further improving depth map estimated quality and suppressing noise, adopt following formula to carry out depth map reprocessing:
Wherein
w x ,
w c with
w d the weight that represents respectively position, color and three components of the degree of depth.
Synthesizing of the 7th step, 3D rendering: original 2D image is considered as to left view, according to the degree of depth of estimating, original 2D image conversion is arrived to new position as right view; In order to guarantee the third dimension of visual salient region, before conversion, by following formula, the degree of depth is carried out to nonlinear adjustment:
Wherein
dthe depth map of expression after nonlinear adjustment;
s xrepresent original 2D image
qvisual saliency coefficient at x place, position, its value is positioned at interval [0 1], more approaches 1 expression more remarkable; The neighborhood that N (x) is x place pixel; Obtain by DIBR technology, synthesizing 3D rendering after left and right view.
Claims (7)
1. the single image 2D of RGBD data depth migration turns a 3D method, it is characterized in that: comprise the following steps:
The calculating of step 1, global description's symbol: accord with by calculating the global description of image;
Step 2, retrieval neighbour image: according to described global description's symbol, in the Internet RGBD database, retrieve candidate image, and calculate its coupling cost, K the neighbour's image and the corresponding degree of depth thereof that according to the coupling cost calculating, obtain from small to large original 2D image, wherein K is positive integer;
Step 3, set up dense corresponding relation: by sheet, mate the dense corresponding relation of setting up between original 2D image and K neighbour's image;
Step 4, depth migration: according to described dense corresponding relation by the corresponding depth migration of the K of original 2D image neighbour's image to the original 2D plane of delineation;
Step 5, depth map merge: for each pixel of original 2D image, from the depth map of K migration, choose a best degree of depth as its initial estimation degree of depth;
Step 6, depth map reprocessing: for further improving depth map estimated quality and suppressing noise, the depth map after merging is carried out to reprocessing;
Synthesizing of step 7,3D rendering: according to the visual saliency of original 2D image, depth map is carried out to Nonlinear Processing, synthetic 3D rendering; The formula of described depth map reprocessing is:
Wherein
w x ,
w c with
w d the weight that represents respectively position, color and three components of the degree of depth;
The formula of described Nonlinear Processing is:
Wherein d represents the depth map after nonlinear adjustment;
s xrepresent that original 2D image Q is at the visual saliency coefficient at x place, position, its value is positioned at interval [0 1], the neighborhood that N (x) is x place pixel; Set up dense corresponding relation: by sheet, mate the dense corresponding relation of setting up between original 2D image and K neighbour's image; Detailed process is: the first step, each pixel in Q and C is considered as to the upper left corner coordinate of W * W sheet, and to each such sheet in Q, by the method for random pair, specify its corresponding sheet in C, the pairing cost of two sheets is calculated with following formula:
Second step, to neighborhood transmission, if the pairing cost of this transmission is less than neighborhood sheet, when anter coupling retains and adjust the coupling of neighborhood sheet, otherwise is delivered to the displacement when anter coupling to work as anter by the coupling of neighborhood sheet; The 3rd step, take current matching as starting point, and the index step length searching of usining in a large window makes minimum coupling as the optimum Match when anter; The 4th step, does not return to second step if iteration reaches predetermined number of times, otherwise exits.
2. the single image 2D of a kind of RGBD data depth migration according to claim 1 turns 3D method, it is characterized in that: the calculating of described global description's symbol adopts GIST feature.
3. the single image 2D of a kind of RGBD data depth migration according to claim 2 turns 3D method, it is characterized in that: the computing formula of described coupling cost is:
Wherein Q is original 2D image, the to be matched image of C for retrieving from RGBD database, the GIST characteristic vector of G (x) presentation video.
4. the single image 2D of a kind of RGBD data depth migration according to claim 3 turns 3D method, it is characterized in that: described sheet coupling adopts random search and neighborhood transmission to improve computational speed.
5. the single image 2D of a kind of RGBD data depth migration according to claim 4 turns 3D method, it is characterized in that: the formula of described depth migration is:
Wherein
f(x) be pixel x in Q (
i,
j) correspondence position in image C to be matched;
d c for depth map corresponding to image C to be matched.
6. the single image 2D of a kind of RGBD data depth migration according to claim 5 turns 3D method, it is characterized in that: the formula that described depth map merges is:
Wherein fusion function adopts medium filtering or weighted average filtering.
7. the single image 2D of a kind of RGBD data depth migration according to claim 6 turns 3D method, it is characterized in that: the synthetic employing DIBR technology of described 3D rendering.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310016333.9A CN103024420B (en) | 2013-01-17 | 2013-01-17 | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310016333.9A CN103024420B (en) | 2013-01-17 | 2013-01-17 | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103024420A CN103024420A (en) | 2013-04-03 |
| CN103024420B true CN103024420B (en) | 2014-12-10 |
Family
ID=47972485
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310016333.9A Expired - Fee Related CN103024420B (en) | 2013-01-17 | 2013-01-17 | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103024420B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106548494A (en) * | 2016-09-26 | 2017-03-29 | 浙江工商大学 | A kind of video image depth extraction method based on scene Sample Storehouse |
| CN109191506B (en) * | 2018-08-06 | 2021-01-29 | 深圳看到科技有限公司 | Depth map processing method, system and computer readable storage medium |
| CN110866882B (en) * | 2019-11-21 | 2021-09-07 | 湖南工程学院 | Layered joint bilateral filtering depth map repairing method based on depth confidence |
| CN113034385B (en) * | 2021-03-01 | 2023-03-28 | 嘉兴丰鸟科技有限公司 | Grid generating and rendering method based on blocks |
| CN115170739B (en) * | 2022-07-11 | 2023-09-01 | 杭州时戳信息科技有限公司 | Vehicle three-dimensional design device based on artificial intelligence |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102695070A (en) * | 2012-06-12 | 2012-09-26 | 浙江大学 | Depth consistency fusion processing method for stereo image |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9123115B2 (en) * | 2010-11-23 | 2015-09-01 | Qualcomm Incorporated | Depth estimation based on global motion and optical flow |
-
2013
- 2013-01-17 CN CN201310016333.9A patent/CN103024420B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102695070A (en) * | 2012-06-12 | 2012-09-26 | 浙江大学 | Depth consistency fusion processing method for stereo image |
Non-Patent Citations (7)
| Title |
|---|
| Brandon M. Smith,et al.Stereo Matching with Nonparametric Smoothness Priors in Feature Space.《IEEE Xplore DIGITAL LIBRARY》.2009,485-492. * |
| Connelly Barnes,et al.The PatchMatch randomized matching algorithm for image manipulation.《COMMUNICATIONS OF THE ACM》.2011,第54卷(第11期),103-110. * |
| J. Konrad,et al.Automatic 2D-to-3D image conversion using 3D examples from the internet.《Proc. of SPIE-IS& * |
| J. Konrad,et al.Automatic 2D-to-3D image conversion using 3D examples from the internet.《Proc. of SPIE-IS&T Electronic Imaging》.2012,第8288卷1-12. * |
| Stereo Matching with Nonparametric Smoothness Priors in Feature Space;Brandon M. Smith,et al;《IEEE Xplore DIGITAL LIBRARY》;20091231;全文 * |
| T Electronic Imaging》.2012,第8288卷1-12. * |
| The PatchMatch randomized matching algorithm for image manipulation;Connelly Barnes,et al;《COMMUNICATIONS OF THE ACM》;20111130;第54卷(第11期);103-110 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103024420A (en) | 2013-04-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Guttmann et al. | Semi-automatic stereo extraction from video footage | |
| CN103024420B (en) | 2D-3D (two-dimension to three-dimension) conversion method for single images in RGBD (red, green and blue plus depth) data depth migration | |
| CN103810708B (en) | A kind of laser speckle image depth perception method and device | |
| CN101287143B (en) | Method for converting flat video to tridimensional video based on real-time dialog between human and machine | |
| CN102665086B (en) | Method for obtaining parallax by using region-based local stereo matching | |
| CN105374039B (en) | Monocular image depth information method of estimation based on contour acuity | |
| CN109064505B (en) | Depth estimation method based on sliding window tensor extraction | |
| CN115205489A (en) | Three-dimensional reconstruction method, system and device in large scene | |
| CN108596975A (en) | A kind of Stereo Matching Algorithm for weak texture region | |
| CN102496183A (en) | Multi-view stereo reconstruction method based on Internet photo gallery | |
| Wu et al. | A novel method for semi-automatic 2D to 3D video conversion | |
| CN104159098B (en) | The translucent edge extracting method of time domain consistence of a kind of video | |
| CN104778673B (en) | A kind of improved gauss hybrid models depth image enhancement method | |
| CN115601406A (en) | Local stereo matching method based on fusion cost calculation and weighted guide filtering | |
| CN103679699B (en) | A kind of based on notable figure translation and the solid matching method of combined measure | |
| Fan et al. | Vivid-DIBR based 2D–3D image conversion system for 3D display | |
| CN112270701A (en) | Packet distance network-based parallax prediction method, system and storage medium | |
| Yang et al. | Depth map generation using local depth hypothesis for 2D-to-3D conversion | |
| Sisi et al. | The overview of 2D to 3D conversion system | |
| Mathai et al. | Automatic 2D to 3D video and image conversion based on global depth map | |
| CN103400380A (en) | Single camera underwater target three-dimensional trace stimulation method merged with image matrix offset | |
| CN110610503A (en) | Three-dimensional information recovery method for power disconnecting link based on stereo matching | |
| CN103559702B (en) | Based on the two-dimensional single-view image depth estimation method of wavelet coefficient entropy | |
| GB2585197A (en) | Method and system for obtaining depth data | |
| CN203027382U (en) | RGBD (Red/Green/Blue Data) depth migration equipment for converting single 2D (Two-Dimensional) images into 3D (Three-Dimensional) images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent for invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 315211 Zhejiang Province, Ningbo Jiangbei District Fenghua Road No. 201 Applicant after: Ningbo University of Technology Address before: 315016 Zhejiang Province, Ningbo city Haishu District Cuibai Road No. 89 Ningbo University of Technology College of electronic and Information Engineering Applicant before: Ningbo University of Technology |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20141210 Termination date: 20160117 |
|
| EXPY | Termination of patent right or utility model |