CN103384343A - Image cavity filling method and device thereof - Google Patents
Image cavity filling method and device thereof Download PDFInfo
- Publication number
- CN103384343A CN103384343A CN2013102771165A CN201310277116A CN103384343A CN 103384343 A CN103384343 A CN 103384343A CN 2013102771165 A CN2013102771165 A CN 2013102771165A CN 201310277116 A CN201310277116 A CN 201310277116A CN 103384343 A CN103384343 A CN 103384343A
- Authority
- CN
- China
- Prior art keywords
- view
- macroscopic
- void
- pixels
- depth map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses an image cavity filling method and a device thereof. According to the image cavity filling method and the device thereof, image rendering is performed by utilizing to-be-processed target frames in video frame sequences and a piece of deep map which is corresponding to the to-be-processed target frames, generated large cavities and small cavities are processed during the rendering process, the small cavities are processed at advance, the large cavities are processed through a method of finding pixel blocks which are optimally matched with the to-be-processed large cavities in abundant video frame sequences, and the cavity filling effect is greatly increased.
Description
Technical field
The invention belongs to technical field of image processing, particularly a kind of method and device thereof of filling up the image cavity.
Background technology
In recent years, along with the continuous breakthrough of free 3 D display technology in hardware and software research, free stereo is presented at the fields such as medical domain, advertisement field, military affairs, amusement and all has been subject to increasing favor.
Faster for better effects if, speed that free stereo is shown, emerge the multiple algorithm of playing up to image, the common algorithm of playing up roughly is divided into two classes:
The first kind is that former figure plus depth figure plays up algorithm (2D plus Z), mainly to calculate the parallax value of this scene point in known view and desired medial view according to scene point at depth value corresponding in depth map, calculate again the projected position of scene point in medial view, want scene point is all carried out above-mentioned calculating, just can render medial view.
Equations of The Second Kind is many view renderings algorithm (multi-view), is mainly to render the view of scene visual angle according to known a plurality of views.
When view is played up, cause the generation in cavity because block between scene point.As shown in Figure 4, the corresponding left view viewpoint of C1, the corresponding right view viewpoint of C2.Scene point P1 and P2 are projected in same position P ' in left view L, like this, the scene point P1 from left viewpoint C1 away from will be blocked by P2.Scene point P1 and P2 on right view R again respectively projection in position P1 ', P2 ', like this, if when carrying out playing up of right view based on left view and depth map, the P1 ' point on right view R will the cavity not occur because there is no corresponding information.
at present, solving the main method of empty problem is: utilize the information such as background, texture, neighbor that the cavity is filled up, as Criminisi, A., P é rez, P. ， ﹠amp, Toyama, K. " the Region filling and object removal by exemplar-based image inpainting. Image Processing that delivers, IEEE Transactions on, 13 (9), 1200-1212. " and Po, L. M., Zhang, S., Xu, X. wait " the A new multidirectional extrapolation hole-filling method for Depth-Image-Based Rendering. In Image Processing (ICIP) that delivers, 2011 18th IEEE International Conference on (pp. 2589-2592). IEEE. " etc.Although the title of the method for mentioning in their article is different, be in fact all by the pixel value around the cavity is comprehensively come filling cavity.There is following shortcoming at least in the technology that as above solves empty problem: when solving empty problem, filling technique can not be from solving empty problem in essence, particularly fill up larger when empty, can present and significantly fill up vestige, filling effect is undesirable; Level and smooth preconditioning technique is out of shape object, and the visual quality of the virtual view that obtains is significantly decreased.
Summary of the invention
The object of the present invention is to provide a kind of method and device thereof of filling up the image cavity, effectively filled in the image rendering process because blocking the cavity that produces between scene point.
The technical solution that realizes the object of the invention is: a kind of method of filling up the image cavity, and step is as follows:
S1, in pending sequence of frames of video and depth map corresponding to each frame, the target frame that wish is processed is as the first view, and selects depth map corresponding to this first view;
S2 utilizes the image rendering algorithm by the first view and corresponding depth map thereof, and the virtual visual point image of wanting to ask is played up, and obtains the second view;
S3: according to predetermined threshold value, the size in cavity described in the second view is divided, the cavity of scale in the second view less than set threshold value is considered as being caused by quantization error, be called little cavity, and little cavity is filled up;
S4: according to predetermined threshold value, the cavity of scale in the second view greater than set threshold value is considered as blocking because of scene point the cavity that produces, be called macroscopic-void, and according to the span of this macroscopic-void interior pixels and get the pixel of default number outside this macroscopic-void edge, determine the position of this macroscopic-void in the second view;
S5: except depth map corresponding to the first view, in depth map corresponding to each remaining frame, seek the block of pixels identical with macroscopic-void place location of pixels in the second view;
S6: to the processing of suing for peace of the pixel value of the block of pixels obtained in step S5, find out the depth map under the block of pixels of its neutralization number minimum, and determine its corresponding that frame of video according to this depth map, this frame of video is designated as three-view diagram;
S7: in described three-view diagram, take out with the second view in macroscopic-void position and big or small all identical block of pixels, the block of pixels that will get in three-view diagram copies and fills up macroscopic-void place in the second view.
A kind of device of filling up the image cavity, by data input module, rendering module, first fill up module, computing module, second is filled up module and is formed, this data input module, rendering module, computing module, second are filled up module and are connected successively, and first fills up module is connected with rendering module;
In data input module, in pending sequence of frames of video and depth map corresponding to each frame, elect the target frame that current wish is processed as first view, and select depth map corresponding to this first view;
In rendering module, according to described the first view and its corresponding depth map, utilize the image rendering algorithm, the virtual view of wanting to ask is played up obtained the second view;
In computing module, except depth map corresponding to the first view, in depth map corresponding to each remaining frame, seek the block of pixels identical with the middle macroscopic-void place location of pixels of the second view;
In second filled up module, the block of pixels that computing module is determined copied and fills up in macroscopic-void in the second view;
Described first fills up module is comprised of the empty scale identifying unit that connects successively and little hole-filling unit, in this cavity scale identifying unit, according to default threshold value, determines area in the second view and is called little cavity less than the cavity of set threshold value; In little hole-filling unit, according to the result of determination of described empty scale identifying unit, little cavity is filled up.
The present invention compared with prior art, its remarkable advantage: carry out image rendering by utilizing simple eye view in sequence of frames of video and a depth map of its correspondence, in render process, the big or small cavity that produces is processed respectively, first process little cavity, recycling is from remaining sequence of frames of video, the method of the block of pixels of searching and macroscopic-void optimum Match to be dealt with is processed macroscopic-void, has greatly strengthened the effect of filling up in cavity.
Below in conjunction with accompanying drawing, the present invention is described in further detail.
Description of drawings
Fig. 1 is the cavity figure that the needs that obtain after playing up are filled up.
Fig. 2 is the design sketch that adopts after traditional complementing method (Image Inpainting method) is filled up.
Fig. 3 is the design sketch that adopts after the method that proposes in the present invention is filled up.
Fig. 4 produces the schematic diagram in cavity in the rendering image process in prior art.
Fig. 5 is the flow chart that the present invention fills up the method in image cavity.
Fig. 6 is the structural representation that the present invention fills up the device in image cavity.
Embodiment
Referring to Fig. 5, the present invention fills up the method in image cavity, is to fill up for the cavity that each two field picture of Video Rendering is produced, and step is as follows:
S1: in pending sequence of frames of video and depth map corresponding to each frame, the target frame that wish is processed is as the first view, and selects depth map corresponding to this first view.
S2: utilize the image rendering algorithm by the first view and corresponding depth map thereof, the virtual visual point image of wanting to ask is played up, obtain the second view.The reason of blocking due to the quantization error of numerical computations in render process and scene point itself will produce cavity not of uniform size in the second view that obtains.
in this step, utilizing virtual view to produce the method that algorithm carries out image rendering according to the first view and corresponding depth map thereof has multiple, as operable rendering intent, Liang Zhang is arranged, Lawrence, B., Demin Wang, Vincent, A. the article of delivering on (CVMP 2005) The 2nd IEE European Conference " Comparison study on feature matching and block matching for automatic 2d to 3d video conversion, Visual Media Production, 2005. " in the virtual view generation method used.State in the use in the process of method, require in the first corresponding depth map of view, the pixel value at the background information place of corresponding the first view should be zero.
S3: according to predetermined threshold value, the size in cavity described in the second view is divided, the cavity of scale in the second view less than set threshold value is considered as being caused by quantization error, be called little cavity, and utilize existing algorithm that little cavity is filled up.
In this step, the type of dividing the cavity in the second view need to preset a threshold value as the criteria for classifying.can adopt prior art to process, as the Xuyuan Xu at Hong Kong City University of Hong Kong, method (the Acoustics of a kind of " dividing big or small cavity based on the size of playing up adjacent depth value saltus step in depth map used " is proposed in the article of " A FOREGROUND BIASED DEPTH MAP REFINEMENT METHOD FOR DIBR VIEWSYNTHESIS " that Lai-Man Po etc. delivers, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference, page 805-808).The method is passed through formula:
The threshold value T of depth value (being the value of pixel in depth map) saltus step between neighbor pixel is played up in depth map used in setting
dWherein Δ d is that on horizontal direction, the difference between adjacent degree of depth pixel is T
d, t
cBe the distance between reference view and virtual view, f is focal length, z
nAnd z
fBe respectively the nearest and maximum distance in real scene.This threshold value T
dRelevant with the scale in cavity, because T
dIn the situation that h=3 calculates.The h here is the horizontal size in cavity, take pixel as unit.That is to say, the difference of adjacent depth value is greater than threshold value T on horizontal direction
dThe time, the corresponding cavity size in the horizontal direction that occurs should be more than or equal to 3 pixels in playing up the image that obtains, otherwise just less than 3 pixels.When dividing empty type just can this as the criteria for classifying.
Take above-mentioned threshold value as the criteria for classifying, with on horizontal direction less than 3 pixel (threshold value T
dValue can rule of thumb decide, generally can be 3-8 pixel) the cavity be considered as little cavity, then fill with the existing technology of filling up, as Inpainting algorithm integrated in matlab or opencv.
S4: according to predetermined threshold value, the cavity of scale (area) in the second view greater than set threshold value is considered as blocking because of scene point the cavity that produces, be called macroscopic-void, and according to the span of this macroscopic-void interior pixels and get the pixel of default number outside this macroscopic-void edge, determine the position of this macroscopic-void in the second view.
Wherein, the method of determining the position is: for the macroscopic-void in the second view, because this cavity is to be caused by blocking of scene point, so utilized virtual view production algorithm that the second view is played up by the first view and its corresponding depth map after, variation has occured with respect to the first view in the visual angle of cause the second view, so, the point that is originally sheltered from by other scene point in the first view (claiming that here they are to reappear point), can appear in the second view, as shown in Figure 2.Blocked by other scene point the colouring information of having lost oneself because these reappear point in the first view, this colouring information is embodied in the size of these reproductions pixel value after imaging o'clock in the first view, and its span is [0 255].Therefore, these appear at reproduction in the second view and name a person for a particular job and can not played up by the first view and depth map corresponding to the first view, cause presenting the zone of a black on the second view, i.e. macroscopic-void in the second view.The value of the pixel value that this is empty is less, between [0,10].And the value of the pixel value around should the cavity is led to outside [0,10] this scope.So, can by detecting the size of each pixel point value in the second view, determine the position of this cavity in the second view.In the process that detects, the size of at first taking out the second view, namely the value of height and width, be for example [m, n], and wherein m represents that the height of the second view is line number, and n represents that the width of the second view is columns.A line number value and a columns value can uniquely be determined a pixel, and for example point (i, j) represents capable by i and the determined pixel of j row, and the value of i should be within [1, m] scope certainly, and lower limit comprises 1, and the upper limit comprises m.In like manner, the value of j also should comprise bound within [1, n] scope.Then, get one by one columns according to fixing delegation, after columns takes (j=m), changing delegation fixes again, repeat the operation of front, pixel in searching loop the second view, taking out each is traversed the value of pixel and judges that whether this value is [0,10] in the scope that (comprises bound), if satisfy Rule of judgment, just determined the position coordinates (i, j) of this pixel correspondence in the second view simultaneously, after whole pixels in having traveled through the second view, also just determined the particular location of macroscopic-void in the second view in the second view fully.
For the macroscopic-void in the second view, at definite its marginal position coordinate time, because the distribution of these edge pixel points is irregular, and in these edge pixel points, have the value of pixel of fraction outside given judgement scope [0,10], that is to say, this judgement scope be used for to be determined this cavity marginal position coordinate time, in the value of all pixels that should cavity edge can not being completely contained in.Therefore, the pixel that does not satisfy the macroscopic-void edge of decision condition in order not omit these, when stating in the use the searching loop method and detecting, as the pixel (i that a value detected and satisfy the judgement scope, j) after, think that simultaneously its four pixels all around are also the points that satisfies decision condition, so just, enlarged the position range of described macroscopic-void in the second view, all marginal points of described macroscopic-void can be included simultaneously, finally can be found the position of described macroscopic-void in the second view fully.
S5: except depth map corresponding to the first view, in depth map corresponding to each remaining frame, seek the block of pixels identical with the middle macroscopic-void place location of pixels of the second view.
For the method for seeking required block of pixels be: at first, in the block of pixels of getting, the position range of the macroscopic-void that the position range of block of pixels need to find with method therefor in step S4 is identical, and the position coordinates in sequence of frames of video under it of the each point in the block of pixels that namely finds is identical with the position coordinates of each pixel in the second view of determined composition macroscopic-void in step S4; Secondly, in the second view of determining take method described in step S4, the position coordinates of macroscopic-void is as locating information, except depth map corresponding to the first view, in depth map corresponding to each remaining frame of video, navigate to same position coordinates place, the position coordinates of certain pixel in the second view that for example forms macroscopic-void is (i, j), arrive so except depth map corresponding to the first view, reach one by one the pixel value under (i, j) coordinate in depth map corresponding to each remaining frame; In like manner, take the position coordinates that forms all pixels of macroscopic-void in the second view as the value foundation, in the depth map of remainder, after taking out all pixel values, just obtained all with the second view in the equal identical block of pixels in macroscopic-void scale and position.
S6: in step S5, the depth map under the block of pixels of its neutralization number minimum is found out in the processing of suing for peace of the pixel value of all block of pixels of obtaining, and determines that frame of video of its correspondence according to this depth map, and this frame of video is designated as three-view diagram.
To all block of pixels specific practice of processing of suing for peace be: except depth map corresponding to the first view, in depth map corresponding to each remaining frame of video, each width depth map of remainder by the method for obtaining block of pixels described in step S5, is carried out the extraction of block of pixels.Extract with the second view in macroscopic-void scale and position all in the process of identical target pixel block, pixel value corresponding to all pixels that forms this target pixel block all will be traversed one by one.The pixel value that first will traverse is for the first time stored, then carry out the addition summation with second pixel value that traverses, take again afterwards this and value to carry out the addition summation with the 3rd pixel value that traverses, the rest may be inferred, after having traveled through all target pixel points, just obtained the target pixel block obtained in this width depth map corresponding and value.Every width depth map to except depth map corresponding to the first view all repeats aforesaid operations, just obtained all with the second view in macroscopic-void scale and position all identical block of pixels corresponding and value.
The size of these and value is relatively at last found out that and value of its neutralization number minimum.Determine this depth map corresponding with value for the ease of basis and value, in practical operation, the every width depth map with the value correspondence is numbered.For example, other depth maps except depth map corresponding to the first view are numbered I_1 ~ I_m, in every width depth map target pixel block corresponding just be numbered accordingly Sum_1 ~ Sum_m with value.After comparing minimum and value, the depth map of correspondence also just known, the frame of video that depth map is corresponding so also can be determined very intuitively with same method.
S7: in described three-view diagram, take out and the macroscopic-void position described in the second view and big or small all identical block of pixels.The acquisition methods of its block of pixels is identical with acquisition methods described in step S5, and the block of pixels that obtains in this step is only fills up the required optimum Match block of pixels of macroscopic-void in the second view.
S8: in step S7, the optimum Match block of pixels of getting in three-view diagram copies and fills up macroscopic-void place in the second view.
The specific practice in two steps of S7, S8 is: correspond to one by one same position coordinates place in described three-view diagram according to the position coordinates that forms each pixel of macroscopic-void in determined the second view in step S4.And in described three-view diagram, take these coordinates as the value foundation, take out one by one the pixel value at these respective coordinates place, and then with these pixel values, copy to one by one in the second view position coordinates place identical on macroscopic-void, reach the purpose of filling up this macroscopic-void.It finally fills up effect as shown in Figure 3.
Referring to Fig. 6, the present invention fills up the device in image cavity, purpose is for the image cavity is filled up, by data input module, rendering module, first fill up module, computing module, second is filled up module and is formed, this data input module, rendering module, computing module, second are filled up module and are connected successively, and first fills up module is connected with rendering module.
In data input module, in pending sequence of frames of video and depth map corresponding to each frame, elect the target frame that current wish is processed as first view, and select depth map corresponding to this first view.
In rendering module, according to described the first view and its corresponding depth map, utilize the image rendering algorithm, the virtual view of wanting to ask is played up obtained the second view; Wherein, the method for playing up can be chosen a lot of prior aries and play up.
Computing module is chosen unit, similarity calculated, comparing unit, is chosen the unit and form by the macroscopic-void position calculation unit, the compared pixels piece that connect successively.In computing module, except depth map corresponding to the first view, in depth map corresponding to each remaining frame, seek the block of pixels identical with the middle macroscopic-void place location of pixels of the second view.
in described macroscopic-void position calculation unit, result of determination according to empty scale identifying unit, calculate in the second view area greater than the cavity of set threshold value, be the position of macroscopic-void in the second view, namely for the macroscopic-void in the second view, because this cavity is to be caused by blocking of scene point, so utilized virtual view production algorithm that the second view is played up by the first view and its corresponding depth map after, variation has occured with respect to the first view in the visual angle of cause the second view, so, the point that is originally sheltered from by other scene point in the first view (claiming that here they are to reappear point), can appear in the second view, as shown in Figure 2.Blocked by other scene point the colouring information of having lost oneself because these reappear point in the first view, this colouring information is embodied in the size of these reproductions pixel value after imaging o'clock in the first view, and its span is [0 255].Therefore, these appear at reproduction in the second view and name a person for a particular job and can not played up by the first view and depth map corresponding to the first view, cause presenting the zone of a black on the second view, i.e. macroscopic-void in the second view.The value of the pixel value that this is empty is less, between [0,10].And the value of the pixel value around should the cavity is led to outside [0,10] this scope.So, can by detecting the size of each pixel point value in the second view, determine the position of this cavity in the second view.In the process that detects, the size of at first taking out the second view, namely the value of height and width, be for example [m, n], and wherein m represents that the height of the second view is line number, and n represents that the width of the second view is columns.A line number value and a columns value can uniquely be determined a pixel, and for example point (i, j) represents capable by i and the determined pixel of j row, and the value of i should be within [1, m] scope certainly, and lower limit comprises 1, and the upper limit comprises m.In like manner, the value of j also should comprise bound within [1, n] scope.Then, get one by one columns according to fixing delegation, after columns takes (j=m), changing delegation fixes again, repeat the operation of front, pixel in searching loop the second view, taking out each is traversed the value of pixel and judges that whether this value is [0,10] in the scope that (comprises bound), if satisfy Rule of judgment, just determined the position coordinates (i, j) of this pixel correspondence in the second view simultaneously, after whole pixels in having traveled through the second view, also just determined the particular location of macroscopic-void in the second view in the second view fully.
For the macroscopic-void in the second view, at definite its marginal position coordinate time, because the distribution of these edge pixel points is irregular, and in these edge pixel points, have the value of pixel of fraction outside given judgement scope [0,10], that is to say, this judgement scope be used for to be determined this cavity marginal position coordinate time, in the value of all pixels that should cavity edge can not being completely contained in.Therefore, the pixel that does not satisfy the macroscopic-void edge of decision condition in order not omit these, when stating in the use the searching loop method and detecting, as the pixel (i that a value detected and satisfy the judgement scope, j) after, think that simultaneously its four pixels all around are also the points that satisfies decision condition, so just, enlarged the position range of described macroscopic-void in the second view, all marginal points of described macroscopic-void can be included simultaneously, finally can be found the position of described macroscopic-void in the second view fully.
In the compared pixels piece is chosen the unit, in depth map corresponding to each frame of the remainder except depth map corresponding to the first view, choose the block of pixels identical with macroscopic-void position in the second view that the macroscopic-void position calculation unit calculates.
In similarity calculated, choose the result of choosing of unit according to the compared pixels piece, all pixel values in block of pixels are superposeed, obtain corresponding each block of pixels and value.The content of judging the optimum Match block of pixels that the optimum Match block of pixels will satisfy should be this condition of sequence of frames of video background information.because fill up the macroscopic-void that causes because blocking, best effect is fully to go to fill up with the background information of the scene of macroscopic-void corresponding position, and in a lot of video sequences, background information remains unchanged, and the background information in depth map corresponding to each frame all represents with identical numerical value, therefore can be by the macroscopic-void size of first obtaining in depth map and will filling up also identical block of pixels of identical and position coordinates, afterwards these block of pixels being carried out suitable numerical value processes, can obtain to fill up the desired block of pixels size of macroscopic-void identical with macroscopic-void, and the content of block of pixels is the optimum Match block of pixels of video frequency sequence background information, namely choose the result of choosing of unit according to the compared pixels piece, the block of pixels used of choosing is carried out same operation.At first, obtain whole number of pixels in block of pixels to be dealt with; Then the whole pixel values in block of pixels to be dealt with are carried out addition summation, obtain the respective pixel piece and value.
In comparing unit, relatively each block of pixels corresponding and value, select and be worth depth map under minimum block of pixels.So according to the result of similarity calculated, the corresponding task of comparing unit be exactly relatively all block of pixels corresponding with size value; Then the minimum affiliated depth map of block of pixels of basis and value, find frame of video corresponding to this depth map.
In choosing the unit, the depth map of selecting according to comparing unit in frame of video corresponding to this depth map, is chosen the block of pixels identical with macroscopic-void position in the second view.By the calculating of similarity calculated and comparing unit two unit, choose the corresponding task in unit and be exactly in the selected frame of video of comparing unit, choose and fill up the required optimum Match block of pixels of macroscopic-void.
In above-mentioned compared pixels piece is chosen unit, similarity calculated and is chosen the unit, at first, in the block of pixels of getting, the position range of the macroscopic-void that the position range of block of pixels need to find with method therefor in step S4 is identical, and the position coordinates in sequence of frames of video under it of the each point in the block of pixels that namely finds is identical with the position coordinates of each pixel in the second view of determined composition macroscopic-void in step S4; Secondly, in the second view of determining take method described in step S4, the position coordinates of macroscopic-void is as locating information, outside depth map corresponding to the first view, in depth map corresponding to each remaining frame of video, navigate to same position coordinates place, the position coordinates of certain pixel in the second view that for example forms macroscopic-void is (i, j), arrive so except depth map corresponding to the first view, reach the pixel value under (i, j) coordinate in depth map corresponding to each remaining frame; In like manner, take the position coordinates that forms all pixels of macroscopic-void in the second view as the value foundation, in the depth map of remainder, after taking out all pixel values, all block of pixels identical with macroscopic-void scale in the second view have just been obtained.
In second fills up module, the block of pixels of determining the unit of choosing of computing module is copied and fills up in macroscopic-void in the second view;
First fills up module is comprised of the empty scale identifying unit that connects successively and little hole-filling unit, in this cavity scale identifying unit, according to default threshold value, determines area in the second view and is called little cavity less than the cavity of set threshold value.Wherein predetermined threshold value can be with reference to the content of abovementioned steps S3.In little hole-filling unit, according to the result of determination of described empty scale identifying unit, little cavity is filled up, can adopt and existingly fill up technology and fill up.
The present invention carries out image rendering by a depth map that utilizes wish processing target frame in sequence of frames of video and its correspondence, in render process, the big or small cavity that produces is processed respectively, first process little cavity, recycling is from remaining sequence of frames of video, the method of the block of pixels of searching and macroscopic-void optimum Match to be dealt with, macroscopic-void is processed, greatly strengthened the effect of filling up in cavity.The below is that the complementing method (The proposed method) that proposes in the present invention is as shown in table 1 below with traditional complementing method (Image Inpainting method) experimental data relatively.
Image and the PSNR between true picture and the SSIM value of the macroscopic-void position after the different complementing methods of table 1. are processed
| PSNR?/SSIM | Image?Inpainting?method | The?proposed?method |
| Group?Ⅰ | 32.7387/0.7252 | 34.3150/0.8873 |
| Group?Ⅱ | 36.2807/0.7757 | 38.9685/0.8727 |
As known from Table 1, the Group I is other one group of sequence of frames of video of using in actual experiment, and the Group II is the sequence of frames of video in Fig. 1.PSNR can Description Image the peak value letter make ratio, when calculating PSNR in this table, so-called signal is exactly the real pixel value of macroscopic-void position, and the pixel value of the optimum Match block of pixels of namely getting in three-view diagram, noise refer to fill up the pixel value of macroscopic-void position later.The value of PSNR is the bigger the better, and larger explanation noise is less, namely fills up the error of bringing less, fills up more near actual value.The SSIM algorithm is to consider from the structural information angle of image, and its value also is the bigger the better, and the effect after larger explanation is filled up is more near actual value.Maximum is 1, if the value of SSIM by 1 explanation two width images of comparison identical.
Claims (7)
1. method of filling up the image cavity is characterized in that step is as follows:
S1, in pending sequence of frames of video and depth map corresponding to each frame, the target frame that wish is processed is as the first view, and selects depth map corresponding to this first view;
S2 utilizes the image rendering algorithm by the first view and corresponding depth map thereof, and the virtual visual point image of wanting to ask is played up, and obtains the second view;
S3: according to predetermined threshold value, the size in cavity described in the second view is divided, the cavity of scale in the second view less than set threshold value is considered as being caused by quantization error, be called little cavity, and little cavity is filled up;
S4: according to predetermined threshold value, the cavity of scale in the second view greater than set threshold value is considered as blocking because of scene point the cavity that produces, be called macroscopic-void, and according to the span of this macroscopic-void interior pixels and get the pixel of default number outside this macroscopic-void edge, determine the position of this macroscopic-void in the second view;
S5: except depth map corresponding to the first view, in depth map corresponding to each remaining frame, seek the block of pixels identical with the macroscopic-void position in the second view;
S6: to the processing of suing for peace of the pixel value of the block of pixels obtained in step S5, find out the depth map under the block of pixels of its neutralization number minimum, and determine that frame of video of its correspondence according to this depth map, this frame of video is designated as three-view diagram;
S7: in described three-view diagram, take out with the second view in macroscopic-void position and big or small all identical block of pixels, the block of pixels that to get in three-view diagram copies and fills up macroscopic-void place in the second view, namely correspond to one by one same position coordinates place in described three-view diagram according to the position coordinates of each pixel that forms macroscopic-void in determined the second view in step S4, and in described three-view diagram, take these coordinates as the value foundation, take out one by one the pixel value at these respective coordinates place, and then with these pixel values, copy to one by one in the second view position coordinates place identical on macroscopic-void, thereby filled up the macroscopic-void in the second view.
2. the method for filling up the image cavity according to claim 1, it is characterized in that step S4 determines the location of pixels method of macroscopic-void in the second view and is: by detecting the size of each pixel pixel value in the second view, determine the position of this cavity in the second view, namely in the process that detects, at first take out the size of the second view, i.e. the value [m, n] of height and width, wherein m represents that the height of the second view is line number, and n represents that the width of the second view is columns; A line number value and the unique definite pixel of columns value, (i, j) expression is the capable and determined pixel of j row by i, and the value of i should be within [1, m] scope, and the value of j is within [1, n] scope;
Then, get one by one columns according to fixing delegation, after columns takes, be j=m, changing delegation fixes again, repeat the operation of front, pixel in searching loop the second view takes out the pixel value that each is traversed pixel, and judges that whether this value is [0,10] in scope, if satisfy Rule of judgment, just determined the position coordinates (i, j) of this pixel correspondence in the second view simultaneously, after whole pixels in having traveled through the second view, also just determined the particular location of macroscopic-void in the second view in the second view fully;
When stating in the use the searching loop method and detecting, as the pixel (i that a value detected and satisfy the judgement scope, j) after, think that simultaneously its four pixels all around are also the points that satisfies decision condition, so just, enlarged the position range of described macroscopic-void in the second view, all marginal points with described macroscopic-void are included simultaneously, finally find the position of described macroscopic-void in the second view fully.
3. the method for filling up the image cavity according to claim 1 and 2 is characterized in that seeking in step S5 the block of pixels method identical with macroscopic-void place location of pixels in the second view and is:
At first, in the block of pixels of getting, the position range of the macroscopic-void that the position range of block of pixels need to find with method therefor in step S4 is identical, and the position coordinates in sequence of frames of video under it of the each point in the block of pixels that namely finds is identical with the position coordinates of each pixel in the second view of determined composition macroscopic-void in step S4;
Secondly, in the second view of determining take method described in step S4, the position coordinates of macroscopic-void as locating information, outside depth map corresponding to the first view, in depth map corresponding to each remaining frame of video, navigates to same position coordinates place.The position coordinates of certain pixel in the second view that forms macroscopic-void be (i, j), arrives so except depth map corresponding to the first view, reaches the pixel value under (i, j) coordinate in depth map corresponding to each frame of remainder; In like manner, take the position coordinates that forms all pixels of macroscopic-void in the second view as the value foundation, in the depth map of remainder, after taking out all pixel values, just obtained all with the second view in the equal identical block of pixels in macroscopic-void scale and position.
4. the method for filling up the image cavity according to claim 1, it is characterized in that in step S6: to the block of pixels specific practice of processing of suing for peace be: except depth map corresponding to the first view, in depth map corresponding to each remaining frame of video, to each width depth map of remainder by the method for obtaining block of pixels described in step S5, carry out the extraction of block of pixels, extract with the second view in macroscopic-void scale and position all in the process of identical target pixel block, pixel value corresponding to all pixels that forms this target pixel block all will be traversed one by one, the pixel value that first will traverse is for the first time stored, then carry out the addition summation with second pixel value that traverses, the 3rd pixel value that will be somebody's turn to do and be worth and traverse again afterwards carries out the addition summation, the rest may be inferred, after having traveled through all target pixel points, just obtained the target pixel block obtained in this width depth map corresponding and value, every width depth map to except depth map corresponding to the first view all repeats aforesaid operations, just obtained all with the second view in macroscopic-void scale and position all identical block of pixels corresponding and value.
5. the method for filling up the image cavity according to claim 1, it is characterized in that in step S7: according to forming the position coordinates of each pixel of macroscopic-void in determined the second view in step S4, in described three-view diagram take these coordinates as the value foundation, take out one by one the pixel value at respective coordinates place, and this pixel value is copied the same position coordinates place of filling up macroscopic-void described in the second view.
6. device of filling up the image cavity, it is characterized in that by data input module, rendering module, first fill up module, computing module, second is filled up module and is formed, this data input module, rendering module, computing module, second are filled up module and are connected successively, and first fills up module is connected with rendering module;
In data input module, in pending sequence of frames of video and depth map corresponding to each frame, elect the target frame that current wish is processed as first view, and select depth map corresponding to this first view;
In rendering module, according to described the first view and its corresponding depth map, utilize the image rendering algorithm, the virtual view of wanting to ask is played up obtained the second view;
In computing module, except depth map corresponding to the first view, in depth map corresponding to each remaining frame, seek the block of pixels identical with macroscopic-void place location of pixels in the second view;
In second filled up module, the block of pixels that computing module is determined copied and fills up in macroscopic-void in the second view;
Described first fills up module is comprised of the empty scale identifying unit that connects successively and little hole-filling unit, in this cavity scale identifying unit, according to default threshold value, determines area in the second view and is called little cavity less than the cavity of set threshold value; In little hole-filling unit, according to the result of determination of described empty scale identifying unit, little cavity is filled up.
7. the device of filling up the image cavity according to claim 6, it is characterized in that computing module chosen unit, similarity calculated, comparing unit, chosen the unit and form by the macroscopic-void position calculation unit, the compared pixels piece that connect successively, in described macroscopic-void position calculation unit, result of determination according to empty scale identifying unit, calculate in the second view area greater than the cavity of set threshold value, i.e. the position of macroscopic-void in the second view;
In the compared pixels piece is chosen the unit, in depth map corresponding to each frame of the remainder except depth map corresponding to the first view, choose the block of pixels identical with macroscopic-void position in the second view that the macroscopic-void position calculation unit calculates;
In similarity calculated, choose the result of choosing of unit according to the compared pixels piece, all pixel values in block of pixels are superposeed, obtain corresponding each block of pixels and value;
In comparing unit, relatively each block of pixels corresponding and value, select and be worth depth map under minimum block of pixels;
In choosing the unit, the depth map of selecting according to comparing unit in frame of video corresponding to this depth map, is chosen the block of pixels identical with macroscopic-void position in the second view.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310277116.5A CN103384343B (en) | 2013-07-02 | 2013-07-02 | A kind of method and device thereof filling up image cavity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310277116.5A CN103384343B (en) | 2013-07-02 | 2013-07-02 | A kind of method and device thereof filling up image cavity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103384343A true CN103384343A (en) | 2013-11-06 |
| CN103384343B CN103384343B (en) | 2015-08-19 |
Family
ID=49492027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310277116.5A Active CN103384343B (en) | 2013-07-02 | 2013-07-02 | A kind of method and device thereof filling up image cavity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103384343B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104954780A (en) * | 2015-07-01 | 2015-09-30 | 南阳师范学院 | DIBR (depth image-based rendering) virtual image restoration method applicable to high-definition 2D/3D (two-dimensional/three-dimensional) conversion |
| CN106028020A (en) * | 2016-06-21 | 2016-10-12 | 电子科技大学 | Multi-direction prediction based virtual visual-angle image cavity filling method |
| CN108028917A (en) * | 2015-10-16 | 2018-05-11 | 镭亚股份有限公司 | Near-eye display based on multiple beam diffraction grating |
| CN110602476A (en) * | 2019-08-08 | 2019-12-20 | 南京航空航天大学 | Hole filling method of Gaussian mixture model based on depth information assistance |
| WO2020098530A1 (en) * | 2018-11-15 | 2020-05-22 | 腾讯科技（深圳）有限公司 | Picture rendering method and apparatus, and storage medium and electronic apparatus |
| CN111314688A (en) * | 2020-03-16 | 2020-06-19 | 北京迈格威科技有限公司 | Disparity map hole filling method and device and electronic system |
| CN111768353A (en) * | 2020-06-30 | 2020-10-13 | 北京华捷艾米科技有限公司 | Hole filling method and device for three-dimensional model |
| CN116542938A (en) * | 2023-05-09 | 2023-08-04 | 深圳聚源视芯科技有限公司 | Binocular vision-based parallax post-processing system and method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101610423A (en) * | 2009-07-13 | 2009-12-23 | 清华大学 | A kind of method and apparatus of rendering image |
| CN103051908A (en) * | 2012-12-26 | 2013-04-17 | 四川虹微技术有限公司 | Disparity map-based hole filling device |
-
2013
- 2013-07-02 CN CN201310277116.5A patent/CN103384343B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101610423A (en) * | 2009-07-13 | 2009-12-23 | 清华大学 | A kind of method and apparatus of rendering image |
| CN103051908A (en) * | 2012-12-26 | 2013-04-17 | 四川虹微技术有限公司 | Disparity map-based hole filling device |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104954780A (en) * | 2015-07-01 | 2015-09-30 | 南阳师范学院 | DIBR (depth image-based rendering) virtual image restoration method applicable to high-definition 2D/3D (two-dimensional/three-dimensional) conversion |
| CN108028917A (en) * | 2015-10-16 | 2018-05-11 | 镭亚股份有限公司 | Near-eye display based on multiple beam diffraction grating |
| CN108028917B (en) * | 2015-10-16 | 2021-06-29 | 镭亚股份有限公司 | Near-to-eye display based on multibeam diffraction grating |
| CN106028020A (en) * | 2016-06-21 | 2016-10-12 | 电子科技大学 | Multi-direction prediction based virtual visual-angle image cavity filling method |
| WO2020098530A1 (en) * | 2018-11-15 | 2020-05-22 | 腾讯科技（深圳）有限公司 | Picture rendering method and apparatus, and storage medium and electronic apparatus |
| US11488346B2 (en) | 2018-11-15 | 2022-11-01 | Tencent Technology (Shenzhen) Company Limited | Picture rendering method and apparatus, storage medium, and electronic apparatus |
| CN110602476B (en) * | 2019-08-08 | 2021-08-06 | 南京航空航天大学 | Hole filling method of Gaussian mixture model based on depth information assistance |
| CN110602476A (en) * | 2019-08-08 | 2019-12-20 | 南京航空航天大学 | Hole filling method of Gaussian mixture model based on depth information assistance |
| CN111314688B (en) * | 2020-03-16 | 2022-05-17 | 北京迈格威科技有限公司 | Disparity map hole filling method and device and electronic system |
| CN111314688A (en) * | 2020-03-16 | 2020-06-19 | 北京迈格威科技有限公司 | Disparity map hole filling method and device and electronic system |
| CN111768353A (en) * | 2020-06-30 | 2020-10-13 | 北京华捷艾米科技有限公司 | Hole filling method and device for three-dimensional model |
| CN111768353B (en) * | 2020-06-30 | 2023-11-03 | 北京华捷艾米科技有限公司 | Hole filling method and device for three-dimensional model |
| CN116542938A (en) * | 2023-05-09 | 2023-08-04 | 深圳聚源视芯科技有限公司 | Binocular vision-based parallax post-processing system and method |
| CN116542938B (en) * | 2023-05-09 | 2024-04-09 | 深圳聚源视芯科技有限公司 | Binocular vision-based parallax post-processing system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103384343B (en) | 2015-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103384343B (en) | A kind of method and device thereof filling up image cavity | |
| US5734743A (en) | Image processing method and apparatus for block-based corresponding point extraction | |
| CN102902355B (en) | The space interaction method of mobile device | |
| CN106023303B (en) | A method of Three-dimensional Gravity is improved based on profile validity and is laid foundations the dense degree of cloud | |
| US11328479B2 (en) | Reconstruction method, reconstruction device, and generation device | |
| KR100793076B1 (en) | Edge-adaptive stereo/multi-view image matching apparatus and its method | |
| CN101610423B (en) | Method and device for rendering image | |
| EP2595116A1 (en) | Method for generating depth maps for converting moving 2d images to 3d | |
| CN105374039B (en) | Monocular image depth information method of estimation based on contour acuity | |
| US20150138193A1 (en) | Method and device for panorama-based inter-viewpoint walkthrough, and machine readable medium | |
| WO2007052191A2 (en) | Filling in depth results | |
| CN109644280B (en) | Method for generating hierarchical depth data of scene | |
| CN102740096A (en) | Space-time combination based dynamic scene stereo video matching method | |
| CN104065946A (en) | Cavity filling method based on image sequence | |
| Tomiyama et al. | Algorithm for dynamic 3D object generation from multi-viewpoint images | |
| CN110096993A (en) | The object detection apparatus and method of binocular stereo vision | |
| KR20200107217A (en) | Method and apparatus for image processing | |
| CN103049903B (en) | Binocular stereoscopic matching method for stereoscopic vision system | |
| CN108924434B (en) | Three-dimensional high dynamic range image synthesis method based on exposure transformation | |
| CN104537637B (en) | A kind of single width still image depth estimation method and device | |
| Berent et al. | Segmentation of epipolar-plane image volumes with occlusion and disocclusion competition | |
| CN108805841B (en) | Depth map recovery and viewpoint synthesis optimization method based on color map guide | |
| KR102091860B1 (en) | Method and apparatus for image encoding | |
| CN107886533A (en) | Vision significance detection method, device, equipment and the storage medium of stereo-picture | |
| CN103945206A (en) | Three-dimensional picture synthesis system based on comparison between similar frames |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |