CN110602476A - Hole filling method of Gaussian mixture model based on depth information assistance - Google Patents
Hole filling method of Gaussian mixture model based on depth information assistance Download PDFInfo
- Publication number
- CN110602476A CN110602476A CN201910729173.XA CN201910729173A CN110602476A CN 110602476 A CN110602476 A CN 110602476A CN 201910729173 A CN201910729173 A CN 201910729173A CN 110602476 A CN110602476 A CN 110602476A
- Authority
- CN
- China
- Prior art keywords
- texture
- background
- depth
- interval
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 239000000203 mixture Substances 0.000 title claims abstract description 20
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 10
- 230000008569 process Effects 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 4
- 239000012528 membrane Substances 0.000 claims description 3
- 150000001875 compounds Chemical class 0.000 claims description 2
- 230000003044 adaptive effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000015572 biosynthetic process Effects 0.000 description 8
- 238000003786 synthesis reaction Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
- G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a depth information assisted Gaussian mixture model-based hole filling method, which comprises the following steps: (1) acquiring image sequence data comprising a plurality of views, wherein the image sequence data of each view comprises texture frames and depth frames of a plurality of frames; (2) determining a GMM texture background in the image sequence data by adopting a Gaussian mixture model algorithm; (3) dividing the texture frame and the depth frame into a plurality of intervals according to time sequence, and carrying out histogram equalization processing on the depth frame in each interval; (4) calculating a foreground depth correlation algorithm for the texture frame and the depth frame in each interval to correspondingly obtain an FDC interval texture background and an FDC interval depth background; (5) and combining the FDC interval texture background obtained by each interval according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background. The method of the invention can visually observe better cavity filling effect and can obtain obvious objective gain in the reciprocating motion sequence.
Description
Technical Field
The invention relates to the technical field of three-dimensional video hole filling, in particular to a hole filling method of a Gaussian mixture model based on depth information assistance.
Background
Virtual viewpoint synthesis has become a core part of three-dimensional video (3-D) research because it can avoid transmitting a large number of viewpoint images in Free Viewpoint Video (FVV). The most common viewpoint synthesis technique is Depth Image Based Rendering (DIBR) using texture images and their related Depth maps, and although DIBR techniques have been developed more and more fully, the problem of hole filling in virtual views is still a relatively difficult problem.
Generally speaking, there are two main cases of generating a hole, namely a discontinuous area in the virtual view caused by an inaccurate depth value, and a background object occluded in the reference view may become visible in the virtual view, thereby causing a large piece of hole in the virtual view. Image restoration is a commonly used technique for filling up holes, and the pixel values of a hole area are determined mainly according to the spatial texture correlation between adjacent pixels. However, there is usually no spatial texture correlation between the occluded region and the foreground region, and when there is a large number of holes generated by occlusion in the virtual view, the difference between the result of using the normal image for repairing and the actual real view image is often large.
To get better hole filling results, the most popular technique at present is GMM (Gaussian Mixture Model) algorithm. The GMM may generate a stable scene background from the temporal correlation information, where each pixel is modeled and represented by multiple gaussian models that are iteratively updated at a learning rate when new sample data appears. Because occluded regions that cannot be restored in a virtual view are typically background, filling occluded regions with a GMM-generated background is a very efficient method. However, for foreground regions that are periodically rotated or reciprocated, the GMM will often mistakenly treat them as background and fill in the hole regions, since the foreground repeats at the same position over multiple time frames. The composite image obtained in this way will have a larger difference from the image taken by the real camera at this position, and further affect the viewing effect of the 3D video.
Disclosure of Invention
The purpose of the invention is as follows: in order to overcome the defects of the prior art, the invention provides a hole filling method of a Gaussian mixture model based on depth information assistance, and the method can solve the problem of poor hole filling effect caused by the use of the existing GMM model.
The technical scheme is as follows: the invention relates to a depth information assisted Gaussian mixture model-based hole filling method, which comprises the following steps:
(1) using image sequence data comprising a plurality of views, the image sequence data for each view containing a texture frame and a depth frame for a number of frames;
(2) determining a GMM texture background in the image sequence data by adopting a Gaussian mixture model algorithm;
(3) dividing the texture frame and the depth frame into a plurality of intervals according to time sequence, and carrying out histogram equalization processing on the depth frame in each interval;
(4) calculating a foreground depth correlation algorithm for the texture frame in each interval to correspondingly obtain an FDC interval texture background; calculating a foreground depth correlation algorithm for the depth frame after binarization in each interval to correspondingly obtain an FDC interval depth background;
(5) combining the FDC interval texture background obtained from each interval according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background;
further, comprising:
the step (2) of determining the GMM texture background in the image sequence data by using a gaussian mixture model algorithm specifically includes:
(21) all texture frames and depth frames in the image sequence data are used in a GMM process, the texture frames are used for constructing a Gaussian model, the depth frames are used for determining model updating parameters, and the method comprises the following steps for single pixel points corresponding to different frames:
(211) the kth Gaussian model of the texture frame of the tth frame is marked as etak,tModel mean is recorded as μk,tAnd the standard deviation is recorded as σk,tAnd the weight is recorded as ωk,tK is more than or equal to 1 and less than or equal to M, and M is the total number of Gaussian models constructed on each pixel point;
(212) calculating the pixel value X of the current pixel point of the next frame t +1tDifference from mean of each Gaussian model|Xt-μk,t-1If Xt-μk,t-1|＜2.5σk,tIf so, judging that the pixel value of the current pixel point of the next frame belongs to the Gaussian model, and updating the learning rate of the model;
otherwise, if | Xt-μk,t-1|≥2.5σk,tIf so, judging that the pixel value of the current pixel point of the next frame does not belong to any one of the M Gaussian models, deleting the Gaussian model with the minimum weight-variance ratio in the M models, and constructing a new Gaussian model for replacement;
(22) all the compounds obtained in step (21) contain one omegak,tWeight ω of other M-1 models, model 1k,tDetermining the pixel point which is 0 as a pixel point which only contains one Gaussian model;
(23) and (3) newly building a membrane map, recording the position of only one Gaussian model by adopting a pixel value of 0, determining the position as a background position, and recording the position of a plurality of Gaussian model pixels by adopting a pixel value of 255, thereby determining the GMM texture background.
Further, comprising:
in the step (212), the learning rate of the model is updated, and the learning rate updating formula is as follows:
where α is the initial learning rate, ε is the empirical value in the sequence, and is the depth pixel value that can distinguish the foreground and background regions observed by the depth frame, dt(x, y) is a pixel value at a (x, y) position on the t-th frame depth frame, αtIs the updated learning rate.
Further, comprising:
in the step (4), the calculation of the foreground depth correlation algorithm is performed on the texture frame in each interval, and the FDC interval texture background is correspondingly obtained, which specifically includes:
(41) generating an initial FDC texture background by using a first frame in a manner of filling a blank image by adopting a region which is distinguished as the background by a binary depth map in the first frame, namely obtaining a background texture region in a first frame texture map corresponding to the background region of the first frame;
(42) updating an unfilled region in the FDC texture background by using a second frame texture map corresponding to a region in which the second frame in the binary depth map of the second frame becomes the background;
(43) and executing the same process as the second frame on all the texture frames in the subsequent interval so as to obtain the FDC interval texture background of the interval.
Further, comprising:
in the step (4), the depth frame after binarization in each interval is subjected to foreground depth correlation algorithm calculation, and the depth background of the FDC interval is correspondingly obtained.
Further, comprising:
in the step (5), the FDC interval texture background obtained from each interval is combined according to the depth information included in the FDC interval depth background to finally obtain the FDC texture background, including:
for a single pixel point (x, y), the texture background corresponding to the interval depth background with the minimum depth value is used as the final texture background, which is expressed as psi if there isdk(x,y)＝min(ψdi(x, y)), thentf(x,y)＝ψtk(x, y) whereintf(x, y) is the final FDC texture background at point (x, y), ψdi(x, y) is the FDC interval depth background, ψ, corresponding to the ith interval at point (x, y)tkAnd (x, y) is the FDC interval texture background corresponding to the kth interval of the point (x, y). k is the resulting psi with the smallest depth valuediThe number i, i ∈ [1, K ] of (x, y)]Where t and d in the subscript are used to distinguish between texture and depth map frames.
Further, comprising:
in the step (6), adaptively filling all image frame numbers by adopting the GMM texture background and the FDC texture background, including:
filling pixel points containing a Gaussian model with pixel points at positions corresponding to GMM texture backgrounds; otherwise, filling the pixel points at the corresponding positions of the FDC texture background
Has the advantages that: compared with the prior art, the invention has the following remarkable advantages: 1. from the perspective of generating a reliable background, in order to enable a traditional Gaussian mixture model to have a good background generation effect on a periodically moving video sequence, the invention provides a strategy of adjusting the learning rate by adopting depth information; 2. in order to obtain a more accurate foreground and background classification result, all frames in the sequence are subjected to interval division, an FDC interval texture background is generated in each interval, and the backgrounds are organically combined in the result; 3. the method also combines the GMM texture background result and the FDC texture background result in a self-adaptive manner according to the number of Gaussian models of each pixel, further improves the background, is suitable for filling the holes generated by occlusion in the virtual view, and generates better filling effect.
Drawings
FIG. 1 is a sample of inaccurate discrimination results in discriminating foreground and background in the prior FDC method;
FIG. 2 is a general algorithm flow framework for hole filling in the present method;
FIG. 3 is a modified FDC process;
FIG. 4 is a schematic diagram of a modified FDC process in which the original FDC process is used;
fig. 5 shows subjective results of experimental examples: 5a is the virtual viewpoint synthesis result of 29 th frame from Ballet sequence camera 0 to camera 1; 5b is the virtual viewpoint synthesis result of the 6 th frame from the cameras 5 to 6 of the Breakdancers sequence; 5c is a void filling result of 29 th frames from the Ballet sequence camera 0 to the camera 1; 5d are the hole filling results for Breakdancers sequence camera 5 through camera 6 frame 6.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to the invention, based on the principle that an occlusion area which is invisible in a reference view and visible in a virtual view is usually a background area, a sliding window method is added on the basis of FDC, and GMM texture background and FDC texture background are combined according to the number of Gaussian models, so that a reliable background result is obtained, and the purpose of effectively filling holes is achieved.
In the method of filling holes with the resulting background, accurate discrimination of foreground and background is crucial. Although the original FDC method can produce good subjective and objective results in the reciprocating motion sequence, it still has the problem of misclassifying the foreground as the background, as shown in fig. 1, the oval frame area is the area where the foreground is misclassified as the background, and once the foreground area is misclassified as the background, the area will always exist in the background result. This makes the partitioning result of the first frame crucial in gradually acquiring the FDC background in time domain.
In order to avoid the situation, the invention adopts a sliding window mode, divides the texture frame and the depth frame in the sequence into a plurality of intervals according to time sequence, and executes the FDC method for each interval, so that each interval obtains an FDC interval texture background result. The final FDC texture background is determined by combining the multiple FDC interval texture background results according to the depth information, so that the situation that the foreground is mistakenly divided into the background and cannot be recovered under the condition of a single interval (namely, the interval is not divided) can be avoided.
First, in order to avoid misclassification of foreground pixels into background pixels, the learning rate in the GMM is adjusted by using depth information, so that the proportion of the foreground pixels in the model is reduced and the proportion of the background pixels in the model is increased. Furthermore, an improved Foreground Depth Correlation (FDC) algorithm is proposed, which generates background frames by tracking the variation of foreground depth in the time domain. Compared with the existing algorithm, the algorithm uses a sliding window to obtain a plurality of background reference frames, and the reference frames are fused together by using depth information to generate a more accurate background frame. Finally, texture background pixels in the GMM and FDC are adaptively selected to fill the hole. By adopting the method of the invention, better cavity filling effect can be observed by naked eyes, and obvious objective gain can be obtained in the reciprocating motion sequence. The method specifically comprises the following steps:
(1) acquiring image sequence data comprising a plurality of views, wherein the image sequence data of each view comprises a texture frame and a depth frame of a plurality of frames; each view is different, and the number of texture frames and depth frames is the same, the holes of the present invention can be obtained by the following method: the texture frames and the depth frames in the first view correspondingly acquire the virtual texture frames with the same frame number in the second view through a virtual viewpoint synthesis algorithm, and due to the shielding effect of the foreground, the virtual texture frames have large holes at the boundary of the foreground background, so that the viewing effect is influenced.
(2) Determining a GMM texture background in the image sequence data by adopting a Gaussian mixture model algorithm;
an acquisition method of a reference texture background is provided. Firstly, all texture frames and depth frames in a sequence are used in a GMM process, the texture frames are used for constructing a Gaussian model, and the depth frames are used for determining model updating parameters. Each pixel in the image is represented by M gaussian models (typically 3-5 models), and is continually updated according to the pixel value changes of subsequent frames.
For a single pixel, the kth Gaussian model of the tth frame is denoted as ηk,tModel mean is recorded as μk,tAnd the standard deviation is recorded as σk,tAnd the weight is recorded as ωk,tWherein, in the step (A),the first frame texture frame in the sequence is used for constructing an initialized Gaussian model of each pixel;
calculating the difference | X between the pixel value of the current pixel of the next frame and the mean value of each Gaussian modelt-μk,t-1L. If | Xt-μk,t-1|＜2.5σk,tIf so, judging that the pixel value of the current pixel of the next frame belongs to the Gaussian model, and updating the model according to a model updating parameter, wherein the model updating parameter is determined according to the following formula:
where α is the learning rate and is a constant, ε is the empirical value in the sequence and is the depth pixel value observed from the depth frame that can distinguish the foreground and background regions, dt(x, y) is the pixel value at the (x, y) position on the t-th frame depth map, αt(x, y) the updated learning rate is usually different for each pixel of each frame.
For each pixel, the process of model update is as follows:
μk,t←(1-αt)μk,t-1+αXt
ωk,t←(1-αt)ωk,t-1+α
if the pixel value of the current pixel of the next frame does not belong to any Gaussian model, the weight-to-variance ratio omega is deletedk,t/σk,tA minimum of one model and constructing a new model. Such an approach may generate a better background, resulting in a higher filling efficiency.
After the GMM process is completed, some pixels contain a model with the weight of 1, and other M-1 models have the weight of 0, that is, only one model. This means that in all texture frames of the sequence, the pixel values of the same location have not changed, for which pixels it is determined to be background, and will be applied to the GMM texture background ψgIn (1).
And (3) newly building a membrane map, recording the position of only one Gaussian model by adopting a pixel value of 0, determining the position as a background position, and recording the position of a plurality of Gaussian model pixels by adopting a pixel value of 255, thereby determining the GMM texture background.
(3) Dividing the texture frame and the depth frame into a plurality of intervals according to time sequence, and carrying out histogram equalization processing on the depth frame in each interval;
and dividing all texture frames and depth frames in the sequence into K intervals according to time sequence, and processing each depth map in the intervals by using a K-Means method to obtain a binary depth map. Wherein k is 2. The pixel value 0 represents the background and the value 255 represents the foreground.
(4) Calculating a foreground depth correlation algorithm for the texture frame in each interval to correspondingly obtain an FDC interval texture background; calculating a foreground depth correlation algorithm for the depth frame after binarization in each interval to correspondingly obtain an FDC interval depth background;
referring to fig. 4, performing foreground depth correlation algorithm calculation on the texture frame in each interval to obtain the FDC interval texture background correspondingly includes:
(41) generating an initial FDC texture background by using a first frame in a manner of filling a blank image by adopting a region which is distinguished as the background by a binary depth map in the first frame, namely obtaining a background texture region in a first frame texture map corresponding to the background region of the first frame;
(42) updating an unfilled region in the FDC texture background by using a second frame texture map corresponding to a region in which the second frame in the binary depth map of the second frame becomes the background;
(43) and executing the same process as the second frame on all the texture frames in the subsequent interval so as to obtain the FDC interval texture background of the interval.
Once an area in all previous frames classified as foreground is found, in a frame classified as background, this area in the texture frame will be used to recover the FDC interval texture background.
The technical details of obtaining the FDC interval depth background and the FDC interval texture background are the same, but the method has different functions, wherein the former object is a texture frame, and the latter object is a depth frame after binarization.
(5) Combining the FDC interval texture background obtained from each interval according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background;
referring to fig. 3, K obtained interval background depth maps are compared, and for K depth pixel points corresponding to each position, the pixel point with the minimum depth value is selectedFilling the background with texture pixel points at corresponding positions in a texture background image corresponding to the depth image; that is, for a single pixel point (x, y), we use the texture background corresponding to the interval depth background with the minimum depth value as the final texture background, which is expressed as psi if there isdk(x,y)＝min(ψdi(x, y)), thentf(x,y)＝ψtk(x, y) whereintf(x, y) is the final FDC texture background at point (x, y), ψdi(x, y) is the FDC interval depth background, ψ, corresponding to the ith interval at point (x, y)tkAnd (x, y) is the FDC interval texture background corresponding to the kth interval of the point (x, y). k is the resulting psi with the smallest depth valuediThe number i, i ∈ [1, K ] of (x, y)]Where t and d in the subscript are used to distinguish between texture and depth map frames.
The invention determines the final FDC method background by combining the background results of a plurality of FDC intervals according to the depth information, thereby avoiding the situation that the foreground is wrongly divided into the background and can not be retrieved under the condition of a single interval (namely, the interval is not divided).
(6) And adaptively filling all image frame numbers by adopting the GMM texture background and the FDC texture background to finish a reference texture background image, and further filling the cavity area of the virtual texture frame in the virtual view with the corresponding area background.
Filling pixel points containing a Gaussian model with pixel points at positions corresponding to GMM texture backgrounds; otherwise, filling the pixel points at the corresponding positions of the FDC texture background, namely: finally, the GMM texture background and the FDC texture background are combined, and the combined background is marked as psiB. For pixels in GMM with only one model, ψB(x,y)＝ψg(x, y), otherwise ψB(x,y)＝ψtf(x,y)。
To verify the effectiveness of the present invention, the present invention is further described in detail below with reference to a specific embodiment. For ease of explanation, and without loss of generality, the following assumptions are made:
the method proposed by the invention is intended to use the Ballet test sequence in the Microsoft data set for testing, the resolution is 1024 x 768, texture and depth data of 8 different views are contained, and the texture and depth data are all 100 frames. Accompanied by the inner and outer matrix parameters for each camera (camera 0 to camera 7 arranged from right to left), and the largest and smallest true depth value in the sequence.
In this embodiment, the camera 0 view is used to restore the camera 1 view through a virtual viewpoint synthesis algorithm and a hole filling technique. First, 100 frames of virtual texture frames of the camera 1 view are respectively acquired through a virtual viewpoint synthesis algorithm using 100 frames of texture frames and depth frames in the camera 0 view. Due to the occlusion effect of the foreground, these virtual texture frames all have large pieces of holes at the boundary of the foreground background, as shown by the gray areas behind the human in fig. 5a and b, and fig. 5c and d are the hole filling respectively performed by the method of the present invention. In order to make the virtual view have better viewing effect, the invention adopts a mode of self-adaptively selecting the GMM texture background and the FDC texture background for filling the holes.
To obtain a better GMM texture background, depth information is added to adjust the learning rate in the GMM parameter update process. The depth map data in the Ballet test sequence in the Microsoft data set is observed to obtain the empirical depth value epsilon for distinguishing the foreground and the background, which is 60, and the depth data of each frame is combined and then is processed by a formulaTo adjust the learning rate. And (4) using the data of each frame for adjusting Gaussian model parameters, and obtaining the GMM texture background by using the final model parameters.
For the final combination with the FDC texture background, when the final model parameters are obtained, a film map is created, and the positions of the pixels containing only one gaussian model and a plurality of gaussian models are recorded by pixel values 0 and 255.
In order to obtain the FDC texture background, histogram equalization processing is carried out on the depth frame in advance, and FDC operation is carried out on 100 frames of texture frames and the processed depth frame in different regions. Since 100 frames of data exist in a single view in the Ballet sequence, and when the sequence is used for carrying out the FDC operation, the background obtained by measuring about 30 frames does not change any more, the 100 frames of data are divided into 3 intervals for carrying out the FDC operation. And organically combining the obtained 3 interval texture background images through corresponding depth background images to obtain a final FDC texture background.
FDC operation of a single interval as shown in FIG. 4, a binary depth map is obtained by a K-Means method for each frame of depth map in the interval. Firstly initializing a background image to be filled, determining a background area of a first frame of texture image through a first frame of binary depth image, filling the texture area into the background, then determining a background area of a second frame of texture image through a second frame of binary depth image, filling the texture area into the background area which is not filled yet, and repeating the steps for each next frame to gradually complete the background.
And according to the film image obtained by the GMM process, filling the GMM texture background pixel value into the position of the film image pixel value of 0 for the final background image, otherwise, filling the FDC texture background pixel. And after the final background image is obtained, synthesizing the background of the virtual viewpoint by using a virtual viewpoint synthesis algorithm, and filling the hole areas of the 100 frames of virtual texture frames with the backgrounds of the corresponding areas respectively.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made in the embodiments of the present invention without departing from the spirit or scope of the embodiments of the invention. Thus, if such modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to encompass such modifications and variations.
Claims (6)
1. A hole filling method based on a depth information assisted Gaussian mixture model is characterized by comprising the following steps:
(1) acquiring image sequence data comprising a plurality of views, wherein the image sequence data of each view comprises texture frames and depth frames of a plurality of frames;
(2) determining a GMM texture background in the image sequence data by adopting a Gaussian mixture model algorithm;
(3) dividing the texture frame and the depth frame into a plurality of intervals according to time sequence, and carrying out histogram equalization processing on the depth frame in each interval;
(4) calculating a foreground depth correlation algorithm for the texture frame in each interval to correspondingly obtain an FDC interval texture background; calculating a foreground depth correlation algorithm for the depth frame after binarization in each interval to correspondingly obtain an FDC interval depth background;
(5) combining the FDC interval texture background obtained from each interval according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background;
(6) and adaptively generating a reference texture background image by adopting the GMM texture background and the FDC texture background, and further filling the cavity area of the virtual texture frame in the virtual view with the corresponding area background.
2. The method for filling up a hole based on the depth information assisted gaussian mixture model according to claim 1, wherein in the step (2), the GMM texture background in the image sequence data is determined by using a gaussian mixture model algorithm, which specifically comprises:
(21) all texture frames and depth frames in the image sequence data are used in a GMM process, the texture frames are used for constructing a Gaussian model, the depth frames are used for determining model updating parameters, and the method comprises the following steps for single pixel points corresponding to different frames:
(211) the kth Gaussian model of the texture frame of the tth frame is marked as etak,tModel mean is recorded as μk,tAnd the standard deviation is recorded as σk,tAnd the weight is recorded as ωk,tK is more than or equal to 1 and less than or equal to M, and M is the total number of Gaussian models constructed on each pixel point;
(212) calculating the pixel value X of the current pixel point of the next frame t +1tDifference | X from the mean of each Gaussian modelt-μk,t-1If Xt-μk,t-1|＜2.5σk,tThen, the pixel value of the current pixel point of the next frame is determinedBelonging to the Gaussian model and updating the learning rate of the model;
otherwise, if | Xt-μk,t-1|≥2.5σk,tIf so, judging that the pixel value of the current pixel point of the next frame does not belong to any one of the M Gaussian models, deleting the Gaussian model with the minimum weight-variance ratio in the M models, and constructing a new Gaussian model for replacement;
(22) all the compounds obtained in step (21) contain one omegak,tWeight ω of other M-1 models, model 1k,tDetermining the pixel point which is 0 as a pixel point which only contains one Gaussian model;
(23) and (3) newly building a membrane map, recording the position of only one Gaussian model by adopting a pixel value of 0, determining the position as a background position, and recording the position of a plurality of Gaussian model pixels by adopting a pixel value of 255, thereby determining the GMM texture background.
3. The method for filling holes in a gaussian mixture model based on depth information assistance according to claim 2, wherein in step (212), the learning rate of the model is updated according to the formula:
where α is the initial learning rate, ε is the empirical value in the sequence, and is the depth pixel value that can distinguish the foreground and background regions observed by the depth frame, dt(x, y) is a pixel value at a (x, y) position on the t-th frame depth frame, αtIs the updated learning rate.
4. The method for filling up a hole based on the depth information assisted gaussian mixture model according to claim 1, wherein in the step (4), the calculation of the foreground depth correlation algorithm is performed on the texture frame in each interval, and the FDC interval texture background is correspondingly obtained, which specifically includes:
(41) generating an initial FDC texture background by using a first frame in a manner of filling a blank image by adopting a region which is distinguished as the background by a binary depth map in the first frame, namely obtaining a background texture region in a first frame texture map corresponding to the background region of the first frame;
(42) updating an unfilled region in the FDC texture background by using a second frame texture map corresponding to a region in which the second frame in the binary depth map of the second frame becomes the background;
(43) and executing the same process as the second frame on all the texture frames in the subsequent interval so as to obtain the FDC interval texture background of the interval.
5. The method for filling up a hole based on the depth information assisted gaussian mixture model as claimed in claim 4, wherein in the step (5), the FDC interval texture background obtained from each interval is combined according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background, and the method comprises:
for a single pixel point (x, y), the texture background corresponding to the interval depth background with the minimum depth value is used as the final texture background, which is expressed as psi if there isdk(x,y)＝min(ψdi(x, y)), thentf(x,y)＝ψtk(x, y) whereintf(x, y) is the final FDC texture background at pixel point (x, y), ψdi(x, y) is the depth background of FDC interval corresponding to the ith interval of pixel point (x, y), psitk(x, y) is the texture background of the FDC interval corresponding to the kth interval of the pixel point (x, y); k is the resulting psi with the smallest depth valuediThe number i, i ∈ [1, K ] of (x, y)](ii) a Where t and d in the subscript are used to distinguish between texture and depth map frames.
6. The method for filling holes in a gaussian mixture model based on depth information assistance as claimed in claim 2, wherein in the step (6), the adaptive filling of all image frame numbers by using the GMM texture background and the FDC texture background comprises:
filling pixel points containing a Gaussian model with pixel points at positions corresponding to GMM texture backgrounds; otherwise, filling the pixel points at the corresponding positions of the FDC texture background.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910729173.XA CN110602476B (en) | 2019-08-08 | 2019-08-08 | Hole filling method of Gaussian mixture model based on depth information assistance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910729173.XA CN110602476B (en) | 2019-08-08 | 2019-08-08 | Hole filling method of Gaussian mixture model based on depth information assistance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110602476A true CN110602476A (en) | 2019-12-20 |
| CN110602476B CN110602476B (en) | 2021-08-06 |
Family
ID=68853790
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910729173.XA Active CN110602476B (en) | 2019-08-08 | 2019-08-08 | Hole filling method of Gaussian mixture model based on depth information assistance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110602476B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113487660A (en) * | 2021-06-16 | 2021-10-08 | 普联国际有限公司 | Depth information fused moving target detection method, device, medium and equipment |
| WO2022022548A1 (en) * | 2020-07-31 | 2022-02-03 | 阿里巴巴集团控股有限公司 | Free viewpoint video reconstruction and playing processing method, device, and storage medium |
| JP2023524326A (en) * | 2021-04-08 | 2023-06-12 | グーグル エルエルシー | Neural blending for novel view synthesis |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0330114A (en) * | 1989-06-27 | 1991-02-08 | Toshiba Corp | Servo equipment and servo method for data recording and reproducing device |
| US7277118B2 (en) * | 1999-08-09 | 2007-10-02 | Fuji Xerox Co., Ltd. | Method and system for compensating for parallax in multiple camera systems |
| CN101640809A (en) * | 2009-08-17 | 2010-02-03 | 浙江大学 | Depth extraction method of merging motion information and geometric information |
| US20120106791A1 (en) * | 2010-10-27 | 2012-05-03 | Samsung Techwin Co., Ltd. | Image processing apparatus and method thereof |
| CN103384343A (en) * | 2013-07-02 | 2013-11-06 | 南京大学 | Image cavity filling method and device thereof |
| CN104268851A (en) * | 2014-09-05 | 2015-01-07 | 浙江捷尚视觉科技股份有限公司 | ATM self-service business hall behavior analysis method based on depth information |
| CN108040243A (en) * | 2017-12-04 | 2018-05-15 | 南京航空航天大学 | Multispectral 3-D visual endoscope device and image interfusion method |
| CN108681999A (en) * | 2018-05-22 | 2018-10-19 | 浙江理工大学 | SAR image target shape generation method based on depth convolutional neural networks model |
| CN108828481A (en) * | 2018-04-24 | 2018-11-16 | 朱高杰 | A kind of magnetic resonance reconstruction method based on deep learning and data consistency |
| CN108960080A (en) * | 2018-06-14 | 2018-12-07 | 浙江工业大学 | Based on Initiative Defense image to the face identification method of attack resistance |
| CN109934195A (en) * | 2019-03-21 | 2019-06-25 | 东北大学 | A kind of anti-spoofing three-dimensional face identification method based on information fusion |
-
2019
- 2019-08-08 CN CN201910729173.XA patent/CN110602476B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0330114A (en) * | 1989-06-27 | 1991-02-08 | Toshiba Corp | Servo equipment and servo method for data recording and reproducing device |
| US7277118B2 (en) * | 1999-08-09 | 2007-10-02 | Fuji Xerox Co., Ltd. | Method and system for compensating for parallax in multiple camera systems |
| CN101640809A (en) * | 2009-08-17 | 2010-02-03 | 浙江大学 | Depth extraction method of merging motion information and geometric information |
| US20120106791A1 (en) * | 2010-10-27 | 2012-05-03 | Samsung Techwin Co., Ltd. | Image processing apparatus and method thereof |
| CN103384343A (en) * | 2013-07-02 | 2013-11-06 | 南京大学 | Image cavity filling method and device thereof |
| CN104268851A (en) * | 2014-09-05 | 2015-01-07 | 浙江捷尚视觉科技股份有限公司 | ATM self-service business hall behavior analysis method based on depth information |
| CN108040243A (en) * | 2017-12-04 | 2018-05-15 | 南京航空航天大学 | Multispectral 3-D visual endoscope device and image interfusion method |
| CN108828481A (en) * | 2018-04-24 | 2018-11-16 | 朱高杰 | A kind of magnetic resonance reconstruction method based on deep learning and data consistency |
| CN108681999A (en) * | 2018-05-22 | 2018-10-19 | 浙江理工大学 | SAR image target shape generation method based on depth convolutional neural networks model |
| CN108960080A (en) * | 2018-06-14 | 2018-12-07 | 浙江工业大学 | Based on Initiative Defense image to the face identification method of attack resistance |
| CN109934195A (en) * | 2019-03-21 | 2019-06-25 | 东北大学 | A kind of anti-spoofing three-dimensional face identification method based on information fusion |
Non-Patent Citations (2)
| Title |
|---|
| NA-EUN YANG: "Depth Hole Filling Using the Depth Distribution of Neighboring Regions of Depth Holes in the Kinect SENSOR", 《IEEE》 * |
| 曾嘉: "基于图像稀疏性与多尺度神经网络的图像修复算法研究", 《万方》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022022548A1 (en) * | 2020-07-31 | 2022-02-03 | 阿里巴巴集团控股有限公司 | Free viewpoint video reconstruction and playing processing method, device, and storage medium |
| JP2023524326A (en) * | 2021-04-08 | 2023-06-12 | グーグル エルエルシー | Neural blending for novel view synthesis |
| JP7519390B2 (en) | 2021-04-08 | 2024-07-19 | グーグル エルエルシー | Neural Blending for Novel View Synthesis |
| CN113487660A (en) * | 2021-06-16 | 2021-10-08 | 普联国际有限公司 | Depth information fused moving target detection method, device, medium and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110602476B (en) | 2021-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11017586B2 (en) | 3D motion effect from a 2D image | |
| EP2087466B1 (en) | Generation of depth map for an image | |
| JP3679426B2 (en) | A system that encodes image data into multiple layers, each representing a coherent region of motion, and motion parameters associated with the layers. | |
| CN103702098B (en) | Three viewpoint three-dimensional video-frequency depth extraction methods of constraint are combined in a kind of time-space domain | |
| CN110602476B (en) | Hole filling method of Gaussian mixture model based on depth information assistance | |
| CN109462747B (en) | DIBR system cavity filling method based on generation countermeasure network | |
| Ward et al. | Depth director: A system for adding depth to movies | |
| CN103905813B (en) | Based on the DIBR hole-filling method of background extracting and divisional reconstruction | |
| CN108124489B (en) | Information processing method, apparatus, cloud processing device and computer program product | |
| Choi et al. | Space-time hole filling with random walks in view extrapolation for 3D video | |
| JP2003196661A (en) | Appearance model for visual motion analysis and visual tracking | |
| CN104966286A (en) | 3D video saliency detection method | |
| CN111832745A (en) | Data augmentation method and device and electronic equipment | |
| KR101173559B1 (en) | Apparatus and method for the automatic segmentation of multiple moving objects from a monocular video sequence | |
| CN111223059B (en) | Robust depth map structure reconstruction and denoising method based on guide filter | |
| CN115298708A (en) | Multi-view neural human body rendering | |
| CN104700384B (en) | Display systems and methods of exhibiting based on augmented reality | |
| US20230080639A1 (en) | Techniques for re-aging faces in images and video frames | |
| CN118071932A (en) | Three-dimensional static scene image reconstruction method and system | |
| Jain et al. | Enhanced stable view synthesis | |
| Guo et al. | Adaptive estimation of depth map for two-dimensional to three-dimensional stereoscopic conversion | |
| CN113034675B (en) | Scene model construction method, intelligent terminal and computer readable storage medium | |
| CN106572351A (en) | Video coding code rate control method and device | |
| CN116711301A (en) | Virtual viewpoint generating, rendering and decoding methods and devices, equipment and storage medium | |
| Zhu et al. | An improved Gaussian mixture model based hole-filling algorithm exploiting depth information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |