CN110602476A - Hole filling method of Gaussian mixture model based on depth information assistance - Google Patents

Abstract

The invention discloses a hole filling method based on a Gaussian mixture model assisted by depth information. The method includes: (1) acquiring image sequence data including multiple views, and the image sequence data of each view contains several frames of texture frames and Depth frame; (2) using the mixed Gaussian model algorithm to determine the GMM texture background in the image sequence data; (3) dividing the texture frame and the depth frame into several intervals in time sequence, and analyzing the depth frame in each interval Perform histogram equalization processing; (4) calculate the foreground depth correlation algorithm for the texture frame and depth frame in each interval, and obtain the texture background of the FDC interval and the depth background of the FDC interval; (5) obtain the depth background of each interval The FDC interval texture background is combined with the depth information contained in the FDC interval depth background to finally obtain the FDC texture background. The method of the present invention can visually observe better cavity filling effect, and can obtain significant objective gain in the reciprocating motion sequence.

Description

A Hole Filling Method Based on Gaussian Mixture Model Assisted by Depth Information

technical field

The invention relates to the technical field of three-dimensional video hole filling, in particular to a hole filling method based on a Gaussian mixture model assisted by depth information.

Background technique

Virtual view synthesis has become a central part of three-dimensional video (3-D) research because it enables the avoidance of transmitting a large number of view images in free view video (FVV). The most commonly used view synthesis technology is Depth Image Based Rendering (DIBR, Depth Image Based Rendering) using texture images and their associated depth maps. Although DIBR technology has been developed relatively maturely, the problem of hole filling in virtual views is still a problem. A trickier question.

Generally speaking, there are two main situations that cause holes. One is the discontinuous area in the virtual view caused by inaccurate depth values, and the other is that background objects that are occluded in the reference view may become visible in the virtual view. This results in large holes in the virtual view. Image inpainting is a common hole-filling technique, which mainly determines the pixel value of the hole area according to the spatial texture correlation between adjacent pixels. However, there is usually no spatial texture correlation between the occluded area and the foreground area. In the case of large holes in the virtual view due to occlusion, the difference between the results of ordinary image inpainting and the actual real-world view image is often large. .

In order to obtain better hole-filling results, the most popular technology at present is the GMM (Gaussian Mixture Model, Gaussian Mixture Model) algorithm. GMM can generate a stable scene background based on temporal correlation information, in which each pixel is modeled and represented by multiple Gaussian models, and these Gaussian models are iteratively updated at a certain learning rate when new sample data appears. Because the occluded areas that cannot be recovered in the virtual view are usually the background, it is an effective method to use the background generated by GMM to fill the occluded areas. However, for the foreground areas that undergo periodic rotation or reciprocating motion, because the foreground repeats at the same position in multiple time frames, GMM usually mistakenly regards them as background and fills them into the hollow area. The synthetic image obtained in this way will be quite different from the image captured by the real camera at the position, thereby affecting the viewing effect of the 3D video.

Contents of the invention

Purpose of the invention: In order to overcome the deficiencies of the prior art, the present invention provides a hole filling method based on a Gaussian mixture model assisted by depth information, which can solve the problem of poor hole filling effect caused by the use of the existing GMM model.

Technical solution: The hole filling method based on the Gaussian mixture model assisted by depth information according to the present invention, the method includes:

(1) Using image sequence data comprising multiple views, the image sequence data of each view contains several frames of texture frames and depth frames;

(2) adopting the mixed Gaussian model algorithm to determine the GMM texture background in the image sequence data;

(3) Divide the texture frame and the depth frame into several intervals according to the time sequence, and perform histogram equalization processing on the depth frame in each interval;

(4) Calculate the foreground depth correlation algorithm for the texture frame in each interval, and obtain the FDC interval texture background correspondingly; perform the calculation of the foreground depth correlation algorithm for the binarized depth frame in each interval, and obtain the FDC correspondingly Interval depth background;

(5) The FDC interval texture background obtained in each interval is combined with the depth information contained in the FDC interval depth background to finally obtain the FDC texture background;

Further, include:

Described step (2), adopts mixed Gaussian model algorithm to determine the GMM texture background in described image sequence data, specifically comprises:

(21) All texture frames and depth frames in the image sequence data are used in the GMM process, the texture frames are used to build a Gaussian model, and the depth frames are used to determine model update parameters. For a single pixel corresponding to a different frame, Include the following steps:

(211) The k-th Gaussian model of the t-th texture frame is recorded as η _k,t , the model mean value is recorded as μ _k,t , the standard deviation is recorded as σ _k,t , and the weight is recorded as ω _k,t , 1≤k ≤M, M is the total number of Gaussian models constructed on each pixel;

(212) Calculate the difference between the pixel value X _t of the current pixel point in the next frame t+1 and the mean value of each Gaussian model |X _t -μ _k,t-1 |, if |X _t -μ _k,t-1 | <2.5σ _k,t , it is determined that the pixel value of the current pixel point in the next frame belongs to the Gaussian model, and the learning rate of the model is updated;

Otherwise, if |X _t -μ _k,t-1 |≥2.5σ _k,t , it is determined that the pixel value of the current pixel point in the next frame does not belong to any of the M Gaussian models, and the M models are deleted A Gaussian model with the smallest weight-to-variance ratio, and a new Gaussian model is constructed to replace it;

(22) All the models obtained in step (21) containing a ω _{k, t} = 1, the pixels of the weight ω _{k, t} = 0 of other M-1 models are determined as the pixels containing only one Gaussian model ;

(23) Create a new film map, use pixel value 0 to record the position containing only one Gaussian model, determine it as the background position, and use pixel value 255 to record the position containing multiple Gaussian model pixels, thereby determining the GMM texture background.

Further, include:

In the step (212), the learning rate of the model is updated, and the learning rate update formula is:

Among them, α is the initial learning rate, ε is the experience value in the sequence, which is the depth pixel value that can distinguish the foreground and background regions obtained from the observation of the depth frame, d _t (x, y) is the depth frame of the tth frame located at ( x, y) pixel value, α _t is the updated learning rate.

Further, include:

In described step (4), carry out the calculation of foreground depth correlation algorithm to the texture frame in each interval, correspondingly obtain FDC interval texture background, specifically include:

(41) Use the first frame to generate the initial FDC texture background, and the generation method is to fill the blank image with the area that is divided into the background by the binary depth map in the first frame, that is, to obtain the first frame texture map corresponding to the background area of the first frame The background texture area in ;

(42) Update the unfilled area in the FDC texture background with the second frame texture map corresponding to the region where the second frame in the binary depth map of the second frame becomes the background;

(43) Execute the same process as the second frame for all texture frames in the subsequent interval, and then obtain the FDC interval texture background of this interval.

Further, include:

In the step (4), the calculation of the foreground depth correlation algorithm is performed on the binarized depth frames in each interval, and the depth background of the FDC interval is correspondingly obtained.

Further, include:

In the step (5), the FDC interval texture background obtained by each interval is combined with the depth information contained in the FDC interval depth background to finally obtain the FDC texture background, including:

For a single pixel point (x, y), the texture background corresponding to the interval depth background with the minimum depth value is used as the final texture background, which is expressed as, if ψ _dk (x, y)=min(ψ _di (x ,y)), then ψ _tf (x,y)=ψ _tk (x,y), where ψ _tf (x,y) is the final FDC texture background at point (x,y), ψ _di (x, y) is the FDC interval depth background corresponding to the i-th interval of point (x, y), and ψ _tk (x, y) is the FDC interval texture background corresponding to the k-th interval of point (x, y). k is the serial number i of ψ _di (x,y) obtained with the minimum depth value, i∈[1,K], where t and d in the subscript are used to distinguish texture and depth image frames.

Further, include:

In described step (6), adaptively adopt GMM texture background and FDC texture background to carry out the filling of all image frame numbers, including:

For pixels containing a Gaussian model, fill them with the pixels corresponding to the GMM texture background; otherwise, fill them with the pixels corresponding to the FDC texture background

Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. From the perspective of generating a reliable background, in order to enable the traditional Gaussian mixture model to have a better background generation effect on video sequences with periodic motion , the present invention proposes a strategy of adjusting the learning rate using depth information; 2. In order to obtain more accurate foreground and background classification results, the present invention divides all frames in the sequence into intervals, generates an FDC interval texture background in each interval, and These backgrounds are organically combined in the result; 3. The present invention also combines the GMM texture background result and the FDC texture background result adaptively according to the number of Gaussian models of each pixel. This method further improves the background and is suitable for virtual views Filling the voids caused by occlusion in the medium, and producing a better filling effect.

Description of drawings

Figure 1 is an example of inaccurate results when distinguishing between foreground and background in the original FDC method;

Fig. 2 is the overall algorithm flow frame of this method hole filling;

Fig. 3 is the improved FDC process;

Fig. 4 is the original FDC process in the improved FDC process;

Figure 5 shows the subjective results of the experimental example: 5a is the virtual viewpoint synthesis result of the Ballet sequence camera 0 to camera 1 frame 29; 5b is the virtual viewpoint synthesis result of the Breakdancers sequence camera 5 to camera 6 frame 6; 5c is the Ballet sequence camera Hole filling results of frame 29 from camera 0 to camera 1; 5d is the hole filling result of frame 6 from camera 5 to camera 6 of the Breakdancers sequence.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The present invention is based on the principle that the occlusion area that is invisible in the reference view but visible in the virtual view is usually the background area, adds a sliding window method on the basis of FDC, and combines the GMM texture background and the FDC texture background according to the number of Gaussian models. So as to obtain reliable background results, in order to achieve the purpose of effectively filling holes.

In methods that fill holes with the resulting background, accurate distinction between foreground and background is crucial. Although the original FDC method can produce good subjective and objective results in the reciprocating motion sequence, it still has the problem of misclassifying the foreground as the background. As shown in Figure 1, the elliptical frame area is the area where the foreground is misclassified as the background, and Once the foreground area is misclassified as background, this area will always exist in the background result. This makes the segmentation result of the first frame very important in the process of gradually acquiring the FDC background in the time domain.

In order to avoid this situation, the present invention adopts a sliding window method, divides the texture frame and depth frame in the sequence into multiple intervals in time sequence, and executes the FDC method for each interval, so that each interval obtains an FDC interval texture background result. The final FDC texture background is determined by combining the texture background results of multiple FDC intervals according to the depth information, which can avoid the irreversible situation that the foreground is mistakenly divided into the background in the case of a single interval (that is, an undivided interval).

As shown in Figure 2, the present invention discloses a hole filling method based on a Gaussian mixture model assisted by depth information. First, in order to avoid misclassifying foreground pixels as background pixels, the depth information is used to adjust the learning rate in the GMM, so that The proportion of foreground pixels in the model decreases, and the proportion of background pixels in the model increases. Furthermore, an improved Foreground Depth Correlation (FDC) algorithm is proposed, which generates background frames by tracking changes in foreground depth over time. Compared with existing algorithms, this algorithm uses a sliding window to obtain multiple background reference frames, and fuses these reference frames with depth information to generate a more accurate background frame. Finally, textured background pixels in GMM and FDC are adaptively selected to fill holes. By adopting the method of the present invention, better void filling effects can be observed with the naked eye, and significant objective gain can be obtained in the reciprocating motion sequence. Specifically include:

(1) Gather image sequence data comprising a plurality of views, and the image sequence data of each view all contain texture frames and depth frames of several frames; each view is different, and the number of texture frames and depth frames is the same, the present invention The hole can be obtained by the following method: the texture frame and the depth frame in the first view use the virtual view synthesis algorithm to obtain the virtual texture frame of the same number of frames in the second view. Due to the occlusion effect of the foreground, these virtual texture frames are separated by the foreground There are large voids everywhere, which affects the viewing effect.

(2) adopting the mixed Gaussian model algorithm to determine the GMM texture background in the image sequence data;

Provides a method for obtaining the reference texture background. First, all the texture frames and depth frames in the sequence are used in the GMM process, the texture frames are used to build the Gaussian model, and the depth frames are used to determine the model update parameters. Each pixel in the image is represented by M Gaussian models (generally 3-5 models), which are constantly updated according to the pixel value changes in subsequent frames.

For a single pixel, the kth Gaussian model of the tth frame is recorded as η _k,t , the model mean value is recorded as μ _k,t , the standard deviation is recorded as σ _k,t , and the weight is recorded as ω _k,t , where, The first texture frame in the sequence is used to build an initialized Gaussian model for each pixel;

Calculate the difference |X _t -μ _k,t-1 | between the pixel value of the current pixel in the next frame and the mean value of each Gaussian model. If |X _t -μ _k,t-1 |<2.5σ _k,t , it is determined that the pixel value of the current pixel in the next frame belongs to the Gaussian model, and the model is updated according to the model update parameters, which are set according to Determined by the following formula:

Among them, α is the learning rate, which is a constant, ε is the experience value in the sequence, which is the depth pixel value that can distinguish the foreground and background regions obtained from the observation of the depth frame, d _t (x, y) is the depth map of the tth frame α _t (x, y) is the updated learning rate, which is usually different for each pixel of each frame.

For each pixel, the model update process is as follows:

μ _k,t ←(1-α _t )μ _k,t-1 +αX _t

ω _k,t ←(1-α _t )ω _k,t-1 +α

If the pixel value of the current pixel in the next frame does not belong to any of its Gaussian models, delete the model with the smallest weight-to-variance ratio ω _k,t /σ _k,t and build a new model. This way can generate a better background, making the filling more efficient.

After the GMM process is executed, some pixels contain a model with a weight of 1, and other M-1 models have a weight of 0, that is, there is only one model. This means that in all texture frames of the sequence, the pixel values at the same position do not change, and these pixels are determined to be the background and will be applied to the GMM texture background ψ _g .

Create a new membrane map, use pixel value 0 to record the location of only one Gaussian model, and determine it as the background location, use pixel value 255 to record the location of multiple Gaussian model pixels, so as to determine the GMM texture background.

(3) Divide the texture frame and the depth frame into several intervals according to the time sequence, and perform histogram equalization processing on the depth frame in each interval;

Divide all the texture frames and depth frames in the sequence into K intervals in time sequence, and use the K-Means method to process each depth map in the interval to obtain a binarized depth map. Among them, k=2. A pixel value of 0 represents the background and a value of 255 represents the foreground.

(4) Calculate the foreground depth correlation algorithm for the texture frame in each interval, and obtain the FDC interval texture background correspondingly; perform the calculation of the foreground depth correlation algorithm for the binarized depth frame in each interval, and obtain the FDC correspondingly Interval depth background;

Referring to Figure 4, the calculation of the foreground depth correlation algorithm is performed on the texture frame in each interval, and the corresponding FDC interval texture background is obtained, specifically including:

(41) Use the first frame to generate the initial FDC texture background, and the generation method is to fill the blank image with the area that is divided into the background by the binary depth map in the first frame, that is, to obtain the first frame texture map corresponding to the background area of the first frame The background texture area in ;

(42) Update the unfilled area in the FDC texture background with the second frame texture map corresponding to the region where the second frame in the binary depth map of the second frame becomes the background;

(43) Execute the same process as the second frame for all texture frames in the subsequent interval, and then obtain the FDC interval texture background of this interval.

Once it is found that an area classified as foreground in all previous frames is classified as background in a certain frame, then this area in the texture frame will be used to restore the FDC interval texture background.

The technical details of obtaining the depth background of the FDC interval and the texture background of the FDC interval are the same, but the object of the method is different. The object of the former is a texture frame, and the object of the latter is a binarized depth frame.

(5) The FDC interval texture background obtained in each interval is combined with the depth information contained in the FDC interval depth background to finally obtain the FDC texture background;

Referring to Figure 3, compare the obtained K interval background depth maps, and for the K depth pixels corresponding to each position, take the texture background map corresponding to the depth map where the pixel point with the smallest depth value is located, and the corresponding position texture pixels to fill the background; that is, for a single pixel point (x, y), we use the texture background corresponding to the interval depth background with the minimum depth value as the final texture background, which is expressed as, if ψ _dk (x, y)=min(ψ _di (x,y)), then ψ _tf (x,y)=ψ _tk (x,y), where ψ _tf (x,y) is the final FDC texture background, ψ _di (x, y) is the FDC interval depth background corresponding to the i-th interval of point (x, y), and ψ _tk (x, y) is corresponding to the k-th interval of point (x, y) FDC interval texture background. k is the serial number i,i∈[1,K] of ψ _di (x,y) with the minimum depth value obtained, where t and d in the subscript are used to distinguish texture and depth image frames.

The present invention combines the background results of multiple FDC intervals according to the depth information to determine the final FDC method background, thereby avoiding the irreversible situation that the foreground is mistakenly divided into the background in the case of a single interval (that is, an undivided interval).

(6) Adaptively use the GMM texture background and the FDC texture background to fill all the image frames, complete the reference texture background image, and then fill the hollow areas of the virtual texture frames in the virtual view with the corresponding area backgrounds.

For pixels containing a Gaussian model, fill them with the pixels corresponding to the GMM texture background; otherwise, fill them with the pixels corresponding to the FDC texture background, that is: finally combine the GMM texture background and the FDC texture background, and combine The background is denoted as ψ _B . For the pixels of only one model in GMM, ψ _B (x, y) = ψ _g (x, y), otherwise ψ _B (x, y) = ψ _tf (x, y).

In order to verify the effectiveness of the present invention, the present invention will be further described in detail below in conjunction with a specific embodiment. For the convenience of explanation and without loss of generality, the following assumptions are made:

The method proposed by the present invention intends to use the Ballet test sequence in the Microsoft data set for testing, with a resolution of 1024×768, containing 8 different views of texture and depth data, all of which are 100 frames. At the same time, the inner matrix and outer matrix parameters of each camera are attached (camera 0 to camera 7 are arranged from right to left), as well as the maximum and minimum real depth values in the sequence.

In this embodiment, the view of camera 0 is used to restore the view of camera 1 through a virtual view synthesis algorithm and a hole filling technology. First, use the 100-frame texture frame and depth frame in the camera 0 view to obtain the 100-frame virtual texture frame of the camera 1 view through the virtual view synthesis algorithm. Due to the occlusion effect of the foreground, these virtual texture frames all have large holes at the boundary between the foreground and the background, as shown in the gray area behind the person in Figure 5a and b, and Figure 5c and d are the hole filling performed by the method of the present invention respectively. In order to make the virtual view have a better viewing effect, the present invention fills these holes by adopting the adaptive selection method of acquiring the GMM texture background and the FDC texture background.

In order to obtain a better GMM texture background, depth information will be added to adjust the learning rate during the GMM parameter update process. By observing the depth map data in the Ballet test sequence in the Microsoft dataset, the empirical depth value ε=60 used to distinguish the foreground and background is obtained, combined with the depth data of each frame, through the formula to adjust the learning rate. The data of each frame is used to adjust the Gaussian model parameters, and the final model parameters are used to obtain the GMM texture background.

In order to finally combine with the FDC texture background, when obtaining the final model parameters, create a new film image, and record the positions of pixels containing only one Gaussian model and multiple Gaussian models through pixel values 0 and 255.

In order to obtain the FDC texture background, the histogram equalization processing is performed on the depth frame in advance, and the FDC operation is performed on the 100-frame texture frame and the processed depth frame. Since a single view in the Ballet sequence has 100 frames of data, and when the sequence is used for FDC operation, it is measured that the background does not change at about 30 frames, so the 100 frames of data are divided into 3 intervals for FDC operation. The obtained three interval texture background images are organically combined with the corresponding depth background images to obtain the final FDC texture background.

The FDC operation of a single interval is shown in Figure 4, and the binary depth map is obtained by the K-Means method for each frame depth map in the interval. First initialize a background image to be filled, determine the background area of the first frame texture image through the first frame binary depth image, fill the texture area into the background, and then determine the second frame texture through the second frame binary depth image For the background area of the picture, fill the texture area into the unfilled background area, and repeat this step for each next frame to gradually complete the background.

According to the membrane image obtained by the GMM process, for the final background image, fill in the GMM texture background pixel value at the position where the membrane image pixel value is 0, otherwise, fill in the FDC texture background pixel. After obtaining the final background image, use the virtual viewpoint synthesis algorithm to synthesize the background of the virtual viewpoint, and fill the hollow areas of the 100-frame virtual texture frame with the background of the corresponding area.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

While preferred embodiments of the invention have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is appreciated. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention.

Apparently, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. In this way, if the modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and equivalent technologies, the present invention also intends to include these modifications and variations.

Claims (6)

1. A hole filling method based on a depth information assisted Gaussian mixture model is characterized by comprising the following steps:

(1) acquiring image sequence data comprising a plurality of views, wherein the image sequence data of each view comprises texture frames and depth frames of a plurality of frames;

(2) determining a GMM texture background in the image sequence data by adopting a Gaussian mixture model algorithm;

(3) dividing the texture frame and the depth frame into a plurality of intervals according to time sequence, and carrying out histogram equalization processing on the depth frame in each interval;

(4) calculating a foreground depth correlation algorithm for the texture frame in each interval to correspondingly obtain an FDC interval texture background; calculating a foreground depth correlation algorithm for the depth frame after binarization in each interval to correspondingly obtain an FDC interval depth background;

(5) combining the FDC interval texture background obtained from each interval according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background;

(6) and adaptively generating a reference texture background image by adopting the GMM texture background and the FDC texture background, and further filling the cavity area of the virtual texture frame in the virtual view with the corresponding area background.

2. The method for filling up a hole based on the depth information assisted gaussian mixture model according to claim 1, wherein in the step (2), the GMM texture background in the image sequence data is determined by using a gaussian mixture model algorithm, which specifically comprises:

(21) all texture frames and depth frames in the image sequence data are used in a GMM process, the texture frames are used for constructing a Gaussian model, the depth frames are used for determining model updating parameters, and the method comprises the following steps for single pixel points corresponding to different frames:

(211) the kth Gaussian model of the texture frame of the tth frame is marked as eta_k,tModel mean is recorded as μ_k,tAnd the standard deviation is recorded as σ_k,tAnd the weight is recorded as ω_k,tK is more than or equal to 1 and less than or equal to M, and M is the total number of Gaussian models constructed on each pixel point;

(212) calculating the pixel value X of the current pixel point of the next frame t +1_tDifference | X from the mean of each Gaussian model_t-μ_k,t-1If X_t-μ_k,t-1|＜2.5σ_k,tThen, the pixel value of the current pixel point of the next frame is determinedBelonging to the Gaussian model and updating the learning rate of the model;

otherwise, if | X_t-μ_k,t-1|≥2.5σ_k,tIf so, judging that the pixel value of the current pixel point of the next frame does not belong to any one of the M Gaussian models, deleting the Gaussian model with the minimum weight-variance ratio in the M models, and constructing a new Gaussian model for replacement;

(22) all the compounds obtained in step (21) contain one omega_k,tWeight ω of other M-1 models, model 1_k,tDetermining the pixel point which is 0 as a pixel point which only contains one Gaussian model;

(23) and (3) newly building a membrane map, recording the position of only one Gaussian model by adopting a pixel value of 0, determining the position as a background position, and recording the position of a plurality of Gaussian model pixels by adopting a pixel value of 255, thereby determining the GMM texture background.

3. The method for filling holes in a gaussian mixture model based on depth information assistance according to claim 2, wherein in step (212), the learning rate of the model is updated according to the formula:

where α is the initial learning rate, ε is the empirical value in the sequence, and is the depth pixel value that can distinguish the foreground and background regions observed by the depth frame, d_t(x, y) is a pixel value at a (x, y) position on the t-th frame depth frame, α_tIs the updated learning rate.

4. The method for filling up a hole based on the depth information assisted gaussian mixture model according to claim 1, wherein in the step (4), the calculation of the foreground depth correlation algorithm is performed on the texture frame in each interval, and the FDC interval texture background is correspondingly obtained, which specifically includes:

(41) generating an initial FDC texture background by using a first frame in a manner of filling a blank image by adopting a region which is distinguished as the background by a binary depth map in the first frame, namely obtaining a background texture region in a first frame texture map corresponding to the background region of the first frame;

(42) updating an unfilled region in the FDC texture background by using a second frame texture map corresponding to a region in which the second frame in the binary depth map of the second frame becomes the background;

(43) and executing the same process as the second frame on all the texture frames in the subsequent interval so as to obtain the FDC interval texture background of the interval.

5. The method for filling up a hole based on the depth information assisted gaussian mixture model as claimed in claim 4, wherein in the step (5), the FDC interval texture background obtained from each interval is combined according to the depth information contained in the FDC interval depth background to finally obtain the FDC texture background, and the method comprises:

for a single pixel point (x, y), the texture background corresponding to the interval depth background with the minimum depth value is used as the final texture background, which is expressed as psi if there is_dk(x,y)＝min(ψ_di(x, y)), then_tf(x,y)＝ψ_tk(x, y) wherein_tf(x, y) is the final FDC texture background at pixel point (x, y), ψ_di(x, y) is the depth background of FDC interval corresponding to the ith interval of pixel point (x, y), psi_tk(x, y) is the texture background of the FDC interval corresponding to the kth interval of the pixel point (x, y); k is the resulting psi with the smallest depth value_diThe number i, i ∈ [1, K ] of (x, y)](ii) a Where t and d in the subscript are used to distinguish between texture and depth map frames.

6. The method for filling holes in a gaussian mixture model based on depth information assistance as claimed in claim 2, wherein in the step (6), the adaptive filling of all image frame numbers by using the GMM texture background and the FDC texture background comprises:

filling pixel points containing a Gaussian model with pixel points at positions corresponding to GMM texture backgrounds; otherwise, filling the pixel points at the corresponding positions of the FDC texture background.