CN101873509A - A Method for Eliminating Background and Edge Jitter of Depth Map Sequences - Google Patents

Abstract

The present invention proposes a method and device for eliminating the background and edge jitter of a depth image sequence, wherein the method includes the following steps: reading the background image pair of the source image sequence and calculating the depth image of the background image; calculating the The frame difference of the source image sequence and the background image to obtain a frame difference array; calculate to obtain the depth map of each frame of the source image sequence; judge the foreground and background of the source image depth map according to a preset threshold, and eliminate Background shaking; assigning a value to the foreground of the depth map of the source image to eliminate the shaking of the foreground contour; optimizing the final depth map of the source image through median filtering and bilateral filtering. The method for eliminating background and edge jitter of the depth image sequence proposed by the invention overcomes the deficiencies of the prior art, can effectively eliminate image jitter, eliminate interference noise, and smooth the edge.

Description

A Method for Eliminating Background and Edge Jitter of Depth Map Sequences

technical field

The invention relates to the technical field of computer vision, in particular to a method for eliminating background and edge jitter of a depth map sequence.

Background technique

Three-dimensional images and displays are an important form of expression for future information systems. On the one hand, three-dimensional images and displays have distinct layers and bright colors, and have a strong visual impact, allowing viewers to stay longer and leave a deep impression on them; In addition, the three-dimensional images and displays give people a real and lifelike character, and the characters are ready to appear, and there is an immersive feeling, which has a high artistic appreciation value. Just because three-dimensional images and videos have the characteristics that these plane images and videos do not have, they have broad application prospects in fields such as computer display, television, video, robotics, measurement, inspection, medical treatment, advertising media, and electronic games.

The reason why the human eye sees the world with a three-dimensional sense is that there is a disparity (disparity) because the left eye and the right eye see the world with slightly different perspectives. Parallax refers to the horizontal displacement of two pixels corresponding to the same world point in the left view and right view. The theorem in the field of computer vision points out that the parallax of a certain point is inversely proportional to the depth (depth, that is, the sense of distance) of its corresponding world point; Points have a disparity of 0. The depth values of all points in an image form a depth map.

The development of depth map generation technology based on binocular stereo vision has made it possible to quickly generate high-quality depth maps, but because the existing technology does not consider temporal correlation, when operating on image sequences, each depth The maps are generated independently, which leads to serious background and edge jitters in the generated depth map sequence during continuous playback, which affects the overall presentation effect.

Contents of the invention

The present invention aims to solve at least one of the above-mentioned technical problems.

For this reason, an object of the present invention is to propose a method for eliminating depth image sequence background and edge jitter, which overcomes the deficiencies of the prior art, can effectively eliminate image jitter, eliminate interference noise, and smooth edges.

One aspect of the present invention proposes a method for eliminating the background and edge jitter of a depth map sequence, comprising the following steps: reading the background image pair of the source image sequence and calculating the depth map of the background image; calculating the source image sequence and the frame difference of the background image to obtain a frame difference array; calculate to obtain the depth map of each frame of the source image sequence; judge the foreground and background of the source image depth map according to a preset threshold, and eliminate background jitter; assigning a value to the foreground of the source image depth map to eliminate foreground contour jitter; and optimizing to obtain the final source image depth map through median filtering and bilateral filtering.

In one embodiment of the present invention, the data of each pixel of the background image and the source image are converted into gray values by using a grayscale transformation formula to calculate the background image and the source image depth map.

In an embodiment of the present invention, the calculating the frame difference between the source image sequence and the background image to obtain the frame difference array further includes: reading each frame image pair of the source image sequence; Perform grayscale transformation on each frame of the image sequence; perform a difference between the grayscale-transformed reference view of the source image and the reference view of the background image to obtain the array of frame differences.

In an embodiment of the present invention, said judging the foreground and background of the depth map of the source image according to a preset threshold, and eliminating background jitter further includes: performing threshold judgment on the frame difference array by a preset threshold to distinguish The foreground and background of the depth map of the source image; using the value corresponding to the depth map of the background to assign a value to the background of the depth map of the source image.

In an embodiment of the present invention, the assigning the foreground value of the depth map of the source image to eliminate the jitter of the foreground contour further includes: counting all the pixels in the foreground range; The foreground is assigned the depth value.

In an embodiment of the present invention, said obtaining the final depth map of the source image by means of median filtering and bilateral filtering further includes: performing a step-by-step process on the depth map of the source image sequence to eliminate background jitter and foreground contour jitter. Value filtering to remove the interference noise in the background; bilateral filtering is performed on the depth map from which the interference noise in the background has been removed to smooth edges of the depth map to obtain a final optimized depth map of the source image sequence.

Another aspect of the present invention proposes a device for eliminating the background and edge jitter of a depth map sequence, including: an image acquisition module, which is used to read the background image pair of the source image sequence and calculate the depth map of the background image; The difference array calculation module is used to calculate the frame difference between the source image sequence and the background image to obtain a frame difference array; the calculation module is used to calculate the depth map of each frame of the source image sequence; foreground and background judgment A module for judging the foreground and background of the source image depth map according to a preset threshold, and eliminating background jitter; a foreground image processing module for assigning a value to the foreground of the source image depth map to eliminate foreground contour jitter; image optimization The module is used to obtain the final depth map of the source image through median filtering and bilateral filtering.

The method for eliminating the background and edge jitter of the depth map sequence according to the embodiment of the present invention overcomes the shortcomings of the prior art, and can effectively eliminate image jitter, eliminate interference noise, and smooth the edge. In addition, the present invention is simple to implement and convenient to operate.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 is an overall flowchart of a method for eliminating background and edge jitter of a depth map sequence according to an embodiment of the present invention;

Fig. 2 is the depth map before processing by the method of the embodiment of the present invention;

Fig. 3 is a depth map optimized by using the method of the embodiment of the present invention in Fig. 2;

Fig. 4 is the stereo rendering image of other images obtained by processing with the method of the embodiment of the present invention; and

FIG. 5 is a schematic diagram of an overall structure of a device for eliminating background and edge jitter of a depth map sequence according to an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

The method for eliminating the background and edge jitter of the depth map sequence in the embodiment of the present invention is described below with reference to the accompanying drawings. The method overcomes the shortcomings of the prior art and can effectively eliminate image jitter, eliminate interference noise, and smooth edges. In addition, the present invention is simple to implement and convenient to operate.

As shown in FIG. 1 , it is an overall flowchart of a method for eliminating background and edge jitter of a depth map sequence according to an embodiment of the present invention. Include the following steps:

Step S101, read the background image pair of the source image sequence and calculate the depth map of the background image.

1.1, define the image interface, including: image data pointer, image width, height, bit depth, image format.

1.2. Define an image reading function according to the image interface, use the image reading function to open the background image pair file of the source image sequence from the hard disk, so as to obtain the background image data, and save the background image data in the system memory.

1.3. Convert the background image data into grayscale image data of the corresponding background image by using a grayscale transformation formula.

Specifically, in one embodiment of the present invention, the grayscale transformation is performed on the source image pair by using the RGB (Red Green Blue, red, green, blue) image transformation formula to obtain the corresponding grayscale image pair. Wherein, the RGB image transformation formula in the embodiment of the present invention is as follows:

Y＝0.212671×R+0.715160×G+0.072169×B (1)

Among them, Y is the gray value of each pixel in the gray image, and R, G, and B are the R, G, and B components of each pixel in the planar video frame image, respectively.

1.4, use the stereo matching algorithm of belief propagation to calculate the depth map of the background image of the source image sequence. In one embodiment of the present invention, the depth map of the background image is marked as: reDepth.

Step S102, calculating the frame difference between the source image sequence and the background image to obtain a frame difference array.

2.1. Define the image reading function according to the image interface, use the image reading function to open the image pair file of each frame of the source image sequence from the hard disk, so as to obtain the image data of the source image, and save the image data of the source image in the memory.

2.2. Use the grayscale transformation formula to convert the image data of each frame of the source image sequence into grayscale image data.

2.3. Make the difference between the source image reference view after the grayscale transformation in 2.2 and the background image reference view after the grayscale transformation in 1.4, and take the absolute value to obtain the frame difference array between the source image reference view and the background image reference view .

Specifically, in an embodiment of the present invention, the source image after the gray scale transformation is marked as: depth1, the background image reference view is marked as: depth2, and the frame difference array is marked as: cha. Use the following formula:

cha＝|depth1-depth2| (2)

Get the corresponding frame difference array cha.

Step S103, calculate to obtain the depth map of each frame of the source image sequence.

3.1, get the source image sequence data after the grayscale transformation in 2.2.

3.2, use the stereo matching algorithm of belief propagation proposed in 1.4 to calculate the data in 3.1 to obtain the depth map of each frame of the source image sequence. In one embodiment of the present invention, the depth map of each frame of the source image sequence is recorded as: depth. As shown in FIG. 2 , it is a depth map of each frame image of the source image sequence of the embodiment of the present invention.

Step S104, judging the foreground and background of the depth map of the source image according to a preset threshold, and eliminating background shake.

4.1, define the threshold, and perform threshold judgment on the frame difference array obtained in 2.3. Because the camera does not move, the background is considered to be motionless, so the points smaller than the threshold can be used as the background, and other points as the foreground. In one embodiment of the present invention, the threshold is defined as 20.

4.2. For the area divided into the background, use the value corresponding to the background depth map to assign a value to the source image depth map, using the following formula:

$\left\{\begin{array}{cc}\begin{array}{cc}\mathrm{<span\; class="notranslate">\; if</span>}& \mathrm{<span\; class="notranslate">\; cha</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 20</span>}\end{array}& \mathrm{<span\; class="notranslate">\; depth</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; reDepth</span>}\\ \mathrm{<span\; class="notranslate">\; else</span>}& \mathrm{<span\; class="notranslate">\; depth</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; depth</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

In one embodiment of the present invention, it can be seen from formula (3) that if the frame difference array cha is less than the threshold value 20, the color of the depth map depth of each frame of the source image sequence is assigned as the depth map of the background image The color of reDepth. Otherwise it is foreground, keep the original value.

Step S105, assign a value to the foreground of the depth map of the source image to eliminate the jitter of the foreground contour.

5.1, make statistics on all the pixels in the foreground range.

5.3. The uniform value obtained through statistical calculation is used as the depth value of the foreground, and the uniform value is assigned to the depth map of the foreground to eliminate the jitter of the foreground outline. In one embodiment of the present invention, a unified value is calculated by using a method of summing and averaging all points.

Step S106, obtain the final depth map of the source image through median filtering and bilateral filtering to optimize.

6.1. Median filtering is performed on the depth map of the source image sequence processed in 5.3, and the interference noise in the background can be removed through median filtering.

6.2. Perform bilateral filtering on the depth map of the source image sequence processed in 6.1, and the edges can be smoothed through bilateral filtering.

6.3, Obtain the depth map of each frame of the source image. In an embodiment of the present invention, as shown in FIG. 3 , it is the final optimized depth map shown in FIG. 2 .

In another embodiment of the present invention. As shown in Figure 4, in order to use other image sequences to test the effect of the method for eliminating the background and edge jitter of the depth image sequence proposed by the present invention, it can be seen from Figure 4 that the stereoscopic rendering image still has no obvious background and edge jitter.

Another aspect of the present invention proposes a device for eliminating background and edge jitter of a depth map sequence. The device 100 for eliminating background and edge jitter of a depth map sequence includes an image acquisition module 110, a frame difference array calculation module 120, and a foreground and background judgment module. 130 , a foreground image processing module 140 , and an image optimization module 150 . Wherein, the image acquisition module 110 is used to read the background image pair of the source image sequence and calculate the depth map of the background image, and the frame difference array calculation module 120 is used to calculate the frame difference between the source image sequence and the background image To obtain the frame difference array, the foreground and background judging module 130 is used to judge the foreground and background of the depth map of the source image according to a preset threshold, and eliminate background shaking, and the foreground image processing module 140 is used to process the foreground of the depth map of the source image The value is assigned to eliminate the jitter of the foreground contour, and the image optimization module 150 is used to obtain the final depth map of the source image through median filtering and bilateral filtering.

Wherein, the image acquisition module 110 includes a background image acquisition module 1101 , a gray scale transformation module 1102 , and a calculation module 1103 . The background image acquisition module 1101 is used to obtain the background image, the grayscale transformation module 1102 uses the grayscale transformation formula to change the data of each pixel of the background image and the source image into a grayscale value, and the calculation module 1103 is used to convert The background image and the source image after the gray scale transformation are calculated to obtain the depth map of the background image and the source image.

However, those of ordinary skill in the art know that the image optimization module 150 of the device 100 for eliminating the background and edge jitter of the depth map sequence proposed by the present invention can also be subdivided into a median filter module and a bilateral filter module, and of course there can be other forms Combinations, these should also be considered within the protection scope of the present invention.

The method for eliminating background and edge jitter of the depth image sequence proposed by the invention overcomes the deficiencies of the prior art, can effectively eliminate image jitter, eliminate interference noise, and smooth the edge. In addition, the present invention is simple to implement and convenient to operate.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (8)

1. A method for eliminating depth map sequence background and edge jitter, characterized in that, comprising the following steps: Reading the background image pair of the source image sequence and calculating the depth map of the background image; calculating the frame difference between the source image sequence and the background image to obtain a frame difference array; Calculate to obtain the depth map of each frame of the source image sequence; judging the foreground and background of the depth map of the source image according to a preset threshold, and eliminating background jitter; Assigning a value to the foreground of the source image depth map to eliminate the jitter of the foreground contour; The final depth map of the source image is obtained through optimization by median filtering and bilateral filtering. 2. The method for eliminating the background and edge jitter of the depth map sequence as claimed in claim 1, wherein the data of each pixel of the background image and the source image are changed into grayscale by using a grayscale transformation formula value, to calculate the depth map of the background image and the source image. 3. The method for eliminating depth map sequence background and edge jitter as claimed in claim 2, wherein said calculating the frame difference of said source image sequence and said background image to obtain a frame difference array further comprises: Read each frame image pair of the source image sequence; Perform grayscale transformation on each frame image of the source image sequence; The difference between the source image reference view and the background image reference view after the gray scale transformation is performed to obtain the frame difference array. 4. The method for eliminating depth map sequence background and edge jitter as claimed in claim 3, wherein said judging the foreground and background of said source image depth map according to a preset threshold, and eliminating background jitter further comprises: performing threshold judgment on the frame difference array by using a preset threshold to distinguish the foreground and background of the depth map of the source image; Use the value corresponding to the background depth map to assign a value to the background of the source image depth map. 5. the method for eliminating depth map sequence background and edge jitter as claimed in claim 4, is characterized in that, described foreground assignment to described source image depth map further comprises: Count all pixels within the foreground range; A unified value is obtained through calculation as a depth value to assign the depth value to the foreground. 6. The method for eliminating depth map sequence background and edge jitter as claimed in claim 5, wherein said obtaining the final source image depth map through median filtering and bilateral filtering further comprises: performing median filtering on the depth map of the source image sequence from which background jitter and foreground contour jitter has been eliminated, to remove disturbing noise in the background; Bilateral filtering is performed on the depth map from which interference noise in the background has been removed to smooth edges of the depth map to obtain a final optimized depth map of the source image sequence. 7. A device for eliminating background and edge jitter of a depth map sequence, characterized in that it comprises: An image acquisition module, configured to read the background image pair of the source image sequence and calculate the depth map of the background image; A frame difference array calculation module, configured to calculate the frame difference between the source image sequence and the background image to obtain a frame difference array; A calculation module, used for calculation to obtain the depth map of each frame of the source image sequence; A foreground and background judging module, configured to judge the foreground and background of the source image depth map according to a preset threshold, and eliminate background jitter; The foreground image processing module is used to assign values to the foreground of the source image depth map to eliminate the jitter of the foreground contour; The image optimization module is used to obtain the final depth map of the source image through median filtering and bilateral filtering. 8. The device for eliminating the background and edge jitter of the depth map sequence as claimed in claim 7, wherein the data of each pixel of the background image and the source image are changed into grayscale by using a grayscale transformation formula value, to calculate the depth map of the background image and the source image.

Images