CN111405264B - A 3D Video Comfort Improvement Method Based on Depth Adjustment - Google Patents

Abstract

The invention discloses a 3D video comfort level improving method based on depth adjustment. The problems that the general 3D video is uncomfortable to watch and the visual experience is not good are solved. The invention comprises the following steps: s1: preprocessing the depth maps of the left view map and the right view map to obtain preprocessed maps; s2: carrying out depth filtering on the preprocessed image to obtain a filtering image; s3: carrying out depth de-texturing on the filtering image to obtain a de-textured image; s4: drawing a virtual viewpoint according to the texture-removed picture to obtain a virtual right viewpoint color picture; s5: and replacing the color image of the virtual right viewpoint and the original viewpoint image to obtain the improved 3D video. The method has the advantages of reducing the influence of parallax, gradient change and texture on the viewing experience and improving the overall comfort.

Description

A 3D Video Comfort Improvement Method Based on Depth Adjustment

technical field

The invention relates to the fields of stereoscopic video comfort enhancement and 3D image processing, in particular to a 3D video comfort improvement method based on depth adjustment.

Background technique

With the application and development of 3D technologies such as virtual reality, augmented reality and holographic projection in human life, human beings have an increasing demand for viewing 3D images. At present, the generation and display technology of 3D video is not perfect, and its quality varies. When people watch 3D images and videos of poor quality for a long time, some uncomfortable physiological phenomena will occur. For example, the application number is CN201510275341.4, and the name is a stereoscopic video comfort enhancement method based on the continuous adjustment of parallax changes. It extracts the parallax and motion vector information of each frame of investigation period in the stereoscopic video decoding process, and evaluates the model according to the comfort level. , calculate the average value of stereoscopic video comfort for each inspection period and determine whether the frame in the inspection period is an uncomfortable frame, and the method of adjusting the parallax for the uncomfortable frame is a complicated process, and is generally calculated by subsequent calculation. Parallax information is inaccurate.

SUMMARY OF THE INVENTION

The invention solves the problems of uncomfortable viewing and poor visual experience of general 3D videos, and provides a method for improving the comfort of 3D videos based on depth adjustment.

In order to solve the above-mentioned existing technical problems, the technical solution of the present invention is: a method for improving the comfort of 3D video based on depth adjustment. The 3D video includes a left view point image and a right view point image, and both the left view point image and the right view point image include color images. and depth map, the improvement method includes the following steps:

S1: Preprocess the depth maps of the left view point image and the right view point image to obtain a preprocessed image;

S2: Perform depth filtering on the preprocessing image to obtain a filtered image;

S3: Depth detexture is performed on the filter image to obtain a detextured image;

S4: Perform virtual viewpoint drawing according to the de-textured map to obtain a virtual right viewpoint color map;

S5: Improved 3D video by replacing the virtual right view color map and the original view map.

The parallax of the original depth map may be too large. The preprocessing process is to appropriately adjust the depth value of the depth map to reduce the impact of excessive parallax on the comfort of 3D video. The necessity of depth filtering is that it can solve the discomfort caused by gradient changes in the depth map. When there is too much texture information, it will cause obvious fusion difficulties for the human eye when fusing 3D images. By de-texturing the images, the comfort can be improved. The rendering of virtual viewpoints and the related replacement of the image can improve the overall look and feel experience.

As a preferred solution of the above solution, the preprocessing is to substitute the depth value of each pixel of the depth map into the preprocessed objective function for operation, and the image obtained after the operation is a preprocessed map, and the preprocessed target The function is:

Where Z _pro represents the depth value after preprocessing, Z represents the original depth value, and Zm represents the minimum depth value in the depth image. The depth pixel value shown in the existing depth map is inversely proportional to the depth. The smaller the depth, the larger the depth pixel value, and the closer it is to white. The "0.8" ratio in the formula means that the depth of the part greater than half of the maximum value is moved 80% farther, that is, when the parallax is too large, it is adjusted to a certain extent.

As a preferred solution of the above scheme, the depth filtering is to substitute each pixel of the preprocessing map into the objective function of the depth filtering for operation, and the image obtained after the operation is a filtering map, and the objective function of the depth filtering is:

O(x,y)=∑ _m,n Z(x+m,y+n)*K(m,n)

where O(x, y) represents the pixel output value at the (x, y) position after filtering, (x, y) represents the position of the depth map pixel, Z(x+m, y+n) represents the preprocessing image ( The depth value at the position x+m, y+n). K(m, n) represents the filter kernel, (m, n) represents the coordinates in the schematic diagram of the position coordinates of the filter kernel, and the filter kernel adopts Gaussian filtering. Gaussian filtering is a linear filter, which is a process of weighted averaging of the entire image. Unlike the case where all template coefficients in the mean filter are 1, the template coefficient of the Gaussian filter is changed, and the coefficient is the same as the pixel and the template center. distance is inversely proportional. Gaussian filtering is used because it can smooth the gradient changes and improve discomfort.

其中(x，y)表示3×3高斯核位置坐标图中的坐标，σ表示标准差，采用σ＝0.8进行坐标修正，σ＝0.8是根据有效次试验获取的最佳数值。深度滤波目标函数中的K(m，n)的表达式采用二维高斯函数的表达式，即 K(m，n)＝h(m，n)。As a preferred solution of the above solution, the Gaussian filtering adopts a Gaussian kernel of 3×3 for blurring, and the expression of the two-dimensional Gaussian function corresponding to the Gaussian kernel of 3×3 is:

Where (x, y) represents the coordinates in the 3×3 Gaussian kernel position coordinate graph, σ represents the standard deviation, and σ=0.8 is used for coordinate correction, and σ=0.8 is the best value obtained from valid trials. The expression of K(m, n) in the depth filtering objective function adopts the expression of a two-dimensional Gaussian function, that is, K(m, n)=h(m, n).

As a preferred solution of the above scheme, the coordinate correction includes the following steps:

S51: Substitute 9 coordinates in the 3×3 Gaussian kernel position coordinate map into the two-dimensional Gaussian function, and obtain the corresponding 9 values;

S52: sum 9 values, and the reciprocal of the sum is α;

S53: Multiply all 9 coordinates in the 3×3 Gaussian kernel position coordinate map by α to obtain a new 3×3 Gaussian kernel position coordinate map;

S54 : Substitute the 9 coordinate values in the new 3×3 Gaussian kernel position coordinate map into the objective function of depth filtering to perform corresponding calculation.

The weight calculation is essentially to obtain 9 coefficients, which are substituted into the objective function of depth filtering, which is equivalent to taking each pixel as the center, and multiplying the pixel and the surrounding 8 pixels offset by each coefficient respectively. , and finally sum up the 9 products to obtain the pixel output value at the position of the central pixel point.

As a preferred solution of the above solution, the depth detexture includes the following steps:

S61: use Z=[Z(i, j)] _W×H to represent a given filter map with a resolution of W×H and perform DCT transformation;

The objective function of the DCT transform is:

in:

W和H表示分辨率的两个值，(i，j)表示滤波图中像素点的坐标，(μ，v)表示经过DCT变换后的滤波图中像素点的坐标，u表示二维波的水平方向频率， v为二维波的垂直方向频率。

W and H represent two values of resolution, (i, j) represent the coordinates of the pixel points in the filter image, (μ, v) represent the coordinates of the pixel points in the filter image after DCT transformation, and u represents the two-dimensional wave frequency in the horizontal direction, v is the frequency in the vertical direction of the two-dimensional wave.

S62: Perform inverse DCT transformation on the result of DCT transformation after thresholding to obtain a de-textured map. The objective function defined by the threshold is:

where DM represents the de-textured map, and T is the threshold selected for the experiment.

When there is too much texture information, it will cause obvious fusion difficulties for the human eye when fusing 3D images. Therefore, by de-texturing the image, it is easier for the human eye to fuse the 3D effects, thereby improving comfort.

As a preferred solution of the above scheme, the virtual viewpoint is drawn using DIBR, including the following steps: S71: Correcting LeftNearestDepthValue, LeftFarthestDepthValue, RightNearestDepthValue and RightFarthestDepthValue, the corrected objective function is:

DV _ad =DV _or -DV _or ×0.1

Where DV _ad represents the adjusted maximum depth value and minimum depth value, DV _or represents the original maximum and minimum depth values, LeftNearestDepthValue represents the minimum depth of the left view to the texture map, LeftFarthestDepthValue represents the left view to the maximum depth of the texture map, RightNearestDepthValue represents the minimum depth of the texture map from the right view, and RightFarthestDepthValue represents the maximum depth of the texture map from the right view;

S72: Perform DIBR drawing according to LeftNearestDepthValue, LeftFarthestDepthValue, RightNearestDepthValue and RightFarthestDepthValue to obtain a virtual right-view color map. The parallax is also reduced to a certain extent by rendering the virtual right-view color map 1 through DIBR.

As a preferred solution of the above solution, the replacement with the virtual right-view color map and the original view is to replace the virtual right-view color map, the original left-view color map, the left-view de-textured map and the right-view de-textured map with the original ones. Right-view color map, left-view color map, left-view depth map, and right-view depth map in 3D video. The virtual right-view color map is combined with the original left-view map and the de-textured map of the left- and right-view points and replaced with the original image to achieve overall 3D video comfort improvement.

Compared with the prior art, the beneficial effects of the present invention are:

1. The preprocessing method can reduce the influence of excessive parallax on the comfort of 3D video, and adjust it to a comfortable level;

2. The use of Gaussian filtering can reduce the discomfort caused by gradient changes in the depth map;

3. De-texture processing can improve the influence of too many texture features in the image on the look and feel;

4. The rendering of virtual viewpoints and related replacements achieve an overall improvement in comfort.

Description of drawings

Fig. 1 is the flow chart of the present invention;

FIG. 2 is a schematic diagram of the Gaussian kernel position coordinates of the present invention.

Detailed ways

The technical solutions of the present invention will be further described below through examples and in conjunction with the accompanying drawings.

Embodiment: This embodiment is a method for improving the comfort of 3D video based on depth adjustment. The 3D video includes a left view point image and a right view point image, and both the left view point image and the right view point image include a color image and a depth image. The improvement method includes: The following steps: Step 1: Preprocess the depth maps of the left view point map and the right view point map to obtain a preprocessed map. Preprocessing is to substitute the depth value of each pixel of the depth map into the preprocessed objective function for operation. The image obtained after the operation is the preprocessed map, and the preprocessed objective function is:

Where Z _pro represents the depth value after preprocessing, Z represents the original depth value, and Zm represents the minimum depth value in the depth image. The depth pixel value shown in the existing depth map is inversely proportional to the depth. The smaller the depth, the larger the depth pixel value, and the closer it is to white. The "0.8" ratio in the formula means that the depth of the part greater than half of the maximum value is moved 80% farther, that is, when the parallax is too large, it is adjusted to a certain extent.

Step 2: Perform deep filtering on the preprocessed image to obtain a filtered image. Depth filtering is to substitute each pixel of the preprocessing image into the objective function of depth filtering for operation. The image obtained after the operation is the filtering image, and the objective function of depth filtering is:

O(x,y)=∑ _m,n Z(x+m,y+n)*K(m,n)

where O(x, y) represents the pixel output value at the (x, y) position after filtering, (x, y) represents the position of the depth map pixel, Z(x+m, y+n) represents the preprocessing image ( The depth value at the position x+m, y+n). K(m, n) represents the filter kernel, (m, n) represents the coordinates in the schematic diagram of the position coordinates of the filter kernel, and the filter kernel adopts Gaussian filtering. Gaussian filtering is a linear filter, which is a process of weighted averaging of the entire image. Unlike the case where all template coefficients in the mean filter are 1, the template coefficients of the Gaussian filter change, and the coefficients are the same as the pixel and template center. distance is inversely proportional. Gaussian filtering is used because it can smooth the gradient changes and improve discomfort.

其中(x，y)表示3×3高斯核位置坐标图中的坐标，高斯核位置坐标图如图2所示，σ表示标准差，采用σ＝0.8进行坐标修正，σ＝0.8是根据有效次试验获取的最佳数值。深度滤波目标函数中的K(m，n)的表达式采用二维高斯函数的表达式，即 K(m，n)＝h(m，n)。Among them, Gaussian filtering adopts a Gaussian kernel of 3×3 for blurring, and the expression of the two-dimensional Gaussian function corresponding to the Gaussian kernel of 3×3 is:

Where (x, y) represents the coordinates in the 3×3 Gaussian kernel position coordinate diagram, and the Gaussian kernel position coordinate diagram is shown in Figure 2, σ represents the standard deviation, and σ=0.8 is used for coordinate correction, and σ=0.8 is based on the effective times The best value obtained by experiment. The expression of K(m, n) in the depth filtering objective function adopts the expression of a two-dimensional Gaussian function, that is, K(m, n)=h(m, n).

Among them, the coordinate correction is to substitute the 9 coordinates in the 3×3 Gaussian kernel position coordinate map into the two-dimensional Gaussian function, obtain the corresponding 9 values, and then sum the 9 values, and the reciprocal of the sum is α, Then multiply the 9 coordinates in the 3×3 Gaussian kernel position coordinate graph by α to obtain a new 3×3 Gaussian kernel position coordinate graph, and finally substitute the 9 coordinate values in the new 3×3 Gaussian kernel position coordinate graph into the The objective function of the depth filter to calculate accordingly. The weight calculation is essentially to obtain 9 coefficients, which are substituted into the objective function of depth filtering, which is equivalent to taking each pixel as the center, and multiplying the pixel and the surrounding 8 pixels offset by each coefficient respectively. , and finally sum up the 9 products to obtain the pixel output value at the position of the central pixel point.

Step 3: Deeply detexture the filter image to obtain a detextured image. Depth detexturing consists of the following steps:

1. Use Z=[Z(i,j)] _W×H to represent a given filter map with a resolution of W×H and perform DCT transform;

The objective function of the DCT transform is:

in:

W和H表示分辨率的两个值，(i，j)表示滤波图中像素点的坐标，(μ，v)表示经过DCT变换后的滤波图中像素点的坐标，u表示二维波的水平方向频率， v为二维波的垂直方向频率。

W and H represent two values of resolution, (i, j) represent the coordinates of the pixel points in the filter image, (μ, v) represent the coordinates of the pixel points in the filter image after DCT transformation, and u represents the two-dimensional wave frequency in the horizontal direction, v is the frequency in the vertical direction of the two-dimensional wave.

2. Perform inverse DCT transformation on the result of DCT transformation after thresholding to obtain a de-textured image. The threshold-limited objective function is:

where DM represents the de-textured map, and T is the threshold selected for the experiment.

When there is too much texture information, it will cause obvious fusion difficulties for the human eye when fusing 3D images. Therefore, by de-texturing the image, it is easier for the human eye to fuse the 3D effects, thereby improving comfort.

Step 4: Draw a virtual viewpoint according to the de-textured map to obtain a virtual right viewpoint color map. The virtual viewpoint drawing adopts DIBR drawing, including the following steps:

1. Revise LeftNearestDepthValue, LeftFarthestDepthValue, RightNearestDepthValue and RightFarthestDepthValue. The revised objective function is:

DV _ad =DV _or -DV _or ×0.1

Where DV _ad represents the adjusted maximum depth value and minimum depth value, DV _or represents the original maximum and minimum depth values, LeftNearestDepthValue represents the minimum depth of the left view to the texture map, LeftFarthestDepthValue represents the left view to the maximum depth of the texture map, RightNearestDepthValue represents the minimum depth of the texture map from the right view, and RightFarthestDepthValue represents the maximum depth of the texture map from the right view;

2. Perform DIBR drawing according to LeftNearestDepthValue, LeftFarthestDepthValue, RightNearestDepthValue and RightFarthestDepthValue to obtain a virtual right view color map. The parallax is also reduced to a certain extent by rendering the virtual right-view color map 1 through DIBR.

The last step: replacing the virtual right view color map with the original view image is to replace the right view color map in the original 3D video with the virtual right view color map, the original left view color map, the left view de-texture map and the right view de-texture map , left-view color map, left-view depth map, and right-view depth map to get an improved 3D video. Combining the virtual right-view color map with the original left-view map and the left- and right-view de-textured maps and replacing the original image finally improves the overall comfort of 3D video.

Claims (8)

1. a 3D video comfort improvement method based on depth adjustment, the 3D video includes a left view point image and a right view point image, and the left view point image and the right view point image both include a color map and a depth map, and it is characterized in that, the improvement method includes The following steps: S1: Preprocess the depth maps of the left view point image and the right view point image to obtain a preprocessed image; S2: Perform depth filtering on the preprocessing image to obtain a filtered image; S3: Depth detexture is performed on the filter image to obtain a detextured image; S4: Perform virtual viewpoint drawing according to the de-textured map to obtain a virtual right viewpoint color map; S5: Improved 3D video by replacing the virtual right view color map and the original view map. 2. a kind of 3D video comfort improvement method based on depth adjustment according to claim 1, is characterized in that, described preprocessing is to substitute the depth value of each pixel of the depth map into the objective function of preprocessing and carry out calculation , the graph obtained after the operation is the preprocessing graph, and the objective function of the preprocessing is:

Where Z _pro represents the depth value after preprocessing, Z represents the original depth value, and Zm represents the minimum depth value in the depth image. 3. a kind of 3D video comfort improvement method based on depth adjustment according to claim 1, is characterized in that, described depth filter is to substitute each pixel of preprocessing map into the objective function of depth filter to carry out calculation, calculation The obtained image is the filter image, and the objective function of the depth filter is: O(x,y)=∑ _m,n Z(x+m,y+n)*K(m,n) where O(x, y) represents the pixel output value at the (x, y) position after filtering, (x, y) represents the position of the depth map pixel, Z(x+m, y+n) represents the depth map (x +m, y+n) the depth value at the position; K(m, n) represents the filter kernel, (m, n) represents the coordinates in the schematic diagram of the position coordinates of the filter kernel, and the filter kernel adopts Gaussian filtering. 4 . The method for improving the comfort of 3D video based on depth adjustment according to claim 3 , wherein the Gaussian filtering adopts a Gaussian kernel of 3×3 for blurring, and the Gaussian kernel of 3×3 is used for blurring. 5 . The corresponding two-dimensional Gaussian function is expressed as

Among them, (m, n) represents the coordinates in the schematic diagram of the position coordinates of the 3×3 Gaussian kernel, and σ=0.8 is used for coordinate correction, and σ represents the standard deviation. 5. a kind of 3D video comfort improvement method based on depth adjustment according to claim 4 is characterized in that, described coordinate correction comprises the following steps: S51: Substitute the 9 coordinates in the schematic diagram of the position coordinates of the 3×3 Gaussian kernel into the two-dimensional Gaussian function

Find the corresponding 9 values; S52: sum 9 values, and the reciprocal of the sum is α; S53: Multiply all 9 coordinates in the 3×3 Gaussian kernel position coordinate diagram by α to obtain a new 3×3 Gaussian kernel position coordinate diagram; S54: Substitute the 9 coordinate values in the new 3×3 Gaussian kernel position coordinate diagram into the objective function of depth filtering to perform corresponding calculation. 6. a kind of 3D video comfort improvement method based on depth adjustment according to claim 1, is characterized in that, described depth detexture comprises the following steps: S61: use Z=[Z(i, j)] _W×H to represent a given filter map with a resolution of W×H and perform DCT transformation; The objective function of the DCT transform is:

in:

W and H represent two values of resolution, (i, j) represent the coordinates of the pixel in the filter map, (u, v) represent the coordinates of the pixel in the filter map after DCT transformation, and u represents the two-dimensional wave in the The coordinate value in the horizontal direction in the frequency domain, v is the coordinate value in the vertical direction of the two-dimensional wave in the frequency domain; S62: Perform inverse DCT transformation on the result of DCT transformation after thresholding to obtain a de-textured map. The objective function defined by the threshold is:

where DM represents the de-textured map, and T is the threshold selected for the experiment. 7. a kind of 3D video comfort improvement method based on depth adjustment according to claim 1, is characterized in that, described virtual viewpoint drawing adopts DIBR to draw, comprises the following steps: S71: Corrected LeftNearestDepthValue, LeftFarthestDepthValue, RightNearestDepthValue and RightFarthestDepthValue, the corrected objective function is: DV _ad =DV _or -DV _or ×0.1 Among them, DV _ad represents the maximum depth value and minimum depth value after adjustment, DV _or represents the original maximum and minimum depth values, LeftNearestDepthValue represents the minimum depth of the left view to the texture map, LeftFarthestDepthValue represents the left view to the maximum depth of the texture map, RightNearestDepthValue represents the minimum depth of the texture map from the right view, and RightFarthestDepthValue represents the maximum depth of the texture map from the right view; S72: Perform DIBR drawing according to LeftNearestDepthValue, LeftFarthestDepthValue, RightNearestDepthValue and RightFarthestDepthValue to obtain a virtual right-view color map. 8. a kind of 3D video comfort improvement method based on depth adjustment according to claim 1, is characterized in that, described replacing with virtual right viewpoint color map and original viewpoint map is to replace virtual right viewpoint color map, original left viewpoint color map and original left viewpoint color map. The view color map, left view detexture map and right view detexture map replace the right view color map, left view color map, left view depth map and right view depth map in the original 3D video.

Images