CN106803952B - In conjunction with the cross validation depth map quality evaluating method of JND model - Google Patents

Abstract

The invention discloses a cross-validation depth map quality evaluation method combined with a JND model, which uses the color map corresponding to the depth map and the color map on the auxiliary viewpoint to obtain a difference map; uses the depth map and the color map corresponding to it through 3D ‑Warping is mapped to the number of pixels at each coordinate in the color map on the auxiliary viewpoint to obtain the occlusion mask; then use the occlusion mask to remove the occluded pixels in the difference map to obtain the de-occluded difference map; then Divide the color image on the auxiliary viewpoint into three regions of flat, edge and texture to obtain the region marker map; then introduce the JND model, combined with the region marker map, to obtain the error visual threshold of each pixel in the color image on the auxiliary viewpoint ;Finally, according to the de-occluded difference map and the error visual threshold, the depth error map is obtained, and then the ratio of the error pixels in the depth map is obtained as the quality evaluation value; the advantage is that it can effectively improve the evaluation result and the drawn result. Consistency between the quality of virtual viewpoints.

Description

Cross Validated Depth Map Quality Evaluation Method Combined with JND Model

technical field

The invention relates to an image quality evaluation method, in particular to a cross-validation depth map quality evaluation method combined with a JND (Just-noticeable-distortion, just noticeable distortion) model.

Background technique

In recent years, video technology has developed rapidly, and many new applications have emerged, such as 3D video and Free Viewpoint Video (FVV, Free Viewpoint Video). Compared with traditional two-dimensional video, 3D video provides depth information, resulting in a more realistic visual experience. Depth maps play a fundamental role in many 3D video applications, for example, depth maps can be used to generate arbitrary new viewpoint images by interpolating or extrapolating images at available viewpoints; moreover, a high-quality depth map is an important tool for solving problems in computer vision. provided assistance with challenging problems. The performance of many 3D video applications benefits from the estimation or acquisition of accurate and high-quality depth maps, which can be obtained by matching rectified color images or using depth cameras. In stereo matching technology, due to the influence of occlusion and large uniform area, it often produces inaccurate depth maps. Although the inherent difficulties of stereo matching algorithms can be solved by using depth cameras, the inevitable sensor noise problem still exists. , affecting the accuracy of the depth and the shape of the object.

One of the main development directions of 3D video technology is the free-viewpoint video system based on color plus depth. The free-viewpoint video system based on color plus depth allows users to freely choose any viewpoint for viewing, which enhances human-computer interaction. A key technology to realize the free-viewpoint video system is the virtual viewpoint generation technology. Its main purpose is to overcome the limitation of the camera's ability to obtain the real viewpoint and generate a virtual viewpoint at any position. There are two main factors affecting the quality of the virtual viewpoint: one is the quality of the depth map and the corresponding color image; the other is the virtual viewpoint rendering algorithm. Currently, a depth image based rendering (DIBR, Depth Image Based Rendering) technology is the most widely used virtual viewpoint generation technology in the industry. In the depth map-based rendering technology, depth information is the key to generate high-quality virtual viewpoints. Errors in depth information will lead to parallax errors, resulting in pixel position offsets and object distortions in virtual viewpoints, affecting user perception. Depth information represents the distance information from the corresponding scene to the camera imaging plane, and it quantifies the actual distance value to [0, 255]. Because the depth camera is expensive, most of the depth maps currently used for testing are obtained through depth estimation software. In order to popularize applications and reduce costs, the depth information used for virtual viewpoint rendering is not suitable for depth estimation at the receiving end. It needs to be collected or estimated at the sending end, and then encoded and transmitted to the receiving end. Therefore, the limitation of depth map acquisition algorithm and depth map encoding will lead to inaccurate depth estimation and depth compression distortion.

The core idea of the depth map-based rendering technology is to use depth information and camera parameters to project the pixels in the reference image to the target virtual viewpoint. Generally, it can be divided into two steps. First, reproject the pixels in the original reference viewpoint to the target virtual viewpoint using its depth information Their corresponding three-dimensional space positions; then according to the position of the virtual viewpoint (such as camera translation, rotation parameters, etc.), these three-dimensional space points are re-projected to the virtual camera plane for imaging to obtain the pixels in the virtual viewpoint. When drawing a virtual viewpoint, it is necessary to convert the depth into a parallax. The position of the reference pixel in the virtual viewpoint can be obtained through the parallax. The depth value determines the offset distance of the pixel in the reference viewpoint. If the depth values of adjacent pixels change sharply, a hole will be generated between the two pixels, and the sharper the depth value change, the larger the hole will be. Because the depth value changes greatly at the junction of the foreground and the background, the generation of the hole is generally located at the junction of the foreground and the background. Holes appear in the virtual image when background regions in the reference image that are occluded by foreground objects are visible in the virtual image, while occlusion occurs when background regions that are not occluded by foreground objects in the reference image are not visible in the virtual image.

Most of the virtual viewpoint distortions are pixel position offset and object distortion in the virtual viewpoint, and not all detected distorted areas can be well perceived by human eyes. The image is composed of three parts: edge, texture, and flat area. Different areas of distortion have different effects on human visual effects. Areas with higher texture complexity or similar texture features can often tolerate more distortion, while edges The changes in the vicinity can most cause the visual perception of the human eye. Studies in visual physiology and psychology have found that the characteristics of the human visual system and the masking effect play a very important role in image processing. When the image distortion is less than a certain range, the human eye cannot feel this effect. Based on this, people put forward Just-noticeable-distortion (JND, Just-noticeable-distortion) model. Common masking effects include: 1) Brightness masking characteristics, the human eye has poor judgment on the absolute brightness of the observed object, but has a strong judgment on the relative difference in brightness, and is more sensitive to the noise added to the highlight area; 2) Texture masking characteristics, the human visual system is much more sensitive to image smooth areas than texture areas, and areas with higher texture complexity can often tolerate more distortion.

Due to the widespread use of depth maps, the quality assessment of depth maps becomes crucial and can facilitate many practical applications. For example, in a free-viewpoint video system, detecting depth distortion can help to perform depth enhancement. Through depth enhancement, the quality of virtual viewpoint can be further improved, so that viewers can enjoy a better viewing experience. A simple method for quality assessment of depth maps is to compare the depth map under test with an undistorted reference depth map. This method corresponds to a full-reference depth quality metric, which can accurately measure the accuracy of depth maps. However, in most In practical applications, due to the unavoidable errors of the depth map, the reference depth map without distortion is usually not available, so it is more reasonable to evaluate the depth map with the no-reference evaluation method. The no-reference depth map quality assessment scheme proposed by Xiang et al. detects errors by matching the edges of the color image and the depth map, and calculates the dead pixel rate to evaluate the quality of the depth map, which has a good consistency with the quality of the drawn virtual image. , but this scheme only considers the errors near the edge, ignoring other smooth areas, and only some error pixels are detected, and the different attributes and error distribution of the scene have a great impact on the performance of the scheme. The depth map is not directly used for viewing, but is used as auxiliary information to draw a virtual viewpoint. Therefore, it is necessary to evaluate the quality of the depth map from an application point of view.

Contents of the invention

The technical problem to be solved by the present invention is to provide a cross-validation depth map quality evaluation method combined with the JND model, which does not require a distortion-free reference depth map, and can effectively improve the relationship between the evaluation result and the quality of the drawn virtual viewpoint. consistency.

The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a method for evaluating the quality of a cross-validated depth map combined with a JND model, which is characterized in that it includes the following steps:

① mark the depth map to be evaluated as D _tar , mark the color map corresponding to D _tar as Τ _tar , define another known viewpoint except the viewpoint where D _tar and Τ _tar are located as an auxiliary viewpoint, and define the The color map is marked as T _ref ; then by converting the pixel values of all pixels in D _tar into parallax values, all pixels in Τ _tar are mapped to T _ref through 3D-Warping; wherein, D _tar , Τ The total number of pixels on the vertical direction of _tar and T _ref is M, and the total number of pixels on the horizontal direction of D _tar , Τ _tar and T _ref is N;

②Let E _tar represent the difference map whose size is the same as that of D _tar , and record the pixel value of the pixel whose coordinate position is (x, y) in E _tar as E _tar (x, y), when the auxiliary viewpoint When D _tar and Τ _tar are on the left side of the viewpoint, judge whether y+d _{tar, p} (x, y) is greater than N, if so, then make E _tar (x, y)=0, otherwise, satisfy u=x, v=y+d _{tar, p} (x, y), E _tar (x, y)=|Ι _tar (x, y)-Ι _ref (u, v)|; when the auxiliary viewpoint is at the location of D _tar and Τ _tar When on the right side of the viewpoint, judge whether yd _tar,p (x, y) is less than 1, if yes, then set E _tar (x, y)=0, otherwise, satisfy u=x, v=yd _tar,p (x, y), E _tar (x, y) = |Ι _tar (x, y)-Ι _ref (u, v)|; among them, 1≤x≤M, 1≤y≤N, 1≤u≤M, 1 ≤v≤N, d _{tar, p} (x, y) represents the disparity value converted from the pixel value of the pixel whose coordinate position is (x, y) in D _tar , the symbol "||" is the absolute value symbol, Ι _tar (x, y) represents the luminance component of a pixel whose coordinate position is (x, y) in Τ _tar , and Ι _ref (u, v) represents the brightness of a pixel whose coordinate position is (u, v) in T _ref weight;

③ Let C represent the occlusion mask image whose size is the same as that of D _tar , record the pixel value of the pixel whose coordinate position is (x, y) in C as C(x, y), and record each pixel in C The pixel value of each pixel point is initialized to 0, and the total number of pixels at the coordinate position (u, v) in T ref is mapped to T _ref through 3D-Warping in T _tar and recorded as N _{(u, v)} ; when N When _(u,v) ＝1, set C(x,y)＝0; when N _(u,v) >1, Among them, the value of N _{(u, v)} is 0 or 1 or greater than 1, D _tar (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in D _tar , and max() is to take Maximum value function, 1≤x _(u,v),i ≤M,1≤y _(u,v),i ≤N, (x _(u,v),i ,y _(u,v),i ) means The coordinate position of the i-th pixel in the N _{(u, v) pixel points at (u, v)} at the coordinate position in _{T ref through} 3D-Warping mapping in T _tar , D _tar (x _{(u, v), i} , y _{(u, v), i} ) represents the pixel value of the pixel point whose coordinate position is (x _{(u, v), i} , y _{(u, v), i} ) in D _tar ;

④Use C to remove the occluded pixels in E _tar to obtain the difference map after de-occlusion, which is denoted as E' _tar , and the pixel value of the pixel whose coordinate position is (x, y) in E' _tar is denoted as E ' _tar (x, y), E' _tar (x, y) = E _tar (x, y) × (1-C (x, y));

⑤ Calculate the texture judgment factor of each pixel in T _ref , and record the texture judgment factor of the pixel whose coordinate position is (u, v) in T _ref as z(u, v), Among them, 1≤u≤M, 1≤v≤N, z _h (u, v) represents the texture judgment factor in the horizontal direction of the pixel whose coordinate position is (u, v) in T _ref , z _h (u, v ) value is 1 or 0, z _h (u, v) = 1 means that the pixel at the coordinate position (u, v) in T _ref is a texture pixel point in the horizontal direction, z _h (u, v) = 0 means The pixel at the coordinate position (u, v) in T _ref is a non-texture pixel in the horizontal direction, and z _v (u, v) represents the texture in the vertical direction of the pixel at the coordinate position (u, v) in T _ref Judgment factor, the value of z _v (u, v) is 1 or 0, z _v (u, v) = 1 means that the pixel at the coordinate position (u, v) in T _ref is a texture pixel in the vertical direction, z _v (u, v)=0 indicates that the pixel at the coordinate position (u, v) in T _ref is a non-texture pixel in the vertical direction;

⑥Let T denote the region marker map with the same size as T _ref , record the pixel value of the pixel whose coordinate position is (u,v) in T as T(u,v), and set each of T in T The pixel value of the pixel is initialized to 0; use the Canny operator to detect the edge area in T _ref , assuming that the pixel with the coordinate position (u, v) in T _ref belongs to the edge area, then set T (u, v) = 1; Assuming that the texture judgment factor z(u,v)=1 of the pixel whose coordinate position in T _ref is (u,v), then when T(u,v)=0, determine that the coordinate position in T _ref is (u ,v) belongs to the texture area, and set T(u,v)=2 again; wherein, the value of T(u,v) is 0 or 1 or 2, and T(u,v)=0 represents T _ref The pixel point whose coordinate position is (u, v) belongs to the flat area, T(u, v)=1 means that the pixel point whose coordinate position is (u, v) in T _ref belongs to the edge area, T(u, v)= 2 means that the pixel at the coordinate position (u, v) in T _ref belongs to the texture area;

⑦ Introduce a JND model based on brightness masking and texture masking effects, use the JND model, and calculate the error visual threshold of each pixel in T _ref according to the area to which each pixel in T _ref belongs, and convert the coordinates in T _ref The error visual threshold of the pixel at position (u, v) is denoted as Th(u, v), Among them, max() is the function of taking the maximum value, min() is the function of taking the minimum value, bg(u, v) represents the average background brightness of the pixel at the coordinate position (u, v) in T _ref , mg(u, v) represents the maximum average weighting of the surrounding brightness of the pixel at the coordinate position (u, v) in T _ref , and LA(u, v) represents the brightness masking effect of the pixel at the coordinate position (u, v) in T _ref , f(bg(u,v),mg(u,v))=mg(u,v)×α(bg(u,v))+β(bg(u,v)), α(bg(u ,v))=bg(u,v)×0.0001+0.115, β(bg(u,v))=0.5-bg(u,v)×0.01;

⑧ Let E represent the depth error map whose size is the same as that of D _tar , record the pixel value of the pixel whose coordinate position is (x, y) in E as E(x, y), when E' _tar (x ,y)=0, E(x,y)=0; when E' _tar (x,y)≠0, Wherein, V _{(x, y)} =(u, v) represents a mapping process, (x, y) is the coordinate position of the pixel in T _tar , (u, v) is the coordinate position of the pixel in T _ref , when the viewpoint of T _ref is on the left side of the viewpoint of Τ _tar , satisfy u=x, v=y+d _{tar, p} (x, y); when the viewpoint of T _ref is on the right of the viewpoint of Τ _tar , satisfy u = x, v = yd _{tar, p} (x, y);

⑨ count the total number of pixels with a pixel value of 1 in E, and record it as num _E ; then calculate the ratio of the wrong pixels in D _tar as the quality evaluation value of D _tar , and record it as EPR,

The specific process of converting the pixel values of all pixels in D _tar into parallax values in the described step 1. is: for a pixel whose coordinate position is (x, y) in D _tar , its pixel value is converted into The disparity value is recorded as d _tar,p (x,y), Among them, 1≤x≤M, 1≤y≤N, b represents the baseline distance between cameras, f represents the focal length of the camera, Z _near is the closest actual depth of field, Z _far is the farthest actual depth of field, D _tar (x,y) Indicates the pixel value of the pixel whose coordinate position is (x, y) in D _tar .

The acquisition process of z _h (u, v) and z _v (u, v) in the step ⑤ is:

⑤_1. Calculate the differential signal of each pixel point in T _ref along the horizontal direction, and record the differential signal of the pixel point whose coordinate position is (u, v) in T _ref along the horizontal direction as d _h (u, v), Wherein, I _ref (u, v+1) represents the luminance component of the pixel whose coordinate position is (u, v+1) in T _ref ;

⑤_2. Calculate the characteristic sign of the differential signal of each pixel point in T _ref along the horizontal direction, and record the characteristic sign of d _h (u, v) as symd _h (u, v),

⑤_3, calculate z _h (u, v), Among them, d _hsym (u, v) is an intermediate variable, symd _h (u, v+1) represents the characteristic symbol of the differential signal along the horizontal direction of the pixel at the coordinate position (u, v+1) in T _ref ;

⑤_4. Calculate the differential signal of each pixel point in T _ref along the vertical direction, and record the differential signal of the pixel point whose coordinate position is (u, v) in T _ref along the vertical direction as d _v (u, v), Wherein, I _ref (u+1, v) represents the luminance component of the pixel whose coordinate position in T _ref is (u+1, v);

⑤_5. Calculate the characteristic sign of the differential signal of each pixel point in T _ref along the vertical direction, and record the characteristic sign of d _v (u, v) as symd _v (u, v),

⑤_6, calculate z _v (u, v), Among them, d _vsym (u, v) is an intermediate variable, symd _v (u+1, v) represents the characteristic sign of the differential signal along the vertical direction of the pixel at the coordinate position (u+1, v) in T _ref .

Compared with the prior art, the present invention has the advantages of:

1) The method of the present invention fully considers the role of the depth map in virtual viewpoint rendering. The depth map is not used for direct viewing, but provides pixel position offset information. Therefore, it is more convenient to use the virtual viewpoint distortion caused by depth distortion to mark the depth distortion area. Reasonable.

2) The method of the present invention deeply explores the influence of depth map distortion on the quality of virtual viewpoint. Depth distortion will cause offset and object distortion of pixel positions in the virtual viewpoint drawn by using the depth information, and the brightness value of the corresponding pixel is wrong. Generally, the depth The more serious the distortion, the greater the error of the luminance value of the virtual viewpoint pixel, so that the luminance error value of the virtual viewpoint pixel can be used as the error mark of the corresponding depth pixel to obtain a difference map.

3) The method of the present invention fully considers the occlusion of the boundary pixels when the virtual viewpoint is drawn. After the pixels in the color image are mapped to the auxiliary viewpoint by 3D-Warping, the pixels near the object boundary that are closer to the imaging plane may block the distance from the imaging plane. For pixels far from the imaging plane, since the distortion of the occluded pixels has no effect on the quality of the final virtual viewpoint, these occluded pixels can be marked to obtain an occlusion mask, and the occluded pixels can be removed in the difference map The error mark can make the evaluation result of the depth map quality more consistent with the objective result of the virtual viewpoint quality.

4) The method of the present invention fully considers the visual characteristics of the human eye, divides the color image on the auxiliary viewpoint into three parts of edge, texture and flat area, and uses the JND model based on brightness masking and texture masking effects to obtain the pixel points of different parts. The error visual threshold is used to remove the error marks that are smaller than the corresponding error visual threshold after mapping in the de-occluded difference map to obtain the final depth error map, so that the quality evaluation results of the depth map are more in line with the characteristics of the human eye.

Description of drawings

Fig. 1 is the overall realization block diagram of the inventive method;

Figure 2 is a schematic diagram of the cross-validation process;

Figure 3 is a schematic diagram of occlusion;

Figure 4a is the depth map estimated by the AdaptBP method at the second viewpoint of the Cones sequence;

Figure 4b is a color map corresponding to the depth map shown in Figure 4a;

Figure 4c is a color map of the third viewpoint of the Cones sequence;

Figure 4d is a difference map of the depth map shown in Figure 4a obtained after cross-validation;

Figure 5a is an occlusion mask image obtained by mapping the pixel values of the pixels in the depth map shown in Figure 4a to the third viewpoint of the Cones sequence;

Fig. 5b is a difference map after de-occlusion corresponding to the depth map shown in Fig. 4a;

Fig. 5c is a depth error map corresponding to the depth map shown in Fig. 4a.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

A method for evaluating the quality of a cross-validated depth map combined with a JND model proposed by the present invention, its overall implementation block diagram is shown in Figure 1, and it includes the following steps:

① mark the depth map to be evaluated as D _tar , mark the color map corresponding to D _tar as Τ _tar , define another known viewpoint except the viewpoint where D _tar and Τ _tar are located as an auxiliary viewpoint, and define the The color map is marked as T _ref ; then by converting the pixel values of all pixels in D _tar into parallax values, all pixels in Τ _tar are mapped to T _ref through 3D-Warping; wherein, D _tar , Τ The total number of pixels in the vertical direction of _tar and T _ref is M, and the total number of pixels in the horizontal direction of D _tar , Τ _tar and T _ref is N.

In this specific embodiment, the specific process of converting the pixel values of all pixels in D _tar into parallax values in step 1. is: for a pixel whose coordinate position is (x, y) in D _tar , its pixel The disparity value converted from the value is denoted as d _tar,p (x,y), Among them, 1≤x≤M, 1≤y≤N, b represents the baseline distance between cameras, f represents the focal length of the camera, Z _near is the closest actual depth of field, Z _far is the farthest actual depth of field, D _tar (x,y) Indicates the pixel value of the pixel whose coordinate position is (x, y) in D _tar .

②Let E _tar represent the difference map whose size is the same as that of D _tar , and record the pixel value of the pixel whose coordinate position is (x, y) in E _tar as E _tar (x, y), when the auxiliary viewpoint When D _tar and Τ _tar are on the left side of the viewpoint, judge whether y+d _{tar, p} (x, y) is greater than N, if so, then make E _tar (x, y)=0, otherwise, satisfy u=x, v=y+d _{tar, p} (x, y), E _tar (x, y)=|Ι _tar (x, y)-Ι _ref (u, v)|; when the auxiliary viewpoint is at the location of D _tar and Τ _tar When on the right side of the viewpoint, judge whether yd _tar,p (x, y) is less than 1, if yes, then set E _tar (x, y)=0, otherwise, satisfy u=x, v=yd _tar,p (x, y), E _tar (x, y) = |Ι _tar (x, y)-Ι _ref (u, v)|; among them, 1≤x≤M, 1≤y≤N, 1≤u≤M, 1 ≤v≤N, d _{tar, p} (x, y) represents the disparity value converted from the pixel value of the pixel whose coordinate position is (x, y) in D _tar , the symbol "||" is the absolute value symbol, Ι _tar (x, y) represents the luminance component of a pixel whose coordinate position is (x, y) in Τ _tar , and Ι _ref (u, v) represents the brightness of a pixel whose coordinate position is (u, v) in T _ref portion.

In step ①, the process of mapping all pixels in Τ _tar to T _ref through 3D-Warping and the process of step ② are cross-validation processes. Figure 2 shows a schematic diagram of the cross-validation process, where T _l , T _r , D _l , and D _r correspond to the left-viewpoint color map, right-viewpoint color map, left-viewpoint depth map, and right-viewpoint depth map. When it is necessary to obtain the difference map corresponding to the left-viewpoint depth map, the right-viewpoint color map is used as auxiliary information for cross-validation. The brightness value corresponding to the pixel at the coordinate position (x _l , y _l ) in T _l is I _l1 , using the depth information in D _l , it is mapped to the right-viewpoint color image T _r through the 3D-Warping process. range, the pixel at the coordinate position (x _l , y _l ) in L _d is assigned a value of 0, and if it is mapped to the pixel at the coordinate position (x _lr , y _l ) in the right-view color image T _r , the corresponding The luminance value is Ι _r1 , then assign the difference |Ι _l1 -Ι _r1 | of the luminance values of two pixels to the pixel in L _d whose coordinate position is (x _l , y _l ), and L _d is the left view point The difference map corresponding to the depth map. Similarly, to obtain the difference map corresponding to the right-viewpoint depth map, the left-viewpoint color map T _l is used as auxiliary information for cross-validation, and the difference map _Rd corresponding to the right-viewpoint depth map can be obtained.

Use the depth map estimated by the AdaptBP method at the second viewpoint of the Cones sequence as the depth map to be evaluated, as shown in Figure 4a; Figure 4b is the color map corresponding to the depth map shown in Figure 4a; use the color map of the third viewpoint of the Cones sequence As the color map on the auxiliary viewpoint, it is shown in Figure 4c; the difference map obtained after cross-validation is shown in Figure 4d.

③ Let C represent the occlusion mask image whose size is the same as that of D _tar , record the pixel value of the pixel whose coordinate position is (x, y) in C as C(x, y), and record each pixel in C The pixel value of each pixel point is initialized to 0, and the total number of pixels at the coordinate position (u, v) in T ref is mapped to T _ref through 3D-Warping in T _tar and recorded as N _{(u, v)} ; when N When _(u,v) ＝1, set C(x,y)＝0; when N _(u,v) >1, Among them, the value of N _{(u, v)} is 0 or 1 or greater than 1, D _tar (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in D _tar , and max() is to take Maximum value function, 1≤x _(u,v),i ≤M,1≤y _(u,v),i ≤N, (x _(u,v),i ,y _(u,v),i ) means The coordinate position of the i-th pixel in the N _{(u, v) pixel points at (u, v)} at the coordinate position in _{T ref through} 3D-Warping mapping in T _tar , D _tar (x _(u,v),i ,y _(u,v),i ) represents the pixel value of the pixel point whose coordinate position is (x _(u,v),i ,y _(u,v),i ) in D _tar .

Figure 3 shows a schematic diagram of occlusion. The left reference viewpoint foreground boundary point and background boundary point are denoted from left to right as The foreground boundary point and background boundary point of the right reference viewpoint are denoted from left to right as In the 3D-Warping process, the boundary point in the left reference viewpoint respectively mapped to the virtual viewpoint drawn by it Similarly, the boundary points in the right reference viewpoint respectively mapped to the virtual viewpoint drawn by it In the left reference virtual view, from arrive In this part, both the foreground pixels in the left reference image and the background pixels are mapped. Similarly, in the right reference virtual view, from the arrive In this part, both the foreground pixels in the right reference image and the background pixels are mapped. In these parts, the foreground pixels are occluded from the background pixels. In the difference map obtained after cross-validation, the occluded background pixels will also be marked, but this is not caused by the wrong depth value, so These background pixels need to be removed. After 3D-Warping, for the pixels mapped to the same position, compare their depth values, keep the pixel with the largest depth value, and mark the remaining pixels to obtain an occlusion mask image.

Using the depth map shown in Figure 4a as the depth map to be evaluated, and the color map shown in Figure 4c as the color map on the auxiliary viewpoint, the obtained occlusion mask image is shown in Figure 5a.

④Use C to remove the occluded pixels in E _tar to obtain the difference map after de-occlusion, which is denoted as E' _tar , and the pixel value of the pixel whose coordinate position is (x, y) in E' _tar is denoted as E ' _tar (x, y), E' _tar (x, y) = E _tar (x, y) × (1-C (x, y)).

Fig. 5b is a difference map obtained after removing the occluded pixels in the occlusion mask image shown in Fig. 5a from the difference map shown in Fig. 4d.

⑤ Calculate the texture judgment factor of each pixel in T _ref , and record the texture judgment factor of the pixel whose coordinate position is (u, v) in T _ref as z(u, v), Among them, 1≤u≤M, 1≤v≤N, z _h (u, v) represents the texture judgment factor in the horizontal direction of the pixel whose coordinate position is (u, v) in T _ref , z _h (u, v ) value is 1 or 0, z _h (u, v) = 1 means that the pixel at the coordinate position (u, v) in T _ref is a texture pixel point in the horizontal direction, z _h (u, v) = 0 means The pixel at the coordinate position (u, v) in T _ref is a non-texture pixel in the horizontal direction, and z _v (u, v) represents the texture in the vertical direction of the pixel at the coordinate position (u, v) in T _ref Judgment factor, the value of z _v (u, v) is 1 or 0, z _v (u, v) = 1 means that the pixel at the coordinate position (u, v) in T _ref is a texture pixel in the vertical direction, z _v (u, v)=0 indicates that the pixel at the coordinate position (u, v) in T _ref is a non-texture pixel in the vertical direction.

In this specific embodiment, the acquisition process of z _h (u, v) and z _v (u, v) in step ⑤ is:

⑤_1. Calculate the differential signal of each pixel point in T _ref along the horizontal direction, and record the differential signal of the pixel point whose coordinate position is (u, v) in T _ref along the horizontal direction as d _h (u, v), Wherein, I _ref (u, v+1) represents the luminance component of the pixel at the coordinate position (u, v+1) in T _ref .

⑤_2. Calculate the characteristic sign of the differential signal of each pixel point in T _ref along the horizontal direction, and record the characteristic sign of d _h (u, v) as symd _h (u, v),

⑤_3, calculate z _h (u, v), Among them, d _hsym (u, v) is an intermediate variable, symd _h (u, v+1) represents the characteristic symbol of the differential signal along the horizontal direction of the pixel at the coordinate position (u, v+1) in T _ref .

⑤_4. Calculate the differential signal of each pixel point in T _ref along the vertical direction, and record the differential signal of the pixel point whose coordinate position is (u, v) in T _ref along the vertical direction as d _v (u, v), Wherein, I _ref (u+1, v) represents the luminance component of the pixel at the coordinate position (u+1, v) in T _ref .

⑤_5. Calculate the characteristic sign of the differential signal of each pixel point in T _ref along the vertical direction, and record the characteristic sign of d _v (u, v) as symd _v (u, v),

⑤_6, calculate z _v (u, v), Among them, d _vsym (u, v) is an intermediate variable, symd _v (u+1, v) represents the characteristic sign of the differential signal along the vertical direction of the pixel at the coordinate position (u+1, v) in T _ref .

⑥Let T denote the region marker map with the same size as T _ref , record the pixel value of the pixel whose coordinate position is (u,v) in T as T(u,v), and set each of T in T The pixel value of the pixel is initialized to 0; use the Canny operator to detect the edge area in T _ref , assuming that the pixel with the coordinate position (u, v) in T _ref belongs to the edge area, then let T (u, v) = 1; Assuming that the texture judgment factor z(u,v)=1 of the pixel whose coordinate position is (u,v) in T _ref , then when T(u,v)=0, it is determined that the coordinate position in T _ref is (u ,v) belongs to the texture area, and set T(u,v)=2 again; wherein, the value of T(u,v) is 0 or 1 or 2, and T(u,v)=0 represents T _ref The pixel point whose coordinate position is (u, v) belongs to the flat area, T(u, v)=1 means that the pixel point whose coordinate position is (u, v) in T _ref belongs to the edge area, T(u, v)= 2 means that the pixel at the coordinate position (u, v) in T _ref belongs to the texture area.

⑦ Introduce a JND model based on brightness masking and texture masking effects, use the JND model, and calculate the error visual threshold of each pixel in T _ref according to the area to which each pixel in T _ref belongs, and convert the coordinates in T _ref The error visual threshold of the pixel at position (u, v) is denoted as Th(u, v), Among them, max() is the function of taking the maximum value, min() is the function of taking the minimum value, bg(u, v) represents the average background brightness of the pixel at the coordinate position (u, v) in T _ref , bg(u, v) Calculated by a weighted low-pass operator, mg(u,v) represents the maximum average weighting of the surrounding brightness of the pixel whose coordinate position is (u,v) in T _ref , LA(u,v) represents T _ref The brightness masking effect of the pixel with the middle coordinate position (u,v), f(bg(u,v),mg(u,v))=mg(u,v)×α(bg(u,v))+β(bg(u,v)), α(bg(u, v))=bg(u,v)×0.0001+0.115, β(bg(u,v))=0.5−bg(u,v)×0.01.

⑧ Let E represent the depth error map whose size is the same as that of D _tar , record the pixel value of the pixel whose coordinate position is (x, y) in E as E(x, y), when E' _tar (x ,y)=0, E(x,y)=0; when E' _tar (x,y)≠0, Wherein, V _{(x, y)} =(u, v) represents a mapping process, (x, y) is the coordinate position of the pixel in T _tar , (u, v) is the coordinate position of the pixel in T _ref , when the viewpoint of T _ref is on the left side of the viewpoint of Τ _tar , satisfy u=x, v=y+d _{tar, p} (x, y); when the viewpoint of T _ref is on the right of the viewpoint of Τ _tar , satisfy u =x, v=yd _tar,p (x,y).

Fig. 5c is the depth error map of Fig. 4a obtained after removing pixels smaller than the corresponding error visual threshold in Fig. 5b.

⑨ count the total number of pixels with a pixel value of 1 in E, and record it as num _E ; then calculate the ratio of the wrong pixels in D _tar as the quality evaluation value of D _tar , and record it as EPR,

In order to test the performance of the method of the present invention, the depth maps estimated by using a variety of different algorithms provided by the Middlebury database are tested, and four scenes are selected: "Tsukuba", "Venus", "Teddy" and "Cones", for each For each scene, the depth maps estimated by nine different stereo matching algorithms from the second viewpoint are used, and a total of 36 depth maps constitute the evaluation database. The nine different stereo matching algorithms are: AdaptBP, WarpMat, P-LinearS, VSW, BPcompressed, Layered, SNCC, ReliabilityDP and Infection.

Table 1 shows the values of the full-reference objective quality evaluation index PBMP (Percentage of Bad Matching Pixels) for "Tsukuba", "Venus", "Teddy" and "Cones" in the evaluation database. PBMP is used to estimate the depth map with the distortion-free Refer to the depth map for comparison to calculate the error. If the parallax error of a pixel is greater than one pixel width, it is regarded as an error pixel. Due to the use of an undistorted depth map as a reference, PBMP is an accurate and reliable full reference metric.

Table 1 Values of PBMP (%) for different depth maps in the evaluation database

method Tsukuba Venus Teddy Cones AdaptBP 1.37 0.21 7.06 7.92 WarpMat 1.35 0.24 9.30 8.47 P-LinearS 1.67 0.89 12.00 8.44 VSW 1.88 0.81 13.3 8.85 BP compressed 3.63 1.89 13.9 9.85 Layered 1.87 1.85 14.3 14.70 SNCC 6.08 1.73 11.10 9.02 ReliabilityDP 3.39 3.48 16.90 19.90 Infection 9.54 5.53 25.10 21.30

Table 2 shows the quality evaluation values of "Tsukuba", "Venus", "Teddy" and "Cones" in the evaluation database obtained by the method of the present invention. Table 3 has provided the evaluation result of the inventive method and the correlation coefficient of full reference index PBMP, and correlation coefficient has weighed the consistency degree of both, and the value of Pearson coefficient and linear regression coefficient all is that the value closer to 1 is better. It can be seen from Table 3 that the results obtained by the method of the present invention are in good agreement with PBMP, indicating that the method of the present invention can accurately detect depth errors and evaluate the quality of depth maps.

Table 2 The quality evaluation value EPR (%) of different depth maps in the evaluation database

Table 3 Correlation between quality evaluation value EPR and PBMP

Tsukuba Venus Teddy Cones Pearson Coefficient 0.94 0.90 0.84 0.97 linear regression coefficient 0.89 0.80 0.71 0.93

Table 4 shows the correlation coefficient between the evaluation results of the method of the present invention and the quality of the virtual viewpoint, and the quality of the virtual viewpoint is measured by the objective evaluation index mean square error MSE. Because the virtual view synthesis is based on the depth map, the poorer the depth quality will lead to more errors in the virtual view, which indicates that MSE should increase with the increase of the quality evaluation value EPR, using MSE and quality evaluation value EPR The linear regression coefficient for indicates the accuracy of the measure. In "Tsukuba", "Venus", "Teddy" and "Cones", the linear regression coefficients of the quality evaluation values EPR and MSE all exceeded 0.75. In particular, the linear regression coefficient exceeded 0.92 in "Tsukuba". This shows that the quality evaluation value EPR has a good agreement with the quality of the virtual viewpoint.

Table 4 Correlation between quality evaluation value EPR and virtual viewpoint quality

Tsukuba Venus Teddy Cones linear regression coefficient 0.93 0.76 0.84 0.91

Claims (3)

1. a method for evaluating the quality of a cross-validation depth map in conjunction with a JND model, characterized in that it may further comprise the steps: ① mark the depth map to be evaluated as D _tar , mark the color map corresponding to D _tar as Τ _tar , define another known viewpoint except the viewpoint where D _tar and Τ _tar are located as an auxiliary viewpoint, and define the The color map is marked as T _ref ; then by converting the pixel values of all pixels in D _tar into parallax values, all pixels in Τ _tar are mapped to T _ref through 3D-Warping; wherein, D _tar , Τ The total number of pixels on the vertical direction of _tar and T _ref is M, and the total number of pixels on the horizontal direction of D _tar , Τ _tar and T _ref is N; ②Let E _tar represent the difference map whose size is the same as that of D _tar , and record the pixel value of the pixel whose coordinate position is (x, y) in E _tar as E _tar (x, y), when the auxiliary viewpoint When D _tar and Τ _tar are on the left side of the viewpoint, judge whether y+d _{tar, p} (x, y) is greater than N, if so, then make E _tar (x, y)=0, otherwise, satisfy u=x, v=y+d _{tar, p} (x, y), E _tar (x, y)=|Ι _tar (x, y)-Ι _ref (u, v)|; when the auxiliary viewpoint is at the location of D _tar and Τ _tar When on the right side of the viewpoint, judge whether yd _tar,p (x, y) is less than 1, if yes, then set E _tar (x, y)=0, otherwise, satisfy u=x, v=yd _tar,p (x, y), E _tar (x, y) = |Ι _tar (x, y)-Ι _ref (u, v)|; among them, 1≤x≤M, 1≤y≤N, 1≤u≤M, 1 ≤v≤N, d _{tar, p} (x, y) represents the disparity value converted from the pixel value of the pixel whose coordinate position is (x, y) in D _tar , the symbol "| |" is the absolute value symbol, Ι _tar (x, y) represents the luminance component of a pixel whose coordinate position is (x, y) in Τ _tar , and Ι _ref (u, v) represents the brightness of a pixel whose coordinate position is (u, v) in T _ref weight; ③ Let C represent the occlusion mask image whose size is the same as that of D _tar , record the pixel value of the pixel whose coordinate position is (x, y) in C as C(x, y), and record each pixel in C The pixel value of each pixel point is initialized to 0, and the total number of pixels at the coordinate position (u, v) in T ref is mapped to T _ref through 3D-Warping in T _tar and recorded as N _{(u, v)} ; when N When _(u,v) ＝1, set C(x,y)＝0; when N _(u,v) >1, Among them, the value of N _{(u, v)} is 0 or 1 or greater than 1, D _tar (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in D _tar , and max() is to take Maximum value function, 1≤x _(u,v),i ≤M,1≤y _(u,v),i ≤N, (x _(u,v),i ,y _(u,v),i ) means The coordinate position of the i-th pixel in the N _{(u, v) pixel points at (u, v)} at the coordinate position in _{T ref through} 3D-Warping mapping in T _tar , D _tar (x _{(u, v), i} , y _{(u, v), i} ) represents the pixel value of the pixel point whose coordinate position is (x _{(u, v), i} , y _{(u, v), i} ) in D _tar ; ④Use C to remove the occluded pixels in E _tar to obtain the difference map after de-occlusion, which is denoted as E' _tar , and the pixel value of the pixel whose coordinate position is (x, y) in E' _tar is denoted as E ' _tar (x, y), E' _tar (x, y) = E _tar (x, y) × (1-C (x, y)); ⑤ Calculate the texture judgment factor of each pixel in T _ref , and record the texture judgment factor of the pixel whose coordinate position is (u, v) in T _ref as z(u, v), Among them, 1≤u≤M, 1≤v≤N, z _h (u, v) represents the texture judgment factor in the horizontal direction of the pixel whose coordinate position is (u, v) in T _ref , z _h (u, v ) value is 1 or 0, z _h (u, v) = 1 means that the pixel at the coordinate position (u, v) in T _ref is a texture pixel point in the horizontal direction, z _h (u, v) = 0 means The pixel at the coordinate position (u, v) in T _ref is a non-texture pixel in the horizontal direction, and z _v (u, v) represents the texture in the vertical direction of the pixel at the coordinate position (u, v) in T _ref Judgment factor, the value of z _v (u, v) is 1 or 0, z _v (u, v) = 1 means that the pixel at the coordinate position (u, v) in T _ref is a texture pixel in the vertical direction, z _v (u, v)=0 indicates that the pixel at the coordinate position (u, v) in T _ref is a non-texture pixel in the vertical direction; ⑥Let T denote the region marker map with the same size as T _ref , record the pixel value of the pixel whose coordinate position is (u,v) in T as T(u,v), and set each of T in T The pixel value of the pixel is initialized to 0; use the Canny operator to detect the edge area in T _ref , assuming that the pixel with the coordinate position (u, v) in T _ref belongs to the edge area, then let T (u, v) = 1; Assuming that the texture judgment factor z(u,v)=1 of the pixel whose coordinate position is (u,v) in T _ref , then when T(u,v)=0, it is determined that the coordinate position in T _ref is (u ,v) belongs to the texture area, and set T(u,v)=2 again; wherein, the value of T(u,v) is 0 or 1 or 2, and T(u,v)=0 represents T _ref The pixel point whose coordinate position is (u, v) belongs to the flat area, T(u, v)=1 means that the pixel point whose coordinate position is (u, v) in T _ref belongs to the edge area, T(u, v)= 2 means that the pixel at the coordinate position (u, v) in T _ref belongs to the texture area; ⑦ Introduce a JND model based on brightness masking and texture masking effects, use the JND model, and calculate the error visual threshold of each pixel in T _ref according to the area to which each pixel in T _ref belongs, and convert the coordinates in T _ref The error visual threshold of the pixel at position (u, v) is denoted as Th(u, v), Among them, max() is the function of taking the maximum value, min() is the function of taking the minimum value, bg(u, v) represents the average background brightness of the pixel at the coordinate position (u, v) in T _ref , mg(u, v) represents the maximum average weighting of the surrounding brightness of the pixel at the coordinate position (u, v) in T _ref , and LA(u, v) represents the brightness masking effect of the pixel at the coordinate position (u, v) in T _ref , f(bg(u,v),mg(u,v))=mg(u,v)×α(bg(u,v))+β(bg(u,v)), α(bg(u, v))=bg(u,v)×0.0001+0.115, β(bg(u,v))=0.5-bg(u,v)×0.01; ⑧ Let E represent the depth error map whose size is the same as that of D _tar , record the pixel value of the pixel whose coordinate position is (x, y) in E as E(x, y), when E' _tar (x ,y)=0, E(x,y)=0; when E' _tar (x,y)≠0, Wherein, V _{(x, y)} =(u, v) represents a mapping process, (x, y) is the coordinate position of the pixel in T _tar , (u, v) is the coordinate position of the pixel in T _ref , when the viewpoint of T _ref is on the left side of the viewpoint of Τ _tar , satisfy u=x, v=y+d _{tar, p} (x, y); when the viewpoint of T _ref is on the right of the viewpoint of Τ _tar , satisfy u = x, v = yd _{tar, p} (x, y); ⑨ count the total number of pixels with a pixel value of 1 in E, and record it as num _E ; then calculate the ratio of the wrong pixels in D _tar as the quality evaluation value of D _tar , and record it as EPR, 2. the method for evaluating the quality of the cross-validation depth map in conjunction with the JND model according to claim 1, is characterized in that in described step 1. the concrete process that the pixel value of all pixel points in D _tar is converted into parallax value is: : For the pixel point whose coordinate position in D _tar is (x, y), the disparity value obtained by converting its pixel value is recorded as d _{tar, p} (x, y), Among them, 1≤x≤M, 1≤y≤N, b represents the baseline distance between cameras, f represents the focal length of the camera, Z _near is the closest actual depth of field, Z _far is the farthest actual depth of field, D _tar (x,y) Indicates the pixel value of the pixel whose coordinate position is (x, y) in D _tar . 3. according to claim 1 and 2 described cross validation depth map quality evaluation method in conjunction with JND model, it is characterized in that described step 5. in the acquisition process of z _h (u, v) and z _v (u, v) for: ⑤_1. Calculate the differential signal of each pixel point in T _ref along the horizontal direction, and record the differential signal of the pixel point whose coordinate position is (u, v) in T _ref along the horizontal direction as d _h (u, v), Wherein, I _ref (u, v+1) represents the luminance component of the pixel whose coordinate position is (u, v+1) in T _ref ; ⑤_2. Calculate the characteristic sign of the differential signal of each pixel point in T _ref along the horizontal direction, and record the characteristic sign of d _h (u, v) as symd _h (u, v), ⑤_3, calculate z _h (u, v), Among them, d _hsym (u, v) is an intermediate variable, symd _h (u, v+1) represents the characteristic symbol of the differential signal along the horizontal direction of the pixel at the coordinate position (u, v+1) in T _ref ; ⑤_4. Calculate the differential signal of each pixel point in T _ref along the vertical direction, and record the differential signal of the pixel point whose coordinate position is (u, v) in T _ref along the vertical direction as d _v (u, v), Wherein, I _ref (u+1, v) represents the luminance component of the pixel whose coordinate position in T _ref is (u+1, v); ⑤_5. Calculate the characteristic sign of the differential signal of each pixel point in T _ref along the vertical direction, and record the characteristic sign of d _v (u, v) as symd _v (u, v), ⑤_6, calculate z _v (u, v), Among them, d _vsym (u, v) is an intermediate variable, symd _v (u+1, v) represents the characteristic sign of the differential signal along the vertical direction of the pixel at the coordinate position (u+1, v) in T _ref .