CN101888566B - Estimation method of distortion performance of stereo video encoding rate - Google Patents

Abstract

The invention provides an estimation method of distortion performance of a stereo video encoding rate, comprising the following steps: acquiring a multi-view video and acquiring a corresponding multi-view depth map according to the multi-view video; respectively obtaining a multi-view video encoding rate distortion model and a multi-view depth map encoding rate distortion model; respectively obtaining the relationship between multi-view video encoding distortion and virtual view rendering distortion as well as the relationship between the multi-view depth map encoding distortion and the virtual view rendering distortion; and finally establishing a virtual view encoding rendering rate distortion model in a stereo video based on the relationships. By the analysis method of virtual view encoding rendering rate distortion in the stereo video provided by the invention, the distortion performance of the virtual view encoding rendering rate in the stereo video can be accurately and rapidly established so as to provide model guide and solution for problems such as selection of coding parameters, code rate distribution and the like in the stereo video.

Description

Estimation method of distortion performance of stereo video encoding rate

Technical field

The present invention relates to technical field of image processing, particularly a kind of estimation method of distortion performance of stereo video encoding rate to virtual viewpoint rendering.

Background technology

Along with the continuous development of multimedia technology, the traditional two-dimensional video media can not satisfy the needs of human vision, and people hope to see and have more the sense of reality and interactive video frequency program, the 3 D video that also promptly has highly three-dimensional feeling of immersion.The 3 D video technology of rising gradually in recent years becomes the important part of multimedia technology just gradually like brand-new video medias such as multi-view point video, free viewpoint video and three-dimensional video-frequencies.So-called free viewpoint video is meant that the beholder is through freely selecting to watch viewpoint to watch the video media of three-dimensional scenic.

Two Main Stage have been experienced in the free viewpoint video development, and the phase I is known as multi-view point video, and promptly spectators can the viewpoint of selection oneself needs watch from the multi-view point video of playing, and form third dimension through the parallax between the different points of view video.Second stage is called as three-dimensional video-frequency, and promptly the beholder can freely select to watch viewpoint to watch to have relief video.In order to obtain multi-view point video, just need to adopt the multichannel video camera to gather Same Scene, and the multi-channel video that collects is sent to client, and select corresponding multi-view point video to play based on spectators' needs.And in order to obtain to have the three-dimensional video-frequency of free view-point; Need on the basis of multi-view point video, further introduce the geological information of scene; Play up through multi-view point video data and geological information and to obtain the virtual view that spectators select at random, thereby let spectators obtain the stronger third dimension of watching.

Be different from traditional single channel two-dimensional video data, three-dimensional video-frequency is made up of multichannel two-dimensional video data and scene geometric information, so the mass data of multi-view point video is far longer than the traditional two-dimensional video data to the demand of transmission bandwidth.In addition, how to compress scene geometric information effectively and also become the right new challenge of stereo scopic video coding demand side.Therefore, in order to realize effective transmission of stereoscopic video, need to come the coding techniques of design of High Efficiency to the three-dimensional video-frequency characteristic.In early days, the researcher is applied to traditional video compression technology multi-view video compressed, adopts traditional single channel video compression technology that the corresponding video of each viewpoint in the multi-view point video is compressed separately.This scheme becomes one of three-dimensional video-frequency compression solution the earliest.

Yet, there is stronger correlation between the multi-view point video sequence, even adopt the most H.264/AVC coding techniques, still can't compress the redundancy between the different points of view effectively.For this reason, the researcher has designed multi-view video compressed more efficiently scheme.This scheme is extended to traditional single channel video coding technique on the multi-channel video, in the predictive coding that adds on the basis of time domain prediction coding between viewpoint, has further compressed the redundancy between the multi-view point video viewpoint, has improved multi-view video compressed distortion performance.In addition, to three-dimensional video-frequency scene geometric information based on depth map, i.e. multi-view depth graphic sequence, the researcher adopts the multiple view video coding scheme to realize its efficient compression equally.Along with the continuous development of three-dimensional video-frequency technology, new three-dimensional video-frequency compress technique is development fast.At present, one of the key technology in market is moved towards in the application that become three-dimensional video-frequency of the stereo scopic video coding technology of real-time high-efficiency.

On the basis of three-dimensional video-frequency high efficient coding scheme,, need condition adjustment coding parameter Network Based in order to realize effective transmission of stereoscopic video.In traditional video coding, under the situation of known network bandwidth, can estimate decoding quality based on the rate-distortion model of coding, thereby adjust coding parameter better.Therefore, the rate-distortion model analysis to three-dimensional video-frequency high efficient coding scheme is to realize the effectively important step of transmission of three-dimensional video-frequency.

The shortcoming that prior art exists is also not to be directed against the estimation scheme of the stereo video encoding rate distortion performance of virtual viewpoint rendering at present.

For example; At application number is 200810163801; Name is called in a kind of patent application of method for encoding stereo video of network self-adapting, has only provided the adaptive coding scheme under the three-dimensional video-frequency various network bandwidth, does not analyze to the distortion performance of stereo scopic video coding.At application number is 200810126528; Name is called in the patent application of a kind of stereo video coding-decoding method, Apparatus and system; Though disclose a whole set of complete stereo scopic video coding scheme, device and system, do not provided the analysis of stereo video encoding rate distortion performance equally.At application number is 200710164747; Name is called in a kind of patent application of the bit rate control method towards multi-view point video; Though announced the rate-distortion model of multiple view video coding, be not directed against the stereo video encoding rate distortion performance of virtual viewpoint rendering.

Summary of the invention

The object of the invention is intended to solve above-mentioned technological deficiency at least, has proposed a kind of estimation method of distortion performance of stereo video encoding rate to virtual viewpoint rendering.

For achieving the above object, one aspect of the present invention has proposed a kind of estimation method of distortion performance of stereo video encoding rate, may further comprise the steps: obtain multi-view point video, and obtain corresponding multi-view depth figure according to said multi-view point video; Obtain multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model respectively according to said multi-view point video and said multi-view depth figure; Obtain the relation between said multiple view video coding distortion and the virtual viewpoint rendering distortion respectively according to said multi-view point video and said multi-view depth figure, and the relation between distortion of said multi-view depth graph code and the virtual viewpoint rendering distortion; According to the relation between said multiple view video coding distortion and the virtual viewpoint rendering distortion, and the relation between distortion of said multi-view depth graph code and the virtual viewpoint rendering distortion obtains the relation between virtual viewpoint rendering distortion and multi-view point video and the multi-view depth figure coded quantization parameter QP separately; Obtain to draw the relation between needed encoder bit rate of said virtual view and the said QP according to said multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model; And add up the drafting distortion of said virtual view under the different Q P and draw the needed encoder bit rate of said virtual view to obtain the rate-distortion model of virtual view in the stereo scopic video coding.

Through virtual viewpoint rendering encoding rate distortion analysis method in the three-dimensional video-frequency of the present invention's proposition; Can estimate virtual viewpoint rendering encoding rate distortion performance in the three-dimensional video-frequency quickly and accurately; Thereby instruct and solution for problems such as stereo scopic video coding parameters of choice and Data Rate Distribution provide model, further improved the efficient of stereo scopic video coding.

Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize through practice of the present invention.

Description of drawings

Above-mentioned and/or additional aspect of the present invention and advantage are from obviously with easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:

The three-dimensional video-frequency system block diagram that Fig. 1 provides for the embodiment of the invention;

The multiple view video coding predict figure that Fig. 2 provides for the embodiment of the invention;

Fig. 3 is the estimation method of distortion performance of stereo video encoding rate flow chart of the embodiment of the invention;

The virtual viewpoint rendering schematic diagram that Fig. 4 provides for the embodiment of the invention;

The depth map quaternary tree decomposing schematic representation that Fig. 5 provides for the embodiment of the invention.

Embodiment

Describe embodiments of the invention below in detail, the example of said embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Be exemplary through the embodiment that is described with reference to the drawings below, only be used to explain the present invention, and can not be interpreted as limitation of the present invention.

Because in the three-dimensional video-frequency, virtual view is to draw through multi-view point video and scene geometric information, so the virtual viewpoint rendering quality is relevant with the coding quality of multi-view point video and scene geometric information.For this reason; The present invention sets up multi-view point video through needs and scene geometric information coding associating rate-distortion model obtains the encoding rate distortion model to the virtual viewpoint rendering video, thereby instructs problems such as stereo scopic video coding parameters of choice and Data Rate Distribution better.

The present invention has mainly proposed a kind of multi-view point video and depth map encoding associating rate-distortion model method of estimation of practicality.This method can be estimated the relation between multi-view point video and depth map encoding quality and the virtual viewpoint rendering quality effectively, thereby is that problems such as selection of stereo scopic video coding parameter and Data Rate Distribution provide theoretical direction under the condition of the given network bandwidth.

The applied environment of the embodiment of the invention is following: the three-dimensional video-frequency system block diagram that the embodiment of the invention is used is as shown in Figure 1.Wherein, the video sequence that is used for stereo scopic video coding adopts the standard testing video sequence of the name of high-definition format for " Breakdancer "; The pixel of this high-definition format video sequence is 1024 * 768; Decoder adopts the H.264/SVC reference software JMVC (JointMulti-view Video Coding, multiple view video coding) of (Multi-view Video Coding, multi-view point video extended version) standard; The frame number of encoder GOP (Group of Pictures, image sets) is 8; The time domain prediction coding of coding adopts the Hierarchical B predict of (stratification bi-directional predictive coding frame is called for short stratification B frame), and the coded prediction structure chart is as shown in Figure 2.

Server end obtains multi-view point video, and obtains multi-view depth figure based on multi-view point video, and in stereo scopic video coding, multi-view point video and multi-view depth figure is encoded and be transferred to client through network.Client is decoded to the multi-view point video and the multi-view depth graph code code stream that receive, and the virtual view of selecting according to spectators generates corresponding virtual view video and it is shown to spectators through three-dimensional display.

In this enforcement sample, virtual viewpoint rendering adopts two-path video and the depth map adjacent with virtual view to draw.Concrete, the viewpoint 4 of this enforcement sample employing " Breakdancer " sequence and viewpoint 6 these two-path videos are as the multi-view point video list entries.Wherein viewpoint 4 is called left reference view, and viewpoint 6 is called right reference view.The span of multi-view point video and multi-view depth graph code quantization parameter QP is the integer between 0 to 51.The parameter of the virtual view that generates is identical with the parameter of viewpoint 5 in " Breakdancer " sequence.

As shown in Figure 3, be the estimation method of distortion performance of stereo video encoding rate flow chart of the embodiment of the invention, in this embodiment, multiple view video coding adopts and realized originally based on conventional video coding standard multiple view video coding extended edition H.264/AVC.Intracoded frame (I frame), forward-predictive-coded frames (P frame) and the bi-directional predictive coding frame (B frame) of each image sets in each viewpoint all adopt identical quantization parameter (QP) to encode.This method may further comprise the steps:

Step S301 sets up the multiple view video coding rate-distortion model.Under the situation of given multiple view video coding quantization parameter, estimate multiple view video coding code check and distortion.After the value of given coded quantization parameter QP, at first pass through Q _Step=2 ^(QP-4)/6Obtain quantization parameter Q _StepValue.For example, when QP=28, Q _Step=2 ^(28-4)/6=16.

This step can further be subdivided into following two steps:

Step S101, the code check r of multiple view video coding _cCalculation expression do Wherein, parameter Q _StepCalculation expression be Q _Step=2 ^(QP-4)/6, parameter a, b, c need specifically be provided with according to the different video sequence.Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because encoder bit rate r _cWith quantization parameter Q _StepBetween relational expression do

Therefore can come match to obtain parameter a through the method for linear regression be 24.37, b for-0.303 with c be 0.02, also promptly for its encoder bit rate of left and right sides viewpoint r _cWith quantization parameter Q _StepRelational expression between x is: For example, work as Q _Step=16 o'clock, r _c=0.096bpp.

Step S102, the distortion computation expression formula of multiple view video coding is PSNR _c=q _c* QP+p _c, parameter q wherein _cAnd p _cNeed specifically be provided with according to the different video sequence.At this moment, under the condition of given multiple view video coding quantization parameter QP, can obtain multiple view video coding code check r _cWith distortion PSNR _cBetween corresponding relation.

Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because the distortion computation expression formula of multiple view video coding is PSNR _c=q _c* QP+p _c, come match to obtain parameter q through the method for linear regression _cBe-0.37 and p _cBe 49.30, also promptly for its decoding quality of left and right sides viewpoint PSNR _cAnd the relation between the coded quantization parameter QP can be expressed as PSNR _c=-0.37 * QP+49.30.When QP=28, PSNR _c=-0.37 * 28+49.30=38.94dB.

Step S302 sets up the depth map encoding rate-distortion model.Under the situation of given multi-view depth graph code quantization parameter, estimate the corresponding relation between multi-view point video depth map encoding code check and the distortion.In this step, multi-view depth figure also adopts with multiple view video coding described in the step S301 and encodes.Similarly, I frame, P frame and the B frame of each image sets in each viewpoint also all adopt identical QP to encode.With above-mentioned step, Q when the value of coded quantization parameter QP is 28 _Step=16.

This step can further be subdivided into following two steps:

The code check r of S221, multi-view depth graph code _dCalculation expression be r _d=k/Q _Step+ t, wherein parameter Q _StepCalculation expression be Q _Step=2 ^(QP-4)/6, parameter k, t need specifically be provided with according to the different depth graphic sequence.

Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because encoder bit rate r _dWith quantization parameter Q _StepBetween relational expression be r _d=k/Q _Step+ t, therefore can through the method for linear regression come match obtain parameter k be 0.9996 with t be 0.0040, also promptly for its encoder bit rate of left and right sides viewpoint r _dWith quantization parameter Q _StepBetween relational expression be r _d=0.9996/Q _Step+ 0.0040.For example, work as Q _Step=16 o'clock, r _d=0.0665bpp.

The distortion computation expression formula of S222, multiple view video coding is PSNR _d=q _d* QP+p _d, parameter q wherein _dAnd p _dNeed specifically be provided with according to the different depth graphic sequence.At this moment, under the condition of given multi-view depth graph code quantization parameter QP, can obtain multi-view depth graph code code check r _dWith distortion PSNR _dBetween corresponding relation.

Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because the distortion computation expression formula of multi-view depth graph code is PSNR _d=q _d* QP+p _d, come match to obtain parameter q through the method for linear regression _dBe-0.65 and p _dBe 63.75, also promptly for its decoding quality of left and right sides viewpoint PSNR _dAnd the relation between the coded quantization parameter QP can be expressed as PSNR _d=-0.65 * QP+63.75.When QP=28, PSNR _d=-0.65 * 28+63.75=45.55dB.

Next, the present invention need set up the relational model between virtual view distortion and multiple view video coding distortion and the depth map encoding distortion.In three-dimensional video-frequency; Virtual viewpoint rendering is the position relation that calculates corresponding pixel points between reference view and virtual view through the depth map of reference view, and brightness value and the chromatic value through corresponding pixel points in the video sequence of a plurality of reference views comes the brightness value and the chromatic value of weighted calculation virtual view corresponding pixel points then.Therefore, the distortion of virtual view is because multiple view video coding distortion and the distortion of multi-view depth graph code cause jointly.Therefore in the present invention; For the ease of setting up the functional relation between virtual view distortion and multi-view point video and the multi-view depth graph code parameter; To at first analyze under the undistorted condition of multi-view depth figure the relation between virtual view distortion and the multiple view video coding distortion.Secondly, analyze under the undistorted condition of multi-view point video the relation between virtual view distortion and the distortion of multi-view depth graph code.Set up the associating relational model of virtual view distortion and multi-view point video and the distortion of multi-view depth graph code at last.

Step S303 sets up the relation between multiple view video coding distortion and the virtual viewpoint rendering distortion.In this step, it is undistorted to establish multi-view depth figure, and this moment, the distortion of virtual view was only caused by the multiple view video coding distortion.Might as well establish this moment and generate virtual view C _vNeed M reference view

The virtual viewpoint rendering image is the weighted average of M reference view through the virtual visual point image of depth map drafting.Therefore, when i reference view The video coding distortion do The time, reference view

The video coding distortion for virtual viewpoint rendering image fault E _CWContribution do

Wherein,

Be the weight coefficient of i reference view video coding distortion for drawing virtual view image distortion contribution.Consider when video and depth map through reference view drawn the virtual view video; Phenomenon such as can occur blocking, therefore when calculating virtual view distortion

, also need the error that shield portions is corresponding deduct.Can establish through reference view

The ratio of pixel number of drawing the whole two field picture of number and reference view of the pixel that is blocked in the process of virtual view does

Pass through this moment

Draw the distortion that virtual view produced

Calculation expression should be modified to

Therefore, the present invention can be expressed as through the virtual visual point image distortion that M reference view weighting drafting obtains $E_{\mathrm{CW}}=\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{E}_{\mathrm{CW}}^{{\mathrm{Ref}}_i}.$

Particularly, as shown in Figure 4 in an embodiment of the present invention, for drawing the schematic diagram of virtual view.It is undistorted in this step, to establish multi-view depth figure, and this moment, the distortion of virtual view was only caused by the multiple view video coding distortion.The coding distortion of supposing left and right sides reference view video is respectively With

When the video coding parameter of left reference view was QP=28, the computational methods of the coding distortion of left reference view video did

The video coding distortion of left side reference view is for virtual viewpoint rendering image fault E _CWContribution do

In like manner, when the video coding parameter of right reference view is QP=28, the video coding distortion of right reference view The video coding distortion of right reference view is for virtual viewpoint rendering image fault E _CWContribution do $E_{\mathrm{CW}}^R=w_R{E}_C^R.$

Because virtual view is identical to the distance of left and right sides reference view, therefore, weight coefficient w _LAnd W _RSatisfy w _L=w _R=0.5.When considering the video of drawing virtual view through the video and the depth map of reference view; Phenomenon such as can occur blocking, therefore when calculating virtual view distortion

, also need the error that shield portions is corresponding deduct.Can know through experimental analysis; For " Breakdancer " sequence; Drawing the number of the pixel that is blocked in the process of virtual view and the average proportions of the pixel number of the whole two field picture of reference view through viewpoint 4 and viewpoint 6 is 0.15, and can be expressed as owing to the caused virtual visual point image distortion of the left and right sides reference view coding distortion this moment:

$E_{\mathrm{CW}}=E_{\mathrm{CW}}^L+E_{\mathrm{CW}}^R=0.15\×0.5\×(E_C^L+E_C^R)=0.15\×0.5\×(8.3+8.3)=1.25.$ In one embodiment of the invention, for example when the QP value is 28, E _CW=1.25.

Step S304 sets up the relation between depth map encoding distortion and the virtual viewpoint rendering distortion.In this step, suppose that multi-view point video is undistorted, this moment, the distortion of virtual view was only caused by the distortion of multi-view depth graph code.According to the principle of drawing virtual view image, virtual viewpoint rendering is through pixel in depth information and the camera parameter acquisition virtual view and the mapping relations between the respective pixel in the reference view.This process is equivalent to obtaining in the virtual view parallax between the respective pixel in each pixel and reference view.When distortion appearred in depth map, in the virtual viewpoint rendering process squinted in the position of respective pixel, also is that distortion appears in parallax.At this, can suppose that depth value is D _jThe distortion of depth map be Δ D _j, the distortion of parallax does so

Wherein, α is a constant relevant with camera parameter.

In order to weigh the influence that the parallax distortion brings the virtual view quality better; At first need the video frame images of reference view

be decomposed the corresponding predefined thresholding of the not poor mistake of depth value variance of pixel in each zone

that obtains after feasible the decomposition.In order to realize above-mentioned decomposition, in the present invention, can with

The corresponding depth map of video frame images carry out quaternary tree and decompose, make and decompose each area B of back _jInterior depth value variance is no more than predefined thresholding D _ThFor

Can obtain the average quantization distortion Δ D of this regional depth value according to given depth map encoding quantization step _jWith average depth D _jThen through Δ D _jAnd D _jObtain the average distortion of parallax || Δ d _j||.Obtaining || Δ d _j|| after and since on the virtual viewpoint rendering image that caused of parallax distortion with reference view in

The distortion in corresponding zone, zone is designated as

Then

Calculation expression do

Wherein,

Computing formula do ${\mathrm{\ψ}}_C^j=\frac{1}{2{\left(2\mathrm{\π}\right)}^2}({\∫\∫S}_C^j\left(\mathrm{\ω}\right){\mathrm{\ω}}_1^2\mathrm{D\ω}+{\∫\∫S}_C^j\left(\mathrm{\ω}\right){\mathrm{\ω}}_2^2\mathrm{D\ω}),$ On the reference view video frame images The fourier transform matrix that the matrix that the brightness value in zone or chromatic value are formed is corresponding, ω=[ω ₁, ω ₂] do

The angular frequency of zone level and vertical direction.Therefore, for reference view

The distortion that its depth map quantification is introduced With said step S303, virtual view C _vBe through M reference view

Weighting is drawn and is obtained.Through

Draw in the process of virtual view, the ratio of point that is not blocked and the whole number of pixels of whole two field picture does

In generating virtual visual point image,

Weight coefficient do

So

Corresponding depth map encoding distortion is to by the caused virtual view distortion of depth map encoding distortion E _DWContribution do

Also be $E_{\mathrm{DW}}=\underset{i=1}{\overset{M}{\mathrm{\Σ}}}{\mathrm{\η}}_{{\mathrm{Ref}}_i}\×w_{{\mathrm{Ref}}_i}\×E_{\mathrm{DW}}^{{\mathrm{Ref}}_i}.$

Particularly, the present invention supposes that multi-view point video is undistorted, and this moment, the distortion of virtual view was only caused by the distortion of multi-view depth graph code.When depth value is D _jThe distortion of depth map be Δ D _jThe time, the distortion of parallax does

Wherein, for the viewpoint 4 and the viewpoint 6 of " Breakdancer " sequence, the constant alpha relevant with camera parameter is 8.46 in the following formula.In order to weigh the influence that the parallax distortion brings the virtual view quality better; At first need the video frame images of left and right sides reference view be decomposed, the corresponding depth value variance of pixel in each zone that obtains after feasible the decomposition is no more than predefined thresholding.In order to realize above-mentioned decomposition, among the present invention, can carry out the quaternary tree decomposition by the depth map that the video frame images of left and right sides reference view is corresponding, make and decompose each zone, back

Interior depth value variance is no more than predefined thresholding D _Th=10, wherein, N _RBe the maximum region number after each depth map decomposition.As shown in Figure 5, be the quaternary tree decomposition result for depth map of the embodiment of the invention.

For

Can obtain the average quantization distortion Δ D of this regional depth value according to given depth map encoding quantization step _j=2 with average depth D _j=92.Then, through Δ D _jAnd D _jCan obtain the average distortion of parallax

It for size 8 * 8 zone

With its brightness is example, establishes its luminance matrix to be:

$\left[\begin{array}{cccccccc}62& 63& 62& 61& 63& 58& 62& 63\\ 61& 62& 62& 63& 61& 64& 65& 64\\ 62& 61& 57& 61& 63& 59& 63& 63\\ 67& 61& 58& 62& 61& 61& 66& 64\\ 63& 62& 62& 58& 60& 62& 63& 62\\ 61& 62& 61& 61& 61& 63& 60& 58\\ 62& 62& 61& 62& 62& 62& 60& 62\\ 64& 63& 61& 62& 62& 62& 60& 61\end{array}\right].$

Its Fourier transform of the corresponding matrix

in order to calculate the parallax distortion caused by the virtual viewpoint image rendering with

region corresponding to the region distortion also need to calculate

values .Because

Computing formula do ${\mathrm{\ψ}}_C^{L_j}=\frac{1}{2{\left(2\mathrm{\π}\right)}^2}({\∫\∫S}_C^{L_j}\left(\mathrm{\ω}\right){\mathrm{\ω}}_1^2\mathrm{D\ω}+{\∫\∫S}_C^{L_j}\left(\mathrm{\ω}\right){\mathrm{\ω}}_2^2\mathrm{D\ω}),$ And

The angular frequency of regions perpendicular and horizontal direction does

Therefore,

So, Value do

In like manner, also can obtain other zones and go up the drawing virtual view image distortion of being introduced owing to the depth map quantization error, that is, Equally, for each zone in the right reference view The zone can obtain this regional distortion through the aforementioned calculation method At this moment, the virtual view quantizing distortion that quantizes to be introduced owing to its depth map of right reference view does

S303 is the same with step, draws in the process of virtual view through viewpoint 4 and viewpoint 6, and the average proportions of point that is not blocked and the whole number of pixels of whole two field picture is 0.15, and the weight coefficient of left and right sides reference view is w _L=w _R=0.5, so the depth map encoding distortion of left and right sides reference view is to by the caused virtual view distortion of depth map encoding distortion E _DWCan be expressed as:

$E_{\mathrm{DW}}=0.15\×0.5\×(E_{\mathrm{DW}}^L+E_{\mathrm{DW}}^R)=0.15\×0.5\×(32.24+32.16)=4.83.$

Step S305, the associating above-mentioned relation is set up three-dimensional video-frequency virtual view encoding rate distortion model.In order to set up the rate-distortion model of virtual view, need encoder bit rate and the distortion model of multi-view point video that obtains in above-mentioned four steps and multi-view depth figure are merged, and finally obtain the rate-distortion model of virtual view coding.In this step,, need be divided into following two steps and realize in order to set up the encoding rate distortion model of virtual viewpoint rendering:

Step S501 at first need set up the associating distortion of virtual view and the relation between the coded quantization parameter.Because the multi-view point video distortion mainly causes pixel brightness and the colourity numerical value on the virtual visual point image to produce distortion; And multi-view depth figure distortion mainly causes the pixel position on the virtual visual point image to squint; Therefore caused distortion is separate respectively for virtual visual point image for multi-view point video distortion and multi-view depth figure distortion, also is virtual visual point image distortion E _TCan be expressed as the virtual viewpoint rendering distortion E that is caused by the multi-view point video distortion _CWThe virtual viewpoint rendering distortion E that distortion is caused with multi-view depth figure _DWAnd form.

Further; Through about in the video and process that depth map generates virtual view of two viewpoints; Because still possibly there is a spot of empty pixel in the discontinuity of depth map in virtual visual point image, mends to paint through image and fill up these empty caused errors and use E _ORepresent.Because image has certain continuity, can be through obtaining to be blocked pixel i with the pixel adjacent pixels point that is blocked _OBrightness or chromatic value distribute

This pixel owing to mend is painted the brightness that brought or the error of colourity can be expressed as Wherein y is pixel i _OBrightness or colourity maybe span, y _InBe pixel i _OBrightness when benefit is painted or colourity value.So, image is mended and to be painted the virtual visual point image error of being brought and can be expressed as

Wherein, i _OBe the picture element that is blocked,

Be the picture element i that is blocked _OBrightness or the chromatic value of neighbor pixel distribute, y is pixel i _OBrightness or the span of colourity, y _InBe the picture element i that is blocked _OBrightness when benefit is painted or chromatic value.At this moment, the error E of virtual viewpoint rendering image _TCalculation expression be E _T=E _CW+ E _DW+ E _O

Particularly, for the virtual view in this sample, E _O=0.13.Therefore, the error E of virtual viewpoint rendering image _TCalculation expression be E _T=E _CW+ E _DW+ E _O=1.25+4.83+0.13=6.21 also is that the virtual visual point image quality is 40.2dB.

Step S502 sets up the relation between needed encoder bit rate of virtual viewpoint rendering and the coded quantization parameter.Because in the three-dimensional video-frequency system of definition before this, multi-view point video and multi-view depth figure are encoded respectively by the multiple view video coding device, its coded quantization parameter is also independently chosen.Therefore, the required code check r of stereo scopic video coding _tBe multiple view video coding code check r _cWith multi-view depth graph code code check r _dSum also is r _t=r _c+ r _d

Particularly, in embodiments of the present invention,

When QP=28, r _t=r _c+ r _d=0.096bpp+0.0665bpp=0.1625bpp.

At last through adding up the distortion E of drawing virtual view image under the different Q P _TWith multi-view point video and the required total bitrate r of multi-view depth graph code _TCan obtain the rate-distortion model of drawing virtual view image in the stereo scopic video coding.

Through virtual viewpoint rendering encoding rate distortion analysis method in the three-dimensional video-frequency of the present invention's proposition; Can estimate virtual viewpoint rendering encoding rate distortion performance in the three-dimensional video-frequency quickly and accurately; Thereby instruct and solution for problems such as stereo scopic video coding parameters of choice and Data Rate Distribution provide model, further improved the efficient of stereo scopic video coding.

Although illustrated and described embodiments of the invention; For those of ordinary skill in the art; Be appreciated that under the situation that does not break away from principle of the present invention and spirit and can carry out multiple variation, modification, replacement and modification that scope of the present invention is accompanying claims and be equal to and limit to these embodiment.

Claims (10)

1. an estimation method of distortion performance of stereo video encoding rate is characterized in that, may further comprise the steps:

Obtain multi-view point video, and obtain corresponding multi-view depth figure according to said multi-view point video;

Obtain multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model respectively according to said multi-view point video and said multi-view depth figure;

Obtain the relation between said multiple view video coding distortion and the virtual viewpoint rendering distortion respectively according to said multi-view point video and said multi-view depth figure, and the relation between distortion of said multi-view depth graph code and the virtual viewpoint rendering distortion;

According to the relation between said multiple view video coding distortion and the virtual viewpoint rendering distortion, and the relation between distortion of said multi-view depth graph code and the virtual viewpoint rendering distortion obtains virtual viewpoint rendering distortion and multi-view point video and the multi-view depth relation between the coded quantization parameter QP separately;

Obtain to draw the relation between needed encoder bit rate of said virtual view and the said QP according to said multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model; The drafting distortion of said virtual view and the needed encoder bit rate of the said virtual view of drafting are to obtain the rate-distortion model of virtual view in the stereo scopic video coding under the statistics different Q P.

2. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, wherein, multiple view video coding adopts and realized originally based on video encoding standard multiple view video coding extended edition H.264/AVC.

3. estimation method of distortion performance of stereo video encoding rate as claimed in claim 2; It is characterized in that intracoded frame, forward-predictive-coded frames and the bi-directional predictive coding frame of each image sets in each viewpoint all adopt identical quantization parameter QP to encode.

4. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, wherein, and under the condition of given multiple view video coding quantization parameter:

The method of estimation of the code check of multiple view video coding does

Wherein, Q _Step=2 ^(QP-4)/6, a is 24.37, b be-0.303 and c be 0.02;

The aberration estimation method of multiple view video coding is PSNR _c=q _c* QP+p _c, wherein, q _cBe-0.37, and p _cBe 49.30.

5. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, wherein, and under the condition of given multi-view depth graph code quantization parameter:

The method of estimation of the code check of multi-view depth graph code is r _d=k/Q _Step+ t, wherein, Q _Step=2 ^(QP-4)/6, k be 0.9996 and t be 0.0040;

The distortion computation expression formula of multiple view video coding is PSNR _d=q _d* QP+p _d, wherein, q _dBe-0.65, and p _aBe 63.75.

6. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, the relation between distortion of said acquisition multiple view video coding and the virtual viewpoint rendering distortion further comprises:

Suppose that the multi-view depth graph code is undistorted, the distortion of said virtual view is only caused by said multiple view video coding distortion, and selects to draw virtual view C _vA required M reference view, then the virtual visual point image distortion does

Wherein,

Wherein,

Be the ratio of the pixel number of the number of the pixel that is blocked and the whole two field picture of reference view,

Be the weight coefficient of i reference view video coding distortion for drawing virtual view image distortion contribution, Be i reference view

The video coding distortion.

7. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, the relation between said acquisition multi-view depth graph code distortion and the virtual viewpoint rendering distortion further comprises:

Suppose that multiple view video coding is undistorted, the distortion of said virtual view is only caused by the distortion of said multi-view depth graph code, and selects virtual view C _vA required M reference view, then the virtual visual point image distortion does

Wherein, Be the ratio of the pixel number of the number of the pixel that is blocked and the whole two field picture of reference view,

Be the weight coefficient of i reference view video coding distortion for drawing virtual view image distortion contribution,

For on the virtual viewpoint rendering image with reference view in

The distortion in corresponding zone, zone.

8. estimation method of distortion performance of stereo video encoding rate as claimed in claim 7 is characterized in that, wherein,

Wherein, Computing formula do ${\mathrm{\ψ}}_C^j=\frac{1}{2{\left(2\mathrm{\π}\right)}^2}({\∫\∫S}_C^j\left(\mathrm{\ω}\right){\mathrm{\ω}}_1^2\mathrm{D\ω}+{\∫\∫S}_C^j\left(\mathrm{\ω}\right){\mathrm{\ω}}_2^2\mathrm{D\ω}),$

On the reference view frame of video

The matrix that brightness value or the chromatic value in zone formed carries out the matrix that obtains behind the Fourier transform, ω=[ω ₁, ω ₂] do

The angular frequency of zone level and vertical direction, || Δ d _j|| be the average distortion of parallax.

9. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1; It is characterized in that; Said according to the relation between multiple view video coding distortion and the virtual viewpoint rendering distortion, and the relation between distortion of said multi-view depth graph code and the virtual viewpoint rendering distortion obtains the distortion of virtual view and the relation between the coded quantization parameter QP further comprises:

The error E of virtual viewpoint rendering image _TBe E _T=E _CW+ E _DW+ E _O, wherein, E _OPaint the caused error of filling cavity for mending through image, $E_O=\underset{i_o}{\mathrm{\Σ}}{e}_{i_o}^o=\underset{i_o}{\mathrm{\Σ}}\underset{y\≠y_{\mathrm{In}}}{\mathrm{\Σ}}{P}_{i_o}^o\left(y\right)\×\left({y-y}_{\mathrm{In}}\right),$

Wherein, i _OBe the said picture element that is blocked,

Be the said picture element i that is blocked _OThe brightness of adjacent pixels point or chromatic value distribute, and y is said pixel i _OBrightness or the span of colourity, y _InBe the said picture element i that is blocked _OBrightness when benefit is painted or chromatic value.

10. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1; It is characterized in that, further comprise according to the relation between said multiple view video coding rate-distortion model and the multi-view depth graph code rate-distortion model acquisition drafting needed encoder bit rate of said virtual view and multiple view video coding code check and the multi-view depth graph code code check:

The required code check of stereo scopic video coding is r _t=r _c+ r _d, wherein, r _cBe multiple view video coding code check, r _dBe multi-view depth graph code code check.

Images