CN101888566A - Estimation method of distortion performance of stereo video encoding rate - Google Patents
Estimation method of distortion performance of stereo video encoding rate Download PDFInfo
- Publication number
- CN101888566A CN101888566A CN 201010222351 CN201010222351A CN101888566A CN 101888566 A CN101888566 A CN 101888566A CN 201010222351 CN201010222351 CN 201010222351 CN 201010222351 A CN201010222351 A CN 201010222351A CN 101888566 A CN101888566 A CN 101888566A
- Authority
- CN
- China
- Prior art keywords
- distortion
- view
- video
- virtual
- video coding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention provides an estimation method of distortion performance of a stereo video encoding rate, comprising the following steps: acquiring a multi-view video and acquiring a corresponding multi-view depth map according to the multi-view video; respectively obtaining a multi-view video encoding rate distortion model and a multi-view depth map encoding rate distortion model; respectively obtaining the relationship between multi-view video encoding distortion and virtual view rendering distortion as well as the relationship between the multi-view depth map encoding distortion and the virtual view rendering distortion; and finally establishing a virtual view encoding rendering rate distortion model in a stereo video based on the relationships. By the analysis method of virtual view encoding rendering rate distortion in the stereo video provided by the invention, the distortion performance of the virtual view encoding rendering rate in the stereo video can be accurately and rapidly established so as to provide model guide and solution for problems such as selection of coding parameters, code rate distribution and the like in the stereo video.
Description
Technical field
The present invention relates to technical field of image processing, particularly a kind of estimation method of distortion performance of stereo video encoding rate at virtual viewpoint rendering.
Background technology
Along with the continuous development of multimedia technology, traditional two-dimensional video medium can not satisfy the needs of human vision, and people wish to see and have more the sense of reality and interactive video frequency program, the 3 D video that also promptly has highly three-dimensional feeling of immersion.The 3 D video technology of rising gradually becomes the important part of multimedia technology just gradually as brand-new video medias such as multi-view point video, free viewpoint video and three-dimensional video-frequencies in recent years.So-called free viewpoint video is meant that the beholder is by freely selecting to watch viewpoint to watch the video media of three-dimensional scenic.
Two Main Stage have been experienced in the free viewpoint video development, and the phase I is known as multi-view point video, and promptly spectators can the viewpoint of selection oneself needs watch from the multi-view point video of playing, and form third dimension by the parallax between the different points of view video.Second stage is called as three-dimensional video-frequency, and promptly the beholder can freely select to watch viewpoint to watch to have relief video.In order to obtain multi-view point video, just need to adopt the multichannel video camera to gather Same Scene, and the multi-channel video that collects is sent to client, and select corresponding multi-view point video to play according to spectators' needs.And in order to obtain to have the three-dimensional video-frequency of free view-point, need on the basis of multi-view point video, further introduce the geological information of scene, play up by multi-view point video data and geological information and to obtain the virtual view that spectators select at random, thereby allow spectators obtain the stronger third dimension of watching.
Be different from traditional single channel two-dimensional video data, three-dimensional video-frequency is made up of multichannel two-dimensional video data and scene geometric information, so the mass data of multi-view point video is far longer than traditional two-dimensional video data to the demand of transmission bandwidth.In addition, how to compress scene geometric information effectively and also become the right new challenge of stereo scopic video coding demand side.Therefore, in order to realize effective transmission of stereoscopic video, need design coding techniques efficiently at the three-dimensional video-frequency characteristic.In early days, the researcher is applied to traditional video compression technology multi-view video compressed, adopts traditional single channel video compression technology that the video of each the viewpoint correspondence in the multi-view point video is compressed separately.This scheme becomes one of three-dimensional video-frequency compression solution the earliest.
Yet, there is stronger correlation between the multi-view point video sequence, even adopt the most H.264/AVC coding techniques, still can't compress the redundancy between the different points of view effectively.For this reason, the researcher has designed multi-view video compressed more efficiently scheme.This scheme is extended to traditional single channel video coding technique on the multi-channel video, in the predictive coding that adds on the basis of time domain prediction coding between viewpoint, has further compressed the redundancy between the multi-view point video viewpoint, has improved multi-view video compressed distortion performance.In addition, at three-dimensional video-frequency scene geometric information based on depth map, i.e. multi-view depth graphic sequence, the researcher adopts the multiple view video coding scheme to realize its efficient compression equally.Along with the continuous development of three-dimensional video-frequency technology, new three-dimensional video-frequency compress technique is development fast.At present, one of the key technology in market is moved towards in the application that become three-dimensional video-frequency of the stereo scopic video coding technology of real-time high-efficiency.
On the basis of three-dimensional video-frequency high efficient coding scheme,, need adjust coding parameter according to network condition in order to realize effective transmission of stereoscopic video.In traditional video coding, under the situation of known network bandwidth, can estimate decoding quality according to the rate-distortion model of coding, thereby adjust coding parameter better.Therefore, the rate-distortion model analysis at three-dimensional video-frequency high efficient coding scheme is to realize the effectively important step of transmission of three-dimensional video-frequency.
The shortcoming that prior art exists is also not have the estimation scheme at the stereo video encoding rate distortion performance of virtual viewpoint rendering at present.
For example, at application number is 200810163801, name is called in a kind of patent application of method for encoding stereo video of network self-adapting, has only provided the different network bandwidth adaptive coding scheme down of three-dimensional video-frequency, analyzing at the distortion performance of stereo scopic video coding.At application number is 200810126528, name is called in the patent application of a kind of stereo video coding-decoding method, Apparatus and system, though disclose a whole set of complete stereo scopic video coding scheme, device and system, do not provided the analysis of stereo video encoding rate distortion performance equally.At application number is 200710164747, name is called in a kind of patent application of the bit rate control method towards multi-view point video, though announced the rate-distortion model of multiple view video coding, do not had stereo video encoding rate distortion performance at virtual viewpoint rendering.
Summary of the invention
Purpose of the present invention is intended to solve above-mentioned technological deficiency at least, has proposed a kind of estimation method of distortion performance of stereo video encoding rate at virtual viewpoint rendering.
For achieving the above object, one aspect of the present invention has proposed a kind of estimation method of distortion performance of stereo video encoding rate, may further comprise the steps: obtain multi-view point video, and obtain corresponding multi-view depth figure according to described multi-view point video; Obtain multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model respectively according to described multi-view point video and described multi-view depth figure; Obtain relation between described multiple view video coding distortion and the virtual viewpoint rendering distortion and the relation between distortion of described multi-view depth graph code and the virtual viewpoint rendering distortion respectively according to described multi-view point video and described multi-view depth figure; According to the relation between described multiple view video coding distortion and the virtual viewpoint rendering distortion, and the relation between distortion of described multi-view depth graph code and the virtual viewpoint rendering distortion obtains the relation between virtual viewpoint rendering distortion and multi-view point video and the multi-view depth figure coded quantization parameter QP separately; According to the relation between described multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model acquisition drafting needed encoder bit rate of described virtual view and the described QP; And add up the drafting distortion of described virtual view under the different Q P and draw the needed encoder bit rate of described virtual view to obtain the rate-distortion model of virtual view in the stereo scopic video coding.
By virtual viewpoint rendering encoding rate distortion analysis method in the three-dimensional video-frequency of the present invention's proposition, can estimate virtual viewpoint rendering encoding rate distortion performance in the three-dimensional video-frequency quickly and accurately, thereby instruct and solution for problems such as the selection of stereo scopic video coding parameter and Data Rate Distribution provide model, further improved the efficient of stereo scopic video coding.
Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize by practice of the present invention.
Description of drawings
Above-mentioned and/or additional aspect of the present invention and advantage are from obviously and easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:
The three-dimensional video-frequency system block diagram that Fig. 1 provides for the embodiment of the invention;
The multiple view video coding predict figure that Fig. 2 provides for the embodiment of the invention;
Fig. 3 is the estimation method of distortion performance of stereo video encoding rate flow chart of the embodiment of the invention;
The virtual viewpoint rendering schematic diagram that Fig. 4 provides for the embodiment of the invention;
The depth map quaternary tree decomposing schematic representation that Fig. 5 provides for the embodiment of the invention.
Embodiment
Describe embodiments of the invention below in detail, the example of described embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Below by the embodiment that is described with reference to the drawings is exemplary, only is used to explain the present invention, and can not be interpreted as limitation of the present invention.
Because in the three-dimensional video-frequency, virtual view is to draw by multi-view point video and scene geometric information, so the virtual viewpoint rendering quality is relevant with the coding quality of multi-view point video and scene geometric information.For this reason, the present invention sets up multi-view point video and scene geometric information coding associating rate-distortion model by needs and obtains encoding rate distortion model at the virtual viewpoint rendering video, thereby instructs problems such as the selection of stereo scopic video coding parameter and Data Rate Distribution better.
The present invention has mainly proposed a kind of multi-view point video and depth map encoding associating rate-distortion model method of estimation of practicality.This method can be estimated the relation between multi-view point video and depth map encoding quality and the virtual viewpoint rendering quality effectively, thereby selects and problems such as Data Rate Distribution provide theoretical direction for stereo scopic video coding parameter under the condition of the given network bandwidth.
The applied environment of the embodiment of the invention is as follows: the three-dimensional video-frequency system block diagram that the embodiment of the invention is used as shown in Figure 1.Wherein, the video sequence that is used for stereo scopic video coding adopts the standard testing video sequence of the name of high-definition format for " Breakdancer "; The pixel of this high-definition format video sequence is 1024 * 768; Decoder adopts the H.264/SVC reference software JMVC (JointMulti-view Video Coding, multiple view video coding) of (Multi-view Video Coding, multi-view point video extended version) standard; The frame number of encoder GOP (Group of Pictures, image sets) is 8; The time domain prediction coding of coding adopts the Hierarchical B predict of (stratification bi-directional predictive coding frame is called for short stratification B frame), and the coded prediction structure chart as shown in Figure 2.
Server end obtains multi-view point video, and obtains multi-view depth figure according to multi-view point video, and in stereo scopic video coding to multi-view point video with multi-view depth figure encodes and give client by Network Transmission.Client is decoded to the multi-view point video and the multi-view depth graph code code stream that receive, and the virtual view of selecting according to spectators generates corresponding virtual view video and it is shown to spectators by three-dimensional display.
In this enforcement sample, virtual viewpoint rendering adopts two-path video and the depth map adjacent with virtual view to draw.Concrete, the viewpoint 4 of this enforcement sample employing " Breakdancer " sequence and viewpoint 6 these two-path videos are as the multi-view point video list entries.Wherein viewpoint 4 is called left reference view, and viewpoint 6 is called right reference view.The span of multi-view point video and multi-view depth graph code quantization parameter QP is the integer between 0 to 51.The parameter of the virtual view that generates is identical with the parameter of viewpoint 5 in " Breakdancer " sequence.
As shown in Figure 3, be the estimation method of distortion performance of stereo video encoding rate flow chart of the embodiment of the invention, in this embodiment, multiple view video coding adopts and realized originally based on conventional video coding standard multiple view video coding extended edition H.264/AVC.Intracoded frame (I frame), forward-predictive-coded frames (P frame) and the bi-directional predictive coding frame (B frame) of each image sets in each viewpoint all adopt identical quantization parameter (QP) to encode.This method may further comprise the steps:
Step S301 sets up the multiple view video coding rate-distortion model.Under the situation of given multiple view video coding quantization parameter, estimate multiple view video coding code check and distortion.After the value of given coded quantization parameter QP, at first pass through Q
Step=2
(QP-4)/6Obtain quantization parameter Q
StepValue.For example, when QP=28, Q
Step=2
(28-4)/6=16.
This step can further be subdivided into following two steps:
Step S101, the code check r of multiple view video coding
cCalculation expression be
Wherein, parameter Q
StepCalculation expression be Q
Step=2
(QP-4)/6, parameter a, b, c need specifically be provided with according to the different video sequence.Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because encoder bit rate r
cWith quantization parameter Q
StepBetween relational expression be
Therefore can come match to obtain parameter a by the method for linear regression be 24.37, b for-0.303 and c be 0.02, also promptly for its encoder bit rate of left and right sides viewpoint r
cWith quantization parameter Q
StepBetween relational expression be:
For example, work as Q
Step=16 o'clock, r
c=0.096bpp.
Step S102, the distortion computation expression formula of multiple view video coding is PSNR
c=q
c* QP+p
c, parameter q wherein
cAnd p
cNeed specifically be provided with according to the different video sequence.At this moment, under the condition of given multiple view video coding quantization parameter QP, can obtain multiple view video coding code check r
cWith distortion PSNR
cBetween corresponding relation.
Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because the distortion computation expression formula of multiple view video coding is PSNR
c=q
c* QP+p
c, come match to obtain parameter q by the method for linear regression
cBe-0.37 and p
cBe 49.30, also promptly for its decoding quality of left and right sides viewpoint PSNR
cAnd the relation between the coded quantization parameter QP can be expressed as PSNR
c=-0.37 * QP+49.30.When QP=28, PSNR
c=-0.37 * 28+49.30=38.94dB.
Step S302 sets up the depth map encoding rate-distortion model.Under the situation of given multi-view depth graph code quantization parameter, estimate the corresponding relation between multi-view point video depth map encoding code check and the distortion.In this step, multi-view depth figure also adopts with multiple view video coding described in the step S301 and encodes.Similarly, I frame, P frame and the B frame of each image sets in each viewpoint also all adopt identical QP to encode.With above-mentioned step, Q when the value of coded quantization parameter QP is 28
Step=16.
This step can further be subdivided into following two steps:
The code check r of S221, multi-view depth graph code
dCalculation expression be r
d=k/Q
Step+ t, wherein parameter Q
StepCalculation expression be Q
Step=2
(QP-4)/6, parameter k, t need specifically be provided with according to the different depth graphic sequence.
Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because encoder bit rate r
dWith quantization parameter Q
StepBetween relational expression be r
d=k/Q
Step+ t, therefore can by the method for linear regression come match obtain parameter k be 0.9996 and t be 0.0040, also promptly for its encoder bit rate of left and right sides viewpoint r
cWith quantization parameter Q
StepBetween relational expression be r
d=0.9996/Q
Step+ 0.0040.For example, work as Q
Step=16 o'clock, r
c=0.0665bpp.
The distortion computation expression formula of S222, multiple view video coding is PSNR
d=q
d* QP+p
d, parameter q wherein
dAnd p
dNeed specifically be provided with according to the different depth graphic sequence.At this moment, under the condition of given multi-view depth graph code quantization parameter QP, can obtain multi-view depth graph code code check r
dWith distortion PSNR
dBetween corresponding relation.
Particularly, in an embodiment of the present invention, for " Breakdancer " sequence, because the distortion computation expression formula of multi-view depth graph code is PSNR
d=q
d* QP+p
d, come match to obtain parameter q by the method for linear regression
dBe-0.65 and p
dBe 63.75, also promptly for its decoding quality of left and right sides viewpoint PSNR
dAnd the relation between the coded quantization parameter QP can be expressed as PSNR
d=-0.65 * QP+63.75.When QP=28, PSNR
d=-0.65 * 28+63.75=45.55dB.
Next, the present invention need set up the relational model between virtual view distortion and multiple view video coding distortion and the depth map encoding distortion.In three-dimensional video-frequency, virtual viewpoint rendering is the position relation that calculates corresponding pixel points between reference view and virtual view by the depth map of reference view, and brightness value and the chromatic value by corresponding pixel points in the video sequence of a plurality of reference views comes the brightness value and the chromatic value of weighted calculation virtual view corresponding pixel points then.Therefore, the distortion of virtual view is because multiple view video coding distortion and the distortion of multi-view depth graph code cause jointly.Therefore in the present invention, for the ease of setting up the functional relation between virtual view distortion and multi-view point video and the multi-view depth graph code parameter, to at first analyze under the undistorted condition of multi-view depth figure the relation between virtual view distortion and the multiple view video coding distortion.Secondly, analyze under the undistorted condition of multi-view point video the relation between virtual view distortion and the distortion of multi-view depth graph code.Set up the associating relational model of virtual view distortion and multi-view point video and the distortion of multi-view depth graph code at last.
Step S303 sets up the relation between multiple view video coding distortion and the virtual viewpoint rendering distortion.In this step, it is undistorted to establish multi-view depth figure, and this moment, the distortion of virtual view was only caused by the multiple view video coding distortion.Might as well establish this moment and generate virtual view C
VNeed M reference view
The virtual viewpoint rendering image is the weighted average of M reference view by the virtual visual point image of depth map drafting.Therefore, when i reference view
The video coding distortion be
The time, reference view
The video coding distortion for virtual viewpoint rendering image fault E
CWContribution be
Wherein,
Be the weight coefficient of i reference view video coding distortion for drawing virtual view image distortion contribution.Consider when video by reference view and depth map are drawn the virtual view video, phenomenon such as can occur blocking, therefore calculating the virtual view distortion
In time, also need the error of shield portions correspondence is deducted.Can establish and pass through reference view
The ratio of drawing the pixel number of the whole two field picture of the number of the pixel that is blocked in the process of virtual view and reference view is
Pass through this moment
Draw the distortion that virtual view produced
Calculation expression should be modified to
Therefore, the present invention can be expressed as by the virtual visual point image distortion that M reference view weighting drafting obtains
Particularly, in an embodiment of the present invention, as shown in Figure 4, for drawing the schematic diagram of virtual view.It is undistorted to establish multi-view depth figure in this step, and this moment, the distortion of virtual view was only caused by the multiple view video coding distortion.The coding distortion of supposing left and right sides reference view video is respectively
With
When the video coding parameter of left reference view was QP=28, the computational methods of the coding distortion of left reference view video were
The video coding distortion of left side reference view is for virtual viewpoint rendering image fault E
CWContribution be
In like manner, when the video coding parameter of right reference view is QP=28, the video coding distortion of right reference view
The video coding distortion of right reference view is for virtual viewpoint rendering image fault E
CWContribution be
Because virtual view is identical to the distance of left and right sides reference view, therefore, weight coefficient w
LAnd w
RSatisfy w
L=w
R=0.5.When considering the video of drawing virtual view by the video and the depth map of reference view, phenomenon such as can occur blocking, therefore calculate the virtual view distortion
In time, also need the error of shield portions correspondence is deducted.Analyze as can be known by experiment, for " Breakdancer " sequence, the average proportions of drawing the pixel number of the number of the pixel that is blocked in the process of virtual view and the whole two field picture of reference view by viewpoint 4 and viewpoint 6 is 0.15, this moment since the caused virtual visual point image distortion of left and right sides reference view coding distortion can be expressed as:
Step S304 sets up the relation between depth map encoding distortion and the virtual viewpoint rendering distortion.In this step, suppose that multi-view point video is undistorted, this moment, the distortion of virtual view was only caused by the distortion of multi-view depth graph code.According to the principle of drawing virtual view image, virtual viewpoint rendering is by pixel in depth information and the camera parameter acquisition virtual view and the mapping relations between the respective pixel in the reference view.This process is equivalent to and will obtains in the virtual view parallax between the respective pixel in each pixel and reference view.When distortion appearred in depth map, the position of respective pixel was offset in the virtual viewpoint rendering process, also was that distortion appears in parallax.At this, can suppose that depth value is D
jThe distortion of depth map be Δ D
j, the distortion of parallax is so
Wherein, α is a constant relevant with camera parameter.
In order to weigh the influence that the parallax distortion brings the virtual view quality better, at first need reference view
Video frame images decompose each zone that obtains after make decomposing
In the predefined thresholding of the not poor mistake of depth value variance of pixel correspondence.In order to realize above-mentioned decomposition, in the present invention, can with
The depth map of video frame images correspondence carry out quaternary tree and decompose, make and decompose each area B of back
jInterior depth value variance is no more than predefined thresholding D
ThFor
Can obtain the average quantization distortion Δ D of this regional depth value according to given depth map encoding quantization step
jWith average depth D
jThen by Δ D
jAnd D
jObtain the average distortion of parallax || Δ d
j||.Obtaining || Δ d
j|| after and since on the virtual viewpoint rendering image that caused of parallax distortion with reference view in
The distortion in corresponding zone, zone is designated as
Then
Calculation expression be
Wherein,
Computing formula be
On the reference view video frame images
The fourier transform matrix of the matrix correspondence that the brightness value in zone or chromatic value are formed, ω=[ω
1, ω
2] be
The angular frequency of zone level and vertical direction.Therefore, for reference view
The distortion that its depth map quantification is introduced
With described step S303, virtual view C
VBe by M reference view
Weighting is drawn and is obtained.By
Draw in the process of virtual view, the ratio of point that is not blocked and the whole number of pixels of whole two field picture is
In generating virtual visual point image,
Weight coefficient be
So
Corresponding depth map encoding distortion is to by the caused virtual view distortion of depth map encoding distortion E
DWContribution be
Also be
Particularly, the present invention supposes that multi-view point video is undistorted, and this moment, the distortion of virtual view was only caused by the distortion of multi-view depth graph code.When depth value is D
jThe distortion of depth map be Δ D
jThe time, the distortion of parallax is
Wherein, for the viewpoint 4 and the viewpoint 6 of " Breakdancer " sequence, the constant alpha relevant with camera parameter is 8.46 in the following formula.In order to weigh the influence that the parallax distortion brings the virtual view quality better, at first need the video frame images of left and right sides reference view is decomposed, the depth value variance of the pixel correspondence in each zone that obtains after feasible the decomposition is no more than predefined thresholding.In order to realize above-mentioned decomposition, among the present invention, the depth map of the video frame images correspondence of left and right sides reference view can be carried out quaternary tree and decompose, make and decompose each zone, back
Interior depth value variance is no more than predefined thresholding D
Th=10, wherein, N
RBe the maximum region number after each depth map decomposition.As shown in Figure 5, be the quaternary tree decomposition result for depth map of the embodiment of the invention.
For
Can obtain the average quantization distortion Δ D of this regional depth value according to given depth map encoding quantization step
j=2 and average depth D
j=92.Then, by Δ D
jAnd D
jCan obtain the average distortion of parallax
It for size 8 * 8 zone
With its brightness is example, establishes its luminance matrix to be:
The pairing matrix of its Fourier transform is
For calculate on the virtual viewpoint rendering image that the parallax distortion caused with
The distortion in corresponding zone, zone
Also need to calculate
Value.Because
Computing formula be
And
The angular frequency of regions perpendicular and horizontal direction is
Therefore,
So,
Value be
In like manner, also can obtain other zones and go up the drawing virtual view image distortion of being introduced owing to the depth map quantization error, that is,
Equally, for each zone in the right reference view
The zone can obtain this regional distortion by the aforementioned calculation method
At this moment, the virtual view quantizing distortion that quantizes to be introduced owing to its depth map of right reference view is
S303 is the same with step, draws in the process of virtual view by viewpoint 4 and viewpoint 6, and the average proportions of point that is not blocked and the whole number of pixels of whole two field picture is 0.15, and the weight coefficient of left and right sides reference view is w
L=w
R=0.5, so the depth map encoding distortion of left and right sides reference view is to by the caused virtual view distortion of depth map encoding distortion E
DWCan be expressed as:
Step S305, the associating above-mentioned relation is set up three-dimensional video-frequency virtual view encoding rate distortion model.In order to set up the rate-distortion model of virtual view, multi-view point video that need obtain in above-mentioned four steps and the encoder bit rate of multi-view depth figure and distortion model merge, and finally obtain the rate-distortion model of virtual view coding.In this step,, need be divided into following two steps and realize in order to set up the encoding rate distortion model of virtual viewpoint rendering:
Step S501 at first needs to set up the associating distortion of virtual view and the relation between the coded quantization parameter.Because the multi-view point video distortion mainly causes pixel brightness and the colourity numerical value on the virtual visual point image to produce distortion, and multi-view depth figure distortion mainly causes the pixel position on the virtual visual point image to be offset, therefore caused distortion is separate respectively for virtual visual point image for multi-view point video distortion and multi-view depth figure distortion, also is virtual visual point image distortion E
TCan be expressed as the virtual viewpoint rendering distortion E that is caused by the multi-view point video distortion
CWThe virtual viewpoint rendering distortion E that distortion is caused with multi-view depth figure
DWAnd form.
Further, by about in the video and process that depth map generates virtual view of two viewpoints, still may there be a spot of empty pixel in discontinuity owing to depth map in virtual visual point image, mend to paint by image and fill up these empty caused error E
ORepresent.Because image has certain continuity, can be by obtaining to be blocked pixel i with the pixel adjacent pixels point that is blocked
OBrightness or chromatic value distribute
This pixel owing to mend is painted the brightness that brought or the error of colourity can be expressed as
Wherein y is pixel i
OBrightness or colourity may span, y
InBe pixel i
OBrightness when benefit is painted or colourity value.So, image is mended and to be painted the virtual visual point image error of being brought and can be expressed as
Wherein, i
OBe the picture element that is blocked,
Be the picture element i that is blocked
OThe brightness of neighbor pixel or chromatic value distribute, y is pixel i
OBrightness or the span of colourity, y
InBe the picture element i that is blocked
OBrightness when benefit is painted or chromatic value.At this moment, the error E of virtual viewpoint rendering image
TCalculation expression be E
T=E
CW+ E
DW+ E
O
Particularly, for the virtual view in this sample, E
O=0.13.Therefore, the error E of virtual viewpoint rendering image
TCalculation expression be E
T=E
CW+ E
DW+ E
O=1.25+4.83+0.13=6.21 also is that the virtual visual point image quality is 40.2dB.
Step S502 sets up the relation between needed encoder bit rate of virtual viewpoint rendering and the coded quantization parameter.Because in the three-dimensional video-frequency system of definition before this, multi-view point video and multi-view depth figure are encoded respectively by the multiple view video coding device, its coded quantization parameter is also independently chosen.Therefore, the required code check r of stereo scopic video coding
TBe multiple view video coding code check r
CWith multi-view depth graph code code check r
DSum also is r
T=r
C+ r
D
Particularly, in embodiments of the present invention,
When QP=28, r
T=r
C+ r
D=0.0625+0.096=0.1625bpp.
At last by adding up the distortion E of drawing virtual view image under the different Q P
TWith multi-view point video and the required total bitrate r of multi-view depth graph code
TCan obtain the rate-distortion model of drawing virtual view image in the stereo scopic video coding.
By virtual viewpoint rendering encoding rate distortion analysis method in the three-dimensional video-frequency of the present invention's proposition, can estimate virtual viewpoint rendering encoding rate distortion performance in the three-dimensional video-frequency quickly and accurately, thereby instruct and solution for problems such as the selection of stereo scopic video coding parameter and Data Rate Distribution provide model, further improved the efficient of stereo scopic video coding.
Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, modification, replacement and modification that scope of the present invention is by claims and be equal to and limit to these embodiment.
Claims (10)
1. an estimation method of distortion performance of stereo video encoding rate is characterized in that, may further comprise the steps:
Obtain multi-view point video, and obtain corresponding multi-view depth figure according to described multi-view point video;
Obtain multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model respectively according to described multi-view point video and described multi-view depth figure;
Obtain relation between described multiple view video coding distortion and the virtual viewpoint rendering distortion and the relation between distortion of described multi-view depth graph code and the virtual viewpoint rendering distortion respectively according to described multi-view point video and described multi-view depth figure;
According to the relation between described multiple view video coding distortion and the virtual viewpoint rendering distortion, and the relation between distortion of described multi-view depth graph code and the virtual viewpoint rendering distortion obtains virtual viewpoint rendering distortion and multi-view point video and the multi-view depth relation between the coded quantization parameter QP separately;
According to the relation between described multiple view video coding rate-distortion model and multi-view depth graph code rate-distortion model acquisition drafting needed encoder bit rate of described virtual view and the described QP; The drafting distortion of described virtual view and the needed encoder bit rate of the described virtual view of drafting are to obtain the rate-distortion model of virtual view in the stereo scopic video coding under the statistics different Q P.
2. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, wherein, multiple view video coding adopts and realized originally based on video encoding standard multiple view video coding extended edition H.264/AVC.
3. estimation method of distortion performance of stereo video encoding rate as claimed in claim 2, it is characterized in that intracoded frame, forward-predictive-coded frames and the bi-directional predictive coding frame of each image sets in each viewpoint all adopt identical quantization parameter QP to encode.
4. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, wherein, and under the condition of given multiple view video coding quantization parameter:
The method of estimation of the code check of multiple view video coding is
Wherein, Q
Step=2
(QP-4)/6, a, b, c are according to the concrete parameter that is provided with of different video sequence;
The aberration estimation method of multiple view video coding is PSNR
c=q
c* QP+p
c, wherein, q
cAnd p
cBe parameter according to the concrete setting of different video sequence.
5. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, wherein, and under the condition of given multi-view depth graph code quantization parameter:
The method of estimation of the code check of multi-view depth graph code is r
d=k/Q
Step+ t, wherein, Q
Step=2
(QP-4)/6, k, t are according to the concrete parameter that is provided with of different depth graphic sequence;
The aberration estimation method of multiple view video coding is PSNR
d=q
d* QP+p
d, wherein, q
dAnd p
dBe parameter according to the concrete setting of different depth graphic sequence.
6. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, the relation between distortion of described acquisition multiple view video coding and the virtual viewpoint rendering distortion further comprises:
Suppose that the multi-view depth graph code is undistorted, the distortion of described virtual view is only caused by described multiple view video coding distortion, and selects to draw virtual view C
vA required M reference view, then the virtual visual point image distortion is
Wherein,
Wherein,
Be the ratio of the pixel number of the number of the pixel that is blocked and the whole two field picture of reference view,
Be the weight coefficient of i reference view video coding distortion for drawing virtual view image distortion contribution,
Be i reference view
The video coding distortion.
7. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1 is characterized in that, the relation between described acquisition multi-view depth graph code distortion and the virtual viewpoint rendering distortion further comprises:
Suppose that multiple view video coding is undistorted, the distortion of described virtual view is only caused by the distortion of described multi-view depth graph code, and selects virtual view C
vA required M reference view, then the virtual visual point image distortion is
Wherein,
Be the ratio of the pixel number of the number of the pixel that is blocked and the whole two field picture of reference view,
Be the weight coefficient of i reference view video coding distortion for drawing virtual view image distortion contribution,
For on the virtual viewpoint rendering image with reference view in
The distortion in corresponding zone, zone.
8. estimation method of distortion performance of stereo video encoding rate as claimed in claim 7 is characterized in that, wherein,
Wherein,
Computing formula be
On the reference view frame of video
The brightness value in zone or the matrix formed of chromatic value carry out the matrix that obtains behind the Fourier transform, ω=[ω
1, ω
2] be
The angular frequency of zone level and vertical direction, || Δ d
j|| be the average distortion of parallax.
9. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1, it is characterized in that, described according to the relation between multiple view video coding distortion and the virtual viewpoint rendering distortion, and the relation between distortion of described multi-view depth graph code and the virtual viewpoint rendering distortion obtains the distortion of virtual view and the relation between the coded quantization parameter QP further comprises:
The error E of virtual viewpoint rendering image
TBe E
T=E
CW+ E
DW+ E
O, wherein, E
OPaint the caused error of filling cavity for mending by image,
Wherein, i
OBe the described picture element that is blocked,
Be the described picture element i that is blocked
OThe brightness of adjacent pixels point or chromatic value distribute, and y is described pixel i
OBrightness or the span of colourity, y
InBe the described picture element i that is blocked
OBrightness when benefit is painted or chromatic value.
10. estimation method of distortion performance of stereo video encoding rate as claimed in claim 1, it is characterized in that, further comprise according to the relation between described multiple view video coding rate-distortion model and the multi-view depth graph code rate-distortion model acquisition drafting needed encoder bit rate of described virtual view and multiple view video coding code check and the multi-view depth graph code code check:
The required code check of stereo scopic video coding is r
T=r
C+ r
D, wherein, r
CBe multiple view video coding code check, r
DBe multi-view depth graph code code check.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010222351 CN101888566B (en) | 2010-06-30 | 2010-06-30 | Estimation method of distortion performance of stereo video encoding rate |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010222351 CN101888566B (en) | 2010-06-30 | 2010-06-30 | Estimation method of distortion performance of stereo video encoding rate |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101888566A true CN101888566A (en) | 2010-11-17 |
| CN101888566B CN101888566B (en) | 2012-02-15 |
Family
ID=43074245
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010222351 Active CN101888566B (en) | 2010-06-30 | 2010-06-30 | Estimation method of distortion performance of stereo video encoding rate |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101888566B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158710A (en) * | 2011-05-27 | 2011-08-17 | 山东大学 | Depth view encoding rate distortion judgment method for virtual view quality |
| CN102387368A (en) * | 2011-10-11 | 2012-03-21 | 浙江工业大学 | Fast selection method of inter-view prediction for multi-view video coding (MVC) |
| CN102413353A (en) * | 2011-12-28 | 2012-04-11 | 清华大学 | Method for allocating code rates of multi-view video and depth graph in stereo video encoding process |
| CN102572439A (en) * | 2012-03-14 | 2012-07-11 | 清华大学深圳研究生院 | Method for determining optimal multi-viewpoint video coding mode for coding |
| CN102572440A (en) * | 2012-03-15 | 2012-07-11 | 天津大学 | Multi-viewpoint video transmission method based on depth map and distributed video coding |
| CN102595166A (en) * | 2012-03-05 | 2012-07-18 | 山东大学 | Lagrange factor calculation method applied for depth image encoding |
| CN102769749A (en) * | 2012-06-29 | 2012-11-07 | 宁波大学 | Post-processing method for depth image |
| CN104284196A (en) * | 2014-10-28 | 2015-01-14 | 天津大学 | Bit distribution and rate control algorithm of color and depth video joint encoding |
| CN104506856A (en) * | 2015-01-14 | 2015-04-08 | 山东大学 | Method of estimating quality of virtual view applicable to 3D (Three-dimensional) video system |
| CN104662897A (en) * | 2012-09-25 | 2015-05-27 | 日本电信电话株式会社 | Image encoding method, image decoding method, image encoding device, image decoding device, image encoding program, image decoding program, and recording medium |
| CN106162198A (en) * | 2016-08-31 | 2016-11-23 | 重庆邮电大学 | 3 D video depth map encoding based on the most homogeneous piece of segmentation and coding/decoding method |
| CN106210741A (en) * | 2016-09-10 | 2016-12-07 | 天津大学 | A kind of based on the deep video encryption algorithm of dependency between viewpoint |
| CN109076205A (en) * | 2016-04-21 | 2018-12-21 | 奥崔迪合作公司 | Bimodulus depth estimator |
| CN109891463A (en) * | 2016-10-25 | 2019-06-14 | 索尼公司 | Image processing equipment and image processing method |
| TWI663875B (en) * | 2018-06-21 | 2019-06-21 | 威盛電子股份有限公司 | Video processing method and device thereof |
| CN110536137A (en) * | 2019-08-30 | 2019-12-03 | 无锡北邮感知技术产业研究院有限公司 | A kind of LOOK LEFT video traffic prediction method and device in 3D video |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1471319A (en) * | 2002-07-22 | 2004-01-28 | 中国科学院计算技术研究所 | Association rate distortion optimized code rate control method and apparatus thereof |
| CN101466038A (en) * | 2008-12-17 | 2009-06-24 | 宁波大学 | Method for encoding stereo video |
| CN101729891A (en) * | 2009-11-05 | 2010-06-09 | 宁波大学 | Method for encoding multi-view depth video |
-
2010
- 2010-06-30 CN CN 201010222351 patent/CN101888566B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1471319A (en) * | 2002-07-22 | 2004-01-28 | 中国科学院计算技术研究所 | Association rate distortion optimized code rate control method and apparatus thereof |
| CN101466038A (en) * | 2008-12-17 | 2009-06-24 | 宁波大学 | Method for encoding stereo video |
| CN101729891A (en) * | 2009-11-05 | 2010-06-09 | 宁波大学 | Method for encoding multi-view depth video |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158710B (en) * | 2011-05-27 | 2012-12-26 | 山东大学 | Depth view encoding rate distortion judgment method for virtual view quality |
| CN102158710A (en) * | 2011-05-27 | 2011-08-17 | 山东大学 | Depth view encoding rate distortion judgment method for virtual view quality |
| CN102387368A (en) * | 2011-10-11 | 2012-03-21 | 浙江工业大学 | Fast selection method of inter-view prediction for multi-view video coding (MVC) |
| CN102413353A (en) * | 2011-12-28 | 2012-04-11 | 清华大学 | Method for allocating code rates of multi-view video and depth graph in stereo video encoding process |
| CN102413353B (en) * | 2011-12-28 | 2014-02-19 | 清华大学 | Method for allocating code rates of multi-view video and depth graph in stereo video encoding process |
| CN102595166A (en) * | 2012-03-05 | 2012-07-18 | 山东大学 | Lagrange factor calculation method applied for depth image encoding |
| CN102572439A (en) * | 2012-03-14 | 2012-07-11 | 清华大学深圳研究生院 | Method for determining optimal multi-viewpoint video coding mode for coding |
| CN102572439B (en) * | 2012-03-14 | 2014-02-12 | 清华大学深圳研究生院 | Method for determining optimal multi-viewpoint video coding mode for coding |
| CN102572440A (en) * | 2012-03-15 | 2012-07-11 | 天津大学 | Multi-viewpoint video transmission method based on depth map and distributed video coding |
| CN102769749A (en) * | 2012-06-29 | 2012-11-07 | 宁波大学 | Post-processing method for depth image |
| CN102769749B (en) * | 2012-06-29 | 2015-03-18 | 宁波大学 | Post-processing method for depth image |
| CN104662897A (en) * | 2012-09-25 | 2015-05-27 | 日本电信电话株式会社 | Image encoding method, image decoding method, image encoding device, image decoding device, image encoding program, image decoding program, and recording medium |
| CN104284196A (en) * | 2014-10-28 | 2015-01-14 | 天津大学 | Bit distribution and rate control algorithm of color and depth video joint encoding |
| CN104284196B (en) * | 2014-10-28 | 2017-06-30 | 天津大学 | The colored bit with deep video combined coding is distributed and rate control algorithm |
| CN104506856A (en) * | 2015-01-14 | 2015-04-08 | 山东大学 | Method of estimating quality of virtual view applicable to 3D (Three-dimensional) video system |
| CN109076205A (en) * | 2016-04-21 | 2018-12-21 | 奥崔迪合作公司 | Bimodulus depth estimator |
| CN109076205B (en) * | 2016-04-21 | 2021-03-09 | 奥崔迪合作公司 | Dual mode depth estimator |
| CN106162198A (en) * | 2016-08-31 | 2016-11-23 | 重庆邮电大学 | 3 D video depth map encoding based on the most homogeneous piece of segmentation and coding/decoding method |
| CN106162198B (en) * | 2016-08-31 | 2019-02-15 | 重庆邮电大学 | 3 D video depth map encoding and coding/decoding method based on irregular homogeneous piece of segmentation |
| CN106210741A (en) * | 2016-09-10 | 2016-12-07 | 天津大学 | A kind of based on the deep video encryption algorithm of dependency between viewpoint |
| CN106210741B (en) * | 2016-09-10 | 2018-12-21 | 天津大学 | A kind of deep video encryption algorithm based on correlation between viewpoint |
| CN109891463A (en) * | 2016-10-25 | 2019-06-14 | 索尼公司 | Image processing equipment and image processing method |
| CN109891463B (en) * | 2016-10-25 | 2023-08-29 | 索尼公司 | Image processing apparatus and image processing method |
| TWI663875B (en) * | 2018-06-21 | 2019-06-21 | 威盛電子股份有限公司 | Video processing method and device thereof |
| CN110536137A (en) * | 2019-08-30 | 2019-12-03 | 无锡北邮感知技术产业研究院有限公司 | A kind of LOOK LEFT video traffic prediction method and device in 3D video |
| CN110536137B (en) * | 2019-08-30 | 2021-12-10 | 无锡北邮感知技术产业研究院有限公司 | Left view video flow prediction method and device in 3D video |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101888566B (en) | 2012-02-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101888566B (en) | Estimation method of distortion performance of stereo video encoding rate | |
| CN103179405B (en) | A kind of multi-view point video encoding method based on multi-level region-of-interest | |
| CN101690249B (en) | Method and system for encoding a 3D video signal, method and system for decoder for a 3D video signal | |
| Kamolrat et al. | 3D motion estimation for depth image coding in 3D video coding | |
| CN102413353B (en) | Method for allocating code rates of multi-view video and depth graph in stereo video encoding process | |
| CN101466038B (en) | Method for encoding stereo video | |
| CN101835056B (en) | Allocation method for optimal code rates of texture video and depth map based on models | |
| DE102013015821B4 (en) | System and method for improving video coding using content information | |
| CN101390396A (en) | Method and apparatus for encoding and decoding multi-view video to provide uniform picture quality | |
| Kang et al. | Adaptive geometry-based intra prediction for depth video coding | |
| CN102970540A (en) | Multi-view video code rate control method based on key frame code rate-quantitative model | |
| CN102685532A (en) | Coding method for free view point four-dimensional space video coding system | |
| CN101867816A (en) | Stereoscopic video asymmetric compression coding method based on human-eye visual characteristic | |
| CN101584220B (en) | Method and system for encoding a video signal, encoded video signal, method and system for decoding a video signal | |
| CN101841726B (en) | Three-dimensional video asymmetrical coding method | |
| US20140133554A1 (en) | Advanced video coding method, apparatus, and storage medium | |
| De Silva et al. | Intra mode selection method for depth maps of 3D video based on rendering distortion modeling | |
| Jung et al. | Disparity-map-based rendering for mobile 3D TVs | |
| Yang et al. | Adaptive bit allocation for 3D video coding | |
| Scandarolli et al. | Attention-weighted rate allocation in free-viewpoint television | |
| Wang et al. | Difference detection based early mode termination for depth map coding in MVC | |
| Sun et al. | An efficient quality metric for DIBR-based 3D video | |
| CN103379349A (en) | Viewpoint composite predictive encoding method, decoding method, corresponding devices and code stream | |
| Ziegler | Digital stereoscopic imaging and applications. A way towards new dimensions. The RACE II Project DISTIMA | |
| CN102006469B (en) | Three-dimensional element image based multi-level mixed predictive coding structure parallel implementation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |