CN101895752A - Video transmission method, system and device based on visual quality of images - Google Patents
Video transmission method, system and device based on visual quality of images Download PDFInfo
- Publication number
- CN101895752A CN101895752A CN 201010227359 CN201010227359A CN101895752A CN 101895752 A CN101895752 A CN 101895752A CN 201010227359 CN201010227359 CN 201010227359 CN 201010227359 A CN201010227359 A CN 201010227359A CN 101895752 A CN101895752 A CN 101895752A
- Authority
- CN
- China
- Prior art keywords
- video
- frame
- visual
- visual quality
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 115
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000005540 biological transmission Effects 0.000 title claims abstract description 28
- 230000006835 compression Effects 0.000 claims description 20
- 238000007906 compression Methods 0.000 claims description 20
- 238000013139 quantization Methods 0.000 claims description 12
- 238000009877 rendering Methods 0.000 claims description 11
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 230000001105 regulatory effect Effects 0.000 claims description 2
- 238000011002 quantification Methods 0.000 abstract 1
- 230000006978 adaptation Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention provides a free-viewpoint video or multi-viewpoint video transmission method based on visual quality, which comprises the following steps: a server receives original or encoded/decoded free-viewpoint video or multi-viewpoint video signals; the server calculates the visual quality of virtual viewpoint images; the server determines quantification parameters (QPs) for encoding the free-viewpoint video or multi-viewpoint video signals based on the visual quality of the virtual viewpoint images; and the server encodes and compresses the free-viewpoint video or multi-viewpoint video signals again based on the determined QPs and sends the encoded and compressed signals to one or more clients. The transmission method can adjust the QPs based on the visual quality of the virtual viewpoint images, therefore the images obtained by the clients are of higher-level quality.
Description
Technical field
The present invention relates to the image technique field, particularly a kind of free view-point or multi-view point video transmission method, system and device based on visual quality.
Background technology
Free viewpoint video is a kind of brand-new visual form, is after flat digital television, three-dimensional digit TV, the visual form that can provide abundant man-machine interaction, shows neatly with plane and three-dimensional form.Along with the arrival in mobile multimedia epoch, increasing people utilizes terminal equipment access communications networks such as mobile phone, PDA(Personal Digital Assistant) and notebook computer, and by the terminal mobility and the network coverage, enjoys ubiquitous audiovisual entertainment.Combine between free viewpoint video with flexible interactive means and the immanent communication network, will expand new, more attractive business platform for following 3G mobile communication, IPv6.
Current various countries main flow research institution, university be all to having launched research based on the free viewpoint video under the various technical conditions, and many companies have all carried out the exploration of practicability, commercialization to this field, forward position.But these schemes mainly are at the big characteristics of the data volume of free viewpoint video business, adopt some to come the free viewpoint video data are compressed such as measure such as video compression standard, the multi-view video compressed platform of JMVM H.264.These methods all default network have sufficient ability to handle free viewpoint video stream at any time.This obviously is rational inadequately.As everyone knows, have the isomerism factor based on the multimedia system of mobile internet, promptly program source all exists differently aspect compress mode, time and spatial resolution and the information rate, and also there is notable difference in the disposal ability between the different terminals.So, the method for coding and decoding video and procotol simple combination is difficult to various terminals are proved effective, especially the restriction in the network available resources will cause under the situation of significant impact the quality of Video service.
Therefore, the picture quality that how further improves client reconstruct under the Internet resources condition of limited becomes problem demanding prompt solution.
Summary of the invention
Purpose of the present invention is intended to solve at least one of above-mentioned technological deficiency, has proposed a kind of free viewpoint video or multi-view point video transmission method, system and device that is used for based on visual quality.
For achieving the above object, one aspect of the present invention proposes a kind of free viewpoint video or multi-view point video transmission method based on visual quality, may further comprise the steps: server receives original or free viewpoint video after encoding and decoding or multi-view point video signal; Described server calculates the visual quality of virtual visual point image; Described server is determined the quantization parameter QP of above-mentioned free pilot video or multi-viewpoint video signal code according to the visual quality of described virtual visual point image; According to the described QP that determines described free viewpoint video or multi-view point video signal are carried out encoding compression again with described server, and send to one or more clients.
The present invention has also proposed a kind of free viewpoint video or multi-view point video transmission system based on visual quality on the other hand, comprise server and one or more client, described server, be used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and the visual quality of calculating virtual visual point image, and the quantization parameter QP that determines above-mentioned free pilot video or multi-viewpoint video signal code according to the visual quality of described virtual visual point image, the described QP that determines with basis carries out encoding compression again to described free viewpoint video or multi-view point video signal, and sends to one or more clients; Described client is used to receive described server transmitted stream, and according to the virtual view that the user selects, utilizes video and corresponding parallax to carry out the drafting of virtual view, and drawing result is shown to the user.
Further aspect of the present invention has also proposed a kind of server, comprise encoder and QP adjusting module, described encoder, be used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and described free viewpoint video or multi-view point video signal are carried out encoding compression, and send to one or more clients according to quantization parameter QP; Described QP adjusting module is used to calculate the visual quality of virtual visual point image, and adjusts described QP according to the visual quality of described virtual visual point image.
Further aspect of the present invention has also proposed a kind of client, comprise selection of weight determination module, decoding and error concealment module and viewpoint and virtual viewpoint rendering module, whether described weight determination module is used to judge whether the frame code stream that receives should be rejected and is the error frame that can not decode; Described decoding and error concealment module, be used for when the frame that receives is error frame, then with its frame of video with all viewpoints of the moment, the pairing code check of parallax frame is all given up, and the frame of video and the parallax frame of the corresponding viewpoint of previous moment be copied to current time, and the repetition copy can be correctly decoded reconstruction until the frame of current time, and perhaps the weight of current time frame is greater than the weight of previous moment frame; Described viewpoint is selected and the virtual viewpoint rendering module, is used for the drawing viewpoints virtual view according to decoded result and user's selection.
The present invention can adjust QP according to the visual quality of virtual visual point image, thereby makes the resulting picture quality of client have higher level.In a preferred embodiment of the invention, can be optimized QP simultaneously, thereby can further guarantee the quality of the image that client obtains according to visual quality and network congestion situation.
Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize by practice of the present invention.
Description of drawings
Above-mentioned and/or additional aspect of the present invention and advantage are from obviously and easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:
Fig. 1 is the frame diagram by the viewpoint transmission system of the embodiment of the invention;
Fig. 2 is the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention one;
Fig. 3 is the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention two;
Fig. 4 is the structure chart of the server of the embodiment of the invention;
Fig. 5 is the structure chart of the client of the embodiment of the invention.
Embodiment
Describe embodiments of the invention below in detail, the example of described embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Below by the embodiment that is described with reference to the drawings is exemplary, only is used to explain the present invention, and can not be interpreted as limitation of the present invention.
At present, based on network free viewpoint video system, the video image quality that its client is watched is mainly determined by factors such as true visual point image reconstruction quality, the degree of depth/disparity map image reconstruction quality, virtual viewpoint rendering algorithms.Specifically comprise:
(a) quantization parameter (QP) that when the reconstruction quality of client mainly carries out video compression by coding side, adopted of true visual point image determine that the QP value is big more, reconstruction quality is low more; Otherwise, high more.
(b) true visual point image also is subjected to the influence of error control algorithm at the reconstruction quality of client.
(c) the image reconstruction quality of the degree of depth/disparity map is identical with the situation of true visual point image.
(d) picture quality of virtual view is subjected to the reconstruction quality of true visual point image reconstruction quality, the degree of depth/disparity map, the influence of virtual viewpoint rendering algorithm simultaneously.The reconstruction quality of true visual point image and the degree of depth/disparity map is high more, and the virtual viewpoint rendering algorithm that is adopted is made an appointment, and then the visual quality of virtual visual point image is high more.
The objective of the invention is to improve in the past free viewpoint video or multi-view video system and do not consider the factor of visual quality of images in design, thereby proposed a kind of free view-point transmission method and system, this method and system can have following multinomial function:
1, can utilize the visual quality of images evaluation method to carry out multi-view point video and parallax carries out QP and optimizes coding;
2, the weight grade can be set and carry out network congestion situation Adaptive Transmission;
3, can carry out the client error control by judging network and code stream weight.
By above-mentioned several functions, the present invention can guarantee that the resulting picture quality of client has higher level.And this invention at be that client device type and computing capability variation, the network bandwidth have fluctuation, client have high requirement to visual quality of images environment.But need to prove that above-mentioned 3 functions both can adopt simultaneously, also can separately adopt; for example only QP is optimized according to visual quality of images; perhaps only QP is adjusted according to the network congestion situation, or the like, these all should be included within protection scope of the present invention.
As shown in Figure 1, be the frame diagram by the viewpoint transmission system of the embodiment of the invention.Server 101 adopts the multiple view video coding scheme of Multilevel B frame that all vision signals, disparity map sequence are compressed, and all video code flows, parallax code stream are transmitted to client 102.Client 102 is decoded after receiving the code stream that server 101 sends, and shows the image of the selected viewpoint of current client.When customer selecting arrives virtual view, then utilize video to carry out virtual viewpoint rendering, and drawing result is shown to the client with corresponding parallax.In embodiments of the present invention, what above-mentioned parallax was represented is the difference with same pixel horizontal coordinate between two visual point images of identical semanteme, and the parallax of all pixels constitutes the disparity map of this viewpoint in visual point image; Above-mentioned virtual view is any one viewpoint between two camera viewpoints, and does not have camera on this viewpoint scene is taken.In one embodiment of the invention, server 101 adopts the scheme of quality Identification policy optimization video coding and parallax coding.In a preferred embodiment of the invention, server 101 also needs the congested degree of grid of reference, and degree and the visual quality of images according to network congestion is optimized QP simultaneously.Client 102 adopts recognition strategy that it is judged after the code stream that receives by Network Transmission, carries out image reconstruction according to the result who judges then.
Embodiment one,
As shown in Figure 2, be the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention one, this embodiment mainly is optimized QP according to visual quality for images S.This method may further comprise the steps:
Step S201, gain freedom viewpoint video or multi-view point video signal wherein, comprise vision signal and disparity map sequence etc.
Step S202, the picture quality S of calculating virtual view.In one embodiment of the invention, can calculate virtual visual point image quality S according to virtual visual point image quality evaluating method based on Contourlet.Concrete implementation step is as follows:
(2a1), be that N * N carries out piecemeal to resulting virtual visual point image by block size, obtain every row in the image thus, every row have I, a J piece.Get N=2
n, n 〉=6 wherein.
(2a2), to each the piece B in the virtual visual point image
V Ij(0≤i≤I, 0≤j≤J), utilize the block matching method based on the MSE minimum criteria finds corresponding best matching blocks B by horizontal direction respectively in its left and right sides visual point image
L IjAnd B
R Ij, B wherein
L IjIn left visual point image, B
R IjIn right visual point image.
(2a3), to resulting B
V IjCarry out the Contourle wavelet decomposition with the best matching blocks that obtains, to obtain U high-frequency sub-band piece B
U, ij X(u=1,2 ..., U) with a low frequency piece C
Ij X, { L}, the element in the set are representing virtual view, right viewpoint, left visual point image respectively to X ∈ for V, R herein.
(2a4), each high-frequency sub-band piece B that decomposition is obtained
U, ij XFurther adopt u-layer directional filter component to separate, obtain 2
uIndividual subband piece C
U, o, ij X(o=1,2 ..., 2
u).
(2a5), calculate each subband piece C
U, o, ij XVariances sigma
U, o, ij X
(2a6), at σ
U, o, ij XThe basis on further obtain the σ of entire image
U, o X=∑
I, j(σ
U, o, ij X).
(2a7), calculate virtual visual point image and left and right visual point image at the error parameter δ that decomposes with layer between the ordinal number
U, o=min{| σ
U, o V-σ
U, o R|, | σ
U, o V-σ
U, o L|.
(2a8), the error parameter to each layer is weighted integration Δ=∑
U, o2
-uδ
U, o
(2a9), calculate the visual quality of virtual visual point image
C wherein
0Be normal real number regulatory factor, scope is C
0∈ [1,10], suggested range is C
0∈ [6,8]; S is a real number, represents the subjective visual quality do score value, codomain be (0,1].The S score value is high more, and visual quality is good more.
Step S203 adjusts QP according to the visual quality S of virtual view, and wherein, if visual quality S is high more, then the QP value of Xuan Zeing is more little, if instead visual quality S is low more, then the QP value of Xuan Zeing is big more.As the optional scope of QP is [a, b], then QP can adopt with
SRelevant linear or non-linear etc. mapping relations are one by one adjusted, and its basic principle is that the S value is high more, and the QP value is low more, but no matter which kind of method it adopts, and all belongs to the protection range of this patent.
Step S204 carries out compressed encoding according to the QP value of selecting to free viewpoint video or multi-view point video signal, and video code flow and parallax code stream is sent to client 102.In embodiments of the present invention, be not only applicable to online coding, also be applicable to the off-line coding, specifically comprise:
1, at online coding, its characteristics are that the parameter of video compression coding is dynamically adjusted according to the situation of network congestion, client image quality requirements feedback, to video and parallax signals Real Time Compression and be sent to client.
2, at the off-line coding, its characteristics are that server adopts various coding parameters that video and parallax signals are compressed in advance, leave code stream in server with according to the different feedbacks of network with client, in time extract code stream and are sent to client.
Wherein, in online coding and off-line coding, how to select QP in following embodiment, to be described in detail, do not repeat them here.
Embodiment two,
This embodiment is the preferred embodiments of the present invention, not only according to visual quality of images QP is optimized, and also needs according to the degree of network congestion QP to be carried out the self adaptation adjustment.As shown in Figure 3, be the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention two, this embodiment specifically may further comprise the steps:
Step S301, server obtains original or free viewpoint video after encoding and decoding or multi-view point video signal, wherein, comprises vision signal and disparity map sequence etc.
Step S302, server calculates the picture quality S of virtual view and the degree of network congestion.
Step S303, server is adjusted QP according to the picture quality S of virtual view and the degree of network congestion.Specifically comprise:
(3b) at real-time system, its characteristics are that the parameter of video compression coding is dynamically adjusted according to the situation of network congestion, client image quality requirements feedback, to video and parallax signals Real Time Compression and be sent to client.Specifically comprise:
(3b1) to be embodied in the coded quantization parameter QP that is adopted in video coding and the parallax cataloged procedure be that common constraint is carried out according to the picture quality S of the network bandwidth and virtual view to this optimisation strategy, the big more compression efficiency of QP value is high more, and the reconstruction quality of decoding end image is low more; Otherwise compression efficiency is low more, and reconstruction quality is high more.The QP span is different according to the coding standard that system adopted.
(3b2) QP that adopted of parallax coding is smaller or equal to the QP that is adopted with the viewpoint video coding, i.e. QP
Vi〉=QP
Di(vi is an i viewpoint, and di is the disparity map of i viewpoint).
(3b3) QP that video coding adopted
ViAccording to the evaluation result of the picture quality S of virtual view, be the reconstruction quality that guarantees the client video, its span be [0, QP
Upper-bound].
(3b4) the QP allowed band of a certain coding and decoding video platform of hypothesis is the integer (a, b 〉=0) between [a, b], and wherein more little QP value represents compression efficiency low more.S=1 when making QP=a, the S=0 during QP=b, [S
Bound, 1] and be user-defined virtual view mass range, then QP
Upper-bound=b* (1-S
Bound).
(3b5) QP that adopted of viewpoint video coding
ViExcept the restriction of satisfying (3b4), for adapting to the fluctuation of the network bandwidth, the QP that video coding adopted
ViCarry out trickle adjustment in real time according to current bandwidth conditions, adjustment amount is defined as QP
Vi *, the codomain scope be [0, b-QP
Upper-bound].Bandwidth is more little, QP
Vi *Big more; Otherwise, QP
Vi *More little.
(3b6) QP
Vi *Definite method be that encoding platform adopts QP
Vi+ QP
Vi *The code check rate that is exported after encoding
(out)Less than network bandwidth rate
(band)
(3b7) Δ QP
i=QP
Vi-QP
DiCarry out the self adaptation adjustment according to network bandwidth condition, the network bandwidth is low more, congestion situation is serious more, Δ QP
iBe worth big more.Client requires the visual quality of drawing virtual view image high more, Δ QP
iBe worth more little; Otherwise, Δ QP
iBe worth big more.
(3b8) for guaranteeing the drafting visual quality of client virtual visual point image, Δ QP
iHave span [0, Δ QP
Upper-bound].
(3b9) QP that adopted of disparity map coding
DiAccording to Δ QP
i=QP
Vi-QP
DiAnd [0, QP
Upper-bound] and [0, Δ QP
Up-bound] common definite, obtain QP thus
DiScope be [QP
Di Lower-bound, QP
Di Upper-bound], and make QP
Di=QP
Di Upper-bound
(3c) at off-line system, its characteristics are that server adopts various coding parameters that video and parallax signals are compressed in advance, leave code stream in server with according to the different feedbacks of network with client, in time extract code stream and are sent to client.Wherein, off-line system and on-line system select the mode of QP similar, and different is that on-line system is a line compression, and off-line system is to select according to the QP that determines.In off-line system, this optimisation strategy is embodied in the coded quantization parameter that is adopted in video coding and the parallax cataloged procedure and uses the interior all QP of [QPlower-bound, QPupper-bound] scope to encode respectively, and code stream is preserved at server.Promptly have each viewpoint video resulting Δ QP=QPupper-bound-QPlower-bound group code stream under all QP encoding conditions in [QPlower-bound, QPupper-bound] scope respectively at server, and corresponding Δ QP group parallax code stream.The big more compression efficiency of QP value is high more, and the reconstruction quality of decoding end image is low more; Otherwise compression efficiency is low more, and reconstruction quality is high more.The QP span is different according to the coding standard that system adopted.
Step S304, server carries out free viewpoint video or multi-view point video signal are carried out video coding again according to the QP that determines, and adopts at coded frame weight size and carry out the adaptive strategy that code stream sends.Specifically be that server carries out special processing according to the size of weight to partial frame, comprising:
(3d1) the weight setting is according in the Multilevel B frame encoding scheme, is positioned at the low more frame of level number, and its weight is high more; Otherwise, being positioned at the high more frame of level number, its weight is low more.Level number be integer 0,1,2 ..., W}, its corresponding weight then be W, W-1, W-2 ..., 0}.
(3d2) network congestion grade is made as { B corresponding to the level number number
0, B
1, B
2..., B
W, B wherein
0Congestion condition is higher than B
1, successively decrease successively subsequently, to B
WThe time network congestion situation minimum.
(3d3) corresponding to the different congestion level B of network
x, (a) for real-time system, to the weight scope be W, W-1, W-2 ..., all frames of W-x} are encoded, and send code stream to client; (b) for off-line system, extract the weight scope and be W, W-1, W-2 ..., the code stream of all frame correspondences of W-x}, and send to client.
Step S305, client adopts recognition strategy that it is judged after the code stream that receives by Network Transmission, carries out image reconstruction according to judged result then.In one embodiment of the invention, client also needs to give server with feedback information such as network congestion situations.Specifically comprise:
1, recognition strategy is mainly used in and judges that whether this is given up and whether is the error frame that can not decode for the frame code stream receive.Concrete implementation step is described below according to following link:
(3e1) strategy of giving up in the frame recognition strategy determines jointly according to current network congestion condition and the presently used equipment operational capability of user.
(3e2), the network congestion situation is defined as { B corresponding to the setting of server
0, B
1, B
2..., B
W, B wherein
0Congestion condition is higher than B
1, successively decrease successively subsequently, to B
WThe time network congestion situation minimum.
(3e3) be in different congestion level B when network
xThe time, client only receive the weight scope for W, W-1, W-2 ..., all frame code streams of W-x}.If also have the code stream of other weights to be sent to client, then all given up.
(3a4) its operational capability of the presently used equipment of user also to client receive the weight scope W, W-1, W-2 ..., W-x} influences to some extent, operational capability is poor more, the weight scope is more little.
2, decoding and error concealment
(3f1) when the frame that receives be error frame, in the time of can not correctly rebuilding the image of current time, then it is all given up with constantly frame, the pairing code check of parallax frame of all viewpoints, and the frame and the parallax frame of the corresponding viewpoint of previous moment is copied to current time.
(3f2) repeat the copy of (3f1), can be correctly decoded reconstruction until the frame of current time, perhaps till the weight of the weight of current time frame greater than the previous moment frame.
3, viewpoint is selected and virtual viewpoint rendering
Drawing viewpoints virtual view according to decoded result and user's selection.
As shown in Figure 1, this system comprises server 101 and one or more client 102.Server 101 is used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and the visual quality of calculating virtual visual point image, and determine quantization parameter QP according to the visual quality of described virtual visual point image, the described QP that determines with basis carries out encoding compression again to described free viewpoint video or multi-view point video signal, and sends to one or more clients.Client 102 is used to receive described server transmitted stream, and according to the virtual view that the user selects, utilizes video and corresponding parallax to carry out the drafting of virtual view, and drawing result is shown to the user.
As shown in Figure 4, be the structure chart of the server of the embodiment of the invention.This server 101 comprises encoder 200 and QP adjusting module 203.Encoder 200 is used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and according to quantization parameter QP described free viewpoint video or multi-view point video signal is carried out encoding compression again, and sends to one or more clients.QP adjusting module 203 is used to calculate the visual quality of virtual visual point image, and adjusts described QP according to the visual quality of described virtual visual point image.
In one embodiment of the invention, server 101 also comprises user and network condition feedback module 204, is used to receive user's feedback, and the feedback of network congestion situation.QP adjusting module 203 also is used for adjusting described QP according to described network congestion situation.
In one embodiment of the invention, encoder 200 comprises online encoder 201 and off-line encoder 202.
In one embodiment of the invention, server 101 comprises that also weight is provided with module 205, is used for selecting the frame of respective weights to send or encoding according to the network congestion grade.
As shown in Figure 5, be the structure chart of the client of the embodiment of the invention.Client comprises weight determination module 301, decoding and error concealment module 302 and viewpoint selection and virtual viewpoint rendering module 303.Whether weight determination module 301 is used to judge whether the frame code stream that receives should be rejected and is the error frame that can not decode.Decoding is used for when the frame that receives is error frame with error concealment module 302, then with its frame of video with all viewpoints of the moment, the pairing code check of parallax frame is all given up, and the frame of video and the parallax frame of the corresponding viewpoint of previous moment be copied to current time, and repetition copy, frame until current time can be correctly decoded reconstruction, and perhaps the weight of current time frame is greater than the weight of previous moment frame.Viewpoint selection and virtual viewpoint rendering module 303 are used for the drawing viewpoints virtual view according to decoded result and user's selection.
The present invention can adjust QP according to the visual quality of virtual visual point image, thereby makes the resulting picture quality of client have higher level.In a preferred embodiment of the invention, can be optimized QP simultaneously, thereby can further guarantee the quality of the image that client obtains according to visual quality and network congestion situation.
Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, modification, replacement and modification that scope of the present invention is by claims and be equal to and limit to these embodiment.
Claims (15)
1. free viewpoint video or multi-view point video transmission method based on a visual quality is characterized in that, may further comprise the steps:
Server receives original or free viewpoint video after encoding and decoding or multi-view point video signal;
Described server calculates the visual quality of virtual visual point image;
Described server is determined the quantization parameter QP of described free pilot video or multi-viewpoint video signal code according to the visual quality of described virtual visual point image; With
Described server carries out encoding compression according to the described QP that determines again to described free viewpoint video or multi-view point video signal, and sends to one or more clients.
2. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 1, it is characterized in that described server adopts multi-view point video encoding method or single view method for video coding that described free viewpoint video or multi-view point video signal are compressed.
3. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 2 is characterized in that, the visual quality that described server calculates virtual visual point image further comprises:
Is that N * N carries out piecemeal to resulting virtual visual point image by block size, and wherein, every row has I piece, whenever shows J piece, wherein, and N=2
n, n 〉=6 wherein;
To each the piece B in the described virtual visual point image
V Ij, utilize block matching method based on the MSE minimum criteria, in its left and right sides visual point image, find corresponding best matching blocks B respectively by horizontal direction
L IjAnd B
R Ij, wherein, B
L IjIn left visual point image, V
R IjIn right visual point image;
To resulting B
V IjCarry out the Contourle wavelet decomposition with the best matching blocks that obtains, to obtain U high-frequency sub-band piece B
U, ij X, wherein, u=1,2 ..., U and a low frequency piece C
Ij X, wherein, X ∈ V, and R, L}, wherein, V is that virtual visual point image, R are that right visual point image, L are left visual point image;
Each high-frequency sub-band piece B that decomposition is obtained
U, ij XFurther adopt u-layer directional filter component to separate, obtain 2
uIndividual subband piece C
U, o, ij X, wherein, o=1,2 ..., 2
u
Calculate each subband piece C respectively
U, o, ij XVariances sigma
U, o, ij X
According to σ
U, o, ij XObtain the σ of entire image
U, o X=∑
I, j(σ
U, o, ij X);
Calculate described virtual visual point image and left and right visual point image at the error parameter δ that decomposes with layer between the ordinal number
U, o=min{| σ
U, o V-σ
U, o R|, | σ
U, o V-σ
U, o L|;
Error parameter to each layer is weighted integration Δ=∑
U, o2
-uδ
U, o
Calculate the visual quality of described virtual visual point image
C wherein
0Be normal real number regulatory factor, scope is C
0∈ [1,10].
4. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 3 is characterized in that, described server determines that according to the visual quality of described virtual visual point image quantization parameter QP further comprises:
The visual quality S of described virtual visual point image is high more, and then the QP of Xuan Zeing is low more.
5. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 4 is characterized in that the optional scope of QP is [a, b], then QP can adopt with
SRelevant linear or nonlinear mapping relations are one by one adjusted, and its basic principle is that the S value is high more, and the QP value is low more.
6. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 3 is characterized in that, described server adopts online coding or off-line coded system that described free viewpoint video signal or multi-view point video are encoded.
7. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 4 is characterized in that, also comprise:
Described server is adjusted described QP according to the network congestion situation, and wherein, the network congestion situation is serious more, the Δ QP of adjustment
iBe worth big more.
8. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 7 is characterized in that, also comprise:
Described server is selected the frame of respective weights to send according to the network congestion grade or is encoded.
9. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 8 is characterized in that, also comprise:
Whether described client is judged whether the frame code stream that receives should be rejected and is the error frame that can not decode;
When if the frame that receives is error frame, with the frame of video of all viewpoints constantly, all give up by the pairing code check of parallax frame with it for then described client, and the frame of video and the parallax frame of the corresponding viewpoint of previous moment is copied to current time;
Described client repeats copy, can be correctly decoded reconstruction until the frame of current time, and perhaps the weight of current time frame is greater than the weight of previous moment frame.
10. free viewpoint video or multi-view point video transmission system based on a visual quality is characterized in that, comprise server and one or more client,
Described server, be used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and the visual quality of calculating virtual visual point image, and the quantization parameter QP that determines above-mentioned free pilot video or multi-viewpoint video signal code according to the visual quality of described virtual visual point image, the described QP that determines with basis carries out encoding compression again to described free viewpoint video or multi-view point video signal, and sends to one or more clients;
Described client is used to receive described server transmitted stream, and according to the virtual view that the user selects, utilizes video and corresponding parallax to carry out the drafting of virtual view, and drawing result is shown to the user.
11. a server is characterized in that, comprises encoder and QP adjusting module,
Described encoder is used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and according to quantization parameter QP described free viewpoint video or multi-view point video signal is carried out encoding compression again, and sends to one or more clients;
Described QP adjusting module is used to calculate the visual quality of virtual visual point image, and adjusts the described QP of above-mentioned free pilot video or multi-viewpoint video signal code according to the visual quality of described virtual visual point image.
12. server as claimed in claim 11 is characterized in that, also comprises user and network condition feedback module, is used to receive user's feedback, and the feedback of network congestion situation;
Described QP adjusting module also is used for adjusting described QP according to described network congestion situation.
13., it is characterized in that described encoder comprises online encoder and off-line encoder as claim 11 or 12 described servers.
14. server as claimed in claim 11 is characterized in that, comprises that also weight is provided with module, is used for selecting the frame of respective weights to send or encoding according to the network congestion grade.
15. a client is characterized in that, comprises selection of weight determination module, decoding and error concealment module and viewpoint and virtual viewpoint rendering module,
Whether described weight determination module is used to judge whether the frame code stream that receives should be rejected and is the error frame that can not decode;
Described decoding and error concealment module, be used for when the frame that receives is error frame, then with its frame of video with all viewpoints of the moment, the pairing code check of parallax frame is all given up, and the frame of video and the parallax frame of the corresponding viewpoint of previous moment be copied to current time, and the repetition copy can be correctly decoded reconstruction until the frame of current time, and perhaps the weight of current time frame is greater than the weight of previous moment frame;
Described viewpoint is selected and the virtual viewpoint rendering module, is used for the drawing viewpoints virtual view according to decoded result and user's selection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010227359 CN101895752B (en) | 2010-07-07 | 2010-07-07 | Video transmission method, system and device based on visual quality of images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010227359 CN101895752B (en) | 2010-07-07 | 2010-07-07 | Video transmission method, system and device based on visual quality of images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101895752A true CN101895752A (en) | 2010-11-24 |
| CN101895752B CN101895752B (en) | 2012-12-19 |
Family
ID=43104796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010227359 Expired - Fee Related CN101895752B (en) | 2010-07-07 | 2010-07-07 | Video transmission method, system and device based on visual quality of images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101895752B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013086654A1 (en) * | 2011-12-15 | 2013-06-20 | Thomson Licensing | Method and apparatus for video quality measurement |
| CN106170081A (en) * | 2016-06-28 | 2016-11-30 | 上海米影信息科技有限公司 | A kind of wireless dummy reality server, system and data compression transmission method thereof |
| CN106688233A (en) * | 2016-12-08 | 2017-05-17 | 深圳市大疆创新科技有限公司 | Method and apparatus for transmitting image, and unmanned aerial vehicle |
| CN108780499A (en) * | 2016-03-09 | 2018-11-09 | 索尼公司 | The system and method for video processing based on quantization parameter |
| CN113344069A (en) * | 2021-05-31 | 2021-09-03 | 成都快眼科技有限公司 | Image classification method for unsupervised visual representation learning based on multi-dimensional relation alignment |
| CN115225961A (en) * | 2022-04-22 | 2022-10-21 | 上海赛连信息科技有限公司 | No-reference network video quality evaluation method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1471319A (en) * | 2002-07-22 | 2004-01-28 | 中国科学院计算技术研究所 | Association rate distortion optimized code rate control method and apparatus thereof |
| CN101277454A (en) * | 2008-04-28 | 2008-10-01 | 清华大学 | Method for generating real time tridimensional video based on binocular camera |
| CN101466038A (en) * | 2008-12-17 | 2009-06-24 | 宁波大学 | Method for encoding stereo video |
| WO2010024919A1 (en) * | 2008-08-29 | 2010-03-04 | Thomson Licensing | View synthesis with boundary-splatting |
-
2010
- 2010-07-07 CN CN 201010227359 patent/CN101895752B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1471319A (en) * | 2002-07-22 | 2004-01-28 | 中国科学院计算技术研究所 | Association rate distortion optimized code rate control method and apparatus thereof |
| CN101277454A (en) * | 2008-04-28 | 2008-10-01 | 清华大学 | Method for generating real time tridimensional video based on binocular camera |
| WO2010024919A1 (en) * | 2008-08-29 | 2010-03-04 | Thomson Licensing | View synthesis with boundary-splatting |
| CN101466038A (en) * | 2008-12-17 | 2009-06-24 | 宁波大学 | Method for encoding stereo video |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013086654A1 (en) * | 2011-12-15 | 2013-06-20 | Thomson Licensing | Method and apparatus for video quality measurement |
| CN103999461A (en) * | 2011-12-15 | 2014-08-20 | 汤姆逊许可公司 | Method and apparatus for video quality measurement |
| US9961340B2 (en) | 2011-12-15 | 2018-05-01 | Thomson Licensing | Method and apparatus for video quality measurement |
| CN108780499A (en) * | 2016-03-09 | 2018-11-09 | 索尼公司 | The system and method for video processing based on quantization parameter |
| CN108780499B (en) * | 2016-03-09 | 2024-02-02 | 索尼公司 | System and method for video processing based on quantization parameters |
| CN106170081A (en) * | 2016-06-28 | 2016-11-30 | 上海米影信息科技有限公司 | A kind of wireless dummy reality server, system and data compression transmission method thereof |
| CN106688233A (en) * | 2016-12-08 | 2017-05-17 | 深圳市大疆创新科技有限公司 | Method and apparatus for transmitting image, and unmanned aerial vehicle |
| WO2018103034A1 (en) * | 2016-12-08 | 2018-06-14 | 深圳市大疆创新科技有限公司 | Image transmission method, apparatus, and unmanned aerial vehicle |
| CN113344069A (en) * | 2021-05-31 | 2021-09-03 | 成都快眼科技有限公司 | Image classification method for unsupervised visual representation learning based on multi-dimensional relation alignment |
| CN113344069B (en) * | 2021-05-31 | 2023-01-24 | 成都快眼科技有限公司 | Image classification method for unsupervised visual representation learning based on multi-dimensional relation alignment |
| CN115225961A (en) * | 2022-04-22 | 2022-10-21 | 上海赛连信息科技有限公司 | No-reference network video quality evaluation method and device |
| CN115225961B (en) * | 2022-04-22 | 2024-01-16 | 上海赛连信息科技有限公司 | No-reference network video quality evaluation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101895752B (en) | 2012-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101895753B (en) | Network congestion degree based video transmission method, system and device | |
| Guo et al. | Distributed multi-view video coding | |
| Belyaev et al. | Robust vehicle-to-infrastructure video transmission for road surveillance applications | |
| CN101895752B (en) | Video transmission method, system and device based on visual quality of images | |
| JP2006528870A (en) | System and method for foregoed video coding and transcoding for mono or stereo images | |
| Li et al. | Robust transmission of JPEG2000 encoded images over packet loss channels | |
| Shen et al. | Foveation-based wireless soft image delivery | |
| Zhu et al. | Error-resilient and error concealment 3-D SPIHT for multiple description video coding with added redundancy | |
| Meddeb et al. | ROI-based rate control using tiles for an HEVC encoded video stream over a lossy network | |
| US10469824B2 (en) | Hybrid digital-analog coding of stereo video | |
| WO2001095512A1 (en) | System and method for object-oriented video processing | |
| Xiang et al. | Robust multiview three-dimensional video communications based on distributed video coding | |
| Yip et al. | Joint source and channel coding for H. 264 compliant stereoscopic video transmission | |
| CN116647704A (en) | Video coding and decoding joint error control method for HEVC data option | |
| CN105306946A (en) | Quality scalable video coding method based on mean square error threshold | |
| Li et al. | Soft transmission of 3D video for low power and low complexity scenario | |
| Wang et al. | Important macroblock distinction model for multi-view plus depth video transmission over error-prone network | |
| Gao et al. | Texture-distortion-constrained joint source-channel coding of multi-view video plus depth-based 3D video | |
| Guo et al. | Free viewpoint switching in multi-view video streaming using Wyner-Ziv video coding | |
| Xu et al. | Optimization-Based Pseudo-Analog Transmission Scheme for Wireless Feature Map Transmission | |
| CN102811349A (en) | No-feedback multi-description distributed video coder-decoder capable of adjusting redundancy automatically | |
| CN102355582B (en) | Method for optimizing rate distortion model for three-dimensional video coding | |
| Kim et al. | Error-resilient 3-D wavelet video coding with duplicated lowest sub-band coefficients and two-step error concealment method | |
| Ouddane et al. | Asymmetric stereoscopic images coding using perceptual model | |
| Kung et al. | Packet video transmission over wireless channels with adaptive channel rate allocation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20121219 |