CN101895752B - Video transmission method, system and device based on visual quality of images - Google Patents
Video transmission method, system and device based on visual quality of images Download PDFInfo
- Publication number
- CN101895752B CN101895752B CN 201010227359 CN201010227359A CN101895752B CN 101895752 B CN101895752 B CN 101895752B CN 201010227359 CN201010227359 CN 201010227359 CN 201010227359 A CN201010227359 A CN 201010227359A CN 101895752 B CN101895752 B CN 101895752B
- Authority
- CN
- China
- Prior art keywords
- point image
- video
- visual
- visual point
- viewpoint video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a free-viewpoint video or multi-viewpoint video transmission method based on visual quality, which comprises the following steps: a server receives original or encoded/decoded free-viewpoint video or multi-viewpoint video signals; the server calculates the visual quality of virtual viewpoint images; the server determines quantification parameters (QPs) for encoding the free-viewpoint video or multi-viewpoint video signals based on the visual quality of the virtual viewpoint images; and the server encodes and compresses the free-viewpoint video or multi-viewpoint video signals again based on the determined QPs and sends the encoded and compressed signals to one or more clients. The transmission method can adjust the QPs based on the visual quality of the virtual viewpoint images, therefore the images obtained by the clients are of higher-level quality.
Description
Technical field
The present invention relates to the image technique field, particularly a kind of free view-point or multi-view point video transmission method, system and device based on visual quality.
Background technology
Free viewpoint video is a kind of brand-new visual form, is after flat digital television, three-dimensional digit TV, the visual form that abundant man-machine interaction can be provided, shows neatly with plane and three-dimensional form.Along with the arrival in mobile multimedia epoch, increasing people utilizes terminal equipment access communications networks such as mobile phone, PDA(Personal Digital Assistant) and notebook computer, and by terminal mobility and network coverage property, enjoys ubiquitous audiovisual entertainment.Combine between free viewpoint video with flexible interactive means and the immanent communication network, will expand new, more attractive business platform for following 3G mobile communication, IPv6.
Current various countries main flow research institution, university be all to having launched research based on the free viewpoint video under the various technical conditions, and many companies have all carried out the exploration of practicability, commercialization to this field, forward position.But these schemes mainly are to the professional big characteristics of data volume of free viewpoint video, adopt some to come the free viewpoint video data are compressed such as measure such as video compression standard, the multi-view video compressed platform of JMVM H.264.These methods all default network have sufficient ability to handle free viewpoint video stream at any time.This obviously is rational inadequately.As everyone knows, have the isomerism factor based on the multimedia system of mobile internet, promptly program source all exists differently aspect compress mode, time and spatial resolution and the information rate, and also there is notable difference in the disposal ability between the different terminals.So, the method for coding and decoding video and procotol simple combination is difficult to proved effective in various terminals, especially the restriction in the network available resources will cause under the situation of significant impact the quality of Video service.
Therefore, the picture quality that how under the Internet resources condition of limited, further improves client reconstruct becomes problem demanding prompt solution.
Summary of the invention
The object of the invention is intended to solve at least one of above-mentioned technological deficiency, has proposed a kind of free viewpoint video or multi-view point video transmission method, system and device that is used for based on visual quality.
For achieving the above object, one aspect of the present invention proposes a kind of free viewpoint video or multi-view point video transmission method based on visual quality, may further comprise the steps: server receives original or free viewpoint video after encoding and decoding or multi-view point video signal; Said server calculates the visual quality of virtual visual point image; Said server is confirmed the quantization parameter QP of above-mentioned free pilot video or multi-viewpoint video signal code according to the visual quality of said virtual visual point image; According to the said QP that confirms said free viewpoint video or multi-view point video signal are carried out encoding compression with said server again, and send to one or more clients.
The present invention has also proposed a kind of free viewpoint video or multi-view point video transmission system based on visual quality on the other hand; Comprise server and one or more client; Said server; Be used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and calculate the visual quality of virtual visual point image, and confirm the quantization parameter QP of above-mentioned free pilot video or multi-viewpoint video signal code according to the visual quality of said virtual visual point image; The said QP that confirms with basis carries out encoding compression again to said free viewpoint video or multi-view point video signal, and sends to one or more clients; Said client is used to receive said server transmitted stream, and according to the virtual view that the user selects, utilizes video and corresponding parallax to carry out the drafting of virtual view, and drawing result is shown to the user.
Further aspect of the present invention has also proposed a kind of server; Comprise encoder and QP adjusting module; Said encoder; Be used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and said free viewpoint video or multi-view point video signal carried out encoding compression, and send to one or more clients according to quantization parameter QP; Said QP adjusting module is used to calculate the visual quality of virtual visual point image, and adjusts said QP according to the visual quality of said virtual visual point image.
Further aspect of the present invention has also proposed a kind of client; Comprise selection of weight determination module, decoding and error concealment module and viewpoint and virtual viewpoint rendering module; Whether said weight determination module is used to judge whether the frame code stream that receives should be rejected and is the error frame that can not decode; Said decoding and error concealment module; Be used for when the frame that receives is error frame, then with its frame of video with all viewpoints of the moment, the pairing code check of parallax frame is all given up; And the frame of video and the parallax frame of the corresponding viewpoint of previous moment be copied to current time; And the repetition copy, until the correct decoding and rebuilding of frame ability of current time, perhaps the weight of current time frame is greater than the weight of previous moment frame; Said viewpoint is selected and the virtual viewpoint rendering module, is used for the drawing viewpoints virtual view based on decoded result and user's selection.
The present invention can adjust QP according to the visual quality of virtual visual point image, thereby makes the resulting picture quality of client have higher level.In a preferred embodiment of the invention, can be optimized QP simultaneously, thereby can further guarantee the quality of the image that client obtains according to visual quality and network congestion situation.
Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize through practice of the present invention.
Description of drawings
Above-mentioned and/or additional aspect of the present invention and advantage are from obviously with easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:
Fig. 1 is the frame diagram by the viewpoint transmission system of the embodiment of the invention;
Fig. 2 is the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention one;
Fig. 3 is the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention two;
Fig. 4 is the structure chart of the server of the embodiment of the invention;
Fig. 5 is the structure chart of the client of the embodiment of the invention.
Embodiment
Describe embodiments of the invention below in detail, the example of said embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Be exemplary through the embodiment that is described with reference to the drawings below, only be used to explain the present invention, and can not be interpreted as limitation of the present invention.
At present, based on network free viewpoint video system, the video image quality that its client is watched is mainly determined by factors such as true visual point image reconstruction quality, the degree of depth/disparity map image reconstruction quality, virtual viewpoint rendering algorithms.Specifically comprise:
(a) quantization parameter (QP) that when the reconstruction quality of client mainly carries out video compression by coding side, adopted of true visual point image determine that the QP value is big more, reconstruction quality is low more; Otherwise, high more.
(b) true visual point image also receives the influence of error control algorithm at the reconstruction quality of client.
(c) the image reconstruction quality of the degree of depth/disparity map is identical with the situation of true visual point image.
(d) picture quality of virtual view receives the reconstruction quality of true visual point image reconstruction quality, the degree of depth/disparity map, the influence of virtual viewpoint rendering algorithm simultaneously.The reconstruction quality of true visual point image and the degree of depth/disparity map is high more, and the virtual viewpoint rendering algorithm that is adopted is made an appointment, and then the visual quality of virtual visual point image is high more.
The objective of the invention is to improve in the past free viewpoint video or multi-view video system and in design, do not consider the factor of visual quality of images, thereby proposed a kind of free view-point transmission method and system, this method and system can have following multinomial function:
1, can utilize the visual quality of images evaluation method to carry out multi-view point video and parallax carries out QP and optimizes coding;
2, the weight grade can be set and carry out network congestion situation Adaptive Transmission;
3, can carry out the client error control through judging network and code stream weight.
Through above-mentioned several functions, the present invention can guarantee that the resulting picture quality of client has higher level.And what this invention was directed against is that client device type and computing capability variation, the network bandwidth have fluctuation, client have high requirement to visual quality of images environment.But need to prove that above-mentioned 3 functions both can adopt simultaneously, also can separately adopt; For example only QP is optimized according to visual quality of images; Perhaps only QP is adjusted according to the network congestion situation, or the like, these all should be included within protection scope of the present invention.
As shown in Figure 1, be the frame diagram by the viewpoint transmission system of the embodiment of the invention.Server 101 adopts the multiple view video coding scheme of Multilevel B frame that all vision signals, disparity map sequence are compressed, and all video code flows, parallax code stream are transmitted to client 102.Client 102 is decoded after receiving the code stream that server 101 sends, and shows the image of the selected viewpoint of current client.When customer selecting arrives virtual view, then utilize video to carry out virtual viewpoint rendering, and drawing result is shown to the client with corresponding parallax.In embodiments of the present invention, what above-mentioned parallax was represented is the difference with same pixel horizontal coordinate between two visual point images of identical semanteme, and the parallax of all pixels constitutes the disparity map of this viewpoint in visual point image; Above-mentioned virtual view is any viewpoint between two camera viewpoints, and does not have camera on this viewpoint scene is taken.In one embodiment of the invention, server 101 adopts the scheme of quality Identification policy optimization video coding and parallax coding.In a preferred embodiment of the invention, server 101 also needs the congested degree of grid of reference, and degree and the visual quality of images according to network congestion is optimized QP simultaneously.Client 102 adopts recognition strategy that it is judged after the code stream that receives through Network Transmission, carries out image reconstruction according to the result who judges then.
Embodiment one,
As shown in Figure 2, be the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention one, this embodiment mainly is optimized QP according to visual quality for images S.This method may further comprise the steps:
Step S201, gain freedom viewpoint video or multi-view point video signal wherein, comprise vision signal and disparity map sequence etc.
Step S202, the picture quality S of calculating virtual view.In one embodiment of the invention, can calculate virtual visual point image quality S according to virtual visual point image quality evaluating method based on Contourlet.The practical implementation step is following:
(2a1), be that N * N carries out piecemeal to resulting virtual visual point image by block size, obtain every row in the image thus, every row have I, a J piece.Get N=2
n, n>=6 wherein.
(2a2), to each the piece B in the virtual visual point image
V Ij(0≤i≤I, 0≤j≤J), utilize the block matching method based on the MSE minimum criteria finds corresponding best matching blocks B by horizontal direction respectively in its left and right sides visual point image
L IjAnd B
R Ij, B wherein
L IjIn left visual point image, B
R IjIn right visual point image.
(2a3), to resulting B
V IjCarry out the Contourlet wavelet decomposition with the best matching blocks that obtains, to obtain U high-frequency sub-band piece B
U, ij X(u=1,2 ..., U) with a low frequency piece C
Ij X, { L}, the element in the set are representing virtual view, right viewpoint, left visual point image respectively to X ∈ for V, R here.
Each the high-frequency sub-band piece B that (2a4), decomposition is obtained
U, ij XFurther adopt u-layer directional filter component to separate, obtain 2
uIndividual subband piece C
U, o, ij X(o=1,2 ..., 2
u).
(2a5), calculate each subband piece C
U, o, ij XVariances sigma
U, o, ij X
(2a6), at σ
U, o, ij XThe basis on further obtain the σ of entire image
U, o X=∑
I, j(σ
U, o, ij X).
(2a7), calculate virtual visual point image and left and right visual point image and decomposing the error parameter δ between the ordinal number with layer
U, o=min{| σ
U, o V-σ
U, o R|, | σ
U, o V-σ
U, o L|.
(2a8), the error parameter to each layer carries out weighting integration Δ=∑
U, o2
-uδ
U, o
(2a9), calculate the visual quality of virtual visual point image
C wherein
0Be normal real number regulatory factor, scope is C
0∈ [1,10], suggested range is C
0∈ [6,8]; S is a real number, represents the subjective visual quality do score value, codomain be (0,1].The S score value is high more, and visual quality is good more.
Step S203 adjusts QP based on the visual quality S of virtual view, and wherein, if visual quality S is high more, then the QP value of Xuan Zeing is more little, if instead visual quality S is low more, then the QP value of Xuan Zeing is big more.Like the optional scope of QP is [a, b], then QP can adopt with
SRelevant linear or non-linear etc. mapping relations are one by one adjusted, and its basic principle is that the S value is high more, and the QP value is low more, but no matter which kind of method it adopts, and all belongs to the protection range of this patent.
Step S204 carries out compressed encoding according to the QP value of selecting to free viewpoint video or multi-view point video signal, and video code flow and parallax code stream is sent to client 102.In embodiments of the present invention, be not only applicable to online coding, also be applicable to the off-line coding, specifically comprise:
1, to online coding, its characteristics are that the parameter of video compression coding is dynamically adjusted according to the situation of network congestion, client image quality requirements feedback, to video and parallax signals Real Time Compression and be sent to client.
2, to the off-line coding, its characteristics are that server adopts various coding parameters that video and parallax signals are compressed in advance, leave code stream in server with according to the different feedbacks of network with client, in time extract code stream and are sent to client.
Wherein, in online coding and off-line coding, how to select QP in following embodiment, to be described in detail, repeat no more at this.
Embodiment two,
This embodiment is the preferred embodiments of the present invention, not only according to visual quality of images QP is optimized, and also need carry out the self adaptation adjustment to QP according to the degree of network congestion.As shown in Figure 3, be the free viewpoint video or the multi-view point video transmission method flow chart of the embodiment of the invention two, this embodiment specifically may further comprise the steps:
Step S301, server obtains original or free viewpoint video after encoding and decoding or multi-view point video signal, wherein, comprises vision signal and disparity map sequence etc.
Step S302, server calculates the picture quality S of virtual view and the degree of network congestion.
Step S303, server is adjusted QP according to the picture quality S of virtual view and the degree of network congestion.Specifically comprise: (3b) to real-time system, its characteristics are that the parameter of video compression coding is dynamically adjusted according to the situation of network congestion, client image quality requirements feedback, to video and parallax signals Real Time Compression and be sent to client.Specifically comprise:
(3b) to real-time system, its characteristics are that the parameter of video compression coding is dynamically adjusted according to the situation of network congestion, client image quality requirements feedback, to video and parallax signals Real Time Compression and be sent to client.Specifically comprise:
(3b1) this optimisation strategy is embodied in picture quality S that the coded quantization parameter QP that is adopted in video coding and the parallax cataloged procedure is based on the network bandwidth and virtual view and retrains jointly and carry out; The big more compression efficiency of QP value is high more, and the reconstruction quality of decoding end image is low more; Otherwise compression efficiency is low more, and reconstruction quality is high more.The QP span is different according to the coding standard that system adopted.
(3b2) QP that adopted of parallax coding is smaller or equal to the QP that is adopted with the viewpoint video coding, i.e. QP
Vi>=QP
Di(vi is an i viewpoint, and di is the disparity map of i viewpoint).
The QP that (3b3) video coding adopted
ViAccording to the evaluation result of the picture quality S of virtual view, be the reconstruction quality that guarantees the client video, its span be [0, QP
Upper-bound].
(3b4) the QP allowed band of a certain coding and decoding video platform of hypothesis is the integer (a, b >=0) between [a, b], and wherein more little QP value represents compression efficiency low more.S=1 when making QP=a, the S=0 during QP=b, [S
Bound, 1] and be user-defined virtual view mass range, then QP
Upper-bound=b* (1-S
Bound).
(3b5) QP that adopted of viewpoint video coding
ViExcept the restriction of satisfying (3b4), for adapting to the fluctuation of the network bandwidth, the QP that video coding adopted
ViCarry out trickle adjustment in real time according to current bandwidth conditions, adjustment amount is defined as QP
Vi *, the codomain scope be [0, b-QP
Upper-bound].Bandwidth is more little, QP
Vi *Big more; Otherwise, QP
Vi *More little.
(3b6) QP
Vi *Definite method be that encoding platform adopts QP
Vi+ QP
Vi *The code check rate that is exported after encoding
(out)Less than network bandwidth rate
(band)
(3b7) Δ QP
i=QP
Vi-QP
DiCarry out the self adaptation adjustment according to network bandwidth condition, the network bandwidth is low more, congestion situation is serious more, Δ QP
iBe worth big more.Client requires the visual quality of drawing virtual view image high more, Δ QP
iBe worth more little; Otherwise, Δ QP
iBe worth big more.
(3b8) for guaranteeing the drafting visual quality of client virtual visual point image, Δ QP
iHave span [0, Δ QP
Upper-bound].
(3b9) QP that adopted of disparity map coding
DiAccording to Δ QP
i=QP
Vi-QP
DiAnd [0, QP
Upper-bound] and [0, Δ QP
Up-bound] common definite, obtain QP thus
DiScope be [QP
Di Lower-bound, QP
Di Upper-bound], and make QP
Di=QP
Di Upper-bound
(3c) to off-line system, its characteristics are that server adopts various coding parameters that video and parallax signals are compressed in advance, leave code stream in server with according to the different feedbacks of network with client, in time extract code stream and are sent to client.Wherein, off-line system and on-line system select the mode of QP similar, and different is that on-line system is a line compression, and off-line system is to select according to the QP that confirms.In off-line system, this optimisation strategy is embodied in the coded quantization parameter that is adopted in video coding and the parallax cataloged procedure and uses the interior all QP of [QPlower-bound, QPupper-bound] scope to encode respectively, and code stream is preserved at server.Promptly have each viewpoint video resulting Δ QP=QPupper-bound-QPlower-bound group code stream under all QP encoding conditions in [QPlower-bound, QPupper-bound] scope respectively at server, and corresponding Δ QP group parallax code stream.The big more compression efficiency of QP value is high more, and the reconstruction quality of decoding end image is low more; Otherwise compression efficiency is low more, and reconstruction quality is high more.The QP span is different according to the coding standard that system adopted.
Step S304, server carries out free viewpoint video or multi-view point video signal are carried out video coding again according to the QP that confirms, and adopts to coded frame weight size and carry out the adaptive strategy that code stream sends.Specifically be that server carries out special processing according to the size of weight to partial frame, comprising:
(3d1) the weight setting is according in the Multilevel B frame encoding scheme, is positioned at the low more frame of level number, and its weight is high more; Otherwise, being positioned at the high more frame of level number, its weight is low more.Level number be integer 0,1,2 ..., W}, its corresponding weight then be W, W-1, W-2 ..., 0}.
(3d2) network congestion grade is made as { B corresponding to the level number number
0, B
1, B
2..., B
W, B wherein
0Congestion condition is higher than B
1, successively decrease successively subsequently, to B
WThe time network congestion situation minimum.
(3d3) corresponding to the different congestion level B of network
x, (a) for real-time system, to the weight scope be W, W-1, W-2 ..., all frames of W-x} are encoded, and send code stream to client; (b) for off-line system, extract the weight scope and be W, W-1, W-2 ..., the code stream of all frames correspondences of W-x}, and send to client.
Step S305, client adopts recognition strategy that it is judged after the code stream that receives through Network Transmission, carries out image reconstruction according to judged result then.In one embodiment of the invention, client also need be given server with feedback information such as network congestion situations.Specifically comprise:
1, recognition strategy is mainly used in and judges that whether this is given up and whether is the error frame that can not decode for the frame code stream receive.The practical implementation step is described below according to following link:
(3e1) strategy of giving up in the frame recognition strategy determines according to current network congestion condition and the presently used equipment operational capability of user jointly.
(3e2), the network congestion situation is defined as { B corresponding to the setting of server
0, B
1, B
2..., B
W, B wherein
0Congestion condition is higher than B
1, successively decrease successively subsequently, to B
WThe time network congestion situation minimum.
(3e3) be in different congestion level B when network
xThe time, client only receive the weight scope for W, W-1, W-2 ..., all frame code streams of W-x}.If also have the code stream of other weights to be sent to client, then all give up.
(3a4) its operational capability of the presently used equipment of user also to client receive the weight scope W, W-1, W-2 ..., W-x} influences to some extent, operational capability is poor more, the weight scope is more little.
2, decoding and error concealment
(3f1) when the frame that receives be error frame, in the time of can not correctly rebuilding the image of current time, then it is all given up with constantly frame, the pairing code check of parallax frame of all viewpoints, and the frame and the parallax frame of the corresponding viewpoint of previous moment is copied to current time.
(3f2) copy of repetition (3f1) is until the correct decoding and rebuilding of frame ability of current time, perhaps till the weight of the weight of current time frame greater than the previous moment frame.
3, viewpoint is selected and virtual viewpoint rendering
Drawing viewpoints virtual view based on decoded result and user's selection.
As shown in Figure 1, this system comprises server 101 and one or more clients 102.Server 101 is used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal; And the visual quality of calculating virtual visual point image; And confirm quantization parameter QP according to the visual quality of said virtual visual point image; The said QP that confirms with basis carries out encoding compression again to said free viewpoint video or multi-view point video signal, and sends to one or more clients.Client 102 is used to receive said server transmitted stream, and according to the virtual view that the user selects, utilizes video and corresponding parallax to carry out the drafting of virtual view, and drawing result is shown to the user.
As shown in Figure 4, be the structure chart of the server of the embodiment of the invention.This server 101 comprises encoder 200 and QP adjusting module 203.Encoder 200 is used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and according to quantization parameter QP said free viewpoint video or multi-view point video signal is carried out encoding compression again, and sends to one or more clients.QP adjusting module 203 is used to calculate the visual quality of virtual visual point image, and adjusts said QP according to the visual quality of said virtual visual point image.
In one embodiment of the invention, server 101 also comprises user and network condition feedback module 204, is used to receive user's feedback, and the feedback of network congestion situation.QP adjusting module 203 also is used for adjusting said QP according to said network congestion situation.
In one embodiment of the invention, encoder 200 comprises online encoder 201 and off-line encoder 202.
In one embodiment of the invention, server 101 comprises that also weight is provided with module 205, is used for selecting the frame of respective weights to send or encoding according to the network congestion grade.
As shown in Figure 5, be the structure chart of the client of the embodiment of the invention.Client comprises weight determination module 301, decoding and error concealment module 302 and viewpoint selection and virtual viewpoint rendering module 303.Whether weight determination module 301 is used to judge whether the frame code stream that receives should be rejected and is the error frame that can not decode.Decoding is used for when the frame that receives is error frame with error concealment module 302; Then with its frame of video with all viewpoints of the moment; The pairing code check of parallax frame is all given up, and the frame of video and the parallax frame of the corresponding viewpoint of previous moment is copied to current time, and repeats copy; Until the correct decoding and rebuilding of frame ability of current time, perhaps the weight of current time frame is greater than the weight of previous moment frame.Viewpoint is selected to be used for the drawing viewpoints virtual view according to decoded result and user's selection with virtual viewpoint rendering module 303.
The present invention can adjust QP according to the visual quality of virtual visual point image, thereby makes the resulting picture quality of client have higher level.In a preferred embodiment of the invention, can be optimized QP simultaneously, thereby can further guarantee the quality of the image that client obtains according to visual quality and network congestion situation.
Although illustrated and described embodiments of the invention; For those of ordinary skill in the art; Be appreciated that under the situation that does not break away from principle of the present invention and spirit and can carry out multiple variation, modification, replacement and modification that scope of the present invention is accompanying claims and be equal to and limit to these embodiment.
Claims (11)
1. free viewpoint video or multi-view point video transmission method based on a visual quality is characterized in that, may further comprise the steps:
Server receives original or free viewpoint video after encoding and decoding or multi-view point video signal;
Said server calculates the visual quality of virtual visual point image, comprising:
Is that N * N carries out piecemeal to resulting virtual visual point image by block size, and wherein, every row has I piece, whenever shows J piece, wherein, and N=2
n, n>=6 wherein,
To each the piece B in the said virtual visual point image
V Ij, wherein, 1≤i≤I and i are integer, 1≤j≤J and j are integer, utilize the block matching method based on least mean-square error MSE criterion, in its left and right sides visual point image, find corresponding best matching blocks B by horizontal direction respectively
L IjAnd B
R Ij, wherein, B
L IjIn left visual point image, B
R IjIn right visual point image,
To resulting B
V IjCarry out the Contourlet wavelet decomposition with the best matching blocks that obtains, to obtain U high-frequency sub-band piece B
U, ij X, wherein, u=1,2 ..., U and a low frequency piece C
Ij X, wherein, X ∈ V, and R, L}, wherein, V is that virtual visual point image, R are that right visual point image, L are left visual point image,
Each high-frequency sub-band piece B that decomposition is obtained
U, ij XFurther adopt u-layer directional filter component to separate, obtain 2
uIndividual subband piece C
U, o, ij X, wherein, o=1,2 ..., 2
u,
Calculate each subband piece C respectively
U, o, ij XVariances sigma
U, o, ij X,
According to σ
U, o, ij XObtain the σ of entire image
U, o X=∑
I, j(σ
U, o, ij X),
Calculate said virtual visual point image and left and right visual point image and decomposing the error parameter δ between the ordinal number with layer
U, o=min{| σ
U, o V-σ
U, o R|, | σ
U, o V-σ
U, o L|,
Error parameter to each layer carries out weighting integration Δ=∑
U, o2
-uδ
U, o,
Calculate the visual quality of said virtual visual point image
C wherein
0Be normal real number regulatory factor, scope is C
0∈ [1,10];
Said server is confirmed the quantization parameter QP of said free viewpoint video or multi-viewpoint video signal code according to the visual quality of said virtual visual point image, and wherein, the visual quality S of said virtual visual point image is high more, and the QP that then selects is low more; With
Said server carries out encoding compression according to the said QP that confirms to said free viewpoint video or multi-view point video signal again, and sends to one or more clients.
2. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 1; It is characterized in that said server adopts multi-view point video encoding method or single view method for video coding that said free viewpoint video or multi-view point video signal are compressed.
3. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 1 is characterized in that the optional scope of QP is [a; B] between integer; A wherein, b >=0, then QP can adopt the linear or nonlinear one by one mapping relations relevant with S to adjust; Its basic principle is that the S value is high more, and the QP value is low more.
4. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 1 is characterized in that, said server adopts online coding or off-line coded system that said free viewpoint video signal or multi-view point video are encoded.
5. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 1 is characterized in that, also comprise:
Said server is adjusted said QP according to the network congestion situation, and wherein, the network congestion situation is serious more, the Δ QP of adjustment
iBe worth big more.
6. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 5 is characterized in that, also comprise:
Said server congestion level Network Based is selected the frame of respective weights to send or is encoded.
7. free viewpoint video or multi-view point video transmission method based on visual quality as claimed in claim 6 is characterized in that, also comprise:
Whether said client is judged whether the frame code stream that receives should be rejected and is the error frame that can not decode;
When if the frame that receives is error frame, with the frame of video of all viewpoints constantly, all give up by the pairing code check of parallax frame with it for then said client, and the frame of video and the parallax frame of the corresponding viewpoint of previous moment is copied to current time;
Said client repeats copy, and until the correct decoding and rebuilding of frame ability of current time, perhaps the weight of current time frame is greater than the weight of previous moment frame.
8. free viewpoint video or multi-view point video transmission server based on a visual quality is characterized in that, comprise encoder and QP adjusting module,
Said encoder is used to receive original or free viewpoint video after encoding and decoding or multi-view point video signal, and according to quantization parameter QP said free viewpoint video or multi-view point video signal is carried out encoding compression again, and sends to one or more clients;
Said QP adjusting module is used to calculate the visual quality of virtual visual point image, comprising:
Is that N * N carries out piecemeal to resulting virtual visual point image by block size, and wherein, every row has I piece, whenever shows J piece, wherein, and N=2
n, n>=6 wherein,
To each the piece B in the said virtual visual point image
V Ij, wherein, 1≤i≤I and i are integer, 1≤j≤J and j are integer, utilize the block matching method based on least mean-square error MSE criterion, in its left and right sides visual point image, find corresponding best matching blocks B by horizontal direction respectively
L IjAnd B
R Ij, wherein, B
L IjIn left visual point image, B
R IjIn right visual point image,
To resulting B
V IjCarry out the Contourlet wavelet decomposition with the best matching blocks that obtains, to obtain U high-frequency sub-band piece B
U, ij X, wherein, u=1,2 ..., U and a low frequency piece C
Ij X, wherein, X ∈ V, and R, L}, wherein, V is that virtual visual point image, R are that right visual point image, L are left visual point image,
Each high-frequency sub-band piece B that decomposition is obtained
U, ij XFurther adopt u-layer directional filter component to separate, obtain 2
uIndividual subband piece C
U, o, ij X, wherein, o=1,2 ..., 2
u,
Calculate each subband piece C respectively
U, o, ij XVariances sigma
U, o, ij X,
According to σ
U, o, ij XObtain the σ of entire image
U, o X=∑
I, j(σ
U, o, ij X),
Calculate said virtual visual point image and left and right visual point image and decomposing the error parameter δ between the ordinal number with layer
U, o=min{| σ
U, o V-σ
U, o R|, | σ
U, o V-σ
U, o L|,
Error parameter to each layer carries out weighting integration Δ=∑
U, o2
-uδ
U, o,
Calculate the visual quality of said virtual visual point image
C wherein
0Be normal real number regulatory factor, scope is C
0∈ [1,10],
And adjust the said QP of above-mentioned free viewpoint video or multi-viewpoint video signal code according to the visual quality of said virtual visual point image, and wherein, the visual quality S of said virtual visual point image is high more, and the QP that then selects is low more.
9. server as claimed in claim 8 is characterized in that, also comprises user and network condition feedback module, is used to receive user's feedback, and the feedback of network congestion situation;
Said QP adjusting module also is used for adjusting said QP according to said network congestion situation.
10. like claim 8 or 9 described servers, it is characterized in that said encoder comprises online encoder and off-line encoder.
11. server as claimed in claim 8 is characterized in that, comprises that also weight is provided with module, is used for selecting the frame of respective weights to send or encoding according to the network congestion grade.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010227359 CN101895752B (en) | 2010-07-07 | 2010-07-07 | Video transmission method, system and device based on visual quality of images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010227359 CN101895752B (en) | 2010-07-07 | 2010-07-07 | Video transmission method, system and device based on visual quality of images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101895752A CN101895752A (en) | 2010-11-24 |
| CN101895752B true CN101895752B (en) | 2012-12-19 |
Family
ID=43104796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010227359 Active CN101895752B (en) | 2010-07-07 | 2010-07-07 | Video transmission method, system and device based on visual quality of images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101895752B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015505196A (en) * | 2011-12-15 | 2015-02-16 | トムソン ライセンシングＴｈｏｍｓｏｎ Ｌｉｃｅｎｓｉｎｇ | Method and apparatus for video quality measurement |
| US10499056B2 (en) * | 2016-03-09 | 2019-12-03 | Sony Corporation | System and method for video processing based on quantization parameter |
| CN106170081B (en) * | 2016-06-28 | 2017-12-12 | 上海米影信息科技有限公司 | A kind of wireless dummy reality server, system and its data compression transmission method |
| WO2018103034A1 (en) * | 2016-12-08 | 2018-06-14 | 深圳市大疆创新科技有限公司 | Image transmission method, apparatus, and unmanned aerial vehicle |
| CN113344069B (en) * | 2021-05-31 | 2023-01-24 | 成都快眼科技有限公司 | Image classification method for unsupervised visual representation learning based on multi-dimensional relation alignment |
| CN115225961B (en) * | 2022-04-22 | 2024-01-16 | 上海赛连信息科技有限公司 | No-reference network video quality evaluation method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1471319A (en) * | 2002-07-22 | 2004-01-28 | 中国科学院计算技术研究所 | Association rate distortion optimized code rate control method and apparatus thereof |
| CN101277454A (en) * | 2008-04-28 | 2008-10-01 | 清华大学 | Method for generating real time tridimensional video based on binocular camera |
| CN101466038A (en) * | 2008-12-17 | 2009-06-24 | 宁波大学 | Method for encoding stereo video |
| WO2010024919A1 (en) * | 2008-08-29 | 2010-03-04 | Thomson Licensing | View synthesis with boundary-splatting |
-
2010
- 2010-07-07 CN CN 201010227359 patent/CN101895752B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1471319A (en) * | 2002-07-22 | 2004-01-28 | 中国科学院计算技术研究所 | Association rate distortion optimized code rate control method and apparatus thereof |
| CN101277454A (en) * | 2008-04-28 | 2008-10-01 | 清华大学 | Method for generating real time tridimensional video based on binocular camera |
| WO2010024919A1 (en) * | 2008-08-29 | 2010-03-04 | Thomson Licensing | View synthesis with boundary-splatting |
| CN101466038A (en) * | 2008-12-17 | 2009-06-24 | 宁波大学 | Method for encoding stereo video |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101895752A (en) | 2010-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101895753B (en) | Network congestion degree based video transmission method, system and device | |
| Guo et al. | Distributed multi-view video coding | |
| CN101895752B (en) | Video transmission method, system and device based on visual quality of images | |
| Belyaev et al. | Robust vehicle-to-infrastructure video transmission for road surveillance applications | |
| JP2006528870A (en) | System and method for foregoed video coding and transcoding for mono or stereo images | |
| Zhu et al. | Error-resilient and error concealment 3-D SPIHT for multiple description video coding with added redundancy | |
| US8411743B2 (en) | Encoding/decoding system using feedback | |
| Fujihashi et al. | FreeCast: Graceful free-viewpoint video delivery | |
| Verdicchio et al. | Embedded multiple description coding of video | |
| US10469824B2 (en) | Hybrid digital-analog coding of stereo video | |
| WO2001095512A1 (en) | System and method for object-oriented video processing | |
| Hemami | Digital image coding for robust multimedia transmission | |
| Yip et al. | Joint source and channel coding for H. 264 compliant stereoscopic video transmission | |
| Gao et al. | Texture-Distortion-Constrained joint source-channel coding of multi-view video plus depth-based 3D video | |
| Wang et al. | Important macroblock distinction model for multi-view plus depth video transmission over error-prone network | |
| Li et al. | Soft transmission of 3D video for low power and low complexity scenario | |
| Guo et al. | Free viewpoint switching in multi-view video streaming using Wyner-Ziv video coding | |
| Tan et al. | Adaptive frame level rate control in 3D-HEVC | |
| Kim et al. | Error-resilient 3-D wavelet video coding with duplicated lowest sub-band coefficients and two-step error concealment method | |
| Kung et al. | Packet video transmission over wireless channels with adaptive channel rate allocation | |
| Ouddane et al. | Asymmetric stereoscopic images coding using perceptual model | |
| Chang et al. | Accurate bitrate model and greedy-based rate controller for low delay video transmission | |
| Huo et al. | Tree-structured multiple description coding for multiview mobile tv and camera-phone networks | |
| Akhtarkavan et al. | Multiple descriptions video coding using coinciding lattice vector Quantizer for H. 264/AVC and motion JPEG2000 | |
| Akhtarkavan et al. | Multiple descriptions coding for H. 264/AVC using coinciding A 2 lattice vector quantizer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |