CN101888553B - Scalable video coding method and device - Google Patents
Scalable video coding method and device Download PDFInfo
- Publication number
- CN101888553B CN101888553B CN 201010216832 CN201010216832A CN101888553B CN 101888553 B CN101888553 B CN 101888553B CN 201010216832 CN201010216832 CN 201010216832 CN 201010216832 A CN201010216832 A CN 201010216832A CN 101888553 B CN101888553 B CN 101888553B
- Authority
- CN
- China
- Prior art keywords
- frame
- video
- low resolution
- video frame
- resolution video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention relates to a scalable video coding method and a scalable video coding device. Particularly, the invention describes the scalable video coding method and a hierarchical video representation. The hierarchical video representation acquires a better video quality by using a more effective bit stream representation and scalability and meets the standards of various codecs from a base layer to a reinforcement layer so as to bring least modification to the conventional hardware or systems in the field.
Description
Technical field
Present invention relates in general to video coding, particularly scalable video (SVC).
Background technology
Scalability is the desired characteristics of many multimedia application, for example, and end-to-end (P2P) live video stream, multipart video-meeting and point-to-point video session.For example, in P2P content stream, the content provider expects to provide by different prices the content of multimedia of different quality, and for example, the low resolution preview versions of content of multimedia can be free.Can according to spatial scalability, quality scalability, time scalability or even the scalability of combination weigh scalability, the scalability of this combination generally refers to the combination of space, quality and time scalability.The subclass representative that spatial scalability and time scalability have been described bit stream respectively has the situation of source contents of picture size (spatial resolution) or the frame per second (temporal resolution) of reduction.Quality scalability has been described the spatial and temporal resolution identical with full bit stream and has been possessed the situation sincere than minimum living, and wherein fidelity factor refers to signal to noise ratio (snr) usually off the record.
Under the situation of P2P network, also produce demand to scalability, wherein, depend on such as factors such as data-handling capacity, display capabilities and network conditions, different users can have the different operation ability.For example, under the situation of real-time video broadcasting, expect that each user can both receive and see at least some things for good user experience.This just requires multimedia application is telescopic.
It is telescopic not making video bit stream, and on the contrary, a plurality of video bit streams that the different quality level can be provided also satisfy changeable terminal capability or network condition simultaneously to satisfy user's various demands or operation.Yet this will have too many independent vide data and transmit simultaneously, and can not effectively utilize the efficient of user's input and output bandwidth.In other words, the whole video quality that receives will not be best.
Therefore; Scalability is important characteristic; And through such as the method based on all or part of low layer (low-qualityer video) high level of encoding (video with better quality), scalable video allows the truncation part video bit stream so that video bit stream adapts to various situations, and said method is as being disclosed in: Y.Cui and K.Nahrstedt; Layered peer-to-peer streaming; Proc.NOSSDAV ' in June, 03,2003, by reference its integral body is incorporated herein at this.The video that different layer representatives has different quality is to satisfy various demands.
Video encoding standard was supported the video content scalability a lot of years.There are many existing means to be adopted by standard, such as:
H.262/MPEG-2 video; As be disclosed in: Generic Coding of Moving Pictures andAssociated Audio Information Part 2: video; ITU-T Rec.H.262 and ISO/IEC13818-2 (MPEG-2 video); ITU-T and ISO/IEC JTCl, in November, 1994.
H.263, as be disclosed in: Video Coding for Low Bitrate Communication, ITU-TRec.H.263, ITU-T, version 1:1995 November, version 2: in January, 1998, version 3: in November, 2000;
The MPEG-4 vision; As be disclosed in: Coding of audio-visual object-Part 2: vision; ISO/IEC 14492-2 (MPEG-4 vision), ISO/IEC JTC 1, version 1:1999 April; Version 2: in February, 2000, version 3: in May, 2004 (hereinafter " audio frequency and video object coding-second portion "); And
H.264/MPEG-4AVC, as be disclosed in: J.Reichel, H.Schwarz, T.Wiegand, G.J.Sullivan and M.Wien, Joint Draft 11 of SVC Amendment, joint video team, Doc.JVT-X201, in July, 2007.
By reference the integral body of the above document of quoting is incorporated herein at this.
Various effort are poured in scalable video.United States Patent (USP) 6,639,943 have described fine granularity scalable (FGS) coding of the enhancement layer in the scalable video, comprising the new Code And Decode of hierarchical coding video.Usually at first encode basic layer.Based on basic layer residual coding enhancement layer residual.
U.S. Patent application 2007/0160133 has been described the generation that allows 4 bit streams of space and quality scalability.First and basic bit stream, basic layer is the QCIF video of 96Kbps.Also produce the bit stream that another one has identical QCIF resolution according to this basic layer.The target bit rate of this layer is 192Kbps.For obtaining higher visual quality, also produced the bit stream of another one CIF resolution.Because owing to the restriction of bit stream makes its quality maybe be good inadequately, so the CIF bit stream of the additional higher bit rate of further having encoded is to obtain best resolution and SNR quality.The method has combined spatial scalability and FGS.Yet the resolution of enhancement layer still depends on the interior of low-resolution layer and inserts.
U.S. Patent application 2006/0233241 and 2004/0264567 has been described the use wavelet transformation and has been adopted the acquisition scalability.The method is divided into 4 sub-frame with 1 frame, and uses estimation to compress the similitude between them.
United States Patent (USP) 7,292,635 have described the scalable data coding method that utilizes wavelet transformation.With filtering application in single image influencing coding efficiency, and any scalability is not provided.
Above-mentioned list of references has disclosed: the similitude that some existing methods use the 3-D wavelet transformation when scalability is provided, to remove room and time as much as possible.S.J.Choi and J.W.Woods, Motion-compensated 3-D subband coding of video, IEEE Trans discloses another example.Image processing, the 8th volume, the 2nd phase, the 155-167 page or leaf, is incorporated herein its integral body at this by reference in February, 1999.Some existent method advocate to use the information of low layer, comprise residual error, movable information and reconstruction frames, and these information provide reference for reducing high-rise coding entropy." audio frequency and video object coding-second portion " discloses another example.
Yet above-mentioned method seldom has by industry and adopts widely, because many restrictions of the complexity that their application is realized.For example, because existing method requires the modification fully of existing coder to support scalability, so caused complexity.Have no modification, although can utilize high performance hardware, yet existing scalable video coding method limited subscriber uses existing coder only to check basic layer.In fact, under standard situation H.264, have been found that and H.264 adopt very difficulty of scalable video, especially aspect space and quality retractility.
In addition, because existing method requires to depend on the Code And Decode that other lower-level frame are carried out lower-level frame, so also caused complexity.In other words, can not independently decode video bit stream with each low resolution of reconstruct.
In this area, still keeping improved technology requirement to using scalability to come encoded video, special, in order to meet existing coder, do not revise or minimal modification arranged and have.
Summary of the invention
First aspect of the present invention is to the user who is equipped with existing decoder (as H.264) visual quality as much as possible and layer to be provided.The present invention does not carry out complicated modification to existing coder just provides spatial scalability and quality scalability.
Second aspect of the present invention is for scalable video coding method high code efficiency to be provided.Conventional method use different quantized step-length produces has equal resolution but the layer of different SNR values; Different with conventional method, the present invention is through using the low pass filter with the controlled cut-off frequency of order and produce and have equal resolution but the frame of different SNR values to having high-resolution frame.
The third aspect of the invention provides scalable video coding method, and this method allows not rely on the subclass of other low-resolution video bags with regard to decodable all low-resolution video bags.Therefore, reduced the complexity of coding, even and in other low-resolution video bags, corrupted data is arranged or lose, still can regain the subclass of all such low-resolution video bags reliably.
An aspect further is to improve whole visual qualities for the user.For example, through using filtering but not use and quantize to eliminate detailed information, thereby suppress blocking artifact.Because basic layer need not rely on not sampling operation or wavelet inverse transformation to combine basic layer information and enhanced layer information; Just can be used to predict the higher level that also is considered to enhancement layer; So through allowing basic layer (low resolution; The double sampling version of primitive frame) more difference is arranged, also can obtain better visual quality.Therefore, has more sharp keen characteristic (better video quality) by the basic layer video that the present invention produced than the basic layer video that obtains from existing multiple dimensioned coder method.
Also aspect of the present invention is a video flowing that several layers can be provided satisfying various application demands, and still keeps compression efficiency and standard compliant scalable video coding method.
The present invention to from the high-quality frame of video of the high-resolution of input video (HRHQ frame of video) application of low-pass filters to produce the low-quality frame of video of high-resolution.The low-quality frame of video of this high-resolution (HRLQ frame of video) is carried out double sampling to produce a plurality of low resolution video frame (LR frame of video).This low pass filter has the cut-off frequency higher than anti aliasing frequency, so that through suppressing blocking artifact and improving visual quality for low resolution video frame keeps more information.Simultaneously, this cut-off frequency is set to avoid in low resolution video frame, will having the problem of too many aliasing under certain restriction.With the mode that adapts at least one existing coder standard this low resolution video frame of encoding, and one or more this low resolution video frame is with reference to other low resolution video frame.
This work is different with existing encoding scheme.Current flexible coding method is revised encoder usually, so their conformance with standard no longer just.The present invention overcome Video Decoder (as H.264/AVC or MPEG4) do not support the restriction of scalability (space and quality scalability), and the basic layer of only decoding that can be correct.Therefore, the present invention allows system to have more decoded layer, comprises basic layer and enhancement layer, simultaneously, has obtained better quality owing to having reduced blocking artifact.
The present invention uses useful to any multi-media network.It is effective especially to the network application with non-homogeneous situation, and for example, P2P (end-to-end) video flowing, P2P file are downloaded and three screens (mobile device, PC and TV) are used.In P2P used, each user came work as server, so that the computing capability of I/O bandwidth between user and the user and PC is different.The telescopic video content allows more effectively to use all resources, is each user possible video quality that offers the best simultaneously.
The present invention also can be applicable to pass the media content transmission platform of mobile network or broadband network, for example, is used for the iShare P2P levelling platform of real-time stream and video request program (VoD).The present invention can provide IPTV application, internet video stream and multimedia communication to use, such as video conference, video exchange and enterprise's video server.
Can also the present invention be transferred to video coder user as the Personal Numbering module, for example, multimedia system developer team or solution provider.
Through illustrating of following execution mode, other aspects of the present invention are also disclosed.
Description of drawings
According to following accompanying drawing, below with these and other purposes, aspect and execution mode of more detailed this claimed invention of description, wherein:
Fig. 1 has described the commonplace components with scalable video ability according to some execution mode.
Fig. 2 has described the flow chart that during scalable video, generates LR frame of video and HRLQ frame of video from the HRHQ frame of video according to some execution mode.
Fig. 3 has described the diagram that in the reduction process, generates the LR frame of video from the HRLQ frame of video according to some execution mode.
Fig. 4 has described the flow chart according to the scalable video coding method of some execution mode.
Fig. 5 has described according to the coded reference between the LR frame of video of some execution mode and has concerned.
Fig. 6 A-6E shows cataloged procedure.
Fig. 7 has described in network to user's transmission of video sequence.
Embodiment
Fig. 1 has described the commonplace components with scalable video ability according to some execution mode of the present invention.This commonplace components 130 has one or more processors 110, and it carries out the function of for example controlling and handling.This commonplace components 130 further comprises one or more memory cell, and it stores one or more programs.This program is configured to carried out by said one or more processors 110, and comprises according to the instruction with this disclosed scalable video coding method of the present invention.
Fig. 2 has described the flow chart that during scalable video, generates LR frame of video and HRLQ frame of video from the HRHQ frame of video according to some execution mode.The source video also is considered to the layer of high-resolution high-quality (HRHQ).HRHQ frame of video 210 is the frame of video at the HRHQ layer.Scalable encoder can generate several streams by video simultaneously from the source.For example, low pass filter 220 is applied to HRHQ frame of video 210 to generate the frame of video 230 of high-resolution low quality (HRLQ).At an execution mode of the video bit stream that is used for having low bit rate, low pass filter 220 is [1 ,-2,0,8 ,-10 ,-20,74,154,74 ,-20 ,-10,8,0 ,-2,1]/256.At an execution mode of the video bit stream that is used for having high bit rate, low pass filter 220 is [1 ,-2,0,8 ,-10 ,-20,74,154,74 ,-20 ,-10,8,0 ,-2,1]/256.At another execution mode of the video bit stream that is used for having high bit rate, low pass filter 220 is optional and can ignore.Carry out double sampling to generate one group of low resolution (LR) frame of video 250 by 240 pairs of HRLQ frame of video of double sampling device 230; In this example, the member of this group LR frame of video comprises the LR frame of video with the pixel that is labeled as " 1 ", the LR frame of video with the pixel that is labeled as " 2 ", LR frame of video with the pixel that is labeled as " 3 " and the LR frame of video with the pixel that is labeled as " 4 ".
For example, having from those that at least one LR video bit stream that generates from the LR frame of video of pixel each period, that be labeled as " 1 " is construed to is basic layer, and the coding result of LR frame of video 250 only depends on other frame of video in the LR layer.Can and need not support the decoder of scalability to decode this basic layer through any conformance with standard.Other LR video bit streams; For example; Have bit stream from those, and have bit stream, be called as enhancement layer from the LR frame of video generation of pixel each period, that be labeled as " 3 " from those from the LR frame of video generation of pixel each period, that be labeled as " 2 ".Spatial scalability permission system comprises at least 2 layers, and wherein, basic layer has lower resolution.With after enhanced layer information combines, can make up and have identical content but more high-resolution video.Quality scalability permission system comprises at least 2 layers, and wherein, basic layer has lower SNR value.With after enhanced layer information combines, can rebuild and have identical content and equal resolution but the better video of SNR.In some embodiments, through repeat one or many double sampling process to existing LR frame of video, can obtain one or more additional LR frame of video.In each double sampling process, can be with application of low pass filters to will be by the frame of video of double sampling to limit bit rate before the double sampling.
In one embodiment, level all is 2 with vertical scale factor.In LR frame of video 250, allow to occur some aliasings.Therefore, low pass filter 220 does not use the anti aliasing cut-off frequency of being less than or equal to 0.5 π, and uses the cut-off frequency that is higher than 0.5 π.In one embodiment, have at video bit stream under the situation of low bit rate, 0.6 π is used as the cut-off frequency of low pass filter.In another embodiment, have at video bit stream under the situation of high bit rate, 0.9 π looks the cut-off frequency as low pass filter.In another embodiment, have at video bit stream under the situation of high bit rate, do not use low pass filter.Yet cut-off frequency can not be too high and be maintained in certain limited field, so that the aliasing effect in the low resolution video frame can be too not remarkable.For example, have at video bit stream under the situation of high bit rate, if low pass filter uses 0.6 π as cut-off frequency, then video output will be very fuzzy.For example, have at video bit stream under the situation of low bit rate, if because low pass filter uses 0.9 π as cut-off frequency, then by quantization parameter control bit rate, so video output will be full of blocking artifact.
Fig. 3 has described the diagram that in double sampling, generates the LR frame of video from the HRLQ frame of video according to some execution mode of the present invention.In one embodiment, level all is 2 with vertical scale factor.The resolution of supposing double sampling process HRLQ frame of video 300 before is A * B, and in this particular example, it is 6 * 6.This HRLQ frame of video 300 is divided into a plurality of 2 * 2 piece.Each piece of 2 * 2 comprises 4 pixels, and each pixel all adheres to different groups separately.For illustrative purposes, with identical figure notation each pixel from same group.For example; In 2 * 2 of in HRLQ frame of video 300 each, the pixel that is positioned at the upper left corner is marked as " 1 ", and the pixel that is positioned at the upper right corner is marked as " 2 "; The pixel that is positioned at the lower left corner is marked as " 3 ", and the pixel that is positioned at the lower right corner is marked as " 4 ".The double sampling process is that each 4 * 4 selection from HRLQ frame of video 300 belong to not all pixels on the same group, and they are gathered the LR frame of video.Therefore, the resolution of the LR frame of video of being made up of all pixels " 1 " 310 will be A/2 * B/2, and in this particular example, it is 3 * 3.Similarly; After the double sampling, the LR frame of video of forming by the pixel of all being labeled as " 2 " 320, the LR frame of video of forming by the pixel of all being labeled as " 3 " 330 and will also have the resolution of A/2 * B/2 respectively by the LR frame of video 340 that the pixel of all being labeled as " 4 " is formed.
During decoding, if can only obtain LR frame of video 310, then still can under the situation that does not rely on other LR frame of video, make up the LR video, and only just also can make up high-resolution video through interior slotting LR frame of video 310 in each period.
Further double sampling is possible with ordering.In some embodiments, further the level of double sampling and vertical scale factor can be for any greater than 1 integers, and need be not identical with the scale factor of former double sampling.Before double sampling, can application of low pass filters be arrived the LR frame of video, for example, LR frame of video 310,320,330,340.For example, LR frame of video 310 further double samplings are become the LR frame of video 311 of a plurality of low resolution.The raster scan sequence of LR frame of video 310 is followed in the ordering of these LR frame of video 311.LR frame of video 320 further double samplings are become the LR frame of video 321 of a plurality of low resolution.The raster scan sequence of LR frame of video 320 is followed in the ordering of LR frame of video 321.LR frame of video 330 and 340 is repeated similar process.LR frame of video 311 will be encoded, and LR frame of video 321 will be encoded subsequently, or the like.
Fig. 4 has described the flow chart according to the scalable video coding method of some execution mode.This scalable video coding method coding source video; It comprises by 420 represented double sampling high-resolution video frames; To generate a plurality of low resolution video frame; The quantity of low resolution video frame is 1 between the scope of N, and wherein N is the integer more than or equal to 2, and each in N low resolution video frame is at each given period t
iOccur during this time, wherein i is from the integer 1 to n the scope, and n is the sum of needed period of coding source video; And also comprise by 430 represented ordering N low resolution video frame, so that the rank order of any given low resolution video frame in should N low resolution video frame of group is from t
1To t
nRemain unchanged in period in the scope.Make up and to organize N low resolution video frame; So that during decoding subsequently should be organized low resolution video frame; Receiving equipment can be selected the subclass x of N low resolution video frame, and it is used for producing the video corresponding to the source video at receiving equipment, and wherein x is from the integer 1 to N the scope.
For this application aims, the particular sequence order of statement " rank order " expression low resolution video frame.For example, as shown in Figure 3, the rank order of first frame is generally " 1 ", and second frame is " 2 ", and the 3rd frame is " 3 ", and the 4th frame is " 4 ".When during the rank order of frame 1-4 is being encoded, remaining unchanged, at each period t
i, frame 1-4 is always by 1,2,3,4 orderings.The quantity that of course it is to be understood that low resolution video frame arrives between the scope of N 1, and wherein N is the integer more than or equal to 2, and t
iArrive between the scope of n 1, wherein n is the sum that is used for the period of coding source video.
For obtaining the more frame of video of low resolution, can repeat double sampling 420 and ordering subsequently 430 to low resolution video frame, make like this in each repetitive operation, can produce additional enhancement layer (those are the frame of video of low resolution more).Can obtain a plurality of enhancement layers through repetitive operation repeatedly.
In other words; This scalable video coding method can be selectively further each in the double sampling N low resolution video frame; Producing one group of M low resolution video frame, it is corresponding in N the low resolution video frame each, and wherein M is the integer more than or equal to 2; And can selectively sort to M low resolution video frame of this group, so that the rank order that should organize each member in M the low resolution video frame is at each period t
iAll remain unchanged, keep rank order simultaneously corresponding to N low resolution video frame of M low resolution video frame of this group.This scalable video coding method can should be organized M low resolution video frame to produce according to selectable further each in the double sampling M low resolution video frame of indication.
Before double sampling 420, can be with application of low pass filters to each high-resolution video frame, to generate the low-quality frame of video of high-resolution.For example, if the bit rate of high-resolution video frame is high, then possibly need not to high-resolution video frame application of low-pass filters.If the bit rate of high-resolution video frame is low, then to high-resolution video frame application of low-pass filters, to strengthen the visual quality of decoded video.
As 440 represented, this scalable video coding method is encoded to low resolution video frame, and in one embodiment, with the mode that meets existing coder (for example, H.264 as described below) low resolution video frame is encoded.
The HRLQ layer also is considered to first enhancement layer, perhaps the intermediate bit rate layer.The present invention encodes according to two purposes: one is to eliminate redundancy as much as possible, and another is to keep existing coder constant.For example; For fear of many modifications of carrying out, need just can not encode by the following characteristic that scalable extension H.264 provides to the HRLQ layer to encoder and decoder: for form up-sampling that prediction carries out the low resolution video frame of rebuilding, to the prediction of macro block model, related kinematic parameter and residual signal.
As all being described in 2 the example at vertical scale factor, generating the process of LQ layer acquisition HRLQ layer from the HR layer with level.Through obtain each of HRLQ frame of video to the HR of correspondence frame of video application of low-pass filters.Because the cut-off frequency of low pass filter is near 0.5 π, so in this frame of video, there are many redundancies.However, owing to two reasons, through the up-sampling to the LR frame of video, the HRLQ frame of video is expendable:
At first, although the cut-off frequency of low pass filter is set to about 0.5 π, be impossible yet make the performance of low pass filter idealized.Therefore, in the output of low pass filter, still there are many radio-frequency components.These radio-frequency components are present in the LR frame of video as aliasing, and can not it be eliminated through simple top sampling method.
Secondly, in spatial domain, exist, for example, insert or bilinear interpolation in the bicubic based on the most convenient polynomial and interpolating method rapidly.Such interpolating method has poor efficiency at high-frequency region.
Therefore, based on LR frame of video, also using the method identical as purpose remaining Pixel Information to be encoded and transmit, with as the low-quality enhancement layer still of the high-resolution among the present invention with that time result's reconstruction as basic layer.
Traditional scalable video coding method up-sampling LR frame of video; And the consequential signal of up-sampling and the difference between the HRLQ frame of video are encoded; Different with this traditional scalable video coding method; The present invention encodes according to following mode in order to be fit to the HRLQ frame of video, and wherein the HRLQ frame of video is a band-limited signal:
Use the example that equals 2 like the scale factor among Fig. 3, in the HR of period t frame of video by P
t(not shown) representes, low pass filter the HRLQ frame of video of period t output by
300 expressions, and the LR frame of video that forms respectively through selecting same group pixel from HRLQ frame of video 300 is represented as follows: by
310 expressive notations are the pixel of " 1 ", by
320 expressive notations are the pixel of " 2 ", by
330 expressive notations are the pixel of " 3 ", by
340 expressive notations are the pixel of " 4 ".This frame of video sorts by following mode:
Ordering is with
:
Direct coding HRLQ sequence of frames of video not, but the HRLQ sequence of frames of video in the coding ordered sequence.Fig. 5 shows the coded reference relation between the LR frame of video.According to following concerning, the ordered sequence of LR frame of video is encoded according to the coded reference between the LR frame of video of some execution mode of the present invention:
LR frame of video with the pixel that is labeled as " 1 " is used as basic layer; And be considered to basic layer video frame; For example, in
frame of video 512 of period t with in period t-1
frame of video 511.Other basic frames of only use difference of each basic frame periods as a reference.For example,
frame of video 512 can be used
frame of video 511 as a reference.If receiver receives only basic frame, then still possibly rebuild low-resolution sequence.
Also other LR frame of video
522,
532 that have mark " 2 ", " 3 " and " 4 " are respectively carried out scalable coding with
542, and these LR frame of video are considered to enhancement layer frame usually.In other words;
frame of video is as basic layer;
frame of video is as first enhancement layer;
frame of video is as second enhancement layer, and
frame of video is as the 3rd enhancement layer.Can only use one or more other frame of video of the layer of being less than or equal to itself LR frame of video of encoding as a reference.Use the encode frame of first enhancement layer of the frame of one or more other first enhancement layers; For example, frame of video 521 as a reference for 522 uses
of
frame of video.Use the encode frame of second enhancement layer of the frame of one or more other second enhancement layers; For example, frame of video 531 as a reference for 532 uses
of
frame of video.Use the encode frame of the 3rd enhancement layer of the frame of one or more other the 3rd enhancement layers; For example, frame of video 541 as a reference for 542 uses
of
frame of video.
In addition; Also can use the encode frame of first enhancement layer of the frame of one or more basic units; For example;
frame of video 522 is used
frame of video 512 as a reference, and
frame of video 521 is used
, and frame of video 511 as a reference.Also can use the encode frame of second enhancement layer of the frame of frame and/or one or more other first enhancement layers of one or more basic layers; For example;
frame of video 532 is used
frame of video 522 and/or
frame of video 512 as a reference, and
frame of video 531 is used
frame of video 521 and/or
, and frame of video 511 as a reference.Also can use the frame of one or more basic layers; And/or the frame of one or more other first enhancement layers; And/or the encode frame of the 3rd enhancement layer of the frame of one or more second enhancement layers; For example;
frame of video 542 is used
frame of video 532 and/or
frame of video 522 and/or
frame of video 512 as a reference, and frame of video 511 as a reference for
frame of video 541 use
frame of video 531 and/or
frame of video 521 and/or
.Therefore,, then under interior slotting help, still can rebuild the LR frame of video, only if it is the frame that suffers the basic layer of packet loss if can not rebuild a LR frame of video owing to packet loss.
Fig. 6 A-6E shows cataloged procedure.In one embodiment, in each period 4 LR frame of video are arranged.Each LR frame of video is 1/4th of a HRLQ frame of video, and according to their spatial distribution, difference Different L R frame of video is constantly sorted out.For example, each the LR frame of video with the upper left corner that is derived from the HRLQ frame of video classifies as a LR frame of video.Each LR frame of video in the upper right corner that is derived from the HRLQ frame of video is classified as the 2nd LR frame of video.Each LR frame of video in the lower left corner that is derived from the HRLQ frame of video is classified as the 3rd LR frame of video.Each LR frame of video in the lower right corner that is derived from the HRLQ frame of video is classified as the 4th LR frame of video.In the first step, shown in Fig. 6 A, will be encoded to I frame (independent frame) 611 in the LR frame of video of moment t.
In second step, shown in Fig. 6 B, will be encoded to P frame (predictive frame) 612 in the 2nd LR frame of video of moment t, will be encoded to P frame 613 in the 3rd LR frame of video of moment t, and will be encoded to P frame 614 in the 4th LR frame of video of moment t.In the 2nd LR frame of video of moment t with reference to a LR frame of video at moment t.In other words, P frame 612 is with reference to I frame 611.In the 3rd LR frame of video of moment t with reference in the 2nd LR frame of video of moment t with in the LR frame of video of moment t.In other words, P frame 613 is with reference to P frame 612 and I frame 611.In the 4th LR frame of video of moment t with reference in the 3rd LR frame of video of moment t, in the 2nd LR frame of video of moment t with in the LR frame of video of moment t.In other words, P frame 614 is with reference to P frame 613, P frame 612 and I frame 611.If in the information dropout of some LR frame of video of different periods or damage, then still can be from the reconstruction HRLQ frame of video of remaining LR frame of video part.For example, when when the 2nd all LR frame of video of different periods and the 3rd LR frame of video are all lost, still possibly rebuild half the HRLQ video sequence from a LR frame of video and the 4th LR frame of video.
In the 3rd step, shown in Fig. 6 C, will be encoded to P frame 631 in the LR frame of video of moment t+2, will be encoded to B frame (bi-directional predicted frames) 621 in the LR frame of video of moment t+1.Thereby, consider all first frame of video from the different periods, then can rebuild basic layer.
In the 4th step, shown in Fig. 6 D, will be encoded to P frame 632 in the 2nd LR frame of video of moment t+2.To be encoded to P frame 633 in the 3rd LR frame of video of moment t+2.To be encoded to P frame 634 in the 4th LR frame of video of moment t+2.In the 2nd LR frame of video of moment t+2 with reference in the LR frame of video of moment t+2 with in the 2nd LR frame of video of moment t.In other words, P frame 632 is with reference to P frame 631 and P frame 612.In the 3rd LR frame of video of moment t+2 with reference in the 2nd LR frame of video of moment t+2, in the LR frame of video of moment t+2 with in the 3rd LR frame of video of moment t.In other words, P frame 633 is with reference to P frame 632, P frame 631 and P frame 613.In the 4th LR frame of video of moment t+2 with reference in the 3rd LR frame of video of moment t+2, in the 2nd LR frame of video of moment t+2, in the LR frame of video of moment t+2 with in the 4th LR frame of video of moment t.In other words, P frame 634 is with reference to P frame 633, P frame 632, P frame 631 and P frame 614.
In the 5th step, shown in Fig. 6 E, will be encoded to B frame 622 in the 2nd LR frame of video of moment t+1.To be encoded to B frame 623 in the 3rd LR frame of video of moment t+1.To be encoded to B frame 624 in the 4th LR frame of video of moment t+1.In the 2nd LR frame of video of moment t+1 with reference in the LR frame of video of moment t+1, in the 2nd LR frame of video of moment t with in the 2nd LR frame of video of moment t+2.In other words, B frame 622 is with reference to B frame 621, P frame 612 and P frame 632.In the 3rd LR frame of video of moment t+1 with reference in the 2nd LR frame of video of moment t+1, in the LR frame of video of moment t+1, in the 3rd LR frame of video of moment t with in the 3rd LR frame of video of moment t+2.In other words, B frame 623 is with reference to B frame 632, B frame 631, P frame 633 and P frame 613.In the 4th LR frame of video of moment t+1 with reference in the 3rd LR frame of video of moment t+1, in the 2nd LR frame of video of moment t+1, in the LR frame of video of moment t+1, in the 4th LR frame of video of moment t with in the 4th LR frame of video of moment t+2.In other words, B frame 644 is with reference to B frame 633, B frame 632, B frame 631, P frame 614 and P frame 634.
In some embodiments, also can one or more additional high-resolution high-quality (HRHQ) layers be transferred to decoder, so that one or more extra plays to be provided.For example, for the I frame, additional HRHQ layer is transferred to decoder.From higher resolution frame, extract high-resolution low quality (HRLQ) frame of rebuilding.Difference further is encoded to an I frame, and difference adds that the reconstruction of the HRLQ frame of video of reconstruction also is the reconstruction of higher resolution frame.
Fig. 7 has described in network to user's transmission of video sequence.In network 750, for example the P2P network has many users 760, and each user 760 can have different powers and functions.Their powers and functions are because following factor and difference: for example, and the variation of network condition and each user's 760 various computing ability.Therefore, the transmission of video sequence need be fit to different users.The video that is transmitted is high-resolution (HR) video sequence 710.Scalable coding is applied to this HR video sequence 710, to convert this HR video sequence 710 into a plurality of extensible video streams 730.Further handle by 740 pairs of extensible video streams 730 of priority configuration.This priority configuration 740 is through coming to be these various priority that are responsible for assigning for the different priority of each bag mark.Through selectable reception priority is 1,2,5,6 bag, and the user 760 with high bandwidth or computing capability can visit the video content of first water.Through selectable reception priority is 1,2,3,4 bag, and the user 760 with lower bandwidth or computing capability can visit low-qualityer video content.Through selectable reception priority is 1,2 bag, and the user 760 with low-down bandwidth or computing capability can visit low-qualityer video content.
In addition, because the space similitude of four LR frames, error concealing method also can be applicable among the present invention preferably.In one embodiment, at coding side, high-resolution video frame (HR frame of video) double sampling is become a plurality of less frame of video.Between these less frame of video, but has the less frame of video of one group of self-demarking code at least.But, but when decoding end received other the less frame of video except that self-demarking code, through using some error concealing method, these frame of video can be used to strengthen the quality of the frame of video of self-demarking code.
Equal at scale factor under 2 the situation, the low pass filter with cut-off frequency 0.5 π is the HR frame of video smoothly.If low pass filter is desirable; The LR frame of video of then having only one group of pixel; For example
frame of video can be carried all low frequency information.Need not use the LR frame of video of the pixel of other groups to rebuild the HRLQ frame of video.Because expectation has HRLQ frame of video quality preferably, so being chosen as, the cut-off frequency of low pass filter is higher than 0.5 π, in the HRLQ layer, keeping more information, and the LR frame of video of the pixel of other groups will become more useful to the reconstruction of HRLQ frame of video.Yet the cut-off frequency of low pass filter can not be too high, and it is too many to surpass 0.5 π, and this is because in the LR frame of video, too many aliasing will be arranged, and because the picture noise of aliasing performance, so aliasing is with the adverse influence coding efficiency.
For the selection of the cut-off frequency of low pass filter, can consider the above thing of considering, although this cut-off frequency still depends on video quality very much.In addition,, even maybe not need use filtering operation, because this capture device possibly eliminated high-frequency information to them for some sequences of catching by low-quality camera.Video for having compressed with medium bit rate possibly not need filtering operation, because such compression is also as a kind of filtering yet.
If the visual quality of HRLQ layer is not high enough, then can produce another one HR layer based on existing layer.Can use existing method, for example disclosed these methods in following document: A.Segall and G.J.Sullivan, Spatial scalability, IEEE journal; The Circuits System video technique, the 17th volume, the 9th phase; The 1121-1135 page or leaf, in September, 2007, and D.Marpe H.Schwarz and T.Wiegand; SVC Core Experiment 2.1:Inter-Layer Prediction of Motion and ResidualData, ISO/IEC JTC 1/SC 29/WG 11, Doc; M11043, is incorporated herein its integral body at this by reference in June, 2004.
In addition, if the source frame of video does not have too many content, then need not use filtering operation to this frame of video.Therefore, can keep video quality, and the HRLQ layer will be as the HR layer.Above-mentioned determined three layers of scalable stream system will taper to two-layer resolution scalable stream system.
Execution mode of the present invention can software, the form of the combination of hardware, applied logic or software, hardware and applied logic is implemented.Software, applied logic and/or hardware can reside in IC chip, module or the memory.If desired, part software, hardware and/or applied logic can reside on the IC chip, and part software, hardware and/or applied logic can reside in the module, and part software, hardware and/or applied logic can reside in the memory.In an exemplary execution mode, applied logic, software or instruction set are kept in any one traditional permanent computer-readable medium.
Described process of this specification and logic flow can be implemented by one or more programmable processors, and said processor is carried out one or more computer programs, to work through handling the input data and generating output.Also can pass through dedicated logic circuit, for example, FPGA (field programmable gate array) or ASIC (application-specific integrated circuit (ASIC)) implement said process and logic flow.
Described device of this specification or equipment can be realized through handling the input data and generating output by programmable processor, computer, SOC(system on a chip) or their combination.Device or equipment can comprise dedicated logic circuit, for example, and FPGA (field programmable gate array) or ASIC (application-specific integrated circuit (ASIC)).Except that hardware; Device or equipment also can comprise code; This code is that computer program is created execution environment, for example, (for example forms processor firmware, protocol stack, data base management system, operating system, cross-platform running environment; Or the code of the one or more combination in them virtual machine).
The processor that is fit to computer program comprises: for example, and general and special microprocessor, and any one or a plurality of processor of the digital computer of all kinds.Usually, processor is from read-only memory or random access memory or their the two reception instruction and datas.The element of computer generally includes and is used to the processor implementing or execute instruction and one or more memory device that is used for store instruction and data.
As described in this manual computer-readable medium can for anyly can hold, the medium or the instrument of storage, communication, propagation or move instruction; Said instruction by instruction execution system, device or equipment (for example; Computer) use, perhaps and instruction executive system, device or equipment are relevant.Computer-readable medium can comprise computer-readable recording medium; Said computer-readable recording medium can hold or the medium or the instrument of store instruction for any; Said instruction by instruction execution system, device or equipment (for example; Computer) use, perhaps and instruction executive system, device or equipment are relevant.Computer-readable recording medium can comprise nonvolatile memory, medium and the memory devices of form of ownership, for instance, can comprise semiconductor memory devices, for example, and EPROM, EEPROM and flash memory device; Disk, for example, internal hard disk or portable hard drive; CD; And CD-ROM and DVD-ROM dish.
The enough any program languages of ability are write computer program and (also are considered to; For example; Program, software, software application, script or code), comprise compiling type or interpreted language, statement property or procedural language; And it can be configured to any form, comprises as stand-alone program or as module, assembly, subprogram, object or other parts that are suitable in computer environment, using.Computer program can be configured on a computer or many computers, carry out, and these many computer bit perhaps stride across the distribution of a plurality of places and pass through interconnection of telecommunication network in an independent website.
Can in computer system, implement like execution mode and/or characteristic described in this specification; Said computer system comprises that aft-end assembly (for example; Like the data server), perhaps comprise intermediate equipment assembly (for example, application server); (for example perhaps comprise front end assemblies; Client computer with graphical user interface or web browser, the user can be with mutual like the described execution mode of this specification through this web browser), perhaps comprise the one or more combination in any in this aft-end assembly, intermediate equipment assembly or the front end assemblies.Assembly in the system can interconnect through any type of digital data communications medium, for example, and communication network.The example of communication network comprises local area network (LAN) (" LAN ") and wide area network (" WAN "), for example, and the internet.
Computer system can comprise client and server.Client and server is separated by very far away usually each other, and generally carries out alternately through communication network.Dependence operates in the computer program on the computer separately, is related between the client and server, thereby has had mutual client-server relation.
Whole specification comprises a lot of concrete realization details.These concrete realization details are not the restrictions for the scope that is interpreted as the present invention or requires to protect, but as the description to the peculiar characteristic of specific implementations of the present invention.
Some characteristic described in the context of execution mode independently can be combined, and carries out as independent execution mode.On the contrary, also can in a plurality of execution modes, carry out respectively, perhaps in any suitable son combination, carry out in the various characteristics described in the context of independent execution mode.In addition; Although maybe be with feature description for being applied in some combination; Or even in the initial combination required for protection, yet, in some cases; One or more characteristics in the combination of described combination or requirement protection can be got rid of from combination, and combination required for protection may be pointed to, and son makes up or the distortion of son combination.Although in independent claims, listed various aspects of the present invention; Yet other aspects of the present invention comprise that and claim is not the only clear and definite combination of having listed from execution mode and/or other combinations of characteristic of dependent claims with characteristic of independent claims.
Some function described in this specification may be carried out and/or the execution of while each other with different orders.In addition, if desired, one or more functions described above can be optional or can be combined.
Above description provides exemplary execution mode of the present invention, but should not understand with the thinking of restriction.On the contrary, under situation about not breaking away from like defined scope of the present invention in additional claim, it is possible making some changes or modification.
Claims (20)
1. scalable video equipment comprises:
The device of coding source video, its double sampling high-resolution video frame, to generate a plurality of low resolution video frame, the quantity of low resolution video frame is N, and wherein N is the integer more than or equal to 1, and each in N low resolution video frame is at each given period t
iOccur during this time, wherein i is from the integer 1 to n the scope, and n is the sum of this needed period of source video of coding; And
The device of this N low resolution video frame that sorts is so that the rank order of any given low resolution video frame in this N low resolution video frame is from t
1To t
nRemain unchanged in period in the scope;
Make up the device of this N low resolution video frame that generates; So that during decode low-resolution frame of video subsequently; Receiving equipment can be selected the subclass x of this N low resolution video frame; It is used for producing the video corresponding to this source video at this receiving equipment, and wherein x is from the integer 1 to N the scope.
2. scalable video equipment as claimed in claim 1 further comprises:
Each the device that is used for a further double sampling N low resolution video frame, producing one group of M low resolution video frame, it is corresponding in N the low resolution video frame each, and wherein M is the integer more than or equal to 2; And
Be used for device that M low resolution video frame of this group sorted, so that rank order of each member in should M low resolution video frame of group is at each period t
iAll remain unchanged, keep rank order simultaneously corresponding to N low resolution video frame of M low resolution video frame of this group.
3. scalable video equipment as claimed in claim 2 further comprises:
According to each the device that produces in M low resolution video frame of this group further double sampling M low resolution video frame.
4. scalable video equipment as claimed in claim 1 further comprises:
Be used for before the double sampling device to this high-resolution frame application of low-pass filters.
5. scalable video equipment as claimed in claim 2 further comprises:
Be used for before the double sampling device to each application of low-pass filters of N low resolution video frame.
6. scalable video equipment as claimed in claim 3 further comprises:
Be used for before the double sampling device to each application of low-pass filters of M low resolution video frame.
7. scalable video equipment as claimed in claim 1 further comprises:
Be used for encode one or more device of N low resolution video frame through other members with reference to this N low resolution video frame.
8. scalable video equipment as claimed in claim 7, wherein:
In N the low resolution video frame at least one is encoded to independent frame, so that when encoding this independent frame, do not need with reference to other frames.
9. scalable video equipment as claimed in claim 7, wherein:
In N the low resolution video frame at least one is encoded to predictive frame, so that this predictive frame is with reference to other members in this N low resolution video frame.
10. scalable video equipment as claimed in claim 7, wherein:
In N the low resolution video frame at least one is encoded to bi-directional predicted frames, so that this bi-directional predicted frames is with reference to other members in this N low resolution video frame.
11. a scalable video coding method comprises:
The coding source video, it comprises double sampling high-resolution video frame, and to generate a plurality of low resolution video frame, the quantity of low resolution video frame is N, and wherein N is the integer more than or equal to 1, N low resolution
In the frame of video each is at each given period t
iOccur during this time, wherein i is from the integer 1 to n the scope, and n is the sum of this needed period of source video of coding; And
This N low resolution video frame that sorts is so that the rank order of any given low resolution video frame in this N low resolution video frame is from t
1To t
nRemain unchanged in period in the scope;
Make up this N low resolution video frame that generates; So that during decode low-resolution frame of video subsequently; Receiving equipment can be selected the subclass x of this N low resolution video frame; It is used for producing the video corresponding to this source video at this receiving equipment, and wherein x is from the integer 1 to N the scope.
12. scalable video coding method as claimed in claim 11 further comprises:
Further each in the double sampling N low resolution video frame, producing one group of M low resolution video frame, it is corresponding in N the low resolution video frame each, and wherein M is the integer more than or equal to 2; And
M low resolution video frame of this group sorted, so that the rank order that should organize each member in M the low resolution video frame is at each period t
iAll remain unchanged, keep rank order simultaneously corresponding to N low resolution video frame of M low resolution video frame of this group.
13. scalable video coding method as claimed in claim 12 further comprises:
Instruction according to producing M low resolution video frame of this group comes each in the further double sampling M low resolution video frame.
14. scalable video coding method as claimed in claim 11 further comprises:
Before the double sampling to this high-resolution frame application of low-pass filters.
15. scalable video coding method as claimed in claim 12 further comprises:
Each application of low-pass filters before double sampling in N low resolution video frame.
16. scalable video coding method as claimed in claim 13 further comprises:
Each application of low-pass filters before double sampling in M low resolution video frame.
17. scalable video coding method as claimed in claim 11 further comprises:
Through encode one or more in N the low resolution video frame of other members with reference to this N low resolution video frame.
18. scalable video coding method as claimed in claim 17, wherein:
In N the low resolution video frame at least one is encoded to independent frame, so that when encoding this independent frame, do not need with reference to other frames.
19. scalable video coding method as claimed in claim 17, wherein:
In N the low resolution video frame at least one is encoded to predictive frame, so that this predictive frame is with reference to other members in this N low resolution video frame.
20. scalable video coding method as claimed in claim 17, wherein: in N the low resolution video frame at least one is encoded to bi-directional predicted frames, so that this bi-directional predicted frames is with reference to other members in this N low resolution video frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010216832 CN101888553B (en) | 2010-06-30 | 2010-06-30 | Scalable video coding method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010216832 CN101888553B (en) | 2010-06-30 | 2010-06-30 | Scalable video coding method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101888553A CN101888553A (en) | 2010-11-17 |
| CN101888553B true CN101888553B (en) | 2012-01-11 |
Family
ID=43074234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010216832 Active CN101888553B (en) | 2010-06-30 | 2010-06-30 | Scalable video coding method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101888553B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102075734B (en) * | 2010-12-31 | 2013-07-24 | 浙江宇视科技有限公司 | Video service implementation method and equipment and system thereof |
| CN106851319B (en) | 2011-06-10 | 2020-06-19 | 寰发股份有限公司 | Derivation method and derivation device |
| US9131245B2 (en) | 2011-09-23 | 2015-09-08 | Qualcomm Incorporated | Reference picture list construction for video coding |
| CN104853188A (en) * | 2015-03-12 | 2015-08-19 | 康佳集团股份有限公司 | Selection control method and system for SVC rapid coding mode in wireless network |
| CN104768008B (en) * | 2015-03-26 | 2018-01-30 | 哈尔滨工业大学 | A kind of fast encoding method for scalable video |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004502358A (en) * | 2000-06-30 | 2004-01-22 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | Encoding method for video sequence compression |
| CN101127903B (en) * | 2007-09-29 | 2010-12-08 | 华南理工大学 | DCT-based resolution flexible image coding and decoding method |
-
2010
- 2010-06-30 CN CN 201010216832 patent/CN101888553B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN101888553A (en) | 2010-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10791333B2 (en) | Video encoding using hierarchical algorithms | |
| Kim et al. | Low bit-rate scalable video coding with 3-D set partitioning in hierarchical trees (3-D SPIHT) | |
| JP4891234B2 (en) | Scalable video coding using grid motion estimation / compensation | |
| KR100781525B1 (en) | Method and apparatus for encoding and decoding FGS layers using weighting factor | |
| KR101376666B1 (en) | Enhancing image quality | |
| KR100703788B1 (en) | Video encoding method, video decoding method, video encoder, and video decoder, which use smoothing prediction | |
| JP2007520149A (en) | Scalable video coding apparatus and method for providing scalability from an encoder unit | |
| JP6438904B2 (en) | Video signal processing | |
| JP2007174568A (en) | Encoding method | |
| KR101032243B1 (en) | Method and system for scalable bitstream extraction | |
| CN101888553B (en) | Scalable video coding method and device | |
| CN1729696A (en) | System and method for drift-free fractional multiple description channel coding of video using forward error correction codes | |
| US20070160143A1 (en) | Motion vector compression method, video encoder, and video decoder using the method | |
| US20120002726A1 (en) | Method and apparatus for scalable video coding | |
| JP4660408B2 (en) | Encoding method | |
| US20060133483A1 (en) | Method for encoding and decoding video signal | |
| Edwards et al. | High throughput JPEG 2000 for broadcast and IP-based applications | |
| WO2006132509A1 (en) | Multilayer-based video encoding method, decoding method, video encoder, and video decoder using smoothing prediction | |
| JP4385928B2 (en) | Scalable video encoding / decoding method and system | |
| Prabhakaran | Enhanced Hierarchical Multi-resolution Imaging | |
| KR101388591B1 (en) | Fec packet allocating method and apparataus, and server and client apparatus employing the same | |
| Huang et al. | An experimental analysis of DCT-based approaches for fine-grain multiresolution video | |
| Zhang | Robust scalable video compression using multiple description coding | |
| Prasad et al. | Wavelet based scalable video codec | |
| Bauermann et al. | Encoding parameter estimation for RDTC optimized compression and streaming of image-based scene representations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |