Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/573—Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
  • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
  • H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements

Definitions

• the invention relates to the general field of video coding (or video compression), and in embodiments to so-called scalable video coding. It presents a method and system for encoding a video signal wherein, image compression, for instance scalable image compression is performed and multiple temporal predictions are used wherein multiple frames are stored in a memory.
• the invention also relates to the encoded, e.g. scalable video signal and to a method and system for decoding a video signal.
• Image display systems often receive compressed data streams.
• a variety of image compression techniques are known to reduce the amount of image data that must to be stored or transmitted.
• video compression use is made of prediction, wherein the content of a frame or part of a frame is predicted from the content of one or more previous received or generated frames.
• video signal processing the signal is comprised of intracoded and intercoded frames, for instance I-frames, P-frames and B-frames.
• the I-frames are intracoded.
• the P- and B-frames are referred to as intercoded frames.
• Intra-code frames can be reconstructed without any reference to other frames; intercoded frames are reconstructed using data of other frames (forward or backward prediction).
• the P-and B-frames only contain information or changes between the I-frames, often expressed in motion vectors for macroblocks.
• the referral is relatively simple, and at most two frames are referred to, the P frames are forwardly predicted from I frames and the B frames are forwardly and backwardly predicted from I and P-frames.
• motion estimation motion vectors can be found which are used for motion estimation of parts (macroblocks) of a frame.
• Some more complex video compression standards such as the AVC compression standard have a possibility of many multiple predictions.
• a relatively large number of temporal (i.e. forward or backward) predictions are made. Not just the nearest frames in time are considered for making the predictions, but also frames further removed in time.
• In a buffer several frames are stored in a memory to be used for temporal prediction. As time progresses the frames are shifted through a buffer in the memory and the oldest are bumped out of the buffer as a new 'fresh' frame is stored.
• the method in accordance with the invention is characterized in that a prediction frame in memory is overwritten with a separately produced prediction frame.
• “Separately produced” means within the framework of the invention from outside the normal (temporal) procedure for generating a prediction frame.
• Prediction frames are for instant e.g. produced in an enhancement encoding part of an encoder, prediction frames that are produced in a base stream encoding part are produced separately from the enhancement part of the encoder (although still within the encoder, when seen in a larger sense).
• a prediction frame longest in memory the oldest of the prediction frames in memory, or one of a subset of oldest prediction frames. The invention is based on the following insight:
• the overwritten prediction is one of the oldest predictions.
• the oldest (longest in memory) predictions usually are the least important.
• the method may comprise an algorithm to select the least important prediction(s) prior to overwriting. The method in accordance with the invention does not require more bits or major changes to the existing AVC standard.
• the memory is a memory in an enhancement encoder/decoder and the separately produced frame is an upscaled/de-interlaced frame from a base encoder/decoder. This enables, without any syntax change, to use the standard AVC for scalable compression.
• the external frame comprises depth views. This allows generating 3D video multiview coding.
• the enhancement stream provides a higher frame rate, with a ratio different from 2.
• a envisioned application for such a case is a base layer at 50Hz intended for television sets, and an enhancement layer at 75Hz intended for computer LCD panels (or any other advanced display).
• increasing the frame rate to at least 75Hz gives significant improvements while it demands a lower bit-rate increase than going directly to 100Hz.
• 75Hz is the native refresh rate of many displays (mainly in the computer world).
• 75/90Hz enhancement is to be preferred, but can be realized neither with MPEG2 nor with SVC.
• Figs. Ia to Ic show the processing flow of a post-processing method, including a method for encoding (Fig. Ia) and decoding (Fig. Ib) according to an embodiment of the invention, wherein Fig. Ic illustrates a more complex embodiment.
• FIG. 2 to 5 further illustrate the embodiment of the invention as shown in Figs. Ia to Ic.
• Fig. 6 illustrates a second embodiment of the invention.
• Fig. 7 illustrates a further embodiment of the invention wherein the base stream and the enhancement stream have different frame rates.
• Fig. 8 shows the basic principle of a video compression apparatus having prediction memories (typically for temporal prediction) wherein at least one of the less usuable ones is used for storing a prediction of an alternative predicting means.
• the figures are not drawn to scale. Generally, identical components are denoted by the same reference numerals in the figures.
• Some video compressions standards have a possibility of using a multitude of predictions for e.g. motion estimation/motion compensation.
• the inventors have realized that some of these predictions are more redundant or less useful, e.g. when using AVC multiple temporal predictions are use and the IC is predesigned to be able to deal with this.
• the dogma of temporal prediction is bypassed by overwriting a last of these predictions (i.e. usually the least accurate, especially for wild motion) to encode another prediction.
• the decoder behaves similarly.
• FIG. 1 shows a processing flow of an embodiment of our invention used an encoding and decoding method. This is illustrated in the following: Encoder side:
• Figure Ia illustrates an embodiment of the method and system in accordance with the invention.
• VLC Variable Length Coding Base str: Base stream
• AVC Advanced Video coding
• advanced video coding are for instance H.264 and
• FIG. 1a illustrates a method in which use is made of a base stream and an enhancement stream.
• the massive amounts of data inherent in many image transmission method pose significant problems. More particularly, each digital image frame is a still image formed from an array of pixels according to the display resolution of a particular system. As a result, the amounts of raw digital information included are often massive.
• compression schemes are used to compress the data.
• Various video compression standards or processes have been established, including, MPEG- 2, MPEG-4, and H.263.
• scalability techniques There are three axes on which one can deploy scalability. The first is scalability on the time axis, often referred to as temporal scalability. Secondly, there is scalability on the quality axis, often referred to as signal-to-noise scalability or fine-grain scalability. The third axis is the resolution axis (number of pixels in image) often referred to as spatial scalability or layered coding. In layered coding, the bitstream is divided into two or more bitstreams, or layers.
• each layer can be combined to form a single high quality signal.
• the base layer may provide a lower quality video signal
• the enhancement layer provides additional information that can enhance the base layer image.
• Figure Ia illustrates a method and system for providing a layered signal, in this case have a base stream (base str) and an enhancement stream (base str), sometimes also called an enhancement layer.
• the input is split and sent to a base encoder 2 after having passed a low pass filter 1 , for instance a nyquist filter.
• the signal undergoes in the base encoder a disctret cosine transformation (DCT), or any other similar transformation, such as for instance using wavelets, and is quantized (Q; on the resulting data stream variable length coding is performed providing the base stream to be sent and/or stored.
• DCT disctret cosine transformation
• Q on the resulting data stream variable length coding is performed providing the base stream to be sent and/or stored.
• the signal is often comprised of intracoded and intercoded frames, for instance I-frames, P- frames and B-frames.
• the I-frames are infra-coded.
• the P- and B-frames are referred to as intercoded frames.
• Infra-code frames can be reconstructed without any reference to other frames; intercoded frames are reconstructed using data of other frames (forward or backward prediction).
• the P-and B-frames only contain information or changes between the I-frames, often expressed in motion vectors for macroblocks.
• the original signal has to be reconstructed inside the encoder. This is done by a reverse quantization (Q "1 ) and a reverse Discrete Cosine Transform (DCT "1 ).
• the resulting reconstructed frames are used inside the encoder for estimating motion vectors. In a simple arrangement only one reconstructed frame is used for motion estimation and motion compensation. However, in more complex methods and systems a number of reconstructed frames are used.
• the encoder comprises a shift register wherein for comparison the data of a number of reconmstructed frames are stored in a shift register for use in the motion estimation and/or motion estimation, i.e. in prediction.
• Some more complex video compression standards such as the AVC compression standard have a possibility of many multiple predictions. A relatively large number of temporal (i.e. forward or backward) predictions are made. Not just the nearest frames in time are considered for making the predictions, but also frames further removed in time.
• a buffer In a buffer several frame are stored in a buffer to be used for temporal prediction. As time progresses the frames are shifted through the buffer and the oldest are bumped out of the buffer as a new 'fresh' frame is stored. The resulting base stream can be broadcasted, received and via a decoder, displayed as is, although the base stream does not provide a resolution which would be considered as high-definition.
• the system comprises also an enhancement encoder 3.
• the enhancement encoder 3 a similar method is performed as in the base stream encoder 2, with only this difference that the enhancement stream (i.e. the difference between the original signal and the base stream) is treated.
• the enhancement decoder comprises a means for Discrete Consien Transform and qunatization of the enhanced layer signal, and for reconstructing (DCT "1 , Q "1 ).
• a person skilled in the art will be familiar with these methods.
• the inventors have realized that the stored frames do not all have the same relevance, in particular, but not exclusively the least longest remaining frames may well be less important.
• the longer a frame is in store the lower on average the importance of the temporal prediction based on that frame.
• the dogma of temporal prediction is bypassed by using the space reserved for the last of the temporal predictions (i.e. the Oldest frames') for a separate prediction. This will decrease slightly the accuracy of the temporal prediction.
• the overwritten information is relatively redundant/less useful. Overwriting the information therefore very often does not seriously decrease image quality.
• FIG. 1a illustrates an embodiment of the method and system of the invention.
• the oldest data in the memory in the enhancement encoder is overwritten by an upscaled/interlaced frame produced in the base encoder.
• Figure Ia shows a scalable encoder. Because resources are limited to 180i, the scheme according to the invention enables with only minor modification to a silicon 1080p.
• the last prediction R n _i now e.g. contains a smart content adaptive upscaling candidate.
• the last two 'normal' temporal predictions (R n _i, R n _2) could be overwritten be e.g. the predictions (frames) produced by using two different de-interlacing algorithms.
• the oldest frame in memory in the enhancement encoder is overwritten.
• the oldest frame is overwritten.
• one of the last n frames could be overwritten wherein a choice is made which one is overwritten on basis of an estimation of the importance (ranking) of the set of oldest to be overwritten frames, the overwritten frames than need not necessarily be the very oldest frame, but could be the next to oldest frame.
• Figure Ib shows the decoder side.
• VLD stands for variable length decoding.
• a frame 4 produced in the base stream decoder 2' is used to overwrite frame prediction frame R n _i in memory 5' of the enhancement decoder.
• Figure Ic illustrates for a more complex embodiment of the design shown in Figure Ia. The difference is that a filter is used and furthermore that the data relating to the filter and the upscaling are inserted into SEI (Supplemental Enhancement Information). These data is used in a decoder to establish at the decoder end the same parameters for filtering and upscaling, so that parameters used in the method of encoding in the encoder can be used in the decoding as well.
• SEI Supplemental Enhancement Information
• Figure 2 illustrates in a flow diagram the different steps within the enhancement to ensure that the external frame (upscaled base with PTS j ) does overwrite frame R n _i in the memory.
• FIG 2b Another option is given in Figure 2b; it gives an example when it is safe to overwrite a reference frame in memory.
• a frame is only overwritten when it already has been sent to the output (display).
• the number of reference frames is three. This means that the first three frames it is impossible to overwrite frames, since we need three frames.
• a frame has been displayed, that frame can be overwritten in the memory.
• these frames are oldest frames, but not necessarily the oldest frame in memory.
• the already displayed Ig_frame can be overwritten, but not the as yet not displayed Bi frame.
• the rectangle denote frames used for reference for B3 and P_6_(f ⁇ rst rectangle) and B5 and P 7 (last rectangle).
• Figure 3 illustrates an embodiment wherein an upscaled decoded Io base stream frame overwrites in the memory (buffer) of the enhancement encoder/decoder the last reference prediction frame (R 2 ) in the enhancement encoder/decoder, which in this example is P -6 .
• Figure 4 illustrates an embodiment wherein an upscaled decoded P 2 , with time stamp PTS2; base stream frame overwrites in the memory (buffer) of the enhancement encoder/decoder the last reference prediction frame (R 2 ) in the enhancement encoder/decoder, which in this example is P_ ⁇
• Figure 5 illustrates an embodiment in which the base stream has a 72Op output and the enhancement stream a 108Op output.
• An upscaled/deinterlaced base stream frame with time stamp PTSl overwrites frame R n _i in the memory of the AVC enhancement encoder. This allows to introduce 108Op HDTV at full frame rate in a backwards compatible way (to 72Op or 1080i HDTV), which is not possible or at much greater costs with known methods.
• center view C center view
• L left
• R right
• center view C left
• L left
• R right
• center view C left
• L & R views would be hierarchical B frames.
• the inventors have however realized that the basic concept of the invention, i.e. principle of overwriting one of more reference frames to the multiview case, can be used, so that we also don't have to change any syntax, and basically can use regular AVC also for this application.
• a third application enabled by this invention is creating an enhancement stream for a higher frame rate with a different ratio then power of 2.
• a decoded frame of the 50 Hz sequence is put in the reference list of the 75 Hz sequence, according to the general scheme of tis invention.
• This prediction frame from the 50 Hz sequence is a good predictor and helps reaching a better coding efficiency because it is temporally very close to the frame to encode in the 75 Hz version.
• frame i belongs to the enhanced 75 Hz sequence
• frame i ' belongs to the base 50 Hz sequence: frame 1 ' has the same temporal location as frame 1 frame 2' is temporally closer to frame 2 than any frame in the 75 Hz sequence frame 2' is temporally closer to frame 3 than any frame in the 75 Hz sequence - frame 3 ' has the same temporal location as frame 4 etc.
• Some video compression standards use multiple temporal predictions. One or more of the oldest temporal predictions are overwritten with another prediction.
• a prediction used in an anhancement encoder is in embodiment overwritten by a prediction produced in a base stream encoder. In another embodiment a temporal prediction is overwritten by a 3D view.
• the invention relates to a method and system of encoding, as well as to a method and system of decoding, as described above by way of example.
• the invention is also embodied in a video signal comprising encoded video signals and control information comprising e.g. functional parameters for use in the method of decoding.
• the control information may comprise data in accordance with any, or any combination, of the embodiments described above.
• the control information enclosed in the video signal according to the present invention may comprise one or more of the following information: A: general information, i.e. applicable for the whole of the video signal
• Parameters used in the method such parameters may be for instance
• Parameters used throughout the encoding of the video signal such as parameters for filters, upscaling algorithms or algorithms to determine which of the predictions is to be overwritten.
• the video signal is generated dynamically, i.e. certain choices made during encoding are dependent on the video content; the choices made in the encoder may be included into the encoded video signal.
• the encoding method comprises an algorithm to make a decision as to which one of the prediction is to be overwritten for instance by estimating the importance of the to be overwritten prediction possibly in comparison to the prediction that will overwrite, there is a choice of including the details of said selection algorithm(s) into the video signal (and thus including general information into the video signal).
• the decoder comprises said algorithm. If the decoder does not, the particular dynamic information may be sent, i.e. it is specified for a particular part of the video signal (for instance by a flag) that a particular prediction is to be overwritten.
• the video signal may comprise general information as well as dynamic information. All of the above types of information, as well as any other type of information relating to the use of the method according to the invention are called within the framework of the application a parameter. Parameters may thus be simple yes/no parameters (point Al above), parameters indicating a choice within a set of possibilities (point A2 above for instance), or a parameter for control of a step within the method (point A3 above) or dynamic parameters (point B above) in any shape or form.
• Figure 8 specifies again the most generic apparatus (corresponding to most generic method) of the invention.
• a prediction unit 801 to apply a criterion SEL_F() upon several predictions generated by similar prediction units T, T2, which do e.g. a lossy transformation and motion compensation (e.g. in AVC, the temporal prediction can result from a combination of several previous images)
• one of these "standard” modes may be "re-used” by storing data from a separate prediction from another prediction unit PR UN (i.e. in the sub-memory MEM P2, a coarsely quantized first prediction, e.g. a wavelet approximation, is overwritten by another predicted region, by another algorithm from PR UN).
• PR UN i.e. in the sub-memory MEM P2
• a coarsely quantized first prediction e.g. a wavelet approximation
• This new prediction mode can then be easily be introduced in an existing standard (whatever one likes).
• the selection criterion SEL_F() may also be changed.
• the other prediction unit may e.g. use a texture synthesis. If the encoder side recognizes there are e.g. patches (e.g. blocks) of grass, a comparison criterion or prediction may be used which just uses synthetically generated grass (e.g. typically with no residue update). This may then be selected as the best quality/compressed region compared to e.g. the temporal prediction. The decoder will then also apply the grass generation (it need not even receive temporal predictive information for that region). All kinds of improved predictions can be generated by prediction unit PR UN, e.g.
• synchromization data may also be transmitted via the standardized compressed image signal, e.g. in SEIs.
• the invention is also embodied in any computer program product for a method or device in accordance with the invention.
• computer program product should be understood any physical realization of a collection of commands enabling a processor - generic or special purpose-, after a series of loading steps (which may include intermediate conversion steps, like translation to an intermediate language, and a final processor language) to get the commands into the processor, to execute any of the characteristic functions of an invention.
• the computer program product may be realized as data on a carrier such as e.g. a disk or tape, data present in a memory, data travelling over a network connection -wired or wireless- , or program code on paper.
• characteristic data required for the program may also be embodied as a computer program product.
• the method may de used for only a part of the image, or different embodiments of the method of the invention may be used for different parts of the image, for instance using one embodiment for the center of the image, while using another for the edges of the image.
• Use of the verb "to comprise” and its conjugations does not exclude the presence of elements other than those stated in the claims.
• Use of the article "a” or “an” preceding an element does not exclude the presence of a plurality of such elements.
• frame could be a part of a frame if for instance the method in accordance with the invention is done on a part of the frame, for instance only the middle part of the video or only foreground parts of a video.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (2)

Family

ID=38091743

Family Applications (1)

Country Status (7)

Cited By (2)

Families Citing this family (6)

Family Cites Families (16)

• 2007
  • 2007-01-12 EP EP07100470A patent/EP1944978A1/en not_active Ceased
• 2008
  • 2008-01-04 JP JP2009545257A patent/JP5508024B2/ja active Active
  • 2008-01-04 WO PCT/IB2008/050014 patent/WO2008084417A2/en not_active Ceased
  • 2008-01-04 US US12/521,886 patent/US8571101B2/en active Active
  • 2008-01-04 ES ES08700197.0T patent/ES2572885T3/es active Active
  • 2008-01-04 CN CN200880002202.2A patent/CN101584220B/zh active Active
  • 2008-01-04 PL PL08700197.0T patent/PL2105028T3/pl unknown
  • 2008-01-04 EP EP08700197.0A patent/EP2105028B1/en active Active

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events