WO2011127628A1 - Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal - Google Patents

Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal Download PDF

Info

Publication number
WO2011127628A1
WO2011127628A1 PCT/CN2010/000501 CN2010000501W WO2011127628A1 WO 2011127628 A1 WO2011127628 A1 WO 2011127628A1 CN 2010000501 W CN2010000501 W CN 2010000501W WO 2011127628 A1 WO2011127628 A1 WO 2011127628A1
Authority
WO
WIPO (PCT)
Prior art keywords
lower layer
macroblock
frame
layer frame
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2010/000501
Other languages
French (fr)
Inventor
Yu Wen Wu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Thomson Licensing SAS
Original Assignee
Thomson Licensing SAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Licensing SAS filed Critical Thomson Licensing SAS
Priority to PCT/CN2010/000501 priority Critical patent/WO2011127628A1/en
Publication of WO2011127628A1 publication Critical patent/WO2011127628A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/33—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the spatial domain
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/89—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder
    • H04N19/895—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder in combination with error concealment

Definitions

  • the invention is made in the field of error concealment.
  • the invention relates to recovering of a lost enhancement layer macro block encoded in a spatial-scalable video coding signal.
  • Scalable video coding is an approach towards
  • the difference in quality may be related to temporal resolution, to spatial resolution or to signal-to-noise resolution.
  • image frames of a video are encoded in a spatial lower layer and one or more enhanced spatial layers. While the lower layer provides a low resolution version of the video frames, the enhanced spatial layers carry higher resolution versions of the same content .
  • Each UDP packet contains Cyclic " Redundancy Check (CRC) allowing error detection. If a CRC fails, the whole UDP packet is discarded.
  • CRC Redundancy Check
  • RU upsampling
  • each pixel value of the concealed frame is copied from the corresponding pixel of the first frame in Reference Picture List 0 (RefPicListO ) .
  • This algorithm can be invoked for any layer.
  • TD temporary direct motion vector generation
  • the MVs and reference indices of each subblocks of the missing frame are calculated as if they were coded using the "temporal direct mode”.
  • This algorithm can be invoked for any layer.
  • Motion and Residual Upsampling is an inter layer error concealment method wherein information from a lower layer which may be the lower layer are used for error concealment in an enhanced layer.
  • motion and residual upsampling (BLSkip) algorithm, SVC tools are used and the BLSkip mode is set in the enhancement layer.
  • Residual upsampling is also used to upsample the residual of the lower layer for enhanced layer.
  • the motion compensation is done at the enhanced layer using the upsampled motion fields. This algorithm can directly be used for the enhanced layer if there is no packet loss in the lower layer. If the lower layer is also lost, it needs to generate lower layer motion vectors using TD method in lower layer before BLSkip can use motion field upsampling.
  • Another inter layer error concealment method is
  • the lower layer picture is reconstructed and upsampled using the H.264/AVC 6-tap filter for the lost enhanced layer picture. If lower layer packet is also lost, the FC method is used for the enhancement layer, rather than using an upsampled concealed lower layer picture.
  • Lower Layer Upsampling as known from prior art is the availability of a lower layer frame corresponding to the enhancement layer frame for which a macroblock is lost or defect.
  • a method and device for recovering a lost macroblock of an enhancement layer frame of a spatial- scalable video coding signal are proposed, the method comprising the features of claim 1 and the device being adapted fro performing said method.
  • Said method comprises using reconstruction lower layer upsampling for recovering said lost macroblock, wherein a generated lower layer macroblock is upsampled, said
  • Fig. 1 exemplarily depicts SVC coding structure with two dependency layers with different frame rate and Fig. 2 exemplarily depicts the proposed solution of temporal interpolation lowerd SVC error concealment .
  • the invention can be realized on any electronic device comprising a processing device correspondingly adapted.
  • the invention can be realized in a set-top-box, a television, a DVD- and/or BD-player, a mobile phone, a personal computer, a navigation system or a car video system.
  • FIG. 1 The figure shows an enhanced layer EL comprising the image frames El, E2, E3, E4 and E5 and a lower layer LL which comprises the images frames Bl, B3 and B5.
  • the enhanced layer EL is spatially as well as temporarily enhanced. That is, frames El, E3 and E5 comprise the same content as frames Bl, B3 and B5 but at a higher spatial resolution as indicated by the number of quadrangles comprised in the frames. Furthermore, the enhanced layer EL provides the same content with a higher temporal resolution as it comprises the frames E2 and E4 for which no
  • picture E5 may be replaced by an up-scaled version of picture B5.
  • RU can be the packet recovery method which promises the lowest overall visual degradation resulting from concealment error and error propagation.
  • the current invention proposes generating a
  • corresponding lower layer macroblock can be due to the lower layer's lower frame rate or due to packet loss.
  • enhancement layer frame which suffered a macroblock loss is E2(loss).
  • E2(loss) As a collocated lower layer macroblock does not exist, the corresponding lower layer macroblock is
  • a preceding lower layer frame Bl and a subsequent lower layer frame B3 are used for determining optical flow OF to get corresponding pixel motion directions between the preceding lower layer frame Bl and the subsequent lower layer frame B3. Then, the pixel motion directions are clustered.
  • a least squares minimization method is used to get pixel values of said corresponding lower layer macroblock.
  • the clustered pixel motion is used for segmentation of the frame to-be-generated.
  • the corresponding lower layer macroblock is then upsampled to enhancement layer resolution and used as recovery of the lost enhancement layer block.
  • the current invention proposes a temporal interpolation based method to conceal the transmission error of SVC bit stream to keep the continuity of the rendered video. It can be applied to SVC bit streams having different layers with different frame rate but is not restricted to such bit streams.
  • the invention allows for reconstructing the signal of the higher spatial layer by a lower spatial layer signal.
  • the invention can be applied to higher layer frames of any SVC signal of two or more layers.
  • the invention proposes adopting image processing methods to generate non-existing lower layer frames. In doing said image processing on lower layer frames, time cost of image processing can be controlled.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

The invention is made in the field of error concealment. In particular, the invention relates to recovering of a lost enhancement layer macro block of an enhancement layer frame (E2) encoded in a spatial-scalable video coding signal. An inter layer error concealment method known from prior art is Reconstruction Lower Layer Upsampling (RU). Precondition for prior art Reconstruction Lower Layer Upsampling (RU) is the availability of a lower layer frame corresponding to the enhancement layer frame (E2) from which a macroblock is lost. For removing this precondition, the invention proposes generating a lower layer macroblock using macroblocks comprised in lower layer frames (Bl, B3) non-coincident with said enhancement layer frame (E2). The generated lower layer macro block is then upsampled to enhanced layer resolution and used as concealment after upsampling.

Description

Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal
TECHNICAL FIELD
The invention is made in the field of error concealment. In particular, the invention relates to recovering of a lost enhancement layer macro block encoded in a spatial-scalable video coding signal.
BACKGROUND OF THE INVENTION
Scalable video coding (SVC) is an approach towards
provision of same content in different qualities in a single video stream. The difference in quality may be related to temporal resolution, to spatial resolution or to signal-to-noise resolution.
In spatial scalable video coding, image frames of a video are encoded in a spatial lower layer and one or more enhanced spatial layers. While the lower layer provides a low resolution version of the video frames, the enhanced spatial layers carry higher resolution versions of the same content .
When the compressed video is transmitted over error-prone networks, channel errors, e.g. packet loss, are inevitable, and since compressed data are very vulnerable to channel errors, even a single bit error may corrupt a whole
picture. In other cases, such as real-time communication, video is usually transmitted via unreliable UDP packets without any possibility for retransmissions at the
transport layer. Each UDP packet contains Cyclic "Redundancy Check (CRC) allowing error detection. If a CRC fails, the whole UDP packet is discarded. A UDP packet represents a rather large part of the picture and its loss usually results in the discarding of the whole picture.
To tackle the frame loss problem in SVC, in Cheng Y., Xie K, Zhang F., Pandit P., Boyce J.: "Frame Loss Error
Concealment for SVC" , Journal of Zhejiang University
Science A, 2006 7(5): 677-683, the authors proposed four different picture-loss concealment methods: frame copy (FC) and temporal direct motion vector generation (TD) which are intra-layer concealment methods as well as residual
upsampling (BLSkip) , and reconstruction lower layer
upsampling (RU) which are inter-layer concealment methods.
In the "frame copy" (FC) algorithm, each pixel value of the concealed frame is copied from the corresponding pixel of the first frame in Reference Picture List 0 (RefPicListO ) . This algorithm can be invoked for any layer. In the "temporal direct motion vector generation" (TD) algorithm, the MVs and reference indices of each subblocks of the missing frame are calculated as if they were coded using the "temporal direct mode". This algorithm can be invoked for any layer. Motion and Residual Upsampling (BLSkip) is an inter layer error concealment method wherein information from a lower layer which may be the lower layer are used for error concealment in an enhanced layer. In the "motion and residual upsampling" (BLSkip) algorithm, SVC tools are used and the BLSkip mode is set in the enhancement layer.
Residual upsampling is also used to upsample the residual of the lower layer for enhanced layer. However, the motion compensation is done at the enhanced layer using the upsampled motion fields. This algorithm can directly be used for the enhanced layer if there is no packet loss in the lower layer. If the lower layer is also lost, it needs to generate lower layer motion vectors using TD method in lower layer before BLSkip can use motion field upsampling. Another inter layer error concealment method is
Reconstruction Lower Layer Upsampling (RU) . In the
"reconstruction lower layer upsampling" (RU) algorithm, the lower layer picture is reconstructed and upsampled using the H.264/AVC 6-tap filter for the lost enhanced layer picture. If lower layer packet is also lost, the FC method is used for the enhancement layer, rather than using an upsampled concealed lower layer picture.
In "Analysis of Error Propagation in Hybrid Video Coding with application to Error Resilience", N. Farber, K.
Stuhlmuller, B. Girod, Proc. ICIP: 550-554, October 1999, the authors introduce a selection-based EC strategy in which one of the four EC methods is adaptively selected to minimize the visual degradation of the concealed picture.
SUMMARY OF THE INVENTION Precondition for error concealment using Reconstruction
Lower Layer Upsampling (RU) as known from prior art is the availability of a lower layer frame corresponding to the enhancement layer frame for which a macroblock is lost or defect. For overcoming this precondition and allowing for more flexible use of RU, a method and device for recovering a lost macroblock of an enhancement layer frame of a spatial- scalable video coding signal are proposed, the method comprising the features of claim 1 and the device being adapted fro performing said method.
Said method comprises using reconstruction lower layer upsampling for recovering said lost macroblock, wherein a generated lower layer macroblock is upsampled, said
generated lower layer macroblock is generated using
macroblocks comprised in lower layer frames non-coincident with said enhancement layer frame and said lost macroblock is recovered from the upsampled lower layer frame.
The features of further advantageous embodiments are specified in the dependent claims.
BRIEF DESCRIPTION OF THE DRAWINGS
Exemplary embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description. The exemplary embodiments are explained only for elucidating the invention, but not limiting the invention's disclosure, scope or spirit defined in the claims.
In the figures:
Fig. 1 exemplarily depicts SVC coding structure with two dependency layers with different frame rate and Fig. 2 exemplarily depicts the proposed solution of temporal interpolation lowerd SVC error concealment .
EXEMPLARY EMBODIMENTS OF THE INVENTION The invention can be realized on any electronic device comprising a processing device correspondingly adapted. For instance, the invention can be realized in a set-top-box, a television, a DVD- and/or BD-player, a mobile phone, a personal computer, a navigation system or a car video system.
An exemplary first SVC coding structure with error
concealment is depicted in fig. 1. The figure shows an enhanced layer EL comprising the image frames El, E2, E3, E4 and E5 and a lower layer LL which comprises the images frames Bl, B3 and B5.
The enhanced layer EL is spatially as well as temporarily enhanced. That is, frames El, E3 and E5 comprise the same content as frames Bl, B3 and B5 but at a higher spatial resolution as indicated by the number of quadrangles comprised in the frames. Furthermore, the enhanced layer EL provides the same content with a higher temporal resolution as it comprises the frames E2 and E4 for which no
corresponding lower layer frame is comprised in the lower layer LL . Assumed that a macroblock from picture E5 is lost or damaged, as indicated by the dashed line, picture E5 may be replaced by an up-scaled version of picture B5.
But, if a macroblock of picture E2 or picture E4 is lost or damaged there is no corresponding lower layer frame from which a collocated lower layer macroblock can be upsampled to enhancement layer resolution for concealing the lost macro block.
Still, RU can be the packet recovery method which promises the lowest overall visual degradation resulting from concealment error and error propagation.
Hence, the current invention proposes generating a
corresponding lower layer macroblock, which can be
upsampled for recovering the lost enhancement layer
macroblock, if said corresponding collocated lower layer macroblock is not available. Unavailability of the
corresponding lower layer macroblock can be due to the lower layer's lower frame rate or due to packet loss.
This is exemplarily depicted in Fig. 2 wherein the
enhancement layer frame which suffered a macroblock loss is E2(loss). As a collocated lower layer macroblock does not exist, the corresponding lower layer macroblock is
generated using frame interpolation technology. That is, a preceding lower layer frame Bl and a subsequent lower layer frame B3 are used for determining optical flow OF to get corresponding pixel motion directions between the preceding lower layer frame Bl and the subsequent lower layer frame B3. Then, the pixel motion directions are clustered.
Finally, a least squares minimization method is used to get pixel values of said corresponding lower layer macroblock. In an embodiment, the clustered pixel motion is used for segmentation of the frame to-be-generated. The corresponding lower layer macroblock is then upsampled to enhancement layer resolution and used as recovery of the lost enhancement layer block.
In an embodiment, the current invention proposes a temporal interpolation based method to conceal the transmission error of SVC bit stream to keep the continuity of the rendered video. It can be applied to SVC bit streams having different layers with different frame rate but is not restricted to such bit streams. The invention allows for reconstructing the signal of the higher spatial layer by a lower spatial layer signal. The invention can be applied to higher layer frames of any SVC signal of two or more layers. In an embodiment, the invention proposes adopting image processing methods to generate non-existing lower layer frames. In doing said image processing on lower layer frames, time cost of image processing can be controlled.

Claims

CLAIMS:
1. Method for recovering a lost macroblock of an
enhancement layer frame (E2) of a spatial-scalable video coding signal, said method comprises using reconstruction lower layer upsampling (RU) for recovering said lost macroblock, wherein a generated lower layer macroblock is upsampled, said generated lower layer macroblock is
generated using lower layer frames (Bl, B3) non-coincident with said enhancement layer frame (E2) and said lost macroblock is recovered from the upsampled lower layer frame .
2. Method of claim 1, wherein said non-coincident lower layer frames (Bl, B3) comprise a preceding lower layer frame (Bl) and a succeeding lower layer frame (B3) , said method further comprising generating an interpolated frame (B2*) by interpolating between the preceding lower layer frame (Bl) and a succeeding lower layer frame (B3) and selecting, from the interpolated frame (B2*) a base layer macroblock collocated with the lost enhancement layer macroblock .
3. Method of claim 2, wherein the interpolated frame (B2*) is generated by: determining optical flow (OF) to get pixel motion directions by determining corresponding pixels in said preceding lower layer frame (Bl) and said succeeding lower layer frame (B3), clustering the pixel motion
directions, using the clustered pixel motion directions for determining pixel values of the generated lower . layer macroblock using a least square minimizing method.
. Device for performing the method of one of the preceding claims.
PCT/CN2010/000501 2010-04-15 2010-04-15 Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal Ceased WO2011127628A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2010/000501 WO2011127628A1 (en) 2010-04-15 2010-04-15 Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2010/000501 WO2011127628A1 (en) 2010-04-15 2010-04-15 Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal

Publications (1)

Publication Number Publication Date
WO2011127628A1 true WO2011127628A1 (en) 2011-10-20

Family

ID=44798244

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2010/000501 Ceased WO2011127628A1 (en) 2010-04-15 2010-04-15 Method and device for recovering a lost macroblock of an enhancement layer frame of a spatial-scalable video coding signal

Country Status (1)

Country Link
WO (1) WO2011127628A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007080408A2 (en) * 2006-01-11 2007-07-19 Mitsubishi Electric Information Technology Centre Europe B.V. Error concealment for scalable video coding
WO2007080480A2 (en) * 2006-01-09 2007-07-19 Nokia Corporation Error resilient mode decision in scalable video coding
CN101242540A (en) * 2007-02-09 2008-08-13 安凯（广州）软件技术有限公司 Error control method for video decoder chip

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007080480A2 (en) * 2006-01-09 2007-07-19 Nokia Corporation Error resilient mode decision in scalable video coding
WO2007080408A2 (en) * 2006-01-11 2007-07-19 Mitsubishi Electric Information Technology Centre Europe B.V. Error concealment for scalable video coding
CN101242540A (en) * 2007-02-09 2008-08-13 安凯（广州）软件技术有限公司 Error control method for video decoder chip

Similar Documents

Publication Publication Date Title
US7787540B2 (en) Method for scalably encoding and decoding video signal
US7835452B2 (en) Method for encoding and decoding video signal
US8831102B2 (en) Method for predicting a lost or damaged block of an enhanced spatial layer frame and SVC-decoder adapted therefore
EP1809041A1 (en) Error concealement for scalable video coding
US20060133503A1 (en) Method for scalably encoding and decoding video signal
GB2480224A (en) Decoding a Sequence of Digital Images with Error Concealment
WO2006078142A1 (en) Method and apparatus for encoding/decoding video signal using block prediction information
US20080095241A1 (en) Method And Device For Coding And Decoding
WO2006080662A1 (en) Method and apparatus for effectively compressing motion vectors in video coder based on multi-layer
CN101583045B (en) SVC interlayer error concealment method based on OBMC
WO2006104366A1 (en) Method for scalably encoding and decoding video signal
WO2006104364A1 (en) Method for scalably encoding and decoding video signal
Song et al. Error concealment of H. 264/AVC video frames for mobile video broadcasting
Bajic et al. Error concealment for scalable motion-compensated subband/wavelet video coders
JP5681557B2 (en) Video playback method, video playback apparatus, and video playback program
Huu et al. A frame loss concealment solution for spatial scalable hevc using base layer motion
CN102111621B (en) Video data recovery method, device and system
KR101507243B1 (en) Decoding method for SVC using selective adaption of error concealment and decoding system for SVC therefor
WO2006104363A1 (en) Method for scalably encoding and decoding video signal
Kao et al. Motion vector field manipulation for complexity reduction in scalable video coding
KR100938553B1 (en) Boundary processing method and device using neighboring block information in scalable video encoder / decoder
Shelotkar et al. Video error concealment using H. 264/AVC
Nguyen et al. A Frame Loss Concealment Solution for Spatial Scalable HEVC using Base Layer Motion
Lam et al. Improvement on error concealment for H. 264 spatial scalable video coding
Zeng Adaptive spatial–temporal error concealment with embedded side information

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10849638

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10849638

Country of ref document: EP

Kind code of ref document: A1