Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
  • H04N19/33—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the spatial domain
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/187—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
  • H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
  • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Definitions

• This disclosure is related to the field of video coding and compression.
• SVC scalable video coding
• AVC Advanced Video Coding
• HEVC High Efficiency Video Coding
• SHVC Scalable HEVC
• Scalable video coding enables the encoding of high-quality video data that contains one or more subset bitstreams that can each be decoded with a complexity and reconstruction quality similar to existing video encoding and compression techniques.
• the subset bitstreams may be recovered by dropping certain packets from the larger bitstream.
• the subset bitstream may represent a lower spatial resolution (e.g., a smaller screen size), a lower temporal resolution (e.g., a lower frame rate), or a lower signal quality (e.g., lower signal fidelity) as compared to the larger bitstream.
• the data and decoded samples of each of these lower quality aspects (or bitstreams) can be used to predict data or samples of higher qualities/bit rates in order to reduce the bit rate to code the higher qualities. Accordingly, resolution, bit rate, and fidelity may be scaled to reduce bit rates and improve forward compatibility, as described below.
• the adaptive frame/field coding method may be supported in both a base layer (BL) and an enhancement layer (EL), meaning the base layer and enhancement layer picture can be a progressive frame, interlaced frame, a top field picture or bottom field picture.
• BL base layer
• EL enhancement layer
• Such a design is highly complicated and may present a complex solution for interlaced bitstreams.
• the design of scalable video coding may be simplified while taking the advantage of an interlaced base layer stream.
• this disclosure describes methods and techniques related to scalable video coding (SVC).
• SVC scalable video coding
• the apparatus may comprise a memory unit configured to store syntax elements for a multi-layer picture.
• a processor can be operationally coupled to the memory unit and configured to generate a syntax element indicating a phase offset value between a reference layer sample position in the multi-layer picture and a corresponding enhancement layer sample position.
• the phase offset value can represent a phase offset of the luma sample position and a chroma sample position from the reference layer sample position.
• Each of the luma sample position and the chroma sample position can have a horizontal component and a vertical component in an enhancement layer and a corresponding reference layer for the multi-layer picture.
• the processor can further encode a block based on the data encoded in the generated syntax element.
• the method may comprise storing syntax elements for a multi-layer picture.
• the method may further comprise determining at least one phase offset value between a reference layer sample position in the multi-layer picture and a corresponding enhancement layer sample position.
• the method may further comprise generating a syntax element indicating the phase offset value.
• the phase offset value can represent a phase offset of a luma sample position and a chroma sample position of the reference layer position.
• Each of the luma sample position and the chroma sample position can have a horizontal component and a vertical component in an enhancement layer and a corresponding reference layer for the multi-layer picture.
• the method may further comprise encoding a block based on the data encoded in the generated syntax element.
• the apparatus may comprise a receiver configured to receive a bitstream having syntax elements for a multi-layer picture.
• the apparatus may further comprise a memory unit configured to store syntax elements for a multi-layer picture.
• a processor can be operationally coupled to the memory unit and configured to obtain at least one phase offset value associated with the multi-layer picture from the syntax elements.
• the processor may be further configured to derive a reference layer sample position based on the at least one phase offset value.
• the at least one phase offset value can be obtained for each pair of an enhancement layer and a corresponding reference layer for the picture.
• the phase offset can represent a phase offset of a luma sample position and a chroma sample position of the reference layer sample position.
• Each of the luma sample position and the chroma sample position can have a horizontal component and a vertical component.
• the processor may be further configured to decode a block based on the received syntax elements
• the method may comprise receiving bitstream having syntax elements for a multi-layer picture.
• the method may further comprise obtaining at least one phase offset value associated with the multi-layer picture from the syntax elements.
• the method may further comprise deriving a reference layer sample position based on the at least one phase offset value.
• the at least one phase offset value can be obtained for each pair of an enhancement layer and a corresponding reference layer for the picture.
• the phase offset value can represent a phase offset of a luma sample position and a chroma sample position of the reference layer.
• Each of the luma sample position and the chroma sample position can have a horizontal component and a vertical component.
• the method may further comprise decoding a block based on the received syntax elements.
• FIG. 1 is a block diagram that illustrates an example video coding system
• FIG. 2 is a diagram depicting an example of scalabilities in different dimensions;
• FIG. 3 illustrates an example of SVC coding structure;
• FIG. 4 is a graphical representation of a plurality of access units (AU) in a bitstream
• FIG. 5A is a functional block diagram of a multi-layer video encoder
• FIG. 5B is a functional block diagram of a multi-layer video decoder
• FIG. 6 is a graphical representation of symmetrical down sampling of enhancement layer luma samples
• FIG. 7 is a graphical representation of zero-phase down- sampling.
• FIG. 8 is a graphical representation of an implementation of up-sampling locations in zero phase down- sampling.
• the embodiments described in this disclosure generally relate to scalable video coding (SHVC, SVC) and multiview/3D video coding (e.g., multiview coding plus depth, MVC+D).
• SHVC High Efficiency Video Coding
• SVC extension there could be multiple layers of video information.
• the layer at the very bottom level may serve as a base layer (BL), and the layer at the very top (or the highest layer) may serve as an enhanced layer (EL).
• BL base layer
• EL enhanced layer
• the "enhanced layer” is sometimes referred to as an "enhancement layer,” and these terms may be used interchangeably.
• the base layer is sometimes referred to as a "reference layer,” (RL) and these terms may also be used interchangeably.
• All layers in between the base layer and the top layer may serve as either or both ELs or reference layers (RLs).
• RLs reference layers
• a layer in the middle may be an EL for the layers below it, such as the base layer or any intervening enhancement layers, and at the same time serve as a RL for the enhancement layers above it.
• Each layer in between the base layer and the top layer (or the highest layer) is may be used as a reference for inter-layer prediction by a higher layer and may use a lower layer as a reference for inter-layer prediction.
• Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smart phones," video teleconferencing devices, video streaming devices, and the like.
• the video devices may transmit, receive, encode, decode, and/or store digital video information more efficiently by implementing embodiments described herein.
• Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences.
• a video slice e.g., a video frame or a portion of a video frame
• video blocks which may also be referred to as treeblocks, coding units (CUs) and/or coding nodes.
• Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture.
• Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures.
• Pictures may be referred to as frames, and reference pictures may be referred to a reference frames.
• Spatial or temporal prediction results in a predictive block for a block to be coded. Residual data represents pixel differences between the original block to be coded and the predictive block.
• An inter-coded block is encoded according to a motion vector that points to a block of reference samples forming the predictive block, and the residual data indicating the difference between the coded block and the predictive block.
• An intra-coded block is encoded according to an intra-coding mode and the residual data.
• the residual data may be transformed from the pixel domain to a transform domain, resulting in residual transform coefficients, which then may be quantized.
• the quantized transform coefficients initially arranged in a two- dimensional array, may be scanned in order to produce a one-dimensional vector of transform coefficients, and entropy coding may be applied to achieve even more compression.
• Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU- T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual and ITU- T H.264 (also known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) and Multi-view Video Coding (MVC) extensions.
• SVC Scalable Video Coding
• MVC Multi-view Video Coding
• HEVC High Efficiency Video Coding
• JCT-VC Joint Collaboration Team on Video Coding
• VCEG ITU-T Video Coding Experts Group
• MPEG ISO/IEC Motion Picture Experts Group
• HEVC WD The latest HEVC draft specification, and referred to as "HEVC WD" hereinafter, is available from http://phenix.int- evry.fr/jct/doc_end_user/documents/15_Geneva/wgl l/JCTVC-O1003-vl .zip.
• MV-HEVC multi-view extension to HEVC
• 3D-HEVC 3D-HEVC
• SHVC scalable video coding extension to HEVC
• 3D-HEVC WD1 The latest WD of 3D-HEVC, referred to as 3D-HEVC WD1 hereinafter, is available from http://phenix.it-sudparis.eu/jct2/doc_end_user/documents/6_Geneva /wg 11/JCT3V- F1001-v3.zip.
• a recent Working Draft (WD) of SHVC and referred to as SHVC WD3 hereinafter, is available from http://phenix.it- sudparis.eu/jct/doc_end_user/documents/15_Geneva/wgl l/JCTVC-O1008-v3.zip.
• Scalable video coding may be used to provide quality (also referred to as signal-to-noise (SNR)) scalability, spatial scalability and/or temporal scalability.
• a reference layer e.g., a base layer
• the enhancement layer includes additional video information relative to the reference layer such that the reference layer and the enhancement layer together include video information sufficient to display the video at a second quality level higher than the first level (e.g., less noise, greater resolution, better frame rate, etc.).
• An enhanced layer may have different spatial resolution than a base layer.
• the spatial aspect ratio between EL and BL can be 1.0, 1.5, 2.0 or other different ratios in vertical and horizontal directions.
• the spatial aspect of the EL may equal 1.0, 1.5, or 2.0 times the spatial aspect of the BL.
• the scaling factor of the EL may be greater than the BL.
• a size of pictures in the EL may be greater than a size of pictures in the BL. In this way, it may be possible, although not a limitation, that the spatial resolution of the EL is larger than the spatial resolution of the BL.
• inter-layer prediction refers to the SVC extension for H.264 or the SHVC extension for H.265 (as discussed above)
• prediction of a current block may be performed using the different layers that are provided for SVC.
• Such prediction may be referred to as inter-layer prediction.
• Inter-layer prediction methods may be utilized in SVC in order to reduce inter-layer redundancy.
• Some examples of inter-layer prediction may include inter-layer intra prediction, inter-layer motion prediction, and inter-layer residual prediction.
• Inter-layer intra prediction uses the reconstruction of co-located blocks in the base layer to predict the current block in the enhancement layer.
• Inter-layer motion prediction uses motion information (including motion vectors) of the base layer to predict motion in the enhancement layer.
• Inter-layer residual prediction uses the residue of the base layer to predict the residue of the enhancement layer.
• a resampling (or upsampling) process can be applied to the reference layer picture to match the size of the enhancement layer picture for inter-layer prediction.
• an N tap resampling filter can be applied for each color component.
• the sample (or pixel) magnitudes of the reference layer picture can be multiplied by filter coefficients and summed up. Since the size of the reference layer picture and the size of the enhancement layer picture are different, the coordinates of the reference layer samples involved in the filtering process may be defined. For example, the sample location of the reference layer picture that corresponds to the sample location of the current enhancement layer picture can be determined so that sample(s) indicated by the sample location of the reference layer picture can be used in the resampling process.
• High level syntax-only (HLS- only) scalable video coding enables video blocks in a current layer to be predicted using video blocks from a reference layer without introducing low-level changes to the HEVC coding specification.
• HLS- only SVC enables such coding by using existing inter coding techniques with reference layers from the same access unit of the current layer. Some techniques may enable multiple reference layers to be identified for possible use in inter-layer coding.
• SHVC does not support scalable video coding with the base layer containing field pictures coded based on H.264/AVC while the enhancement layer includes frame pictures coded based on SHVC.
• migration from 1080i to 1080p with SHVC can be used, for example, in broadcasting since the H.264/AVC 1080i bitstreams may be widely used.
• this is also useful when different layers have distinct color formats, such as base layer having YUV420 format and an adjacent layer having YUV422 or YUV 444 color format. To support those functionalities, the upsampling process of SHVC can be modified.
• FIG. 1 is a block diagram that illustrates an example video coding system 10 that may utilize techniques in accordance with aspects described in this disclosure.
• video coder refers generically to both video encoders and video decoders.
• video coding or “coding” may refer generically to video encoding and video decoding.
• video coding system 10 includes a source device 12 and a destination device 14.
• the source device 12 generates encoded video data.
• the destination device 14 may decode the encoded video data generated by source device 12.
• the source device 12 can provide the video data to the destination device 14 via a communication channel 16, which may include a computer-readable storage medium or other communication channel.
• the source device 12 and the destination device 14 may include a wide range of devices, including desktop computers, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets, such as so-called “smart” phones, so-called “smart” pads, televisions, cameras, display devices, digital media players, video gaming consoles, in-car computers, video streaming devices, or the like.
• the source device 12 and destination device 14 may be equipped for wireless communication.
• the destination device 14 may receive the encoded video data to be decoded via the communication channel 16.
• Communication channel 16 may comprise a type of medium or device capable of moving the encoded video data from the source device 12 to the destination device 14.
• the communication channel 16 may comprise a communication medium to enable the source device 12 to transmit encoded video data directly to the destination device 14 in real-time.
• the encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14.
• the communication medium may comprise a wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines.
• RF radio frequency
• the communication medium may form part of a packet-based network, such as a local area network, a wide-area network, or a global network, such as the Internet.
• the communication medium may include routers, switches, base stations, or other equipment that may be useful to facilitate communication from the source device 12 to the destination device 14.
• encoded data may be output from an output interface 22 to a storage device.
• the channel 16 may correspond to a storage device or computer-readable storage medium that stores the encoded video data generated by the source device 12.
• the destination device 14 may access the computer-readable storage medium via disk access or card access.
• encoded data may be accessed from the computer-readable storage medium by an input interface 28.
• the computer-readable storage medium may include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or other digital storage media for storing video data.
• the computer-readable storage medium may correspond to a file server or another intermediate storage device that may store the encoded video generated by the source device 12.
• the destination device 14 may access stored video data from the computer-readable storage medium via streaming or download.
• the file server may be a type of server capable of storing encoded video data and transmitting that encoded video data to the destination device 14.
• Example file servers include a web server (e.g., for a website), an FTP server, network attached storage (NAS) devices, or a local disk drive.
• the destination device 14 may access the encoded video data through a standard data connection, including an Internet connection.
• This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server.
• the transmission of encoded video data from the computer-readable storage medium may be a streaming transmission, a download transmission, or a combination of both.
• the embodiments in this disclosure can apply to applications or settings in addition to wireless applications or settings.
• the embodiments may be applied to video coding in support of a of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions, such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.
• the system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.
• the source device 12 includes the video source 18, the video encoder 20, and the output interface 22.
• the destination device 14 includes an input interface 28, video decoder 30, and display device 32.
• the video encoder 20 of the source device 12 may be configured to apply the techniques for coding a bitstream including video data conforming to multiple standards or standard extensions.
• the source device 12 and the destination 14 device may include other components or arrangements.
• the source device 12 may receive video data from an external video source 18, such as an external camera.
• the destination device 14 may interface with an external display device, rather than including an integrated display device.
• the video source 18 of the source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface to receive video from a video content provider.
• the video source 18 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer generated video.
• the source device 12 and the destination device 14 may form so-called camera phones or video phones.
• the captured, pre-captured, or computer-generated video may be encoded by the video encoder 20.
• the encoded video information may be output by the output interface 22 to a communication channel 16, which may include a computer-readable storage medium, as discussed above.
• Computer-readable storage medium may include transient media, such as a wireless broadcast or wired network transmission, or storage media (e.g., non-transitory storage media), such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, or other computer-readable media.
• a network server (not shown) may receive encoded video data from the source device 12 and provide the encoded video data to the destination device 14 (e.g., via network transmission).
• a computing device of a medium production facility such as a disc stamping facility, may receive encoded video data from the source device 12 and produce a disc containing the encoded video data. Therefore, the communication channel 16 may be understood to include one or more computer-readable storage media of various forms.
• the input interface 28 of the destination device 14 can receive information from the communication channel 16.
• the information of the communication channel 16 may include syntax information defined by the video encoder 20, which can be used by video decoder 30, that includes syntax elements that describe characteristics and/or processing of blocks and other coded units, e.g., a group of pictures (GOPs).
• the display device 32 displays the decoded video data to a user, and may include any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.
• CTR cathode ray tube
• LCD liquid crystal display
• OLED organic light emitting diode
• the encoder 20 and the video decoder 30 may operate according to a video coding standard, such as the High Efficiency Video Coding (HEVC) standard, and may conform to the HEVC Test Model (HM).
• HEVC High Efficiency Video Coding
• HM HEVC Test Model
• video encoder 20 and video decoder 30 may operate according to other proprietary or industry standards, such as the ITU-T H.264 standard, alternatively referred to as MPEG 4, Part 10, Advanced Video Coding (AVC), or extensions of such standards.
• the embodiments in this disclosure are not limited to any particular coding standard.
• Other examples of video coding standards include MPEG-2 and ITU-T H.263.
• the video encoder 20 and the video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate MUX-DEMUX units, or other hardware and software, to handle encoding of both audio and video in a common data stream or separate data streams. If applicable, MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).
• MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).
• FIG. 1 is an example of a video coding/decoding system and the embodiments in this disclosure may apply to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices.
• data can be retrieved from a local memory, streamed over a network, or the like.
• An encoding device may encode and store data to memory, and/or a decoding device may retrieve and decode data from memory.
• the encoding and decoding is performed by devices that do not communicate with one another, but simply encode data to memory and/or retrieve and decode data from memory.
• the video encoder 20 and the video decoder 30 each may be implemented as any of a variety of suitable encoder circuitry, such as one or more processors or microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof.
• DSPs digital signal processors
• ASICs application specific integrated circuits
• FPGAs field programmable gate arrays
• a device may store instructions for the software in a non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the embodiments in this disclosure.
• Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective device.
• a device including the video encoder 20 and/or the video decoder 30 may comprise an integrated circuit, a microprocessor, and/or a wireless communication device, such as a cellular telephone.
• FIG. 2 is a diagram depicting an example of scalabilities in different dimensions.
• the scalabilities may be applied to data encoded or transferred from the encoder 21 to the decoder 31, as discussed above.
• the scalabilities are enabled in three dimensions: time, space, and quality, or Signal to Noise Ratio (SNR).
• temporal scalability (T) denoted by the time dimension along the horizontal (x) axis, may support various frame rates, for example, 7.5 Hz, 15 Hz or 30 Hz, as shown.
• the vertical (y) axis shows spatial scalability (S), for example the screen size.
• spatial scalability supports different resolutions such as, for example, Common Intermediate Format (CIF), Quarter Common Intermediate Format (QCIF), and Four Quarter Common Intermediate Format (4CIF).
• CIF Common Intermediate Format
• QCIF Quarter Common Intermediate Format
• 4CIF Four Quarter Common Intermediate Format
• Q SNR
• SNR is represented along the z axis.
• SVC standardizes the encoding of a high-quality video bitstream that also contains one or more subset bitstreams.
• a subset video bitstream may be derived by dropping packets from the larger video bitstream to reduce the bandwidth required for the subset bitstream.
• the subset bitstream can represent a lower spatial resolution (smaller screen), lower temporal resolution (lower frame rate), or lower quality video signal.
• an extractor tool at the decoder may be used to adapt the actual delivered content according to application requirements, which may be dependent, for example, on the client(s) or the transmission channel. As a non-limiting example, this may allow a user to view a lower resolution version of a video extracted from very high resolution transmission on a mobile device.
• FIG. 2 a plurality of cubes 105 are shown, distributed based on T, S, and Q values. As shown, only cubes 105a and 105b are labeled to preserve figure clarity.
• Each of the cubes 105 is representative of pictures or video having the same frame rate (e.g., the temporal level), spatial resolution, and SNR layers.
• Each of the cubes 105 is depicted having a value for T, S, and Q (e.g., 0,0,0) depending on the encoded layer.
• the cube 105a has TO, SO, Q0 indicating a single layer (e.g., the base layer, or layer 0) for each of the three scalabilities.
• the cube 105b annotated as "Tl, S2, Q0," indicating a second layer in T, a third layer in S, and the base layer in Q.
• the pictures having the lowest spatial and quality layer are compatible with a single layer video codec, and the pictures at the lowest temporal level such as the cube 105a, form the temporal base layer, which can be enhanced with pictures at higher temporal levels.
• several spatial and/or SNR enhancement layers can be added to provide further spatial and/or quality scalabilities.
• SNR scalability may also be referred as quality (Q) scalability.
• Each spatial or SNR enhancement layer may be temporally scalable, with the same temporal scalability structure as the base layer.
• the lower layer it depends on is also referred as the base layer (BL) of that specific spatial or SNR enhancement layer.
• FIG. 3 illustrates an example of SVC coding structure.
• a base layer 302 may have the lowest spatial and quality layer (for example, the pictures in layer 0 and layer 1 of FIG. 2, having QCIF resolution and a 7.5 Hz frame rate and 64Kbps bit rate.
• QCIF images or videos are 176 pixels wide and 144 pixels tall (176 x 144 pixels).
• the name Quarter CIF is written as QCIF and the resolution is one quarter the size of CIF resolution (352 x 288 pixels).
• QCIF is smaller than CIF, QVGA, and VGA.
• the base layer 302 may include a plurality of coded slices or frames, depending on the compression protocol.
• the layer 0 302 may include a series of frames, starting with an i-frame "Io."
• Frame Io may also be referred to herein as a frame 320a.
• the frame 320a is an I-frame, or intra-coded picture, for example.
• the subsequent frames 320b-320f may include a B (bi-predicted picture) frame or a P frame (predicted picture) in order to encode the image data.
• the temporal base layer 302 (layer 0) can be enhanced with pictures of higher temporal levels, such as a layer 1 304 (the second layer) increasing the frame rate to 15Hz and bit rate to 96Kbps.
• a layer 1 304 the second layer
• several spatial and/or SNR enhancement layers can be added to provide spatial and/or quality scalabilities.
• an enhancement layer 2 306 may be added, having an increased bit rate of 256Kbps while maintaining the same spatial resolution (CIF) as in the layer 1 304.
• a layer 2 308 (the third layer) can further be added as a CIF representation of the image data with the same resolution as layer 2 304.
• the layer 3 308 may be an SNR enhancement layer.
• each spatial or SNR enhancement layer 306, 308 may be temporally scalable, with the same temporal scalability structure as the base layer 302.
• an enhancement layer can enhance both spatial resolution and frame rate.
• a layer 3 310 (the fourth layer) may provide a 4CIF enhancement layer, which further increases the frame rate from 15 Hz to 30 Hz, doubling for example, the number of slices or frames transmitted.
• 4CIF may generally be defined as being four times the size of CIF, or 704 x 576 pixels.
• FIG. 4 s a graphical representation of a plurality of access units (AU) in a bitstream.
• a plurality of AUs 405 may be formed from the coded slices 320 described in connection with FIG. 3.
• Each of the coded slices 320 in the same time instance are successive in the bitstream order and form one access unit 405 in the context of SVC.
• Those SVC access units 405 then follow the decoding order, which may be different from the display order.
• the decoding order may further be decided, for example, by the temporal prediction relationship between the slices 320 within the AUs 405.
• the high-level syntax provides the encapsulation for the coded video for further processing.
• the high level syntax includes the structure of the bitstream as well as signaling of high-level information that applies to one or more entire slices of pictures of a bitstream.
• the high-level syntax indicates the spatial resolution of the video, which coding tools might be employed, and describes certain random access functionalities of the bitstream.
• the high-level tool decoding processes associated with the syntax elements are also considered to be included in high level syntax.
• high- level syntax decoding processes may include reference picture management and the output of decoded pictures.
• up-sampled collocated reference layer picture(s) may be put into a memory or a memory unit, such as a reference buffer (e.g., a reference frame memory, described below) of the enhancement layer so that inter-layer prediction is achieved in the same way as inter-frame prediction in the same layer.
• the ILR picture may be marked as a long term reference picture.
• the motion vector difference of inter-layer reference may be constrained to 0.
• FIG. 5A is a functional block diagram of a multi-layer video encoder.
• a multi-layer video encoder (“video encoder” or “encoder”) 21 that may implement techniques in accordance with aspects described in this disclosure.
• the video encoder 21 may be configured to process multi-layer video frames, such as for SHVC and multiview coding. Further, the video encoder 21 may be configured to perform any or all of the embodiments in this disclosure.
• the video encoder 21 includes a video encoder 20a and video encoder 20b, each of which may be configured as the video encoder 20 of FIG. 1A, also indicated by the reuse of reference numerals.
• the video encoder 21 is shown including two video encoders 20A and 20B, however the video encoder 21 may include any number of video encoder 20 layers.
• the video encoder 21 may include a video encoder 20 for each picture or frame in an access unit 405 (FIG. 4).
• the access unit 405 that includes four pictures may be processed or encoded by a video encoder that includes four encoder layers 20.
• the video encoder 21 may include more encoder layers than frames in an access unit (e.g., the access unit 405). In some such cases, some of the video encoder layers may be inactive when processing some access units 405.
• the encoders 20a, 20b (collectively, "video encoders 20") of FIG. 5A each illustrate a single layer of a codec. However, some or all of the video encoders 20 may be duplicated for processing according to a multi-layer codec.
• the video encoders 20 may perform intra-layer and inter-layer prediction (sometime referred to as intra-, inter- or inter-layer coding) of video blocks within video slices.
• Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture.
• Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or pictures of a video sequence.
• Inter-layer coding relies on prediction based upon video within a different layer(s) within the same video coding sequence.
• Intra-mode (I mode) may refer to any of several spatial based coding modes.
• Inter-modes, such as unidirectional prediction (P mode) or bi-prediction (B mode) may refer to any of several temporal-based coding modes.
• the video encoders 20 receive a current video block within a video frame to be encoded.
• the video encoder 20 includes a mode select unit 40, a reference frame memory (“RFM") 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy encoding unit 56.
• the mode select unit 40 may include a motion compensation unit, a motion estimation unit, an intra-prediction unit, an inter-layer prediction unit, and a partition unit.
• the individual components are not depicted for brevity, but their individual functions are discussed in relation to the mode select unit 40.
• the RFM 64 may include a decoded picture buffer ("DPB").
• DPB decoded picture buffer
• the DPB is a broad term having its ordinary meaning, and in some embodiments refers to a video codec-managed data structure of reference frames.
• the RFM 64 may be further configured to store an ILR picture 65, labeled in dashed lines.
• the ILR picture 65 may provide a reference for inter-layer prediction.
• the video encoders 20 may also include an inverse quantization unit 58, an inverse transform unit 60, and a summer 62.
• a deblocking filter (not shown in FIG. 5A) may also be included to filter block boundaries to remove blockiness artifacts from reconstructed video.
• the deblocking filter may filter the output of the summer 62. Additional filters (in loop or post loop) may also be used in addition to the deblocking filter. Such filters are not shown for brevity, but if implemented, may filter the output of summer 50 (as an in-loop filter).
• the video encoders 20 receive certain video data 23, such as a video frame or slice to be coded.
• the frame or slice may be divided into multiple video blocks.
• the motion estimation unit and motion compensation unit as a function of the mode select unit 40, may perform inter-predictive coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction.
• the intra-prediction unit may alternatively perform intra- predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction.
• the video encoders 20a, 20b may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.
• the partition unit also within the mode select unit 40, may partition blocks of video data into sub-blocks, based on evaluation of previous partitioning schemes in previous coding passes.
• the video encoder 21 may include a resampling unit 90.
• the resampling unit 90 may, in some cases, upsample a base layer of a received video frame to, for example, create an enhancement layer.
• the resampling unit 90 may upsample particular information associated with the received base layer of a frame, but not other information.
• the resampling unit 90 may upsample the spatial size or number of pixels of the base layer, but the number of slices or the picture order count may remain constant.
• the resampling unit 90 may not process the received video and/or may be optional.
• the mode select unit 40 may perform upsampling.
• the resampling unit 90 is configured to upsample a layer and reorganize, redefine, modify, or adjust one or more slices to comply with a set of slice boundary rules and/or raster scan rules. Although primarily described as upsampling a base layer, or a lower layer in an access unit 405 (FIG. 4), in some cases, the resampling unit 90 may downsample a layer. For example, if during streaming of a video, the bandwidth is reduced, a frame may be downsampled instead of upsampled. Resampling unit 90 may be further configured to perform cropping and/or padding operations, as well.
• the resampling unit 90 may be configured to receive a picture (e.g., the ILR picture 65) or frame (or picture information associated with the picture) from the RFM 64 of the lower layer encoder 20a and to upsample the picture (or the received picture information).
• the received picture is an inter-layer reference picture stored within the RFM 64.
• the upsampled picture may then be provided to the mode select unit 40 of a higher layer encoder 20b configured to encode a picture in the same access unit as the lower layer encoder.
• the higher layer encoder 20b is one layer removed from the lower layer encoder 20a.
• the resampling unit 90 may be omitted or bypassed.
• the picture from the RFM 64a e.g., the ILR picture 65a
• the mode select unit 40b of the video encoder 20b may be provided directly, or at least without being provided to the resampling unit 90, to the mode select unit 40b of the video encoder 20b.
• the reference picture may be provided to the video encoder 20b without any resampling.
• the video encoder 21 downsamples video data to be provided to the lower layer encoder using the downsampling unit 94 before provided the video data to the video encoder 20a.
• the downsampling unit 94 may be a resampling unit 90 capable of upsampling or downsampling the video data.
• the downsampling unit 94 may be omitted.
• the video encoder 21 may further include a multiplexor ("mux") 98.
• the mux 98 can output a combined bitstream from the video encoder 21.
• the combined bitstream may be created by taking a bitstream from each of the video encoders 20a and 20b and alternating which bitstream is output at a given time. While in some cases the bits from the two (or more in the case of more than two video encoder layers) bitstreams may be alternated one bit at a time, in many cases the bitstreams are combined differently.
• FIG. 5B is a functional block diagram of a multi-layer video decoder.
• a multi-layer decoder (“decoder” or “video decoder”) 31 may be configured to process multi-layer video frames, such as for SHVC and multiview coding. Further, the video decoder 31 may be configured to perform any or all of the embodiments in this disclosure.
• the video decoder 31 includes a video decoder 30a and video decoder 30b, each of which may be configured as the video decoder 30 of FIG. 1A. Further, as indicated by the reuse of reference numbers, the video decoders 30a and 30b may include at least some of the systems and subsystems as the video decoder 30. Although the video decoder 31 is illustrated as including two video decoders 30a and 30b, the video decoder 31 may include any number of video decoder 30 layers. In some embodiments, the video decoder 31 may include a video decoder 30 for each picture or frame in an access unit 405 (FIG. 4).
• the access unit 405 that includes four pictures may be processed or decoded by a video decoder that includes four decoder layers.
• the video decoder 31 may include more decoder layers than frames in the access unit 405.
• some of the video decoder layers may be inactive when processing some access units, for example access units 405d, 405e.
• the video decoder 31 may further include an upsampling unit 92.
• the upsampling unit 92 may upsample a base layer of a received video frame to create an enhanced layer to be added to the reference picture list for the frame or access unit 405. This enhanced layer can be stored in the reference frame memory ("RFM") 82 (e.g., in its decoded picture buffer, etc.).
• the upsampling unit 92 can include some or all of the embodiments described with respect to the resampling unit 90 of FIG. 5A.
• the upsampling unit 92 is configured to upsample a layer and reorganize, redefine, modify, or adjust one or more slices to comply with a set of slice boundary rules and/or raster scan rules.
• the upsampling unit 92 may be a resampling unit configured to upsample and/or downsample a layer of a received video frame.
• the RFM 82 may be further configured to store an ILR picture 83, similar to the ILR picture 65.
• the ILR picture 83 is shown in dashed lines indicating its storage within the RFM 82.
• the ILR picture 83 may be further utilized in inter-layer prediction as described above.
• the upsampling unit 92 may be configured to receive a picture or frame (or picture information associated with the picture) from the RFM 82 of the lower layer decoder (e.g., the video decoder 30a) and to upsample the picture (or the received picture information).
• This upsampled picture (e.g., the ILR picture 83a) may then be provided to the mode select unit 71b of a higher layer decoder (e.g., the video decoder 30b) configured to decode a picture in the same access unit 405 as the lower layer decoder.
• the higher layer decoder 30b is one layer removed from the lower layer decoder 30a. In other cases, there may be one or more higher layer decoders between the layer 0 decoder 30a and the layer 1 decoder 30b.
• the upsampling unit 92 may be omitted or bypassed.
• the picture (e.g., the ILR picture 83a) from the RFM 82a of the video decoder 30a may be provided directly, or at least without being provided to the upsampling unit 92, to the mode select unit 71b of the video decoder 30b.
• the upsampling unit 92 may be a resampling unit 90 configured to upsample or downsample a reference picture received from the RFM 82 of the video decoder 30a.
• the video decoder 31 may further include a demultiplexer ("demux") 99.
• the demux 99 can split a multiplexed or encoded video bitstream into multiple bitstreams with each bitstream output by the demux 99 being provided to a different video decoder 30a and 30b.
• the multiple bitstreams may be created by receiving a bitstream and each of the video decoders 30a and 30b receives a portion of the bitstream at a given time.
• the base layer picture and the enhancement layer picture may have different sizes or resolutions.
• the width of the base layer picture is half of that of the enhancement layer picture
• the height of the base layer picture is half of that of the enhancement layer picture.
• the base layer sequence may be generated by applying down- sampling of the original sequences.
• the enhancement layer sequence may be the original sequence.
• up-sampling may be applied to the reconstructed base layer picture. Such a process or method may be applied to one of the ILR pictures 65 (FIG. 5A) or the ILR pictures 83 (FIG. 5B), for example.
• a resampling filter may downsample at different sampling locations.
• two sampling locations are shown in the following FIG. 6 and FIG. 7.
• the diagrams depict example pixel locations of a base layer picture and an enhancement later picture.
• the enhancement layer pixels are indicated by white squares, while the base layer pixels are indicated by solid circles.
• FIG. 6 is a graphical representation of symmetrical down sampling of enhancement layer luma samples.
• an array 600 includes a plurality of enhancement layer luma samples 602 that are down-sampled from a corresponding set of pixels to a lower spatial resolution.
• the exemplary array 600 includes a 4x4 array of enhancement layer luma samples 602.
• the resolution of the down-sampled enhancement layer luma samples 602 is then notionally one-quarter of the resolution, as indicated by a plurality of base layer luma samples 610 (e.g., half of the vertical and half of the horizontal resolution).
• the base layer luma samples 610 are shown in a 2x2 array 605.
• the array 605 is indicated by the dotted arrow and dotted box including the base layer luma samples 610.
• the 4x4 array 600 of enhancement layer luma samples 602 may be derived from an enhancement layer, for example, from one of the enhancement layers 306, 308, 310 (FIG. 3).
• the 4x4 array of the enhancement layer luma samples 602 is down-sampled into the 2x2 array 605 in a base layer.
• the two arrays 600, 605 may have the same center position 620.
• the down-sampled 2x2 array 605 may include the base layer luma samples 610 which may individually have a certain phase offset 650 from the enhancement layer luma samples 602.
• the phase offset 650 may have a horizontal component 650a and a vertical component 650b.
• the horizontal component 650a and vertical component 650b may be referred to collectively herein as the "phase,” “offset,” or “phase offset.”
• the "phase,” or “phase offset” is described below with reference to luma and chroma references in other examples.
• the base layer luma samples 610 may be located in the middle of four surrounding enhancement layer luma samples 602.
• the downsampled luma sample 610a is surrounded by the enhancement layer luma samples 602a, 602b, 602e, 602f.
• the phase offset 650 may then be half of the distance between the adjacent pixels, for example, referenced from the enhancement layer luma sample 602a.
• the base layer luma samples 610 may have an average or mean value or position of the four surrounding enhancement layer pixels 602a, 602b, 602e, 602f.
• FIG. 7 is a graphical representation of zero-phase down-sampling.
• a plurality of enhancement layer luma samples 602 are again shown in a 4x4 array 700.
• the spatial distance, or phase e.g. the phase offset 650
• phase generally refers to the spatial distance between the left-top sample in the enhancement layer and the left-top sample in the base layer; hence the term "zero-phase.”
• the base layer luma sample 710b also coincides with the enhancement layer luma sample 602c. Accordingly, the center position (e.g., the center position 620 of FIG. 6) moves to a center position 750, which is also collocated with the enhancement layer luma sample 602f.
• the square shapes indicate the locations of luma samples 602, 702 taken from the enhancement layer pixels, whereas the round shapes indicate the locations of down- sampled pixels, which may ultimately form the base layer picture.
• FIG. 8 is a graphical representation of an implementation of up-sampling locations in zero phase down- sampling.
• a 4x4 array 800 including a plurality of pixels 802 is shown in a similar manner to the luma samples 602 (FIG. 6) and luma samples 702 (FIG. 7).
• the down- sampling location and phase information for each pixel is needed in the decoding and up- sampling process.
• the enhancement layer pixels 802a, 802b may also need to be up-sampled from the base layer pixels 710 (e.g., the pixels that were down-sampled in FIG. 7). Since the pixel 802a and the pixel 710a are collocated, the phase of the up- sampling filter for generating the pixel 802a is zero (0).
• phase 850 of one half the distance must be applied in the up-sampling filter for generating the pixel 802b.
• 2X spatial scalability with a zero- phase down-sampling filter results in a zero phase for some pixels 802a and one-half phase 850 for the pixels 802b of the up-sampling filter.
• the phase 850 of each pixel, luma sample, and/or chroma sample may be required as described below.
• phase offsets e.g., the phase 650, 850
• phase offsets may be required.
• the luma sample 602b is one quarter the distance between the luma sample 610a and the luma sample 610b.
• the luma sample 602c is three quarters the distance from the luma sample 610a to the luma sample 610b. The same measurements can be applied to the vertical and horizontal phases allowing upsampling from symmetrical phase downsampled pixels.
• a resampling (or upsampling) process can be applied to the reference layer picture to match the size of the enhancement layer picture for inter- layer prediction.
• an N tap resampling filter can be applied for each color component.
• the "N" may generally refer to the number of layers within codecs applied to the transmitted video.
• an eight (8) tap filter may have eight encoder/decoder layers in the respective codec. This is discussed in more detail below in table IThe following sections describe certain syntax and decoding processes for a given resampling process.
• the syntax and decoding process for the resampling process in SHVC may be as follows. (see, e.g., working draft http://phenix.it- sudparis.eu/jct/doc_end_user/documents/15_Geneva/wgl l/JCTVC-O1008-v3.zip).
• Video parameter set extension syntax
• cross_layer_phase_alignment_flag implemented during processes related to upsampling and downsampling video data (e.g., SHVC data).
• the cross_layer_phase_alignment_flag may be limited to having a binary value, for example, zero (0) or one (1).
• the cross_layer_phase_alignment_flag 1 may specify that the locations of the luma sample grids of all layers are aligned at the center sample position of the pictures, as shown in FIG. 6.
• a cross_layer_phase_alignment_flag 0 may specify that the locations of the luma sample grids of all layers are aligned at the top-left sample position of the picture, as shown in FIG. 7.
• the phase or phase offset 650, 850 may vary according to the center position 620 or 750. Accordingly, related syntax elements may vary.
• the resampling process may require certain inputs, including the phase or phase offset (e.g., x and y offest) from a reference point. This may be done for both luma and chroma samples of a video bitstream. Inputs to this process are a variable, cidx, specifying the color component index, and a sample location (xP, yP) relative to the top-left sample (e.g., the sample 602a of FIG. 7) of the color component of the current picture specified by cidx.
• phase or phase offset e.g., x and y offest
• output of this process is a sample location (xRefl6, yRefl6) specifying the reference layer sample location in units of 1/16-th sample relative to the top-left sample of the reference layer picture.
• the top-left sample may be similar to the enhancement layer luma sample 602a of FIG. 6, FIG. 7, and FIG. 8.
• the l/16 _th sample may also correspond to 1/16-th of the distance between two adjacent pixels, or for example, the enhancement layer luma samples 602a and 602b.
• variables xRefl6 and yRef 16 may be derived as follows:
• xRefl6 ( ( ( xP - offsetX ) * ScaleFactorX + addX + ( 1 « 11 ) ) » 12 ) - ( phaseX « 2 )
• the Luma sample interpolation process may require the luma reference sample array (e.g., the array 700 of FIG. 7) rlPicSample L , and a luma sample location (xP, yP) relative to the top-left luma sample of the current picture.
• the luma reference sample array e.g., the array 700 of FIG. 7
• rlPicSample L e.g., the array 700 of FIG. 7
• a luma sample location xP, yP
• the output of this process is an interpolated luma sample value intLumaSample .
• the luma sample value intLumaSample may correspond to the location of the pixel 802b (FIG. 8).
• the value of the interpolated luma sample IntLumaSample is derived by applying the following ordered steps:
• yPosRL Clip3( 0, RefLayerPicHeightlnSamplesY - 1, yRef + n - 1 ) (H-30)
• refW RefLayerPicWidthlnSamplesY
• intLumaSample ( f L [ yPhase, 0 ] * tempArray [ 0 ] +
• intLumaSample Clip3( 0, ( 1 « BitDepthy) - 1 , intLumaSample ) (H-33)
• chroma interpolation may require the chroma reference sample array rlPicSample c , and a chroma sample location (xPc, yPc) relative to the top- left chroma sample of the current picture.
• the chroma interpolation may follow a similar process as the luma interpolation. The output of this process is a interpolated chroma sample value "intChromaS ample.”
• the derivation process for reference layer sample location in resampling is invoked with cidx and chroma sample location (xPc, yPc) given as the inputs and (xRefl6, yRefl6) in units of 1/16-th sample as output.
• shiftl RefLayerBitDepthc - 8
• tempArray[ n ] ( f c [ xPhase, 0 ] * rlPicSample c [ Clip3( 0, refWC - 1, xRef - 1), yPosRL ] + f c [ xPhase, 1 ] * rlPicSample c [ Clip3( 0, refWC - 1, xRef ), yPosRL ] + f c [ xPhase, 2 ] * rlPicSample c [ Clip3( 0, refWC - 1, xRef + 1 ), yPosRL ] +
• SHVC does not support SVC with the base layer including field pictures coded based on H.264/AVC while the enhancement layer(s) includes frame pictures coded based on HEVC. It may, however, be advantageous to migrate from 1080i (interlaced video) to 1080p (progressive video) with SHVC because of frequent use of H.264/AVC 1080i bitstreams for broadcasting.
• one approach for high coding efficiency of interlaced video sequence involves supporting the adaptive frame/field coding in both the base layer and the enhancement layer, such that the base layer (e.g., the layer 0 302 of FIG. 3) and the enhancement layer picture (or pictures) can be a progressive frame, an interlaced frame, a top field picture, or a bottom field picture.
• the base layer e.g., the layer 0 302 of FIG. 3
• the enhancement layer picture or pictures
• a lightweight, efficient scheme to support the interlaced base layer in SHVC should be implemented.
• one or more flags may be signaled to support interlayer prediction from field picture(s) of 1080i to frame picture(s) of 1080p, for example.
• a flag "bottom_field_to_frame_resampling_flag" is signaled to specify the resampling process method to generate the interlayer reference picture.
• this flag may be an indication that the input of the resampling process invoked to generate the interlayer reference pictures for the current picture is a bottom field picture and the output is a frame picture. If this flag is set to or equals 0, this may be an indication that the above described restriction does not apply. In at least one example, this flag may be signaled in the slice segment header, or any other high level syntax part. In another example, the value of this flag may be assigned via an external approach. For example, the flag is provided by the system level application.
• a flag "field_to_frame_resampling_present_flag" may be signaled to control the presence of the bottom_field_to_frame_resampling_flag.
• field_to_frame_resampling_present_flag can be signaled in the Picture Parameter Set (PPS), Video Parameter Set (VPS), Sequence Parameter Set (SPS), or any other high level syntax part.
• PPS Picture Parameter Set
• VPS Video Parameter Set
• SPS Sequence Parameter Set
• the value of this flag could be assigned via an external approach.
• the following flags may be signaled to support interlayer prediction from field picture (e.g., 1080i) to frame pictures (e.g., 1080p).
• a flag "base_layer_field_picture_flag" may be signaled to specify whether the base layer pictures (e.g., layer 0 302 of FIG. 3) are field pictures.
• this flag can be signaled in VPS, SPS, or any other high level syntax part.
• the value of this flag may be assigned via an external approach.
• a flag "base_layer_bot_field_flag" may be signaled to specify whether the base layer picture of the current AU (e.g., the AU 405 of FIG. 4) is a bottom field picture.
• this flag can be signaled in the slice segment header, or any other high level syntax part.
• the value of this flag may be assigned via an external approach.
• a variable "botFieldtoFrameResamplingFlag" may be derived based on the signaled flags and used to calculate reference layer sample location, similar to the methods described above and in connection FIG. 6, FIG. 7, and FIG. 8.
• a bitstream conformance restriction may be implemented such that when bottom_field_to_frame_resampling_flag (or base_layer_bot_field_flag) is equal to 1, then the cross_layer_phase_alignment_flag shall be equal to 0.
• a bitstream conformance restriction may be implemented such that when bottom_field_to_frame_resampling_flag is equal to 1 (or base_layer_bot_field_flag is equal to 1 and the base layer picture is the reference layer picture of the current picture), then the picture width of the current picture shall be equal to that of the reference (base) layer picture and the picture height of the current picture shall be the twice of that of the reference (base) layer picture.
• the bitstream is a mixture of field pictures and frame pictures.
• a different layer has a different color format.
• the first layer e.g., the layer 0 302
• second layer e.g., the layer 1 306
• the base layer may be in YUV420 format
• another layer may be in YUV422 or YUV444 color format.
• phase offset e.g., the phase 850 of FIG. 8
• the encoder instead of deriving the value of phase offset variable phaseX and phaseY by a few signaled flags at decoder (e.g., the decoder 31 of FIG. 5B) side, the encoder (e.g., the encoder 21 of FIG. 5A) can signal the phase offset 850 value.
• the transferred variable values may be used directly to derive the reference layer sample location.
• variables may be signaled for each pair of a current layer and its reference layer, such as, for example, "phase_offset_X_luma,” “phase_offset_X_chroma,” “phase_offset_Y_luma,” and “phase_offset_Y_chroma.”
• These variables may represent the phase offset 850 to derive a horizontal position of a reference layer luma sample, a horizontal position of a reference layer chroma sample, a vertical position of a reference layer luma sample, and a vertical position of a reference layer chroma sample.
• the value of a given variable can be in range of zero (0) to four (4), inclusively.
• the values may be, for example in the range of 0-15 inclusive, or 0-31 inclusive.
• the related syntaxes may be inferred to be equal to 0.
• the reference layer sample location may be derived according to the following approaches.
• variables xRef 16 and yRef 16 may be derived as follows:
• variables xRefl6 and yRef 16 may be derived as follows:
• variables xRefl6 and yRef 16 may be derived as follows:
• xRefl6 ( ( xP - offsetX ) * ScaleFactorX + addX + ( 1 « 11 ) ) » 12
• yRef 16 ( ( yP - offsetY ) * ScaleFactorY + addY + ( 1 « 11 ) ) » 12
• phase offset (e.g, the phase offset 850) at reference layer was merged into the signalled enhancement layer offset 850.
• the appropriate default value may need to be set.
• the default value of phase_offset_X_chroma and phase_offset_Y_chroma is dependent on the relative position between luma and chroma sample position, and the scaling factor of the current layer and the reference layer (or the picture size of the current picture and the reference layer picture).
• the chroma sample is located at the same position to luma sample in horizontal direction and the chroma sample is located at the half pixel ("pel") position to luma sample in vertical direction.
• the default values of the offsets can be set as follows:
• phase_offset_X_luma, phase_offset_X_chroma and phase_offset_Y_luma is equal to 0, and the default value of phase_offset_Y_chroma is equal to: 4 - ( (1 « 18) + ScaleFactorY/2)/ScaleFactorY.
• ScaleFactorY is as defined as follows:
• ScaleFactorY ( ( refLayerPic Height « 16 ) + (currLayerPicHeight » 1 ) ) / currLayerPicHeight.
• currLayerPicHeight and refLayerPicHeight represent the height of the current picture and the height of the reference layer picture which are used to calculate the vertical scaling factor.
• phase_offset_Y_chroma can be set as follows by using the picture size directly:
• the syntax type of a horizontal phase offset (e.g., the phase offset 850b) and a vertical phase offset (e.g., the phase offset 850a) may be different.
• the vertical phase offset syntax may be the same as the syntax in the embodiment described immediately above. That is, two variables may be signaled for each pair of a current layer and its reference layer, including phase_offset_Y_luma and phase_offset_Y_chroma, which may represent the phase offset to derive the vertical position of the reference layer luma sample and the vertical position of the reference layer chroma sample.
• the value of the two variables can be in range of 0 to 15, inclusively, corresponding to the 1/16-th measurements between reference samples above.
• a flag (e.g., horizontal_phase_alignment_flag) may be signaled for all layers in the whole bitstream.
• the horizontal_phase_alignment_flag may be signaled for each pair of a current layer and its reference layer.
• the horizontal_phase_alignment_flag may specify that, at the horizontal direction, the locations of the luma sample grids of the current layer and its reference layer are aligned at the center sample position of the pictures.
• the reference layer sample location may be derived according to the following approaches.
• variables xRefl6 and yRefl6 may be derived as follows:
• phaseX can be derived as follows:
• variables xRefl6 and yRefl6 may be derived as follows:
• the encoder (e.g., the encoder 21 of FIG. 5A) may signal the value of the phase offset 850 and the transferred variable values may be used directly to derive the reference layer sample location.
• the phase offset 850 may be used to adjust the reference layer sample location after the reference layer location has been derived based on the default phase.
• four variables may be signaled for each pair of a current layer and its reference layer, including, for example, phase_offset_X_luma, phase_offset_X_chroma, phase_offset_Y_luma, and phase_offset_Y_chroma.
• These variables may represent the phase offset 850 to derive a horizontal position of a reference layer luma sample, a horizontal position of a reference layer chroma sample, a vertical position of a reference layer luma sample, and a vertical position of a reference layer chroma sample.
• the value of a given variable can be in the range of 0 to 15, inclusively.
• the related syntaxes are not present, they may be inferred to be equal to 0.
• the reference layer sample location may be derived according to the following approaches.
• the values can be set as follows: the default value of phase_offset_X_luma, phase_offset_X_chroma and phase_offset_Y_luma is equal to 0. In some embodiments, the default value of phase_offset_Y_chroma is equal to ( (1 « 16) - ScaleFactorX) » 14;
• phase_offset_Y_chroma 4 - (4*ScaleFactorX » 14), where scaling factor in vertical direction ScaleFactorY is as defined as follows:
• ScaleFactorY ( ( refLayerPicHeight « 16 ) + (currLayerPicHeight » 1 ) ) /
• currLayerPicHeight [00133] Where currLayerPicHeight can refLayerPicHeight represent the height of the current picture and the height of the reference layer picture which are used to calculate the vertical scaling factor.
• phase_offset_Y_chroma can be set as follows by using the picture size directly:
• the default value of phase_offset_X_chroma and phase_offset_Y_chroma is therefore dependent on the relative position between luma and chroma sample position, and the scaling factor of the current layer and the reference layer.
• the default value may be calculated by assuming that, in YUV420 chroma format, the chroma sample is located at the same position to luma sample in horizontal direction and the chroma sample is located at the half pixel position to luma sample in vertical direction.
• the syntax type of a horizontal phase offset 850b and a vertical phase offset 850a may be different.
• the vertical phase offset 850b syntax may be same as the syntax in the embodiment described immediately above. That is, two variables may be signaled for each pair of a current layer and its reference layer, including, for example, "phase_offset_Y_luma" and "phase_offset_Y_chroma,” which represent the phase offset 850 to derive a vertical position of a reference layer luma sample and a vertical position of a reference layer chroma sample.
• the value of the two variables can be in range of 0 to 15, inclusively.
• the related syntaxes are not present, they may be inferred to be equal to 0.
• a flag e.g., horizontal_phase_alignment_flag
• the horizontal_phase_alignment_flag may be signaled for each pair of a current layer and its reference layer.
• the horizontal_phase_alignment_flag may specify that, at the horizontal direction, the locations of the luma sample grids of the current layer and its reference layer are aligned at the center sample position of the pictures. For example, if the horizontal_phase_alignment_flag is equal to 0, this may specify that, at the horizontal direction, the locations of the luma sample grids of the current layer and its reference layer are aligned at the top-left sample position of the pictures.
• its value may be inferred to be equal to 0.
• the reference layer sample location may be derived according to the following approaches.
• xRefl6 ( ( ( xP - offsetX ) * ScaleFactorX + addX + ( 1 « 11 ) ) » 12 ) - ( phaseX « 2 )
• yRef 16 ( ( ( yP - offsetY ) * ScaleFactorY + ( 1 « 11 ) ) » 12 ) - phase_offset_Y
• phaseX can be derived as follows:
• variables xRefl6 and yRef 16 may be derived as follows:
• any of the syntax mentioned above may be signaled in VPS, SPS, PPS or slice segment header, or any other syntax table(s).
• phase offset syntax elements in order to address the case in which a reference layer could be a top field or bottom field, up to two sets of phase offset syntax elements can be signaled for each reference layer. For example, when two sets of phase offsets are signaled for a reference layer, a flag may be signaled in picture level, for example, in the slice header to indicate which set of phase offsets 850 is used for the resampling of the reference picture.
• bitstream conformance restrictions may be implemented, wherein such restrictions are used to limit the value of the phase offset to zero when the scaling factor is 1.
• the conformance restrictions facilitate achieving identical results with two methods: direct copy and using an upsampling process with zero phase filter coefficients.
• phase_offset_X_luma and phase_offset_X_chroma when the ScaleFactorX is equal to 1, the value of phase_offset_X_luma and phase_offset_X_chroma shall be equal to 0; alternatively, when the ScaleFactorX is equal to 1, the syntax element of phase_offset_X_luma and phase_offset_X_chroma shall not be present in the bitstream and inferred to be equal to 0.
• a bitstream conformance restriction may be implemented, such that when the ScaleFactorY is equal to 1, the value of phase_offset_Y_luma and phase_offset_Y_chroma shall be equal to 0; alternatively, when the ScaleFactorY is equal to 1, the syntax element of phase_offset_Y_luma and phase_offset_Y_chroma shall not be present in the bitstream and inferred to be equal to 0.
• the encoder 21 can signal the value of phase offset value for every layer in the VPS to the decoder. Additionally this can be signaled based on the direct dependency of the layers, for example, the dependency association of current and reference layer.
• the encoder can further signal phase offset update in the SPS, PPS or slice header or its extensions conditioned based on the VPS syntax elements.
• VPS the signaling in VPS could be as below: vps_extension( ) ⁇ Descriptor
• phase_offset_present_flag if( phase_offset_present_flag )
• hor_phase_luma specifies the phase offset used for reference layer luma sample location derivation in horizontal direction when reference layer nuh_layer_id is equal to layer_id_in_nuh[ j ]. Additionally, the current layer nuh_layer_id is equal to layer_id_in_nuh[ i ]. When not present, the value of hor_phase_luma[i][j] is inferred to be equal to 0.
• ver_phase_luma specifies the phase offset used for reference layer luma sample location derivation in vertical direction when reference layer nuh_layer_id equal to layer_id_in_nuh[ j ] and the current layer nuh_layer_id equal layer_id_in_nuh[ i ].
• the value of ver_phase_luma [i][j] is inferred to be equal to 0.
• hor_phase_chroma specifies the phase offset used for reference layer chroma sample location derivation in horizontal direction when reference layer nuh_layer_id equal to layer_id_in_nuh[ j ] and the current layer nuh_layer_id equal layer_id_in_nuh[ i ].
• the value of hor_phase_chroma [i][j] is inferred to be equal to 0.
• ver_phase_chroma specifies the phase offset used for reference layer chroma sample location derivation in vertical direction when reference layer nuh_layer_id equal to layer_id_in_nuh[ j ] and the current layer nuh_layer_id equal layer_id_in_nuh[ i ].
• ver_phase_chroma[i][j] is inferred to be equal to
• the encoder 21 can signal the value of phase offset 850 value for every layer to the decoder. Alternatively, this can be signaled in VPS, SPS, PPS or its extensions. The following includes an example using picture level (PPS) signaling:
• hor_phase_luma specifies the phase offset used for reference layer luma sample location derivation in horizontal direction. When not present, the value of hor_phase_luma is inferred to be equal to 0.
• ver_phase_luma specifies the phase offset used for reference layer luma sample location derivation in vertical direction. When not present, the value of ver_phase_luma is inferred to be equal to 0.
• hor_phase_chroma specifies the phase offset used for reference layer chroma sample location derivation in horizontal direction. When not present, the value of hor_phase_chroma is inferred to be equal to 0.
• ver_phase_chroma specifies the phase offset used reference layer chroma sample location derivation in vertical direction. When not present, the value of ver_phase_chroma is inferred to be equal to:
• a number of sets of phase offset may be signaled in VPS or SPS or PPS or its extensions and a syntax element and may be coded at SPS or PPS or Slice header or its extension that codes the index of the set that would be used.
• a syntax element may be signaled to specify the number of sets present.
• Information and signals disclosed herein may be represented using any of a variety of different technologies and techniques.
• data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
• the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials.
• the computer- readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), readonly memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like.
• RAM random access memory
• SDRAM synchronous dynamic random access memory
• ROM readonly memory
• NVRAM non-volatile random access memory
• EEPROM electrically erasable programmable read-only memory
• FLASH memory magnetic or optical data storage media, and the like.
• the techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as propagated signals or waves.
• the program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.
• DSPs digital signal processors
• ASICs application specific integrated circuits
• FPGAs field programmable logic arrays
• a general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term "processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software or hardware configured for encoding and decoding, or incorporated in a combined video encoder-decoder (CODEC).
• CODEC combined video encoder-decoder

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Priority Applications (6)

Applications Claiming Priority (12)

Publications (1)

Family

ID=53522484

Family Applications (1)

Country Status (7)

Cited By (1)

Families Citing this family (36)

Family Cites Families (8)

• 2015
  • 2015-01-15 US US14/598,117 patent/US9906804B2/en active Active
  • 2015-01-16 KR KR1020167021703A patent/KR102357816B1/ko active Active
  • 2015-01-16 JP JP2016546755A patent/JP6453350B2/ja active Active
  • 2015-01-16 CN CN201580004196.4A patent/CN105900431B/zh active Active
  • 2015-01-16 WO PCT/US2015/011828 patent/WO2015109238A1/en not_active Ceased
  • 2015-01-16 EP EP15703166.7A patent/EP3095241B1/en active Active
  • 2015-01-16 CA CA2933206A patent/CA2933206C/en active Active

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events