RU2610589C2 - High-precision upsampling with scalable coding of video images with high bit depth - Google Patents

Abstract

FIELD: computer engineering.SUBSTANCE: invention relates to computer engineering. In a scalable video system method of upsampling image data from a first level to a second level includes determination of parameters of scaling and rounding-off in response to requirements of scalable video system to bit depth. Input data are first filtered according to a first spatial direction using a first parameter of rounding in order to generate first upsampled data. By scaling first upsampled data using a first shift parameter, first intermediate data are generated, which are then filtered according to a second spatial direction using a second parameter of rounding in order to generate second upsampled data. Then by scaling second upsampled data using second shift parameter, second intermediate data are generated. Final upsampled data can be generated by cutting off second intermediate data.EFFECT: technical result consists in maintaining accuracy of upsampling operations.13 cl, 3 dwg

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the priority of provisional application for US patent No. 61/745050, filed December 21, 2012, which by reference is fully incorporated herein.

FIELD OF TECHNOLOGY

[0002] The present invention generally relates to images. In particular, one of the embodiments of the present invention relates to high-precision upsampling in scalable video codecs for video with high bit depth.

BACKGROUND

[0003] Compression of sound and video is a key component in the development, storage, distribution and consumption of multimedia content. Choosing a compression method involves the trade-offs between coding efficiency, coding complexity, and delay. As the ratio of computing power to computing costs increases, it allows the development of more sophisticated compression techniques that make more efficient compression possible. As an example in video compression, the Moving Image Expert Group (MPEG) from the International Organization for Standards (ISO) continued to refine the original MPEG-1 video standard with the release of the MPEG-2, MPEG-4 (part 2) and H.264 encoding standards / AVC (or MPEG-4, part 10).

[0004] Despite the coding efficiency and success of H.264, a new generation of video compression technology, known as High Performance Video Coding (HEVC), is currently under development. HEVC technology, a draft of which is available in document JCTVC-K1003 of the Joint ITU-T / ISO / IEC (JCT-VC) Video Efficiency Coding Team (HEVC) text specification draft 9, October 2012, authors B. Bross, W.-J. Han, G. J. Sullivan, J.-R. Ohm, T. Wiegand, which is incorporated herein by reference in its entirety, is intended to provide compression capabilities improved over the existing H.264 standard (also known as AVC), published as Advanced Video Coding for generic audio-visual services ”, ITU T Rec. H.264, and ISO / IEC 14496-10, which are incorporated herein by reference in their entirety.

[0005] Video signals can be characterized by a variety of parameters, such as bit depth, color space, color gamut, and resolution. Modern televisions and video playback devices (such as Blu-ray players) support a variety of resolutions, including standard definition (e.g. 720x480i) and high definition (HD; e.g. 1,090x1080p). The next-generation resolution format is ultra-high definition (UHD) with a resolution of 3840 × 2160. Ultra-high definition can also be called Ultra HD, UHDTV or Super High-Vision. As used herein, “UHD” means any resolution higher than HD resolution.

[0006] Another feature of a signal characteristic is its dynamic range. The HD dynamic range is the range of intensity (e.g. brightness, luma) in an image, for example, from the darkest to the brightest. For the purposes of this document, the term “dynamic range” (DR) may refer to the ability of the human psychovisual system (HVS) to perceive a range of intensity (eg, brightness, luma) in an image, for example, from the darkest to the brightest. In this sense, DR refers to intensity "in relation to the scene." DR may also relate to the ability of the display device to acceptable or approximately represent an intensity range of a certain width. In this sense, DR refers to the intensity “in relation to the display”. If a certain meaning is not directly defined as having a certain significance anywhere in this document, it should be concluded that this term can be used in any sense, for example, interchangeably.

[0007] As used herein, the term "enhanced dynamic range" (HDR) refers to a width of DR spanning about 14-15 orders of magnitude of the human visual system (HVS). For example, well-adapted people with essentially normal vision (for example, in one or more of the following senses: statistical, biometric, or ophthalmic) have a range of intensity spanning about 15 orders of magnitude. Adapted people can perceive dim light sources in only units of photons. Even more, the same people can perceive the almost painfully bright intensity of the midday sun in the desert, on the sea or in the snow (or even when looking at the sun, but briefly, to avoid damage). Such coverage, however, is available to “fit” people, such as people whose HVS has time to recover and adjust.

[0008] For comparison, DR, in which a person can simultaneously perceive an extensive width in the range of intensities, can be very truncated relative to HDR. For the purposes of this document, the terms “enhanced dynamic range” (EDR), “visual dynamic range”, or “variable dynamic range” (VDR) may individually or interchangeably refer to DR concurrently perceived by HVS. As used herein, EDR may refer to DR spanning 5-6 orders of magnitude. Therefore, the EDR, which is somewhat narrower with respect to the true HDR picture, nevertheless displays a large DR width. As used herein, the term “simultaneous dynamic range” may refer to EDR.

[0009] As used herein, the term "bit depth" of an image or video image refers to the number of bits used to represent or store pixel values of the color component of an image in a video signal. For example, the term “N-bit video image” (for example, N = 8) means that the pixel values of the color component (for example, R, G or B) in this video image can take values in the range from 0 to 2 ^N –1.

[00010] As used herein, the term “high bit depth” means any bit depth value greater than 8 bits (eg, N = 10 bits). It should be noted that although HDR images and video signals are generally associated with high bit depth, a high bit depth image does not necessarily have an extended dynamic range. Therefore, in the context of this document, high bit depth images can be associated with both HDR images and SDR images.

[00011] Several levels can be used to support backward compatibility with legacy playback devices, as well as new display technologies for delivering UHD and HDR (or SDR) video data from a device in an upstream direction to devices in a downstream direction. Given such a layered system, legacy decoders can use the base layer to recreate the HD SDR version of the content. Advanced decoders can use both the basic level and the extension levels to recreate the UHD EDR version of the content in order to present it on more suitable displays. As appreciated by the inventors, desirable here are improved techniques for encoding video images with high bit depth using scalable codecs.

[00012] The approaches described in this section are approaches that could be implemented, but not necessarily the approaches understood and implemented previously. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualifies as prior art solely by virtue of its inclusion in this section. Similarly, unless otherwise indicated, problems identified based on this section in connection with one or more approaches should not be considered to be recognized elsewhere in the prior art.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

[00013] One embodiment of the present invention is illustrated by a non-limiting example in the drawings of the accompanying graphic materials in which like reference numerals refer to like elements and in which:

FIG. 1 is one illustrative embodiment of a scalable coding system in accordance with one embodiment of the present invention.

FIG. 2 is one illustrative embodiment of a scalable decoding system in accordance with one embodiment of the present invention.

FIG. 3 depicts one example of an upsampling process of image data in accordance with one embodiment of the present invention.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS OF THE INVENTION

[00017] This document describes high-precision upsampling for scalable coding of input video images with a high bit depth. Given the parameters relating to the bit depth of the intermediate results, the bit depth of the internal input and the bit depth of the filter accuracy, scaling factors and rounding factors are determined to preserve the accuracy of operations and prevent overflow.

[00018] In the following description, for purposes of explanation, in order to provide a thorough understanding of the present invention, numerous specific details are set forth. However, as will be apparent, the present invention may be practiced without these specific details. In other cases, well-known structures and devices are not described in detail in order to avoid unnecessary difficulty in understanding the invention.

SUMMARY OF THE INVENTION

[00019] Illustrative embodiments of the invention described herein relate to high-precision upsampling for multi-level coding and decoding of high bit depth video signals. In response to the requirements for the bit depth of a video encoding or decoding system, input data, filtering coefficients, scaling and rounding parameters are determined for a separate upsampling filter. The input data is first filtered in the first spatial direction using the first rounding parameter to generate the first data with upsampling. The first intermediate data is generated by scaling the first upsampled data using the first shift parameter. Then, this intermediate data is filtered in a second spatial direction using a second rounding parameter to generate second upsampling data. The second intermediate data is generated by scaling the second upsampled data using a second shift parameter. Final data with upsampling can be generated by clipping the second intermediate data.

High precision split upsampling

[00020] Existing display and playback devices, such as HDTVs, set-top boxes or Blu-ray players, typically support HD 1080p (1920 × 1080 resolution at 60 frames per second) signals. For consumer applications, such signals are currently typically compressed using a bit depth of 8 bits per pixel for each color component in the luma-chroma color format, where, as a rule, chroma components have lower resolution than the luma component (for example, YCbCr or YUV 4: 2: 0 color format). Due to the 8-bit depth and the corresponding narrow dynamic range, such signals are usually referred to as standard dynamic range (SDR) signals.

[00021] As new television standards, such as ultra-high definition (UHD), are developed, it is desirable to encode signals with improved resolution and / or higher bit depth in a scalable format.

FIG. 1 depicts one embodiment of an illustrative implementation of a scalable coding system. In one illustrative embodiment of the invention, the input signal base level (BL) may represent a signal HD SDR, and the input 102 of the extension level (EL) may represent a signal UHD HDR (or SDR) with a high bit depth. BL input 104 is compressed (or encoded) using the BL encoder 105 to generate the BL bitstream 107. The BL encoder 105 may compress or encode the input signal 104 using any of the known or future video compression algorithms, such as MPEG-2, MPEG-4, part 2, H.264, HEVC, VP8 and the like.

[00023] For a given BL input 104, the coding system 100 generates not only the BL bitstream 107, but also the BL signal 112, which represents the BL signal 107 as it will be decoded by the corresponding receiver. In some embodiments, the signal 112 may be generated by a separate BL decoder (110) following the BL encoder 105. In some other embodiments, the signal 112 may be generated by a feedback loop used to perform motion compensation in the BL encoder 105. As depicted in FIG. 1, a signal 112 may be processed by an inter-layer data processing unit 115 to generate a signal suitable for use by the inter-layer prediction process 120. In some embodiments, the inter-layer data processing module 115 zooms in on the signal 112 to match the spatial resolution of the EL input 102 (e.g., from HD resolution to UHD resolution). Following inter-layer prediction 120, a residual signal 127 is computed, which is subsequently encoded by encoder 132 to generate an encoded EL bitstream 132. The BL bit stream 107 and the EL bit stream 132 are typically compressed into a single coded bit stream, which is transmitted to suitable receivers.

[00024] The term “SHVC” refers to a scalable expansion of a new generation of video compression technology known as High Performance Video Coding (HEVC) [1], which makes it possible to significantly compress better than the existing AVC (H.264) standard [2 ]. SHVC is currently being jointly developed by ISO / IEC MPEG and ITU-T WP3 / 16. One of the key features of SHVC is spatial scalability, where the most significant gain is provided by inter-layer texture prediction (for example, 120 or 210). In FIG. 2 shows one example of an SHVC decoder. As part of the inter-layer prediction, the upsampling process (220) performs upsampling, or upconversion, of the pixel data from the base layer (215) to match the pixel resolution for the data received at the extension level (e.g., 202 or 230). In one embodiment of the invention, the upsampling process can be performed by applying an upsampling filter, or an interpolating filter. H.264 Scalable Extension (SVC) software, or SHVC SMuCO.1.1 [3], uses a separate multiphase upsampling / interpolation filter. Although these filters are effective for input with a standard bit depth (for example, for images using 8 bits per pixel for each color component), they can overflow with input data with a high bit depth (for example, images using 10 or more bits per pixel for each color component).

[00025] In two-dimensional upsampling, or interpolation, processes, it is common practice to use separate filters to reduce data processing complexity. Such a filter first performs up-sampling of the image in one spatial direction (for example, horizontal or vertical), and then in another direction (for example, vertical or horizontal). Without loss of generality, the following description assumes that vertical upsampling follows horizontal upsampling. Then the filtering process can be described as

Horizontal upsampling

Vertical upsampling

where eF stores the multiphase upsampling filter coefficients, refSampleArray contains reference discrete values from the reconstructed base level, tempArray stores the intermediate value after the first one-dimensional filtering, predArray stores the final value after the second one-dimensional filtering, xRef and yRef correspond to the relative pixel position for upsampling, nshift stands for the scaling or normalization parameter, offset denotes the rounding parameter, and Clip () denotes the clipping function. For example, for given data x and threshold values A and B in one embodiment of the invention, the function y = Clip (x, A, B) denotes

For example, for N-bit image data, exemplary values A and B may include A = 0 and B = 2 ^Ν –1.

In equation (2), the operation a = b >> c means that b is divisible by 2 ^s (for example, a = b / 2 ^s ) by shifting the binary representation of b to the right by s bits. It should be noted that in equation (1), the clipping or shearing operations are not applied to the first filtering stage. It should also be noted that in this implementation the order of horizontal and vertical filtering does not matter. The use of vertical filtration first and then horizontal filtration leads to the same results as the use of horizontal filtration first and then vertical filtration.

[00026] In SMuCO.01 [3], the filter accuracy (denoted by US_FILTER_PREC) for eF is 6 bits. If the internal bit depth of refSampleArray is 8 bits, then tempArray can be stored within the bit depth of the target implementation (for example, 14 bits or 16 bits). However, if the internal bit depth of refSampleArray is greater than 8 bits (for example, 10 bits), then the output of equation (1) may overflow.

[00027] In one embodiment, this can be prevented by (a) fixing the order of operations in the upsampling process, (b) introducing intermediate scaling operations. In one embodiment, if horizontal filtering is followed by vertical filtering, then upsampling can be implemented as follows.

Horizontal upsampling

Vertical upsampling

[00028] Without loss of generality, let INTERM_BITDEPTH denote the requirement for bit depth (or resolution in bits) for intermediate data processing by a filter; that is, no result can be represented with more bits than INTERM_BITDEPTH (for example, INTERM_BITDEPTH = 16). Let INTERNAL_INPUT_BITDEPTH denote the bit depth used to represent the input video signal in the processor. It should be noted that the bit depth INTERNAL_INPUT_BITDEPTH may be greater than or equal to the original bit depth of the input signal. For example, in some embodiments, 8-bit video input can be internally represented using INTERNAL_INPUT_BITDEPTH = 10. Alternatively, in another example, a 14-bit video image can be represented as a bit depth INTERN AL_INPUT_BITDEPTH = 14.

[00029] In one embodiment, the scaling parameters in equations (3) and (4) can be calculated as

In one embodiment, negative values may not be allowed for nShift1 and nShift2. For example, a negative value for nShift1 indicates that the bit resolution allowed for intermediate results is more than acceptable to prevent overflow; therefore, in the case of negative values, the parameter nShift1 can be equal to zero.

[00030] If rounding is used in (3) and (4) (the greatest complexity, the greatest accuracy), then

iOffset1 = 1 << (nShift1–1), (7)

iOffset2 = 1 << (nShift2–1), (8)

where a = 1 << c denotes the binary shift “1” to the left by s bits, that is, a = 2 ^s .

[00031] Alternatively, if rounding is not used in (3) and (4) (least complexity, least accuracy), then

iOffset1 = 0, (9)

iOffset2 = 0. (10)

[00032] Alternatively, if rounding is used in (3) but not used in (4), then

iOffset1 = 1 << (nShift1–1), (11)

iOffset2 = 0. (12)

[00033] Alternatively, if rounding is used in (4) but not used in (3), then

iOffset1 = 0, (13)

iOffset2 = 1 << (nShift2–1). (fourteen)

[00034] In one illustrative embodiment of the invention, let INTERM_BITDEPTH = 14, US_FILTER_PREC = 6 and INTERNAL_INPUT_BITDEPTH = 8, then based on equations (5) and (6), nShift1 = 0, and nShift2 = 12. In another example, with US_FILTER_PREC = 6, if INTERN AL_INPUT_BITDEPTH = 10 and INTERM_BITDEPTH = 14, then nShift1 = 2, and iOffset1 = 0 or 2 depending on the selected rounding mode. In addition, nShift2 = 10, and iOffset2 = 0 or 2 ⁹ depending on the selected rounding mode.

[00035] It should be noted that when using the implementation depicted in equations (3) and (4), vertical filtering followed by horizontal filtering can lead to different results than horizontal filtering followed by vertical filtering, therefore proper filtering in the decoder can either be fixed and pre-determined for all decoders (for example, by means of a decoding standard or specifications), or in some embodiments of the invention, the proper order can be to code from the encoder to the decoder using the corresponding flag in the metadata.

FIG. 3 depicts one example of an upsampling process of image data in accordance with one embodiment of the present invention. Initially (305), an encoder or decoder in a multi-level coding system determines the proper filtering order (e.g., horizontal filtering followed by vertical filtering) and the scaling and rounding parameters. In one embodiment of the invention, the scaling and rounding parameters can be determined in accordance with equations (5) to (14) based on the required bit depths for intermediate storage (e.g. INTERM_BITDEPTH), filter coefficients (e.g. US_FILTER_PREC) and internal input representation (e.g. , INTERNAL_INPUT_BITDEPTH). At 310, the image data is up-sampled in a first direction (e.g., horizontal). The output from this stage is rounded and scaled before staging using the first shift parameter (e.g., nShift1) and the first rounding parameter (e.g., iOffset1). Then (315) the intermediate results are up-sampled in a second direction (e.g., vertical). The output from this stage is rounded and scaled using a second shift parameter (e.g. nShift2) and a second round parameter (e.g. iOffset2). In conclusion (320), the output of the second stage is cut off before final output or storage.

[00037] The methods described herein may also be applicable to other imaging applications that use separate filtering of image data with a high bit depth, such as downsampling, noise filtering, or frequency domain transform.

Illustrative embodiment of a computer system

[00038] Embodiments of the present invention can be implemented in a computer system, systems configured on an electronic circuit and components, an integrated microcircuit (IC) device, such as a microcontroller, field programmable gate array (FPGA), or other configurable or programmable logic device (PLD) ), in a discrete-time processor or digital signal processor (DSP), a specialized integrated circuit (ASIC) and / or in a device containing one or several ko such systems, devices or components. The computer and / or IC may execute, monitor, or execute instructions related to high-precision upsampling described herein. The computer and / or IC may calculate any of a variety of parameters or values related to the high-precision upsampling described herein. Embodiments of encoding and decoding can be implemented as hardware, software, firmware and various combinations thereof.

[00039] Some implementations of the invention include computer processors executing software instructions that cause the processor to execute a method of the invention. For example, methods related to the above-described high-precision upsampling may be implemented by one or more display processors, an encoder, a set-top box, a code converter, and the like. by executing the software instructions available for these processors. The invention may also be provided in the form of a software product. This software product may include any storage medium that carries a set of computer-readable signals containing instructions that, when executed by a data processor, cause the processor to execute the data of the method of the invention. Software products in accordance with the invention may be in any of a wide variety of forms. This software product may include, for example, physical storage media, such as magnetic storage media, including floppy disks, hard drives, optical storage media, including CD ROMs, DVDs, electronic storage media, including ROM storage devices , flash memory RAM, etc. Machine-readable signals in a software product may optionally be compressed or encrypted.

[00040] Where a component is mentioned above (for example, a software module, processor, assembly, device, circuit, etc.), unless otherwise indicated, a link to this component (including a link to "tools") should be construed as including, as equivalents to this component, any component that performs the function of the described component (for example, which is functionally equivalent), including components that are not structurally equivalent to the disclosed structure, that perform this function in detail Update illustrative embodiments.

Equivalents, extensions, alternatives and more

[00041] Thus, illustrative embodiments of the invention are described with respect to high-precision upsampling for scalable encoding of high bit depth video images. In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Therefore, the only and exclusive indicator of what constitutes an invention is the kit set forth in the claims arising from this application, in the specific form in which the claims follow, including any subsequent revision. Any definitions expressly set forth herein for terms contained in a specified claims should take precedence over the meanings of these terms used in the claims. Thus, no limitation, element, property, characteristic, advantage or attribute, not expressly stated in the claims, should in no way limit the scope of this claims. Accordingly, the description and graphic materials should be considered in an illustrative and not in a limiting sense.

Information sources

 [1] B. Bross, W.-J. Han, G. J. Sullivan, J.-R. Ohm, and T. Wiegand, “High efficiency video coding (HEVC) text specification draft 9”, ITU-T / ISO / IEC Joint Collaborative Team on Video Coding (JCT-VC) document JCTVC-K1003, Oct. 2012.

[2] ITU-T and ISO / IEC JTC 1, “Advanced Video Coding for generic audio-visual services”, ITU T Rec. H.264 and ISO / IEC 14496-10 (AVC).

[3] SMuCO.1.1 software for SHVC (scalable extension of HEVC): https: /hevc.hhi.fraunhofer.de/svn/'svn_SMuCSoftware/tags/0.1.1/.

Claims (29)

1. In a scalable video image system, a method of upsampling image data from a first level to a second level, where the method includes: determination by the processor of scaling and rounding parameters in response to the requirements of the system of scalable video images to the bit depth, while the requirements of the system of scalable video images to the bit depth include intermediate values of the bit depth, the bit depth of the internal input and the bit depth of the accuracy of the filter coefficients; generating first upsampling data by filtering image data from a first level, wherein this filtering of image data is performed in a first spatial direction using a first rounding parameter; generating first intermediate data by scaling the first upsampled data by the first shift parameter; generating second data with increased sampling by filtering the first intermediate data, while filtering the first intermediate data in a second spatial direction using a second rounding parameter; generating second intermediate data by scaling the second upsampled data by the second shift parameter and generating discretized output data for a second layer by clipping the second intermediate data, the determination of the first shift parameter includes adding the bit depth data for the bit depth of the internal input with the bit depth of the filter coefficient accuracy and subtracting the intermediate bit depth values from their sum, while determining the second shift parameter involves subtracting the first shift parameter from the doubled bit depth of the filter coefficient accuracy . 2. The method of claim 1, wherein the scalable video image system comprises a video image encoder. 3. The method of claim 1, wherein the scalable video image system comprises a video image decoder. 4. The method of claim 1, wherein determining the first and second rounding parameters includes calculating iOffset1 = 1 << (nShift1–1) and iOffset2 = 1 << (nShift2–1), iOffset1 is the first rounding parameter, iOffset2 is the second rounding parameter, nShift1 is the first shift parameter and nShift2 is the second shift parameter. 5. The method of claim 1, wherein the first and second rounding parameters are determined to be zero. 6. The method of claim 1, wherein determining the first and second rounding parameters includes calculating iOffset1 = 1 << (nShift1–1) and iOffset2 = 0, iOffset1 is the first rounding parameter, iOffset2 is the second rounding parameter and nShift1 is the first shift parameter. 7. The method of claim 1, wherein determining the first and second rounding parameters includes calculating iOffset1 = 0, and iOffset2 = 1 << (nShift2–1), iOffset1 is the first rounding parameter, iOffset2 is the second rounding parameter and nShift2 is the second shift parameter. 8. The method of claim 1, wherein the first spatial direction is a horizontal direction and the second spatial direction is a vertical direction. 9. The method of claim 1, wherein the first spatial direction is a vertical direction and the second spatial direction is a horizontal direction. 10. The method according to p. 1, in which the first spatial direction is a fixed and predefined technical conditions of the standard for decoding video images. 11. The method of claim 1, wherein the first spatial direction is determined by the encoder and communicated by the encoder to the decoder. 12. A device comprising a processor and configured to perform the method described in paragraph 1. 13. A non-transitory computer-readable storage medium containing computer-readable instructions stored thereon for executing a method according to claim 1.

Images