Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
  • H04N19/36—Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding

Definitions

• the invention is made in the field of coding of videos of high dynamic range.
• the invention is made in the field of coding of videos of high dynamic range
• Videos are generally represented on a limited number of bits (for instance 8, 10, 12 or more bits), corresponding to a limited range of values to represent the luminance signal. Videos represented in such way are called videos of low dynamic range or, shortly, LDR videos. However the human visual system is able to perceive a wider range of luminance. The limited representation most often does not allow for reconstructing correctly small signal variations, in particular in extremely dark or bright video image areas i.e. areas of high or low luminance.
• the HDR (High Dynamic Range) format consists in significantly extending bit-depth of signal representation to integer representation with more bits e.g. 20 to 64 bits, or even to floating
• HDR images or videos can be captured in various ways.
• Digital Single Lens Reflex cameras can use bracketing technique to capture successive images of the same scene with different exposures wherein exposure is the total density of light allowed to fall on the imaging medium (photographic film or image sensor) during the process of taking an image.
• Those images of different exposures are represented as LDR images.
• Under-exposed images capture details in the bright areas whereas over- exposed images capture details in the dark areas, as exemplarily depicted in Fig. 1 for different exposure values EV.
• the produced HDR image/video containing all details those in dark areas as well as those in bright areas .
• Tone Mapping allows representing the image while ensuring a good restitution of the different signal intensity segments, in particular, in high and low intensity ranges. Tone Mapping creates, from an HDR image, a LDR image where all the elements are correctly exposed. The LDR image is much more detailed both in dark areas and in white areas. This is exemplarily depicted in Fig. 2.
• HDR is used, in particular, in post-production. Most if not all special effects tools are dealing with HDR images with a floating point representation. The mixing being natural scene and special effects is also realized in HDR
• Tone Mapping is commonly applied to create a standard, e.g. 8/10/12-bit, master under the control of the Director of Photography.
• the Tone Mapping applied in post processing is commonly an unknown one .
• United States Patent Application 2008/0175494 it is described a method for predicting a high dynamic range image element, said method comprising: receiving low dynamic range image data; receiving high dynamic range image data comprising prediction data and an HDR residual image element; extracting an LDR image value from said LDR image data; modifying said LDR image value based on said prediction data; and combining said modified LDR image value with said HDR residual image element to form an HDR image element.
• the invention therefore proposes representing HDR content using LDR content, LDR residual and global illumination data, instead .
• the processing means for encoding one video of the LDR video and a further LDR video extracted from the HDR video independent from the other video of the LDR video and the HDR video and predictive encoding the other video using the one video as reference, and lossless encoding global illumination data further extracted from the HDR video.
• the encoding device comprises said processing means.
• Said reconstruction method comprises using processing means for decoding an LDR video, the LDR video providing a lower dynamic range depiction of the HDR video content, using the LDR video and a residual for decoding a further LDR video, the further LDR video providing a further lower dynamic range depiction of the HDR video content, decoding global illumination data and using the global illumination data and one of the LDR video and the further LDR video for reconstructing the HDR video.
• the reconstruction device comprises said processing means.
• the invention also proposes a data stream and/or a non- transitory storage medium carrying an HDR video encoded together with an LDR video, the LDR video providing a lower dynamic range depiction of the HDR video content, the HDR video being encoded together with the LDR video according to the proposed encoding method or an embodiment thereof.
• Fig. 1 depicts exemplary images of same content captured with different exposures
• Fig. 2 depicts, on the left, an exemplary low dynamic range image with over exposed areas and, on the right, another exemplary low dynamic range image resulting from tone mapping of a corresponding high dynamic range image will all areas being we11-exposed;
• Fig. 3 depicts a first exemplary framework of the
• Fig. 4 depicts a first exemplary framework of the
• Fig. 5 depicts a first exemplary embodiment of a global illumination data extractor
• Fig. 6 depicts a first exemplary embodiment of a HDR
• Fig. 7 depicts a second exemplary embodiment of a global illumination data extractor
• Fig. 8 depicts a second exemplary embodiment of a HDR video reconstructor corresponding to the exemplary embodiment of a global illumination data extractor depicted in Fig. 7.
• the invention may be realized on any electronic device comprising a processing device correspondingly adapted.
• the invention may be realized in a television, a video phone, a set top box, a gateway, a personal computer, or a digital video camera.
• Some exemplary embodiments of the invention are based on a multi-view (MVC) encoding scheme where a main view is intra-view predictive encoded, e.g. according to H.264/AVC.
• the main view can be decoded by any AVC decoder independent from whether the decoder can further decode an auxiliary view of the MVC scheme.
• an LDR video is encoded in the main view.
• the auxiliary view contains either global illumination data GID or an LDR video
• auxiliary view contains a lossless encoded meta data of the MVC code.
• auxiliary view contains global illumination data GID
• the LDR video image of the main view can be modified based on said global illumination data GID for forming an HDR video image.
• the LDR video image of the main view can be modified based on global illumination data GID for forming an HDR video image.
• the LDR video image of the main view can further be combined with the LDR residual video image in the auxiliary view for forming another LDR video image.
• the LDR video image of the main view can be combined with the LDR residual video image in the auxiliary view for forming a further LDR video image wherein the further LDR video image can be modified based on global illumination data GID for forming an HDR video image .
• the LDR video image can be a tone mapped version of the HDR video image wherein the tone mapping UTM used is irrelevant, e.g. unknown.
• Tone Mapping techniques There are plenty of different Tone Mapping techniques that can be used either in post-production or in real-time live products .
• a further Tone Map KTM can be applied to the HDR video using an appropriate Tone Mapping technique to create the further LDR video which then is a second LDR Tone
• Mapped version of the HDR video Preferably but not
• the second LDR Tone Mapped version of the HDR video is close to the original Tone Mapped version in order to minimize a residual with respect to the original Tone Mapped version.
• MVC encoding means MVCENC are used to encode the original LDR Tone Mapped version of the HDR video as the main view, that is as AVC video, and inter-view predictive encode the further LDR Tone Mapped version of the HDR video as auxiliary view.
• MVC should encode the second view very efficiently resulting in small overhead bitstream.
• global illumination data GID is encoded instead of motion vectors.
• Another example is global illumination data GID encoded as metadata.
• the main view of the resulting data stream is readable and decodable by already deployed AVC decoders to produce the original LDR Tone Mapped video.
• Decoders DEC modified according to the invention can reproduce both, the original LDR Tone Mapped video and the HDR video, by combining in a global illumination combiner GIC the original LDR Tone Mapped video with the residual in the second view and processing the combination result according to the global illumination data GID.
• Such decoder DEC comprises MVC decoding means MVCDEC which decode an incoming stream to regain a main view and a residual of a second view which can be used together with the main view to reconstruct the second view.
• the main view or the second view can be output to source LDR displays.
• the main view or the second view further can be used together with global illumination data GID, decoded using decoding means LLD, in a global illumination combiner GIC to regain the HDR video. Which view is used depends on whether the LDR video used for extracting global
• illumination data GID is encoded as main view or as second view .
• global illumination GID can be extracted using the original LDR Tone Mapped version and the original HDR video. This is exemplarily depicted in Fig. 4.
• ratio of normalized luminance of the HDR video with respect to normalized luminance of the original LDR Tone Mapped version can be used to extract global illumination data GID.
• the second LDR Tone Mapped version is identical to the original LDR Tone Mapped version, thus there is no residual and the global illumination data can be encoded in the second view.
• original LDR Tone Mapped version and the HDR video are sub-sampled SSP prior to extraction.
• ratio of normalized luminance of the HDR video with respect to normalized luminance of the second LDR Tone Mapped version can be used to extract global illumination data wherein the second LDR Tone Mapped version results from a known tone mapping KTM of the original HDR video.
• the normalization NRM is performed with respect to minimum and maximum values extracted from the HDR video.
• the luminance ratio is binarized BIN.
• the Global Illumination data GID is made out of the binarized data and metadata.
• the metadata inform on the min-max values before luminance ratio
• the second LDR Tone Mapped version can be determined by any tone mapping technique.
• a coarse lighting extractor is applied. This is exemplarily depicted in Fig. 7. The principle is the decomposition of the original HDR video into two components: the first component coarsely informs on the lighting of each image and the second component is the result of the coarse lighting removal from the original HDR video, i.e. the second Tone Mapped version of the original HDR video.
• a decomposition computation DEC of the original HDR video is applied.
• An example for this decomposition is square root, but other decompositions are possible.
• the resulting video is then sub-sampled SSP, so that each data represents the luminance of a region (group of pixels) .
• This video represents a Global Illumination Video that is part of the Global Illumination Data GID that is sent to the decoder. This Global Illumination Video is then
• Illumination Video to form the second LDR Tone Mapped video.
• HDR luminance-to-Global Illumination ratio is computed.
• the Global Illumination Data GID is made up with the Global Illumination Video and some metadata.
• the metadata is composed such that a decoder is capable to determine, from the metadata alone or in combination with a coding
• Global Illumination Combiner GIC is used to convolute the Global Illumination Data with the Point Spread Function PSF. The result is then used together with the decoded second LDR Tone Mapped video to form the reconstructed HDR video by inverting the computation made at encoder side. So if at encoder side division is performed, at decoder side multiplication is performed.
• the invention is applicable in many different fields of industry.
• illumination data GID is determined at encoder side using luminance ratios of pixels. This is exemplarily depicted in Fig. 5.
• a global illumination extracting means GIE receives an HDR video and a LDR tone mapped version of said HDR video as input and extracts luminance videos Y from said HDR video and the tone-mapped version thereof.
• the resulting luminance videos are not sub-sampled but normalized only.
• the global illumination data GID can be used together with the LDR tone mapped version of the HDR video for reconstructing said HDR video.
• the b_min, b_max, n_min and n_max are used to debinarize DBN and de-normalize DNM the global illumination data GID. If resolution of the global
• illumination data GID is smaller than resolution of the LDR tone mapped version of the HDR video to-be-reconstructed
• the de-normalized data is up-sampled to the resolution of said the LDR tone mapped version.
• the LDR tone mapped version of the HDR video to-be-reconstructed is normalized using n_min and n_max . Then, the normalized LDR video is pixel-wise multiplied by the de-normalized data of same resolution which results in reconstruction of the HDR video.
• the LDR tone mapped version of the HDR video used in this exemplary embodiment can be an LDR tone mapped version where the tone mapping is unknown. In this case, it can be advantageous to not execute the sub-sampling as this may cause artifacts in the reconstructed HDR image.
• tone mapping of the LDR tone mapped version of the HDR video is known and conveyed to the decoding side possibly occurring artifacts in the reconstructed HDR can be removed.
• global illumination data GID is determined at encoder side using luminance ratios of pixels, too. But in this further exemplary embodiment, a point spread function is used to generate the global illumination data from the HDR video. That is decomposition is applied.
• illumination extracting means GIE receives an HDR video and generates global illumination Data GID there from using decomposition DEC of the HDR video, sub-sampling SSP of the decomposed HDR video and convolution of the sub-sampled decomposed HDR video using a point spread function PSF . Then the global illumination Data GID is convoluted using a point spread function PSF and the HDR video is divided pixel-wise by the convolution result in order to generate an LDR tone mapped version of said HDR video. Global illumination Data GID and the LDR tone mapped version are then encoded and transmitted to a decoder or stored.
• Global illumination Data GID and the LDR tone mapped version of the HDR video to-be- reconstructed are received. Then the global illumination Data GID is convoluted using the same point spread function PSF as at encoder side and the LDR tone mapped version is multiplied pixel-wise by the convolution result in order to reconstruct said HDR video.
• non- transitory storage medium carrying an HDR video of high dynamic range predictive encoded using and LDR video as reference, the LDR video providing a lower dynamic range depiction of the HDR video content.
• the LDR video is further encoded.
• the non-transitory storage medium is carrying an encoded residual of a further LDR video with respect to the LDR video as reference and lossless encoded global illumination data wherein the further LDR video provides a further lower dynamic range depiction of the HDR video content and the global illumination data allows for reconstructing the HDR video using the further LDR video.
• a method for predicting a high dynamic range image element comprises receiving low dynamic range image data and receiving high dynamic range image data comprising global illumination data and an LDR residual image element.
• the method further comprises extracting an LDR image value from said LDR image data; and combining said LDR image value with said LDR residual image element to form an LDR image element.
• said LDR image element is modified based on said global illumination data to form an HDR image element .
• Said method comprises receiving low dynamic range image data; receiving high dynamic range image data comprising prediction data and an HDR residual image element;
• an exemplary embodiment which realizes an encoding of an HDR video of high dynamic range together with an LDR video -wherein the LDR video provides a lower dynamic range depiction of the HDR video content- by using processing means for extracting a further LDR video and corresponding global illumination data from the HDR video, encoding a first LDR video independently and predictive encoding a second LDR video using the first LDR video as reference, and encoding the global illumination data wherein, either, the first LDR video is the LDR video and the second LDR video is the further LDR, or, the first LDR video is the further LDR video and the second LDR video is the LDR video.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Studio Devices (AREA)

Priority Applications (12)

Applications Claiming Priority (2)

Related Child Applications (2)

Publications (1)

Family

ID=47358500

Family Applications (1)

Country Status (11)

Cited By (4)

Families Citing this family (16)

Citations (3)

Family Cites Families (13)

• 2012
  • 2012-01-06 EP EP12305020.5A patent/EP2613532A1/en not_active Withdrawn
  • 2012-12-18 KR KR1020147018394A patent/KR101993945B1/ko active Active
  • 2012-12-18 ES ES12801771T patent/ES2710379T3/es active Active
  • 2012-12-18 TR TR2019/01751T patent/TR201901751T4/tr unknown
  • 2012-12-18 WO PCT/EP2012/076010 patent/WO2013102560A1/en not_active Ceased
  • 2012-12-18 JP JP2014550673A patent/JP6111266B2/ja active Active
  • 2012-12-18 EP EP12801771.2A patent/EP2801191B1/en active Active
  • 2012-12-18 AU AU2012364389A patent/AU2012364389B2/en active Active
  • 2012-12-18 CN CN201811105437.6A patent/CN109274975B/zh active Active
  • 2012-12-18 BR BR112014016417-7A patent/BR112014016417B1/pt active IP Right Grant
  • 2012-12-18 US US14/370,773 patent/US10110921B2/en active Active
  • 2012-12-18 CN CN201280066103.7A patent/CN104041046B/zh active Active
  • 2012-12-18 PL PL12801771T patent/PL2801191T3/pl unknown
• 2018
  • 2018-10-19 US US16/165,159 patent/US10708620B2/en active Active

Patent Citations (3)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events