BR112015019787B1 - PICTURE ENCODER, PICTURE DECODER, PICTURE CODING METHOD, PICTURE DECODING METHOD, PICTURE SIGNAL, AND, MEMORY OBJECT - Google Patents

Abstract

PICTURE ENCODER, PICTURE DECODER, PICTURE CODING METHOD, PICTURE DECODING METHOD, PICTURE SIGNAL, AND, MEMORY OBJECT. To enable classifiers to output visually optimal content from HDR scenes to various display devices with rendering, the inventors invented an image encoder (202) comprising: an input (240) for an input image with high dynamic range (M_HDR); an image classification unit (201) arranged to allow a human color classifier to specify a color mapping from a representation (HDR_REP) of the high dynamic range input image set to a predefined accuracy to a low dynamic range image (Im_LDR) via a human determined color mapping algorithm and arranged to produce data specifying the color mapping (Fi(MP_DH); and an automatic classification unit (203) arranged so to derive a second low dynamic range (GT_IDR) image by applying an automatic color mapping algorithm to one of the input high dynamic range (M_HDR) or low dynamic range (Im_LDR) images (...).

Description

FIELD OF THE INVENTION

[001] The invention relates to apparatus and methods and resulting products as products for storing data or encoded signals, for example as stored in memories, for improved encoding of at least one image or a video with an increased dynamic luminance range, compared to legacy images called images with low dynamic range (LDR).

BACKGROUND OF THE INVENTION

[002] The newly emerging field of high dynamic range (HDR) image capture contrasts with legacy systems that today, by comparison, might be called low dynamic range (LDR) image capture (comprising image or video encoding systems such as PAL or MPEG2, AVC, HEVC or another member of the MPEG family, or similar video standards, e.g. VC1, VC2, etc., or JPEG for still images, etc.)

[003] When talking about HDR, it is necessary to look at the various components of the chain. As this is a very new area of technology where perhaps not everyone is on the same page, we want to quickly establish a reference mindset with some important definitions to avoid misunderstandings. Finally, there is the rendering dynamic range, which the display medium can generate. The rendering dynamic range is generally defined as RDR= luminance_of_brightest_white/luminance_of_darker_black of all pixels in an image (intra-image RDR), or of pixels in successive images (inter-image RDR, for example when the display device is (almost) off, and only the reflection of the surrounding environment is seen in the front glass). However, it is most meaningful when also coupled with a peak_white value (ie, the luminance_value of_brightest_white). LDR renderers are generally in a range defined by peak_white of 100 nits or so and a dynamic range of around 100:1. This is what a TRC display device could have produced, where obviously the luminance_of_darkest_black depends heavily on the lighting in the viewing environment, so one might opt for 40:1 for safety, and also 2:1 might be a practical dynamic range when viewing images on a display device in the sun. The viewing environment that conditions the brightness adaptation of the human observer is related to this, for example, typically 20% of white_peak. Various standards of EBU, SMPTE etc. specify how to classify a video signal so that it can be used in a conventional way, for example so that it is optimal if shown in the prescribed viewing environment. The term classify means to produce an image with altered pixel colors which are changed/specified according to some preference. For example, a camera can automatically classify a camera image in RAW format (which only depends on the specific characteristics of the camera as a linear luminance measuring instrument) given a rendering intent into a directly usable encoding directed to a display device, with which one can manipulate, for example, this type of CRT display device under reference conditions, so that it shows the viewer a clear image.

[004] Often, classification by a human involves more artistic choices. For example, the classifier wants to turn the color of a plant into a beautiful purplish color, but this needs to be specified under reference conditions (both the display device technology and the viewing environment, and in theory also other conditions that affect the viewer's state, such as drug consumption, but of course these are typically largely ignored), as a particular display device may make that color more bluish, in which case the desired artistic effect (of creating a beautiful image) may no longer exist. It is not typical for a camera to automatically create the optimal type of purple, which is why the classifier does this with image processing software. That classifier could be either a photographer or a visual artist working on a film, or even someone working on a television show (potentially even live). Of course, various applications will have varying degrees of classification complexity linked to the technical and/or artistic quality desired for those applications. Typically, the standards above recommend that a rating should be made on a reference monitor with around 100 nits, in a reference environment. The question then is how a color will be rendered and seen in practice. Graphic artists for print publications also generate their work under reference conditions, to have some common ground, and to avoid unnecessary sources of error, for example in print. However, this obviously does not mean that all readers of the book or magazine will read the book under a calibrated D50 lamp, but that they may see more washed out colors when reading in bed under inadequate lighting. The same is true when a movie or television show, or a consumer photo, is shown on a non-reference display device, one of the many different display devices that are available today. For example, the image (rating) can be displayed on a display device with a white_peak of 500 nits. What happens then is that you increase the brightness of all pixel colors with at least linear stretching, which is done by triggering the display device with the classification, i.e. mapping the maximum white (e.g. R=G=B=255 value) to the display device's white_peak (of course there may be additional brightness skew for the various pixel colors in the image if the display device has an electro-optical transfer function). Special native OTF, but usually this is handled internally to make the display device behave like a brighter version of a reference TRC, i.e. with a display gamma of about 2.5).

[005] These standardized LDR ratings (produced in a reference environment, among other things, on a 100-nit reference display device) can now be used (i.e., look reasonably good, i.e., still reasonably similar to how they would look under reference conditions) across a range of display devices and/or ambient conditions around the reference display system (i.e., 100 nits_white_peak, etc.). This is due to the fact that most humans are not so supercritical about the exact (absolute) appearance of colors, as the brain works relatively (for example, depending on the criteria for permissibility, face colors, which are one of the most critical colors, can range from pale to almost white to quite orange, etc. Partly this is also due to the fact that LDR scenes are produced with an "around average" object color strategy (which is achieved, among other things, with well-controlled studio lighting, perhaps not always with the various instantaneous content you currently have), which means that all colors are vivid, and you can even increase the brightness of the image quite a bit, even above the 18% level, with some shadows, but not too deep or important, etc., and this is reproduced very well , both physically and psychologically, in various systems. This is how, for example, naive painters work before discovering complex subjects such as chiaroscuro, etc. Thus, depending on the quality criteria that define acceptable similarity, the LDR_100 nits rating can be used, for example, on display devices from 30 nits to 600 nits, and in viewing environments from 3x less bright to 5x brighter. The latitude for using a classification can be increased by modifying it with a so-called display device transformation. The brightness of a display device and the surrounding environment (related to the Stevens effect and the Bartleson_Brenneman effect) can be corrected to a fair degree much more easily than issues related to the gamma constraints of a display device, and one can typically process the image with gamma functions or similar. For example, when moving a display device from a moderately lit environment to a dark environment (or, in fact, turning off the soft lights in the living room), it goes from an extra gamma of 1.25 to 1.5, i.e., it uses the residual gamma to increase the contrast of the rendered images, as human vision is more sensitive in the dark and therefore perceives the blacks in the rendered image as grayer, which is equivalent to a reduction in the contrast seen, which needs to be compensated for. A similar LDR technology is printing. In this case, you obviously don't have a priori control over the ambient illuminance that determines the white_peak of the print, but at least, as with all reflective objects, the white-black RDR is around 100:1 (depending on the quality of the paper, e.g. glossy versus matte, inks, etc.).

[006] A complication arises when it is necessary to reproduce an image of a scene with enormous dynamic range and, typically, also scene conditions very different from the rendering conditions. For example, in a night scene the eyes might be looking at a dynamic range of the scene, SDR, between 100,000 nits car headlights (or, say, even more for a mercury or high pressure sodium lamp in the scene) versus dark regions in shadows with fractions of a nit. Even in daylight, where it can be more difficult to create dark shadows from the lighting that spreads over everything, an indoor environment can typically be 100x darker than an outdoor environment, and also dark clouds and forest cover, among others, can influence the necessary luminances (whether these are captured or to be rendered), if not in intra-scene playback, at least in inter-frame playback, that is, in temporally successive playback. Citations for the “native dynamic range” of human vision vary between 10,000:1 and 100,000:1, and even 1,000,000:1, as this obviously depends on conditions (e.g., whether it is necessary to see a small darker region in bright areas, or vice versa, whether it is possible to see some small bright object in the dark, whether this is perhaps partially a rhodopsin lightening; whether an amount of glare is considered uncomfortable, etc.; and then , there is obviously also a psychological factor [taking into account things like the importance of certain objects, their perfect or sufficient visibility, emotional impact on the viewer, etc.], which leads to the question of how much of that would need to be rendered on a display device [e.g., a viewer might quickly dismiss an area as "just black", no matter which black, exactly], since the viewer is in an entirely different situation anyway [not really on vacation, or not really interrogated by a policeman shining a light in their face], but if you want a certain amount of realism , which may additionally be a trade-off with other factors, e.g. power consumption, so that one could pragmatically indeed define several dynamic ranges of human vision, e.g. one for a certain type of real scene viewing, and one for television viewing). For example, if a person is adapted to the dark night sky but sees the moon in the corner of the eye, this has less influence on how rods in other parts of the retina can see the faint stars, i.e. the "simultaneous" visible dynamic range will be high. On the other hand, when the eye is bathed in strong daylight (over a large area of its field of vision), it is more difficult to discriminate the darkest colors in a darker interior viewed and lit through a small hole or window, especially if a bright source is adjacent to that dark area. Optical systems will then show various glare phenomena. In fact, the brain at large may not even care about that dark interior, and simply call all those colors psychological blacks. In another example of how light leakage influences and determines the dynamic range of the scene, from the perspective of a human observer, consider a dark, poorly lit bush at night, behind a lamp post. The lamp on the lamp post creates a light scattering profile on the scratches of the viewer's glasses (or, if he does not wear glasses, on the irregularities of his eye lens, for example submicrometer particles, water between the cells, ...), in particular in the form of a halo around the lamp, which reduces the possibility of discrimination between the dark colors of the bush behind it. However, when the observer walks for a few seconds, the lamp moves behind him and out of the eyepiece lens capture zone, and the eyes can quickly adjust to find the predator lurking in the dark.

[007] Therefore, regardless of how one defines the usable dynamic range of a scene for encoding and rendering for human consumption (one could even consider not just encoding the intra-image luminances with a global luminosity scaling factor, but the luminances that actually occur from a sunny tropical environment to the darkest cloudy night), it is clear that more than 100:1 is needed for a faithful or at least plausible rendering of these environments. For example, you'd want the brightest object on a dim display device to be around 10,000 nits, and the darkest 0.01 nit (or at least 0.1 nit), at least if you could, for example, dim the lights if you have totally or predominantly dark scenes in your movie or one or more images.

[008] That's where HDR comes in. Furthermore, when this type of scene is captured, a very complex mathematical mapping is required to approximate the same (or even to be able to render the same) on an LDR display device (this, in fact, is often not really possible). For example, some HDR to LDR mapping algorithms use local adaptation to more or less equalize the lighting field, leaving in the LDR rendering mainly an impression of the reflections of the object, that is, the colors. In view of the light leakage (multiple reflections, scattering, etc.) For example, an indoor scene can have (of course, depending on the depth of the space, the size and position of the windows, the reflectivity of the walls, etc.) a fraction or multiple of about 1/100 of the (i)luminance of the outdoor environment (which is also how you define the daylight factor for lighting buildings). Higher SDRs can be obtained by viewing a sunny outdoor environment from inside a cave through a small slit, etc. Also, on the rendering side on the display device, an HDR streak starts where you start to see new concepts of appearance. For example, on bright display devices such as a 5000 nit SIM2 display device, one can, with the right input images (correctly classified), realistically render the impression of real light bulbs lit, or real sunny landscapes. Unlike the LDR range above, HDR can typically be said to start, for normal living room TV viewing conditions, from a white_peak of around 1,000 nits and above, but more precisely this depends, too, on the exact viewing conditions (e.g. cinema rendering, although with a white_peak of 50 nits, already shows various appearances of HDR). To be even more precise in view of the adaptation of the eyes and the brain, the HDR-like appearance in numerical details would also somewhat depend not only on the physical luminances, but also on the image content, i.e. the chosen classification. However, in any case, there is a clear discrimination between rendering in LDR, which mainly shows an opaque and lightless version of the scene, as if it were lit almost homogeneously and showing only the reflectances of objects, and in HDR, in which an appearance of a full field of illumination is superimposed. If it is possible, then, to render reasonable blacks, for example at 1 nit or below, one can indeed get above an LDR contrast range of kx100:1, where k is typically 2-3 (which, under a specific near-similar paradigm, i.e., with only perhaps a small contrast stretch, the relative rendering of the displayed luminances compared to the scene luminances would correspond to a similar DR in the scene). At the upper end of the glosses, it's partly a matter of taste where the gloss should end, particularly where the added shine just becomes bothersome. It turned out that for classifying various scene types in HDR, 5,000 nits is still somewhat on the lower end, particularly when additional display device limitations such as backlight resolution have to be dealt with. In experiments, it was found that one can definitely go up to 10,000 nits in dark viewing without the glare becoming superfluous or annoying (at least for some viewers). Using a white_peak above 20,000 nits can be a practical technical design consideration as to what to realistically render in terms of luminance, and what to approximate, achieving at least an appearance of brightness. Note that one should not typically drive such a bright display device always at maximum brightness, instead, to create a great HDR experience, one should use the brightest rendering only at certain locations and times, conservatively, and also well chosen in terms of its temporal evolution. One should not only focus on intraimage DR, but also on how the different brightness environments should be rendered in succession, taking human visual adaptation into account.

[009] Another dynamic range is the dynamic range of the CDR camera, which is determined (given the exposure settings) only by the saturation charge of the pixel photodiode, and by the noise on the dark side. When using tricks like multiple exposure or arrays of pixels amenable to different exposure (for example, in cameras with 3 integrated circuits), the CDR becomes limited by the optical elements (for example, scattering by the lens, reflection on the lens or camera body, etc.), but this can also be improved by adequate computational imaging techniques, which try to separate the real illumination from the dark scene regions of erroneous irradiation from diffused light. Of course, when the image source is a computer graphics routine (for example in special effects or in a game application) one can easily create HDR far beyond those limitations. One should ignore the CDR, and just assume it is too high or perhaps a limiting factor, but in a system that is intended to handle very high quality originals situations. In particular, where clipping is introduced, it will be assumed that it is not due to capture by a poor quality camera, but a practical handling of some other limitations in the entire image capture chain, such as the inability of a display device to render very vivid colors.

[010] Now, separately from the RDR of the display device environment, which actually generates the correct distribution of photons in order to stimulate the observer to have the correct sensation (this also depends on the adaptation state of that observer), when talking about handling or encoding of HDR, there is another interesting aspect, which can also be summarized in a dynamic range, which will be called dynamic range of encoding, CODR. A few thought experiments should clarify this important concept. Suppose you draw on a bright white backlit panel with a highly absorbent black marker, so that you have a transmission of 1/16,000 the white of the panel's surroundings (and assuming the surrounding space and the viewer are perfectly absorbing objects). In the world of linear bits (meaning that all values are linearly represented between, say, 0 and 2^B, where A is the power operation and B is the number of bits), for example the camera capturing (its ADC) would therefore need 14 bits to represent this signal. However, as this codec would waste a large amount of codes for values that, in any case, do not occur, it can be said that to faithfully represent that specific signal, theoretically, only 1 bit encoding is needed. Black would be assigned a code 0, and white a 1, and would then be converted to whatever real luminance they correspond to. Note also that a display device need not actually render those values with exactly the same luminances as are present in the scene. In fact, as this signal might not look any better (psychologically and semantically) than a lower DR equivalent of it (indeed, this kind of high-contrast black-and-white drawing might even look weird), one might as well render it on a display device with values of 1 nit and 2000 nits. We observe here for the first time an interesting distinction that is important when talking about HDR coding: the difference between physiological and psychological (or semantic) dynamic range. Human vision consists of two parts, the eyes and the brain. The eyes may need, as a precursor, adequate physiological dynamic range, PDR, to properly stimulate the cones and/or rods (and thus ganglion cells, etc.), but it is ultimately the brain that determines the final appearance of the image or scene (psychological dynamic range, PSDR). Although this does not quite give the exact impression of a very bright region, painters such as Petrus Van Schendel can play with the psychological principles of PSDR to emulate, in an LDR medium, high dynamic range scenes, for example a fire in a dark night cityscape. This, too, is what complex gamma mapping algorithms attempt to do when preconditioning an HDR image for rendering on an LDR display device. The flip side of this principle, however, is that some scenes will look more HDR-like than others, even on an HDR display device (for example, a sunny winter landscape with pale dry bushes and a few trees in the background may appear high-bright but not HDR). For HDR actions, for example pointing a bright light bulb towards the viewer, psychological emulations are generally not as convincing as really bright rendering of regions.

[011] Consider now, along the same lines, a second example: there is an indoor scene with luminances of, say, between 200 nits and 5 nits, and an outdoor scene with luminances of, say, between 1,500 and 20,000 nits. This means that, again, we have two luminance histograms separated by non-existent codes. One could encode them natively in a range of, say, 16 linear bits (where the maximum code corresponds to, say, 32,768 nits), although it would be preferable to use some non-linearity to get enough accuracy in the blacks if there isn't too much capture noise. But one could also code this in a different way. For example, it would be possible to sacrifice 1 bit of precision, and divide an 8-bit non-linear JPEG luminance strip into two adjacent touching parts, the bottom one for the darkest part of the scene above, and the top one for the brightest part (one might not want to cut exactly in the middle, in view of the non-linear JND allocation). If there is concern about losing fine detail when having fewer bits, it may be considered that it may often be better to use available bits for HDR effects instead. This type of allocation would typically correspond to a shift and stretch (non-linear) of the luminance values (L) of the input capture in RAW format to the 8-bit luminance values (Y). Now, one might again ask what is the dynamic range of this type of scene, whether it can be “arbitrarily” compressed or stretched (making the bright external environment even brighter, at least until it becomes, for example, unrealistic), at least in post-processing for rendering. Here, the concept of different appearances can help. One has, in both sub-histograms, a certain number of different luminance values for different pixels or regions that, supposedly, are all or almost all relevant (if not, it is not necessary to encode them, and one can, for example, eliminate one or more bits of precision). Also, the separation (eg measured as a difference in average luminance) of the two histograms when ultimately rendered on a display device has some appearance significance. It is a known fact that human vision discounts lighting to some extent, but not fully (especially if there are two brightness regions), so it is necessary to render/generate these inputs to the eyes, at least to some extent. Therefore, working with different and meaningful appearances of color (or at least brightness or luminosity) of pixels or objects in a renderable scene (for example, when rendered in the best possible display device scenario) provides an understanding of the dynamic range of encoding, CODR, and how it is necessary, consequently, to encode the images in HDR. If the image has different appearances, it's in HDR, and these need to be somehow present in any reasonably faithful encoding.

[012] Since classical image or video coding technologies (e.g. PAL, JPEG, etc.) were primarily dedicated to rendering primarily object luminosity (reflection) in a 100:1 range under originally relatively fixed viewing conditions (a CRT in a domestic environment, not an OLED on the train, or the same consumer having a dedicated dark movie theater in his attic, with instantaneously and dynamically controllable lighting which can adjust to the video content), those systems encoded the video quite fixed, particularly with a fixed universal main coding range which mimics the brightness sensitivity of the eyes, for example V_709=1.099LA0.45-0.099, which is approximately a square root function. However, these systems are not well adapted to handle a wide range of CODRs. In recent years, there have been attempts to encode HDR, either in a native way of encoding linearly with reference to the scene all possible input luminances, as in the OpenEXR system (F. Kainz and R. Bogart: http://www.openexr.com/TechnicalIntroduction.pdf). Or, there are 2-tier systems based on the classic philosophy of scalability. These need at least two images: a base image which will typically be an LDR image that can be used by legacy systems, and an image to reconstruct the one or more main HDR images. One such example is US2012/0314944, which needs the LDR image, a logarithmic boost or ratio image (obtained by dividing the HDR luminances by the LDR luminances obtained after properly classifying an LDR image for LDR rendering systems), and a color clipping correction image per HDR image to be encoded. With a boost image you can boost all regions (depending on the subsampling) from its limited range, whatever the luminance position they should occupy in the HDR range. Note that, for the sake of simplicity, all these operations are described in a luminance view, as one skilled in the art can imagine how these need to be formulated in a luminance view of a particular encoding definition. These multi-pictures are, at least in the coming years, impractical as they need (de)coding ICs with serious upgrades in existing devices, as additional image handling is required in addition to the LDR image.

[013] Recently, and as described in WO2013/046095, a way has been developed to optimize classical video encoding (preferably with minor modifications, preferably with mainly metadata to apply transformations related to two ratings of the same scene for two very different rendering conditions, for example allowing to transform an LDR encoded rating into an HDR rating, or vice versa, and perhaps with some variants having space to store in the metadata some additional small images to make a final adjustment, if this type of additional modification is desired, for example an additive or multiplicative correction on small regions containing an object, for example a brightly lit face in a shot or scene of the film, where the per-pixel corrective factors can then be encoded into e.g. 200 120x60 pixel images to be mapped onto the pixel positions of the current HDR reconstruction via color transform, or even some sub-sampled representation of those small corrective images, to be applied as fine-tuned coarse mappings, described as images) to be capable of encoding images with high dynamic range. In this system, typically a human classifier can determine an optimal mapping function from the input HDR image (main HDR classification) to e.g. LDR encoding 8 or 10 (or 12 or in principle another value at least for the luminance codes, but this value being typically what is reserved for “classic” LDR image encoding) bits, which can be encoded by means of classical video compression (DCT etc.), the optimal mapping function (e.g. a gamma function or similar with optimal gamma coefficient, linear part etc., or a multisegment function, e.g. an S-curve, etc.) typically depending on what the content was in the main HDR (e.g. a dark background, with a very brightly lit region), and how it will be rendered under LDR conditions. This is called simultaneous encoding of an LDR and HDR classification by mapping the HDR classification into an LDR image usable in legacy systems, and encoding the HDR into an LDR container. It was intended to ensure that this technology was backwards compatible, so that the LDR image so generated would provide reasonable results when rendered on, for example, a legacy LDR system (i.e., the image looks reasonably good, if not perfect, typically so that not many people will find the colors of some objects completely wrong). If something like a decrease in accuracy is accepted, the present system can even encode scenes or effects in HDR on legacy 8-bit systems. By reasonable results, it is meant that images rendered in LDR, while perhaps not the best that could theoretically be achieved in terms of artistic appearance, will be acceptable to a content creator and/or viewer, depending of course on the application (e.g. for a cheaper internet-based or mobile phone service, quality restrictions may be of less critical importance). At least the LDR classification will provide good visibility of all or most objects (at least those objects of primary importance to the image or video story) in the imaged scene, when rendered in an LDR system with properties that do not deviate too much from standardized LDR rendering. On the other hand, for HDR display devices, the original main HDR can be approximated in an approximation by mapping with the invertible reverse of the cocoded mapping function from the LDR image to the reconstructed HDR image. This approximation can be defined with mathematical tolerance, for example in terms of just noticeable differences (JND) between the supplied original main HDR and its reconstruction. Typically, any such system will be designed by running tests for a number of additional scenes, actions and situations in typical HDR, to see how different the reconstructed HDR looks (whether this will still be acceptable for certain classes of users, for example content creators for television or film) and validating a class of operations as specific gamma mappings within certain parameter ranges from it. This guarantees that it will always be possible to obtain a certain quality of approximation.

[014] It is an objective of the technologies presented below to offer the classifier even greater versatility in defining at least two classifications, LDR and HDR.

SUMMARY OF THE INVENTION

[015] The above objective is achieved by having an image encoder (202) comprising: - an input (240) for an input image with high dynamic range (M_HDR); - an image classification unit (201) arranged to allow a human color classifier to specify a color mapping from a representation (HDR_REP) of the input image with high dynamic range, defined according to a predefined accuracy, to an image with low dynamic range (Im_LDR), by means of a color mapping algorithm determined by a human being, and arranged to produce data specifying the color mapping (Fi(MP_DH); and - an automatic classification unit (203), arranged in order to derive a second image with low dynamic range (GT_IDR) by applying an automatic color mapping algorithm to one of the input image with high dynamic range (M_HDR) or image with low dynamic range (Im_LDR).

[016] The GT_IDR rating is typically made from either the high dynamic range image (typically the main rating) or the LDR rating, but of course it can also be advantageous to take into account with the same image characteristics, particularly the appearance of brightness or luminosity of various objects, from the other rating (i.e., what the LDR rating should look like if GT_IDR is mapped from M_HDR, so that GT_IDR may represent some kind of balance, but of course it can be formed by all kinds of other conditions/side factors, too). It is assumed that the main HDR image is encoded in whatever format allows this type of encoding (for example this could be OpenEXR, or a system as explained in Figure 7 of this document, generally anything preferred, for example for the grading software manufacturer). That is, the encoding in M_HDR can be of a linear type, with reference to the scene, or already have some interesting code allocation function applied to it, but for the present explanations it can be safely assumed that it is a linear luminance encoding. Typically, this master HDR image will not come straight from the camera (since cameras are just automatic capture tools, with characteristics, for example color filters not similar to the human eye, but more importantly, their circuitry is not like the human brain, what comes out of them by mere recording can be good, but not necessarily great), but an optimal artistic classification (which, for example darkens the background environment of a basement to create an optimal mood state for that scene), however, the classification by a human being could be a simple one functional mapping of the image from a camera somewhere (this then being the main HDR input), for example just to get a first preview on a certain renderer, after which a high quality HDR image is encoded (by means of an LDR image and mapping parameters). An image classification unit typically consists of software running on a computer, which allows color mapping from initial pixel colors to final pixel colors, for example by changing a luminance correlate of those pixels from an initial value to a final value, for example by applying a tone mapping function (e.g. an S curve) to that luminance correlate or, for example color definition curves (such as R,G,B), simultaneously. One skilled in the art needs to understand why the term luminance correlate was used to represent any mathematical encoding that correlates to a luminance of a pixel (when captured in a scene, or rendered in a render of the same) since, given the complexity of color technologies, there are some similar variants of them, such as luminances, values (V), functional definitions for correlates called luminosity, etc. In fact, a linear or nonlinear component of color, such as an amount of red, can also be used as a correlate of luminance. Therefore, the luminance correlate should be understood as any monotonic mapping function between the luminance axis (luminance as defined by the CIE) and another axis, such that any value on that other axis can be immediately converted to a luminance value, and vice versa. Although the formulations for various correlates vary in their precise details, the principle remains the same. However, the term was also introduced to indicate that, although the principles of the present modalities can be defined in luminance mappings, they can in fact be physically constructed by applying mathematical operations to other luminance correlates or, in general, to any color coding. The human color sorter can for example be partially driven by a film director to produce a certain visual appearance for the captured film.

[017] The principle of the above embodiment is that, unlike legacy systems with a fixed mapping function relating an LDR and an HDR classification (for example, encoding any input image, whether or not it has an increased dynamic range, into an LDR encoding), we now have a dual system. This will also typically create two sets of mapping parameters (e.g. luminance correlate mapping functions or in general data defining mathematical transformations performed by software to transform input colors into output colors) rather than just one reversible function to create the image to be rendered (in the case of PCT/2012/054984, an HDR reconstruction to be used on an HDR display device to drive it, either directly or after additional color processing).

[018] In this dual system, there are also two ranks relating to the main HDR. First, there is an automatic classification, which generates a first image in good quality LDR, which is called a second image with low dynamic range, GT_IDR. This can be (partly) influenced by the color classifier (for example, by selecting a preferred function from a set of mapping functions that would normally yield good results on all input images) but typically it is advantageous to have this automatic classification behind the scenes on the instrument, out of sight and concern of the classifier who can then focus on his artistic desires. The point of this technical classification is to create a GT_IDR image that, although perhaps it is not totally great according to the classifier's specific artistic desires, produces an LDR image with good viewing when rendered in an LDR system (with good viewing, meaning not only that any image will be displayed, but the observer can follow most of what is occurring in the movie, because the visibility of all objects is good, even due to some discoloration Great feed, the scene's mood state can be a little altered). In large part, however, this defines its mathematical derivation so that this second image in LDR GT_IDR is technically optimal, since it is easy to reconstruct, from it, a REC_HDR reconstruction of the main HDR with great quality. This means that the loss of information in GT_IDR due to e.g. quantization after using the specific optimal mapping for it, from M_HDR, should be minimal, so that there is a minimum acceptable amount in the reconstructed HDR for all possible typical HDR input images.

[019] Then, on the other hand, there is an artistic classification of the human classifier. He can derive any image he wants, according to his preferences, for viewing in LDR rendering systems. For example, you might have an action take place in a dark basement in a horror movie. The HDR rendering system may be able to render the dark environment as very dark, while still retaining the visibility of most objects (e.g., torture equipment on shelves in shadows, against a wall, or the interior of an adjacent unlit room beyond an open doorway). And at the same time, this might be able to render very bright objects, like a single light bulb flickering on the ceiling of that dark room, or a torch in the hands of a person walking through it. However, the LDR rendering system may have lesser capabilities for rendering the dark environment, particularly as it also needs to leave room in its limited luminance range for the brightest objects, such as the lamp and the face of the person walking under it, and the classifier may want to emulate the brightness by increasing the contrast with the luminances of the surrounding objects, i.e., the dark background. The classifier can, for example, artistically decide to make this background completely black for the LDR classification, Im_HDR. It should be clear that this image with low dynamic range, Im_HDR, can then not be used to reconstruct a REC_HDR with enough information in the background to have all objects visible there. As a generalization of this, it can be noted that the automatic classification unit needs to make sure that no relevant information is lost, so that an HDR reconstruction can still be derived with good approximation accuracy from the encoded LDR image, GT_IDR. Note that the GT_IDR of this LDR image need not, by itself, be defined with the same environment constraints (e.g., 100 nits_white_peak as the intended display device), but could also be, for example, for a 200 nit reference display device.

[020] As can be understood by the person skilled in the art, there are two ways to realize such a system. Either the automatic classification unit that sorts technically does its mapping first, and then the human classifier works on that GT_IDR to create its preferred LDR classification, IM_LDR, or the human classifier first does its Im_LDR classification, and then the automatic classification unit derives a technically more suitable GT_IDR from it to encode all relevant HDR data into an LDR_container format. So, in fact, this logically corresponds to that the human classifier will work, in both cases, with the main HDR representation. In the first case of classification by human being it will be the main HDR itself (infinitely accurate) that will form the starting point. In the second case, the GT_IDR resulting from the automatic technical classification will be a good representation of the main HDR, as it contains most (at least the relevant part) of the main HDR data, either in a different mapped luminance correlate representation (for example, a luminance of a lamp of 10,000 nits in the main HDR can be represented as a luminance code 253 in the GT_IDR). According to a predefined accuracy, again this means that technical limits are placed on how much a REC_HDR reconstruction from the GT_IDR can deviate from the originally given M_HDR. Typically, the person skilled in the art knows that one can (if not just defined according to the preference of human examiners) mathematically characterize these deviations, for example by means of the weighted difference between the pixel colors of REC_HDR and M_HDR. For example, one can use mathematical functions that characterize a difference following human visual principles, for example observing colors in regions and, for example, penalizing differences less if they occur in textured areas, etc. Some semantic objects, for example light bulbs, may be allowed greater differences, as the actual rendered luminance for those objects may be less important. In summary, one skilled in the art will understand that, typically, the technical classification will perform any one of a set of pre-agreed mappings, which for any, or most, typically occurring input M_HDR images will produce reconstruction errors below a certain threshold (which may either be a subjective value agreed upon by the human evaluation panel, or an agreed mathematical value). Typically there will be, for example a set of gamma-like functions (i.e. typically starting with a linear part in the blacks and then making a curve to show an ever-smaller slope of the output versus the input), or three-segment parametric curves to affect the darks/shadows, midtones and brightest sub-ranges of the luminance or luminance correlate axis, all of which behave reasonably, and some of which may result in minor reconstruction errors in a specific luminance region of a type specific to M_HDR. The human classifier can then select this curve. Or, alternatively, the automatic classification unit can select this type of optimal curve, for example by observing the color or luminance histogram of the M_HDR, or performing a more complicated analysis of it (for example, determining where one or more faces are). Thus, the HDR_REP representation of M_HDR according to a predefined accuracy means that this image contains substantially all of the M_HDR data, either in a differently coded way, so that the given M_HDR can be inversely obtained, within a predefined accuracy, i.e. with reconstruction errors that, in the worst case, typically do not exceed an agreed level.

[021] Thus, the human classifier works, therefore, either with the M_HDR, or with the GT_IDR, to obtain its image with low dynamic range Im_LDR preferred, to be used for rendering systems in LDR. It can use any color mapping it wants, from a set of mappings available in the classification software, for example it can adjust a specific global tone mapping (i.e. luminance mapping) or a color mapping function to be applied to all pixels at any spatial position in the image, based only on its input color value. Or, it can use locally tuned mappings. For example, it can, in a specific geometric region of the image (e.g., specified within a rectangular or otherwise defined bounding format), select only those pixels that are brighter than a specific luminance value (or within specified color ranges) and transform only those pixels according to a local color mapping strategy, etc. It will then record all the things it has done in the form of metadata, for example the global change function of the luminance correlate can be recorded in a parametric way (e.g. power function coefficients for three regions of a curve similar to S, such as an endpoint of the shadows, linear parts on each side, a parabolic curvature coefficient, etc.). If these functions are (largely) reversible, the receiving side can then use them to reconstruct, using this output image as an input image, and using the inverse color mapping strategy, the original image from which this output image was obtained, at least within a certain accuracy (after, for example, having introduced quantization and/or DCT artifacts, etc.).

[022] In the modalities in which the human being does the classification first, the human classifier will produce the mapping parameters Fi(MP_DH) from the M_HDR mapping. However, as the automatic classification will still modify the LDR classification, these are not the interesting parameters in the end. The automatic classification unit will derive from this two sets of new parameters. It will derive a different mapping from HDR to the new LDR classification being GT_IDR, with mapping parameters Fi(MP_T). It will also derive new mapping parameters Fi(MP_DL) to create the preferred LDR classification for humans, Im_LDR, from the second technically classified LDR image, GT_IDR. When storing the data necessary for a receiving side to work with the encoded M_HDR image, i.e. particularly allowing the receiver to recreate a REC_HDR reconstruction, a formatter will typically encode the GT_IDR (for texture objects), and two sets of mapping data, Fi(MP_T) and Fi(MP_DL), into a suitable encoding defined in the specification of any signal standard, i.e. typically into picture (or video) TSIG signal metadata. In case the automatic classification occurs first, the human classifier will work on the GT_IDR to produce mapping parameters Fi(MP_DL) and then these will be written to the signal (in addition to the GT_IDR image and Fi(MP_T)).

[023] Depending on which variant the system is, the automatic classification unit will then apply, as a pre-specification, the second image in LDR, GT_IDR, directly from the main HDR, M_HDR, or as a post-correction based on a previously human-classified Im_LDR as input. The term data specifying a color mapping needs to be clear to the skilled person for any of the many possible variants of color mapping. Typically, classification software may store the parameters of the functions it uses and, in particular, may use mapping functions that are preconditioned to be good for encoding. For example, one can design a number of local or global functions that are reversible (within a specified accuracy) when used conservatively, that is, with values within a range, and may become (partially) irreversible when used aggressively by the classifier. An example of this might be a gamma function. Gamma coefficients up to a value of 3.0 can be seen as reversible for a specific system (i.e. starting from a specific initial dynamic range, e.g. CODR, or e.g. with significant important data in several subbands of a defined reference range of 5000 nits, up to a specific reference situation in LDR, e.g. a legacy system definition in LDR, e.g. the specification of the sRGB viewing environment), but gammas above 3.0 can be seen as bass for at least a subband of the input luminance range (ie, for reversible reconstruction). Or, in an extended set to produce a posteriori LDR classifications from auto GT_IDR, there may be functions that do not exist in auto classification, and create significant loss of information in the input HDR information when creating a desired classified LDR image with it. The system can typically operate in a mode or phase where the classifier has limited freedom to create LDR images, but with good technical properties (i.e. close to a well-functioning GT_IDR), and a mode or phase where the classifier has (almost) unlimited freedom, or at least greater freedom in determining its optimal LDR-classified image, Im_LDR.

[024] In advantageous embodiments, the automatic classification unit (203) is arranged so as to determine its automatic color mapping algorithm by meeting a condition that a reconstructed HDR image (REC_HDR) that fits a second predefined accuracy from the input image with high dynamic range (M_HDR) can be calculated by applying a second color mapping algorithm (CMAP_2) to the second image with low dynamic range (GT_IDR).

[025] Thus, the automatic classification unit will maintain the quality of the second image in LDR GT_IDR to allow good reconstruction of the main HDR. It will meet this condition by restricting the functions that can be used to relate M_HDR to GT_IDR. In particular, not a very large amount of (significant) data should be lost by these actions, for example the quantization of color components, for example (R,G,B) or (Y,Cr,Cb), etc. Thus, your mapping functions will typically be selected based on this assessment, whether it was an a priori calculation (e.g. by an algorithm that is pre-tested in the lab so that when operating on HDR images with e.g. certain luminance histogram distribution properties will yield good reconstructability for certain mapping functions or algorithms), or a post-calculation, e.g. in an iterative circuit that selects the best of a number of possible mapping functions. The second predetermined accuracy is the final accuracy, which can be obtained by reconstructing the REC_HDR from the data encoded with the chosen mapping algorithm, that is, by applying the inverse of Fi(MP_T) to the GT_IDR, whose inverse is called the second color mapping algorithm, CMAP_2. For the first modes of automatic classification, this will mean that the automatic classification unit will only determine the mapping between M_HDR and GT_IDR (and the user does not really need to bother with this relationship). Then it will select, for example a suitable gamma function, so that the GT_IDR still has a reasonable approximation with the dark appearance in the M_HDR, but still none of the relevant luminance values are too closely grouped together in a luminance GT_IDR. In the situation of human-first classification, the automatic classification unit still needs to determine a final mapping Fi(MP_T) between M_HDR and GT_IDR. This corresponds to redetermining a new second image classified in LDR, GT_IDR, after the human classifier (but this will not destroy the classification by human being, since the mapping parameters for reconstruction from the GT_IDR are also determined). There may be several strategies for this. For example, the automatic classification unit can query the mapping function, and slightly deviate it in regions that lead to severe data loss, for example due to quantization. With this, the automatic classification unit could study the obtained images (Im_LDR versus GT_IDR, compared to M_HDR) but also the mapping curve itself (noting how much it deviates from mapping curves with generally good performance). Another possibility is that the automatic classification unit selects one of a set of mapping functions that is close to the one selected by the human classifier, and still performs well. From there, mathematical calculation is used to obtain the final system. For example, GT_IDR will be obtained by applying a shift function in the mapping function Fi(MP_DH) from M_HDR to Im_LDR of human classifiers. Indeed, the automatic classification unit can then directly apply this final function to M_HDR, to obtain GT_IDR, directly with minimal error. Im_LDR can be derived from it using the offset function. The skilled person understands how, similarly to other mathematical structures, the automatic classification unit can determine an optimal Fi(MP_T) mapping and correspond to the same a mapping from the GT_IDR to the Im_LDR of the classifier (i.e., Fi(MP_DL)). This is shown schematically in Figure 6, as the application of a technical deformation DEF_TECH to the classification of the human classifier, to obtain the technically classified LDR image GT_IDR. That is, the automatic classification unit can work either starting from the image in LDR Im_LDR and working on a deformation philosophy, and deriving from this Fi(MP_T), or it can directly observe the appearance of the Im_LDR classified by human being, and make an approximation of the same starting from M_HDR, given the technical limitations of its technical classification, leading to an Fi(MP_T), and determine from it an Fi(MP_DL) to derive the classification by human being from GT_IDR (mapping which can then be technically very liberal), etc. Thus, it should be clear to one skilled in the art in what ways the condition can be, and will be, met. Again, the accuracy can be preset to any measure, for example for a quality class of technology (e.g. high quality film for premium users versus low quality HDR encoding which mostly gives the impression but not ultimate quality), for example specifying that certain mappings will, in an HDR image that is a hard case, create artifacts that are no greater than artifacts of a pre-agreed magnitude. Other mapping strategies that do not behave according to the specification should therefore not be used. In any case, other than minutely accurate details in definitions, it should be clear to any infringer whether he is using the double rating technical chain system as described above.

[026] As already shown above, it may be advantageous if at least the automatic classification unit (203) and possibly also the image classification unit (201) are arranged to apply a monotonic mapping function on at least one luminance correlate of pixels in their respective input image, in at least one geometric region of the respective input image corresponding to the same geometric region of the input image with high dynamic range (M_HDR). Having this kind of one-to-one functional definition on uniquely identifiable regions of the image (eg, the entirety of the image), means that at least on an infinite-precision axis, one can easily invert these functions. It is especially advantageous if also the derivatives or slopes of these functions are such that they do not integrate many of the M_HDR luminances into a single Im_LDR code, or at least GT_IDR. Furthermore, these monotonic functions are easy to compute technically, for example with a lookup table. For example, this could take a luminance correlate, such as Y-luminance, as input and output. An example of a frequently occurring HDR scene, which can be made with two spatial regions, is an image with an indoor and outdoor environment, for example shot from inside a car, a room, etc. By "geometric regions corresponding to" is meant that if the region is defined in, say, Im_LDR, then the pixels are identifiable with pixels in M_HDR. For example, if the image has the same geometry (resolution and cropping), the pixel positions can be placed, but in case of geometric transformations, for example scaling, it should also be clear what this means for the technically-versed person.

[027] Although simple systems can, for example, use fixed, pre-agreed Fi(MP_T) mapping functions, which always work correctly, it is advantageous that more advanced systems can optimally determine the mappings themselves, particularly if the automatic classification unit (203) is arranged in such a way as to determine its automatic color mapping algorithm according to a quality criterion that estimates a difference between an amount of information in the luminance correlates of pixels in the input image with high dynamic range (M_HDR) and an amount of information in the luminance correlates of pixels in the second image with low dynamic range (GT_IDR).

[028] One skilled in the art will understand that there are different ways of defining amounts of information, but all involve measuring how much data there is in a representation (especially meaningful data). There may be semantically blind methods, which only measure what colors are available, but not what region or object they come from. For example, one can measure how many of the M_HDR luminances are mapped to a single GT_IDR luminance. If, for example most luminances are only mapped by twos, but in a certain region of the luminance axis M_HDR, 5 digital values of luminance in HDR (or a floating representation of a range of luminances exceeding a certain size) are mapped to a single luminance GT_IDR, this can be seen as a very large loss of information. Thus, the size of spans, or the amount of luminances digitized in M_HDR is an example of a possible amount of information measurements. Of course, these measures can be made smarter, for example by observing how they behave particularly interesting sub-regions of the M_HDR luminance range, or even semantic objects, for example a face. It may be prescribed that, for example, each face should be represented by at least 50 luminance codes in GT_IDR, or each region of a face having N digital luminances in M_HDR (or an equivalent continuous range thereof) should not be represented in GT_IDR by an M quantity of luminance less than half of that N quantity. For example, one can specify how many barely perceptible differences JNDs would correspond to a certain GT_IDR encoding when reconstructed to REC_HDR under a reference HDR viewing environment. And then one can specify that the face should be reconstructable with at least R discriminable JNDs. Or a structure on a face, such as a wrinkle, should change from a darker value inside the wrinkle to a brighter value outside the wrinkle, by a reconstructable step of at most S (say, 3) JNDs. It also introduces the concept of just careable differences JCDs, which can be used for some semantic objects. For example, in a luminaire, it may be sufficient that the luminaire is bright, and yet something of the interior structure (such as a lamp shape) is distinguishable, but neither the exact value of the luminaire nor that of the lamp nor their relative luminances can be of critical importance. In this case, both regions can be coded as being considered accurate if within, say, 1 JCD, which could be, say, 20 JNDs, or specified as a difference or fraction of luminances (for luminances falling within a defined subrange of bright luminances to be used for light rendering). Thus, information criteria can be determined only on the basis of one-dimensional or three-dimensional binning (shape and/or size) of the color data in both images, statistical criteria such as luminance or color histogram, and in particular semantic information of which regions may be most severely deformed (e.g. the human classifier can quickly draw a squiggle over the regions of the image that need to be coded with high accuracy, such as the main action region, which may be especially illuminated during capture, or a face), information geometric features, e.g. edges or shapes of structures in object regions, and how they deform (e.g. sharp visibility, or contrast) under certain classes of mappings, or texture characterizers (e.g. in complex textures a higher amount of artifacts is admissible), or semantic information such as automatic detection of specific objects, or the human characterization of them (by at least approximate marking of a region and class as “least important lamp”), etc. Therefore, the person skilled in the art can understand that there can be several ways to predefine a system of mathematical functions that specify when too much data has been lost, for example reducing the quality of a textureless variable lighting on an object, etc. There may be a single criterion, or a set of criteria that results in a complete analysis of the GT_IDR image, and marks that a certain region of it needs to be redone. With this information, the image classification unit can determine whether a mapping satisfies the technical requirement, or it can determine a new mapping, for example adjusting the old one slightly. For example, if a region of GT_IDR still reconstructs a region (e.g. an object) of M_HDR very roughly, the image classification unit may either redetermine entirely, for example a global mapping (typically, of course, it may only fine-tune the mapping for those M_HDR luminance regions that pose a problem, for example it may increase the derivative of the mapping function Fi(MP_T) downwards, for the problematic luminance subband, which typically corresponds shifting outwards - respectively towards darker values versus brighter values - of the other pixel colors, and adjusting to the new range available for them by gently flexing those parts of the mapping function). Or, the image classification unit may derive an additional local classification to be applied in temporal succession, for example a pre-boost of that region, and save it in a cocoded partial (correction) image, etc. Typically, it is advantageous when the image classification unit, even when creating the GT_IDR with a pre-recognized suitable color mapping strategy, post-determines when the GT_IDR image actually satisfies the condition that REC_HDR is an approximation with sufficient quality.

[029] Advantageously, the automatic classification unit (203) is arranged to determine the monotonic mapping function (Fi(MP_T)) of the luminance correlates of pixels of the input image with high dynamic range (M_HDR) to the luminance correlates of pixels of the second image with low dynamic range (GT_IDR), according to a criterion that determines the respective ranges of luminance correlates of pixels of the input image with high dynamic range (M_HDR) allocated to the respective unique values of a luminance correlate of pixels of the second low dynamic range (GT_IDR) image, where the respective ranges form a set of luminance correlate ranges covering the entire range of possible luminance correlate values for the high dynamic range (M_HDR) input image. This is a simple way to determine information loss, for example due to excessive quantization. For example, a preset range size can be set to map onto a single value versus input M_HDR luminance along the M_HDR luminance axis of all possible values, allowing you to specify that brighter objects can be more coarsely quantized. It may be that they are already approximated with significant error compared to the original captured scene (for example, you don't need to render a car's headlights at exactly 100,000 nits on the HDR display device), so you can accept an additional error in REC_HDR. This criterion can then easily be converted, for example in determining a format of a mapping function, as it should at no point bend so strongly as to map a larger range than is allowed to a single quantized value, given the known settings of the GT_IDR encoder (e.g. MPEG2 quantization values).

[030] The above describes the inner workings of an encoder that can be used in several instruments, for example an intermediate system in an image calculation unit, but it is advantageous if the encoded data obtained is sent out, for example in the form of a signal that can be used by a receiver, that is, the image encoder (202) comprises a data formatter (220) arranged so as to output, in a image signal (TSIG), the second image with low dynamic range (GT_IDR) and at least one of, or both of, the data describing the color mapping (Fi(MP_T)) between the input image with high dynamic range (M_HDR) and the second image with low dynamic range (GT_IDR), and the data describing the color mapping (Fi(MP_DL)) between the image with low dynamic range (Im_LDR) and the second image with low dynamic range (GT_IDR). In principle, not all receivers would need both sets of parameters, but it is advantageous if a receiver receives both, and can then, for example, determine optimally how to use all available information to arrive at a final triggering signal for a specific display device and viewing environment (for example, it could mix the encoded HDR and LDR rating information, to arrive at a new rating, which is called the display device's tuning capability). Note that while the present basic system has been described with only two ratings, within the same system there may be additional ratings, for example a second HDR rating for an ultra-bright HDR display device, or a third LDR rating, or a rating for an MDR display device (of a white_peak intermediate between, say, 100 nits and 5,000 nits references of the LDR and HDR ratings), or for sub_LDR display devices, and these may be interpreted as supplements independently designed, but also according to the philosophies of the invention presented, for example a second technical classification GT_IDR2 can be derived, which is an HDR classification technically derived from M_HDR, and which serves to define the ultraHDR classifications. For example, GT_IDR2 can be derived through simple mathematical stretching of the brightest light regions, but the classifier can correct this by defining additional mapping data Fi(MP_DHH), for example to correct by mapping from GT_IDR2.

[031] The image encoder corresponds to an image decoder (401), which is arranged to receive, via an image signal input (405), an image signal comprising a second image with low dynamic range (GT_IDR), and data describing a first color mapping (Fi(MP_T)) that allows the reconstruction of a reconstruction (REC_HDR) of an image with high dynamic range (M_HDR) based on the second image with low dynamic range (GT) _IDR), and data describing a second color mapping (Fi(MP_DL)) that allows the calculation of a low dynamic range image (Im_LDR) based on the second low dynamic range image (GT_IDR), wherein the image decoder comprises an image derivation unit (403), arranged to derive at least the low dynamic range image (Im_LDR) based on the data describing the second color mapping (Fi(MP_DL)) and the pixel colors encoded in the second image with low dynamic range (GT_IDR). It can be seen from this decoder that it can access mapping parameters to map the colors of an LDR image both upwards to a REC_HDR and “downwards” to obtain an LDR classification, Im_LDR, desirable for content creators. The image derivation unit will have functionality (eg loaded software or hardware parts of an IC) to perform the necessary decoding color mappings (eg pre-arranged). One can also see that the technical classification GT_IDR is a technical classification, as it will have lower mood load (even without comparing to the optimal Im_LDR), since the objects' luminances are not in the optimal location along the luminance axis, and will typically have slightly lower contrast, slightly brighter darks, etc. and, of course, a limited amount of codes for the various object regions in the image.

[032] Advantageously, the image decoder (401) comprises a system configuration unit (402) arranged to determine whether the decoder is connected to, and/or whether to derive an image for, at least one of a high dynamic range display device (411) and a low dynamic range display device (416), wherein the system configuration unit (402) is arranged to configure the image derivation unit (403) to determine at least the reconstruction (REC_HDR), in case of a connection to the display device with high dynamic range (411), and arranged to configure the image derivation unit (403) to determine at least the image with low dynamic range (Im_LDR), in case of a connection to the display device with low dynamic range (416). The present system (i.e. the encoded signal and various decoder types) needs to be able to work with simple decoders that for example receive an HDR encoded like the present LDR_container above in GT_IDR, but this needs only the LDR for an LDR display device. They will then ignore most of the information, and extract only GT_IDR and FI(MP_DL), and calculate Im_LDR from them. The most sophisticated decoders will determine, for example instantaneously with the display device they are connected to, for example wirelessly, and will deliver various combinations of all received encoded information, optimally for the various connected display devices (for example, the same movie sent to the parents in the attic cinema room, and to the child in bed, watching on his portable LDR device).

[033] So, advantageously, the image decoder (401), according to any of the preceding claims, outputs a wired connection (410) or a wireless connection (415) to any connectable display device, and a signal formatter (407) arranged to transmit at least one or both of the reconstruction (REC_HDR) and the image with low dynamic range (Im_LDR) to any connected display device.

[034] Also advantageously, the image derivation unit (403) is arranged to determine an additional image (FURTHGR), based on the reconstruction (REC_HDR) and the image with low dynamic range (Im_LDR), or on the second image with low dynamic range (GT_IDR) and the data describing the first color mapping (Fi(MP_T)) and the data describing the second color mapping (Fi(MP_DL)). This makes it possible to determine the optimal final ratings (e.g. direct trigger signals) for various connected display devices (tuning capability of the display device, e.g. obtaining a value of the surrounding lighting by measuring the display device, etc., and optimizing the display trigger signal with the same).

[035] The image encoder can be understood in various devices, for example its image signal input (405) can be connected to a reading unit (409) arranged to read the image signal from a memory object (102), for example a blu-ray disc.

[036] All embodiments of the above apparatus can be additionally realized as equivalent methods, signals and signal storage products, in various uses or applications, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

[037] These and other aspects of the method and apparatus according to the present invention will be evident and clarified with reference to the implementations and embodiments described later in this document, and with reference to the accompanying drawings, which serve merely as specific non-limiting illustrations, exemplifying the more general concept, and in which dashes are used to indicate that a component is optional, with components without a dash not necessarily essential. Strokes can also be used to indicate which elements, which are explained as being essential, are hidden inside an object, or for intangible things, for example selections of objects/regions (and how these can be shown on a display device).

IN THE DRAWINGS:

[038] Figure 1 schematically illustrates a legacy image or video coding system, as exemplified by a camera with an adjustable knee point;

[039] Figure 2 schematically illustrates a first possible embodiment of the present image coding system, in which the automatic classification unit derives the second automatically classified LDR image GT_IDR based on a previous LDR classification by a human being;

[040] Figure 3 schematically illustrates a second possible embodiment of the present image coding system, in which a second LDR image automatically classified GT_IDR, from an automatic classification unit, serves as the basis for additional classification that defines a final LDR classification Im_LDR by a human classifier;

[041] Figure 4 schematically illustrates a possible variant of a picture decoding system, in which a high-end decoding apparatus reads the data encoded according to any of the present encoding modalities, and derives signals suitable therefrom for several different connected display devices;

[042] Figure 5 schematically illustrates an encoder modality according to the principles of the present invention, which is incorporated in a camera;

[043] Figure 6 schematically illustrates a principle behind a variant of the present coding, shown as a logical graph of color mapping relationships between ranks;

[044] Figure 7 schematically illustrates a way to define the present initial input, which is a main HDR with M_HDR classification, and shows how data from a camera or a computer graphics system can be recorded in this type of mathematical specification of colors, in particular along the range of its luminance correlate;

[045] Figure 8 schematically illustrates an example of a color mapping strategy, specifically a luminance mapping part thereof;

[046] Figure 9 schematically illustrates an example of how to determine whether any mapping function or algorithm has adequate accuracy for REC_HDR reconstruction;

[047] Figure 10 schematically illustrates an example of how to transform an inadequate function into an adequate one;

[048] Figure 11 schematically illustrates some examples of how to manipulate color mapping in a three-dimensional color space;

[049] Figure 12b schematically illustrates how a classifier can interact with a code allocation curve to fine tune it, and in Figure 12a it is schematically shown how regions of the code allocation curve can be selected by (co)interaction with objects in the currently observed image;

[050] Figure 13 schematically illustrates how one can go from a technical classification that, in this example, would already be useful for rendering in LDR, to a classification in LDR with better appearance, through a very simple parametric transformation, performing a good quality contrast stretching and adaptive to the content;

[051] Figure 14 provides an example of how one can handle additional colorimetric limitations in the technical classification, which can then, upon receipt, be used to generate the optimal image for rendering on a specific display device;

[052] Figure 15 schematically illustrates a useful new strategy for saturation processing, which is especially interesting for classification for color representations that have a different luminance structure, for example due to an intended rendering on a display device with different luminance dynamic range;

[053] Figure 16 schematically illustrates an apparatus of the creation and use part for this innovative saturation processing; It is

[054] Figure 17 schematically illustrates just two possible uses when classifications are required for a higher and lower dynamic range rendering situation.

DETAILED DESCRIPTION OF THE DRAWINGS

[055] Figure 1 presents a summary of the ideas behind all the classic image and video encodings, which are called LDR encoding. Suppose you have a professional television camera (although similar considerations apply to consumer still cameras) capturing light with an image sensor 104, for example a CMOS sensor. That light will, after an ADC, be in a linear space that correlates to luminance (in principle being luminance times a scale factor when ignoring noise, ADC non-linearities, etc.) and will be, for example, a so-called raw R,G,B signal (or cyan, magenta, yellow and green, or similar for other sensors, but these will then be matrix converted to RGB, so you can focus on that). The principle behind this LDR capture is that a signal needs to look good on a home TV (which used to consist of a TRC with approximately 100 nits of peak_white, or slightly darker or brighter). In a studio, a director, camera operator or similar person will directly watch the camera output on a reference CRT to verify that the captured program does indeed look good. Capturing this type of program in LDR (indeed, auto exposure as primarily determined by selecting an aperture setting) is determined by the principle of having a suitable brightness rendered for mid-grey. As this average gray is, due to the non-linearity of human vision, directly linked to the whites in similarly illuminated parts of the scene (and assuming a (quasi)linear reproduction, typically up to a scale factor, also in the TRC), it specifically corresponds to objects reflecting approximately 18% of the incident light. The lighting designer adjusts your stage lighting so that around the action the lighting is relatively even (e.g. 3:1 contrast ratio), and perhaps also brightens some corners of the scene to avoid “black holes” in the final render. Now, having a 110 display device with a maximum obtainable white_peak (this is especially true with, for example LCDs with some fixed TL backlighting, but also with a TRC in which the beam intensity could be controlled via a contrast adjustment, for any setting there is still a maximum obtainable brightness), does not mean that this has to exactly match the white of, say, a high reflective paper in the sweet spot of the scene. Due to the ever-existing variation in lighting, especially for moving action in video, this would be impractical, and when one moves the paper to an area of slightly higher illuminance, undesirable clipping can quickly occur. Thus, a little care is needed on the bright side, although for typical LDR scenes and programs, not as much is needed. On the dark side, you simply let the signal fade into camera noise. So on the bright side it renders white at a W position below the white_peak PW. Not too much, preferably, so that it still looks white and not light gray (there is some latitude in this for natural images composed of objects). Also, the mid-grey MG, and the human face colors that frame around it, will then be reasonably visible, as they will be reasonably bright in the render. So that's the minimum requirement of any capture or rating, that you can clearly see the acting of the actors in their faces (and eyes, which could be a bit more shaded due to their protruding location in the eye socket) and, more specifically, the face colors of all those beautiful actors will look attractive (and not too dark or pale). The rest of the colors (eg darker) then automatically become reasonable along the curve. Typically, a curve that is about a square root is used, historically due to the behavior of the TRC (non-linearity of the electron gun modulated by faceplate illumination, etc.), and this curve is still used very usefully because it models the luminosity characterization of human vision (i.e., luminance values are approximately luminosities, the first being the mathematical value encoded in, for example, a Rec. 709 space, and the second being the psychovisual appearance of a human being). Now the TV can do some simple transformations on that curve, for example it can boost all the values by a multiplicative factor. These operations, for example to compensate for a change in the viewing environment, have an impact on the appearance properties of the psychovisual image, such as the contrast of the image, and the contrasts of its objects. The camera can perform similar operations. Now the question is where to optimally place the luminances like MG and how to do it easily. In a simple system, an encoder 101 in the camera can transform all values between Maximum Luminance (Max_Luminance) of the raw signal and whatever is below, by bending them with a square root function defined from that Max_Luminance. Then, all possible values captured by the camera from the scene will be encoded in an Im_LDR image (here shown in an encoded image comprising medium 102, like a blu-ray disc, but could also be a signal transmitted over a cable or over the air) thus generated (typically, the values are quantized to, for example, 8 bits, and other operations can be performed, such as image encoding operations, such as approximation decompositions with discrete cosine transform, DCT). By squaring the values of the encoded luminances, a decoder 103 can retrieve with the display device the original luminances of the scene, as recaptured as luminances rendered by the display device. Now, in this tightly controlled system, there is a degree of freedom to accommodate the slightest variations in a typical LDR scene. With the blind use of this square root allocation, it can happen (if you determine the maximum of the scene on highly illuminated objects, by determining the exposure for those objects in order to still have them captured well) that the middle gray and face colors are too dark on a curve like this. If there are a lot of shiny objects that need to be captured reasonably well, it would be nice to have a curve for code definition, which decreases a bit more slowly starting from the brightest RAW code. This can be done by offering the cameraman a controllable knee point. You can, for example, knee select the input luminance level corresponding to your ideal white point for action, and set this to, say, 90% of the maximum luminance code (corresponding to the peak_of_white on the display device). You then have 10% of codes remaining for encoding all values above that, and you can adjust the slope of the above-the-knee part of the curve, to incorporate, for example, luminances up to a maximum of 600% of the ideal white luminance point. This way, you can adjust this curve corresponding to a simpler or more contrasted lighting of a scene in LDR. If it's a low contrast scene, you can put the knee point close to the maximum luminance, and hardly encode any luminances above the white of the ideal point, and if you want to have a lot of high luminance information, for example in a talk show where you talk about shiny silverware, you can incorporate some of the bright highlights into the encoded luminance signal. This simple system automatically adapts to the best rating of a particular LDR scene on the display device side, i.e. leaves a little extra room for the silver highlights, darkens the darker colors a bit, and pushes some of the visible structure into the brighter objects (severely warped compared to the original scene luminances of those bright objects, and often pastel colored due to the gamma format of RGB systems, but still existing to some degree). However, this type of LDR system quickly cuts out the highest brightness, and is not suitable for encoding, for example, the outside world visible through studio windows, which they don't care about. This sometimes leads to awkward situations, when a cameraman working in the field decides to photograph a person from across their living room, where it is relatively dark. So half of the captured image that shows the brightest parts of the room will be clipped to white when the exposure is correct for the face. While this can still be a minor annoyance on an LDR screen, where whites simply appear as some "white colored objects" but not really bright regions, it leads to a rather odd situation on a 5,000 nit display device, where half of the image is extremely bright.

[056] So, the LDR system, with its philosophy and also the inherent capabilities of its technical construction, is not suitable for capturing in HDR, where at the same time you want to capture a first illuminated part of a scene, and a much more (for example, 100 times) illuminated second part of a scene and, perhaps, simultaneously even a very dark part, etc.

[057] With Figure 2, some of the principles behind the variants of the present invention are now clarified, specifically a specific embodiment of an image encoder 202 incorporated into a classification system. The present automatic sorting unit 203 is shown as being part of a sorting system. This type of system could, for example, be a computer running sorting software, but it could also be a less complex system in which a human being, for example, only occasionally changes some settings of a color mapping from a remote location. A human classifier may specify its desired color mappings via a user interface system 230, which may for example comprise a dedicated classification console, with trackballs etc. Coupled to the software, you can increase, for example, the color saturation of a selected image region, or drag upwards a marked point on a tone mapping curve (for example, red component input (red component_in) versus resulting red component output (red component_out), for all pixels). The present description will focus on what typically happens in a main HDR image already optimally pre-classified (for example, received via input 240 connectable for example, to a data server, or an Internet connection, etc.), although it may also come directly from a camera, which may, for example, have done some classification internally. With Figure 7, an example system is described of how to define these classifications or main HDR images. Any camera capture or classification is, in fact, just a representation of a world scene, which needs to encourage a human being to get a reasonable impression of that world scene, so it need not necessarily be an exactly accurate representation. Indeed, it is always necessary to cross the difficult boundary from scene-as-reference, where the camera simply acts as a linear measurement device, to display-as-reference, where a display device needs to emulate for a human being the original scene in a very different configuration (although some of this complexity need not be handled in the main coding, but can be left to display device transformations). It can be argued whether a core encoding should be able to accurately encode e.g. the sun, where a display device will never be able to accurately render the sun (which, even when possible and sensible, in terms of power consumption, would be very irritating to the viewer on smaller screens). So why not allocate it yourself to, for example, a fixed high luminance code (for example, a pseudo sun of 20,000 nits instead of 1 billion nits). Furthermore, a problem with scene-referenced systems, which are capable of encoding all sorts of values that are not easy to make visible, is that these color spaces are not easy to work with. For example, if a classifier needed to adjust the saturations of some saturated flowers that it can't see on its current sorting display device, it might be making ugly colors for when that image is shown on a better display device, which might show those colors. Perhaps this is an operation that can be fixed later, but one might wonder why this was done in the first place, at least with those colors. In the HDR encoding mode of Figure 7, a large range of luminances are given a lot of value, which can be encoded (/classified) with reasonable accuracy into a luminance range of a high quality HDR reference display device, for example with 10,000 nits of white_peak (corresponding to a MAX_REF luminance value in a full HDR representation HDR_FREP). The idea is that you can have at least the most interesting colors characterized in a high dynamic range specification, and that the classifier can actually see them, and optimally position the luminances of various scene objects, one against the other (e.g. darkening clouds). This range of, for example, 0.01 nit (which you can simply call 0) to 10,000 nits of colors that can be displayed will be the main M_HDR* classification in question, since you can classify it optimally. The idea is that any display device with lower dynamic range can derive its colors to be rendered starting from the specification of the colors within M_HDR* (typically it is extracted from the complete HDR image representation, HDR_FREP, this range as M_HDR input for the present system, for example in Figure 2). This spec will also likely work reasonably well for display devices with higher dynamic range. For example, the classifier can roughly classify some bright lights in the scene so that they at least appear bright on any display device rendering. It can encode, in the M_HDR* reference rating for the HDR reference display device, the brightest light at, say 99% (linear) of MAX_REF, and it can encode another bright light so that it's still bright, but in contrast, definitely less bright, at, say, 80% of MAX_REF. A real 20,000 nit display device could use simple scaling on the codes for those lights, for example boosting both by a factor of 2, which merely corresponds to referencing them similarly (percentage) to their highest white_peak. In that case, both lights might be a little brighter - as they could have been in the original scene - but for the most part that rendering in HDR still looks similar to the reference look on the classifier's 10,000-nit reference monitor. Indeed, if you want to more accurately encode values higher than what can be roughly encoded in M_HDR* (e.g. properly dimming a light so that it is in-range yet similar in appearance, i.e. the exact scene luminance of that light is irrelevant) you can do so in the color space definition of Figure 7, regardless of whether you would like to use those values later on, for a display device with higher dynamic range, in order to render those bright colors more accurately than with a stretch of the encoded values in M_HDR*. Thus, you can still add some really bright (or really dark) colors outside the M_HDR* range of what would typically result in a really good rendering of an HDR scene, and you can typically compress those colors heavily, i.e., represent them with just a couple of values, with the codes in HDR_FREP being highly non-linearly related to the actual luminances in the scene. For example, when looking at a scene with very high contrast, for example a welding scene at night, there can be a long period of time with nothing above the M_HDR* range that encodes the useful objects, and then there are the arc luminances. One could represent those with a strongly posterized arc format (that is, some code) and place them just above MAX_REF. This would already be a way to have reasonable representations of that arc (already with the gamma mapped to colors that are close to those that are typically renderable on an HDR display device) but, if you want, you can also shift in metadata a function of how to shift them to luminances closer to the real luminances in the scene (for example, a luminance shift constant). It is shown a range of capturable luminances cam_1 of an HDR camera that can perform this capture, and part of it is encoded in M_HDR*, for example directly through allocation (possibly with a linear contrast multiplier) of luminance values for luminance values within M_HDR*, or if you can use some mapping function that automatically performs a first type of classification (for example, for example, for example, for example a little of each other the brightest or darker luminances). The brightest luminances captured by the camera are then stored in the overflow range RW_STR_HI up to the maximum MAX_REP of the color encoding. An example was shown where a certain level of darkness is encoded in code 0, and with some mapping even darker luminance can be stored in negative values up to MIN_REP. It was also shown, for example, how special effects professionals can draw in computer graphics, CG, HDR color coding, as bright explosions.

[058] Again referring to Figure 2, the human classifier uses the image classification unit 201, which is arranged to perform any one of a set of color transformations. This can be a limited set of color transform functions following mathematical criteria, for example reversibility (the term reversibility usually means that in a sufficiently accurate color coding, such as with floats, it is possible to reverse the function to redeliver an input image from its output image, after applying the transform; the term color means at least a luminance correlate of the color specification of a pixel, region or object) or, preferably, it is a broad set of functions that allow the classifier to classify the image in any way that you wish. Examples of typical functions are those supported, for example by Da Vinci Resolve, or Adobe Photoshop. Internally, at this stage, it can be assumed that all processing still takes place on the floating [0,0, 1,0] color coefficients encodings, with the precise definition including quantization being activated at a later stage in the image classification unit 201. However, the output LDR image, Im_LDR, will typically already be encoded according to the image or video encoding standard, e.g. for uncompressed signals it may be quantized in a YCrCb color space, or it may be compressed using small waves, etc. The actual formatting of this image, e.g. splitting it into blocks of data, whilst inserting headers and other metadata, will typically be handled by a formatter 220 which outputs a TSIG image signal, e.g. to a memory for images 102. This signal may be stored in that memory according to e.g. One skilled in the art will understand that, similarly, the TSIG image signal can be sent over some data connection, for example wirelessly to a home server with permanent or temporary memory for storage of the TSIG or the one or more images.

[059] In Figure 8, an example of how a classifier can classify starting from an input image to create an output image is presented. It focuses on the relationship between subband brightnesses and, later, presents some examples of how to handle the chromatic components of pixel colors. It is assumed that the pixels of the input HDR image (In) have been encoded with their L_HDR luminances, and that the output LDR values are encodings, so we can call them Y_LDR luminances. Although the present methods are by no means limited to specific bit depths, it is assumed that the luminances are in the range of [0.255]. Now the classifier will study the specific input image to process (for video this will be a key image in a shot of images to be similarly color mapped), and design an optimal color mapping, in the example a multisegment mapping, according to your preference. Suppose you have a covered part (for example, under the shade of trees, where the main actor resides) below the luminance in HDR Lt_1, and some houses in the background that are brighter. So your pixel luminances in HDR will fall above Lt_1, but you don't need the darkest to be directly above Lt_1. Furthermore, there may be a very bright lamp, with luminances above Lt_3. Now this scene is different from the classic LDR scene in question above. You have two interesting scenes, the shadow around the actor (where you want to make the actor very visible, but still clearly darker than most of the scene) and the sunlit houses in the surrounding background. The classifier may, for example, choose to make the region between Lt_11 and Lt_12, which contains the face colors, sufficiently bright and contrasted to make the face clearly visible. It can do this to the detriment of darker colors, which need to be encoded with few Y_LDR code values, and below Lt_13 they will even be clipped to Y_LDR=0. It will also encode/classify with less contrast/accuracy the luminances between Lt_12 and Lt_1, lowering the slope of that segment. This makes room in the Y_LDR range for the sunlit houses, which it records with the curve defined by dragging a CP control point. Lamp colors, after a non-occurring color luminance discontinuity, can be coded just above the Y_H luminance end of the houses, or start a couple of codes (eg 10) above that.

[060] Now, in the LDR container philosophy in question, this color mapping curve can both function as a color appearance optimization curve for the output image, i.e. for example typically an LDR classification derived from an HDR classification, and also function as a code definition curve. Analogous to the 0.45 gamma curves of, for example, MPEG, which define a luminance encoding for each input or rendered luminance, the optimal curve in question defines the allocation of specific code values to the various luminances in the input HDR image. However, therefore, simultaneously the various image objects or their luminance subranges are also already correctly positioned along the luminance axis to drive directly (or perhaps with minor transformation, which may involve pre-correction of physical display device characteristics, such as EOTF manipulation, or small adjustments towards, for example a specific dimmed viewing environment, via a simple display device transformation) an LDR display device.

[061] Thus, some fundamental truths of LDR coding technology were altered or generalized. In particular, by asking what is white in HDR (the white paper in the sweet spot of lighting indoors, where the action takes place, or the white paint of sunny houses outdoors; with human vision also able to handle all of this very deftly semantically), the view of pinning everything to a specific white, or a correlate of it, as “the” middle gray of the scene/image was abandoned. As an alternative to this, color schemes are presented, which can work on their own, whatever the luminance ratio with specific colors, such as some white. For example, there may be the regime for the sunny houses or the light bulb, which may have their own treatment which, now, without necessarily making precise references, may be semantic-relational rather than numerically precise. The example of a bright outdoor environment, or a very bright light, has already been given. Instead of, using a fixed luminance ratio, making the brightness (eg medium, or lower) of the external environment for example 5x brighter, one can simply make it “brighter by a fair amount”. The reasonable amount can then be ultimately determined on the display device side. For example, a very bright 15,000 nit HDR display device might make the outdoor environment 20x brighter, but one that is limited might need to compress all sunny house colors into a higher range that is only 1.5x brighter, on average, than the indoor environment colors, giving only an initial simulation of the fact that it is sunny outside. Similarly, a dark region can be rendered not so exactly with luminances Lx, Ly, etc., but as "only distinguishable". Intelligent HDR decoding and optimization systems can take into account the specifics of the display device and the environment, and can further optimize, starting from the defined classification adjustment, the final trigger signals.

[062] So, secondly this means that it abandons the concept of a single fixed code that defines the curve as a main general range of 0.45 which, being close to human vision, is considered adequate at least for the entire range of LDR luminance. Whether you use any color classification as strict (i.e., the display device needs to try to render as close as possible to how it would look, for example on an LDR reference monitor, i.e., with minimal self-adjustment) or as liberal (where the classification is just a relative orientation, roughly declaring how it can deform the colors to keep something of the artistic intent in, for example, a more limited physical range of luminance), the classifier will be allowed, in some embodiments, even arbitrary code definition functions, which may even be discontinuous functions.

[063] And thirdly, it is stated that there should no longer be a single classified image, but instead the classifications need to be optimized for each rendering situation. And human vision being complex, especially the more rendering systems vary in their properties, the less correctly this can be done with automatic color mappings (especially simple ones), and the more the various classifications must be produced optimally by classifications by humans. In practice, however, it is noticed that for many scenarios, given the necessary investments, it will be sufficient to have only two classifications (a classic one for LDR, and an HDR classification for the systems in HDR), and when a more precise adjustment is necessary, the systems can then get closer to the good classifications by technically interpolating or extrapolating them, based on these two classifications from their included artistic information.

[064] Stopping after classification of Im_LDR, and recording that container image in LDR in an image memory, together with a prediction function to reversibly reconstruct an approximation of the main HDR M_HDR from it (perhaps for some systems a classifier using strictly reversible color mappings is not necessary, since, for lower quality systems it may be sufficient to reconstruct a REC_HDR with significant deviations from M_HDR, as long as the HDR effects derived from the Im_ data LDR still produce a reasonably HDR-like appearance, in which case one could cocode those color mapping functions for HDR reconstruction, which have inverse approximations of the actual color mappings used by the human classifier), it would be fine for systems with e.g. 12 or 14 bit (non-linear) defining Im_LDR encoding (depending on the requirements of the various applications). When deciding on more rigid specifications, for example 8 or 10 bits (which most people would say is difficult to encode images in HDR, but since humans only discriminate a few million colors and depending on the application, such as a video with fast movements and considerable noise, the amount of colors needed may be even less, so if it were the case to encode the most important colors needed correctly in 8 bits, this would be possible), it may be useful to follow the additional steps of the present invention to ensure an improved quality of both the LDR classification and the classification. HDR, while allowing the classifier maximum flexibility as to what those classifications should look like (ie, what luminance subbands (luminance correlate) should all objects fall into).

[065] It is now assumed, in the continuation of the example in Figure 2 in question, that the classifier classifies very liberally, with an arbitrary luminance mapping function, and the luminances in LDR are quantized to only 8 bits. The slope of the mapping curve between Lt12 and Lt_1 can be so low that there are too few codes to faithfully represent those colors, for example of vegetation in the shade. In an LDR render that might not be that unpleasant (in fact it isn't, or the classifier wouldn't have specified the curve that way), however, after reconstructing the HDR luminances for these objects, posterization can impart unpleasantly low texture quality to those objects in such a high quality render.

[066] Therefore, the automatic classification unit 203 will analyze the LDR Im_LDR classification of the human classifier, and identify and solve these problems. There are several ways that unit 203 can do this. It can, for example, simply analyze the images themselves, and compare their spatial regions. For example, it can examine a region in Im_LDR (such as a dark face) and count the number of luminance codes that represent it. The unit's internal rules may specify that any region, or especially a face region, should not be represented with less than MB (e.g. 10) different code values (or, in general, a certain number of colors, typically based on the number of luminances thereof, but similar restrictions can count on the number of different saturations representable, for example). Or, unit 203 may compare the amount of codes in the region in Im_LDR with the amount of different codes (typically luminances, but M_HDR could be encoded with different luminance correlates, too) in the HDR image. If there are many different values in M_HDR, there should be a fair number of luminances in Im_LDR for that region, too. For example, the rule might be that the fraction of Im_LDR luminances versus HDR luminances should not be less than 1/5, or 1/10, etc. Similarly, you can define range-based ratios in the floating-point luminance space for M_HDR. More complex image analysis can be done, for example by studying geometric shapes and estimating how these shapes deviate when represented by fewer colors. For example, a detector can identify misshapen patches in shapes. This is a generalization of a band detector, which checks for rows of a number of pixels having the same posterized value in Im_LDR or, in fact, REC_HDR, where they do not exist, and are in fact smooth functional transitions in M_HDR. Additional information can be obtained from texture estimators, which can determine, for example, the local complexity of image regions, etc. Even if the determination of the automatic GT_IDR classification is not going to be based solely on the analysis of any of the obtainable images (classifications) themselves, it is useful if the automatic classification unit 203 comprises an image analysis unit 213 capable of performing any of the above analyzes, as the resulting data is useful even when specifying a model curve, in a system working on the color mapping curves of the classifications.

[067] For this, a curve determination unit 211 is understood, and some of the possible modalities of the same should be clarified with Figures 9 and 10. In any case, if the automatic classification determination is carried out based on image analysis, mapping algorithms such as luminance mapping functions, or both, or any other analysis or prescription, typically the curve analysis unit 211 will have a unit determining and producing a final color mapping algorithm or function Fi(MP_T) (and possibly also, a unit performing one or more of several color mapping curve algorithms or algorithm analysis, by which the algorithm can be analyzed on its own, or for how it behaves with colors when represented by one or more curves). This automatic mapping is now how GT_IDR can be derived from M_HDR, so a color mapping unit 215 will derive GT_IDR by applying the mapping Fi(MP_T) to M_HDR. It is clear that it is necessary to take into account, in this new formulation, what was the classification by human being Im_LDR, now that everything will be in reference to GT_IDR. Thus, an LDR mapping unit 217 will analyze how Im_LDR can be obtained from GT_IDR, and derive parameters therefrom. If a pixel luminance maps from L_HDR = 2000 to Y_Im_LDR = 180, and to Y_GT_IDR = 200, then a mapping between the latter can be derived. In this type of functional form, Y_Im_LDR values across the range can be derived by applying a function per luminance that multiplies Y_GT_IDR by (Y_Im_LDR/Y_GT_IDR). Similar strategies can be derived for other mappings.

[068] Figure 9 clarifies a computationally simple way to redetermine the classification by human being in a technical classification, by studying the classification curve of the human classifier (either by itself, or with the aid of image analysis, or, potentially, even by interaction with the human classifier). An example of curve deformation will be used, but those skilled in the art will appreciate that a similar analysis can be used to select one from a set of well-functioning curves (one or more CRV_i curves in Figure 2). If the classifier wants smooth, non-contrast behavior in a middle region, and stretched behavior in outer regions (for example, in HDR the predominant regions, where the main action takes place, may need to be silky smooth, but background lamps may be rendered more coarsely, and even banding may not be noticeable or not at all important, at least in some situations), one can select (based on, for example, a calculation of a functional correlation) one among a set of pre-agreed curves, which best matches this behavior, but which doesn't have too high a quantization for the middle part. This type of selection can be guided by further image analysis, such as determining which class an image belongs to (sunny outdoor environment, or nightscape with some bright lights), looking at the histogram distribution and its parameters (e.g. estimated lobe locations and sizes, etc. That is, there may be some predefined 901 curves that offer reasonable behavior, at least from an accuracy point of view. A real image needs to be coordinated around this behavior, taking into account the classifier's wishes for what the classification looks like. Of course, whether an actual quantization is good compared to a theoretically optimal or reasonably functional situation will also depend on how many pixels of a particular color are in a particular image. For example, if the dark part is just a small piece of looking through a grate into a sewer, say 50 x 50 pixels in the bottom right corner of an image, then some quantization might be quite allowable for the classifier, at least for that image, shot, or scene. That is, the various curves can function as a final selection for the technical classification (if there is one, or a few key guide curves - for example, depending on the white point of the intended display device, such as whether the image is primarily intended for 1000-nit or 10,000-nit display devices, or additional characteristics of the rendering environment or image properties - determining quantization accuracy with respect to luminance or luminance range), or they can function as starting points to from which the technical classification curve can be subjected to fine-tuning, until it stretches its code accuracy deformations to the maximum for the most critical image classifications (typically the HDR to be reconstructed) and, from there, one can encode the additional requirements into the other classification appearance (typically an LDR) merely by the transform functions to be applied to that technical classification GT_IDR.

[069] Now, however, an exemplary curve deformation algorithm will be described. A specific modality will be calculated from a quantity of information, which consists of a quantity of used NC codes per luminance subband (and one can digitize a continuous band in M_HDR, too, typically by evenly distributing a few integer codes over that band). A specific test band under study (between L3 and L4) will be observed, but although some modalities may only test certain bands, such as the dark bands, it is advantageous if all M_HDR luminance bands are thus tested. One can similarly formulate if any M_HDR range is mapped to, say, 5 Y_LDR codes, or if a subrange thereof is mapped to a single Y_LDR value.

[070] The method starts from a reference coding function 901 (REF_CODF in Figure 2), which specifies how many codes are needed for each range to be well reconstructed into REC_HDR. One skilled in the art should understand that this depends on, and can be calculated from, technical parameters such as the dynamic range of the reference display device belonging to M_HDR, the intended specifics of the Im_LDR rating, etc. These one or more reference coding functions, although they can be computed in any instantaneous encoder (and can then optionally be transmitted in the TSIG signal in the form of metadata) can typically be pre-computed in a design laboratory of e.g. a classification software manufacturer and hence in an agreed way stored in a memory at least of the encoder (in principle the decoder does not need this information, but it may also have it, e.g. in case functions like Fi(MP_T) are defined as relations to that reference function, but that typically won't be the case, for the sake of simplicity). The encoder may, if it has a few variants, choose one depending on how the final Im_LDR and REC_HDR are to be rendered, and this may happen with some human classifier software selections. Function 901 specifies how many codes are needed for each luminance range. For example, in the example it was decided that only three codes from the 8-bit Y_LDR_min (the minimum number of codes needed, given an allowable reconstruction or a representation error), i.e. actually from Im_LDR, will be used for all HDR luminances darker than L1. So these dark regions will be roughly quantized but still have some structure. If these regions were brightened, severe posterization of the local image objects could occur (either in a reconstructed rendering in HDR, or a modified rendering in LDR via a brightening transform on the display device), but it may have been a final decision to be able to encode enough HDR images into this 8-bit code (if the set of encodable images contains critical images in multiple high-contrast regimes, it typically may have to sacrifice at least some quality). However, many times the dark regions will be rendered so dark that you can't see much detail in the reflections of the viewing environment on the display device's faceplate anyway. To be able to render with reasonable fidelity (given the sensitive human eye when the reconstructed REC_HDR is shown in a dark viewing environment) the dark colors in the range up to L2, the curve prescribes that C2-3 luminance codes at minimum are needed (more codes can be used of course). Defining this curve means that one can encode HDR images up to a maximum LDR of luminance Cmax equal to, say 255 (if the LDR container has 8 bits available for its luminance; note that this can simply be seen as the gamma diamond up to 1.0 being fixed, and quantized with equidistant bins, but the distribution of pixels in the image varies depending on the transformations applied, HDR images for example often have a large percentage of pixels below 0.1 ), which corresponds, if this 8-bit container is actually encoding an HDR image, to a maximum L_HDR luminance of, say, 10,000 nits, depending on the curve. Note that the curve can be adjusted to take into account the relative scaled nature of the rendered luminances, and in the present description it is meant that one can represent HDR luminances of the reference display device between e.g. L1/10 (as a reasonable black still quantifiable at 0 luminance) and 10,000 nits, but one can of course always adapt by scaling so that other luminances are represented. For the sake of simplicity, it can be safely assumed that both L_HDR and Y_HDR have axes between 0.0 and 1.0, and are therefore quantized with some precision.

[071] One skilled in the art will understand that this reference encoding function 901 can be specified by various factors. For example, when the human classifier scribbles over an image region (e.g., M_HDR), which may be, for example, a face region, the image analysis unit 213 can, from this, determine a range of HDR luminances in which the colors of that face fall. She can then respecify the curve so that more luminances are needed to represent it. She can know that, for example if the classifier presses a “face” indication button, how many codes (i.e. quantization bins) are typically needed for any given situation (i.e. a face not as well lit as per the 36% specification, but which for example falls into a darker shadow area of the scene, making its average luminance e.g. 4000 to 5000 nits). This would lead to a prescription of a number of bins (typically on the HDR axis, but could also be on the Y_LDR axis) around the face's current average color point. This can be automatic (eg doubling or modifying the amount of JNDs for reference rendering in HDR covered by this range), or the classifier can directly influence or specify the format of function 901 in that region. The curve can be specified or re-specified (if necessary) in many properties, for example HDR_image measurements. Now regarding the actual luminance mapping curve 902 of the classification that produces Im_LDR (i.e. the classifier actually not considering anything about quantization and data loss and the like, just artistically defining where it wants the colors of its objects to be in the normalized range for this image to get, say, a dreamlike look, with lots of high brightnesses called high keys), it turns out that in the interval [L3,L4] the amount of codes that actually occur, given this mapping curve from the HD classification R principal for the chosen LDR appearance, is less than the minimum amount required NC for that region (one shifts the curve to get a clear overlap, but of course the determination of luminances used can simply be done for any curve). Note that luminances are assumed to be determined equidistantly across bands, but similarly non-linearities can be taken into account, for example by focusing on (in)permissible mappings for single luminances. Normally, however, on the Y_LDR axis there are equidistant bins, so it can be discussed in this sense without losing the generality of the teaching.

[072] Therefore, currently too few codes are used in that range (which could show up as low contrast in the LDR rendering, but probably won't, as the classifier has just optimized this curve, but will typically show up as a rough quantization for the reconstructed HDR images), necessitating stretching the local slope of the 902 curve. There are several ways to do this, for example with elastic error functions which penalize quantization errors over any range. In general, one can do any math taking into account the size of the particular intervals on the one hand and the average luminance/luminance positions of those intervals on the other hand, ie how much the curve deviates from its intended classification. Of course, if the technical classification calls for it, a specific mapping curve shape is required that is far removed from the shape that the classifier wanted for the appearance of the LDR (i.e., the technical requirements of HDR data or the code allocation purpose of the mapping function being too far from the requirements of “LDR” or the appearance purpose of the rendering of the mapping), so the classifier will continue to define its appearance in an additional way, through additional mapping functions. So in principle no strict specification or critical mathematics is needed for this method, but it is clear that some methods will be less complex in terms of computation, or easier to use in terms of how quickly the classifier arrives at the desired coding+sorting result (as the classifier time is expensive, at least for some types of programs).

[073] A simple curve transformation is clarified in Figure 10. If Nc1 is the current amount of luminances allocated to the interval, and Nc2 is the required amount of codes (either minimally required, or somewhat larger), one can stretch that part of the curve, for example by multiplying around the midpoint with Nc2/Nc1. The remainder of the curve needs to be modified, and it is assumed that redistributing the quantization to fit the total amount of codes will already give a satisfactory result. One can, for example, derive the part of the curve above L4, taking this curve, shifting it with OFF_1, and changing its scale, so that the maximum still fits the maximum luminance. By doing this everywhere, you get the technical classification curve 903, which is the Fi(MP_T) color mapping curve. The luminances on the Y_LDR_min axis will then form the GT_IDR image. The system can check if there is another lane which then becomes critical and then for example balance the error between the two regions. For example, ideally Nc2 should have 10 codes, and Nc2* in another range of L_HDR (i.e. also Y_LDR) may ideally need to have 8, but if there is only room for a total of 16 codes, one can distribute the minimum quantization error as 9 codes and 7 codes. Of course, this can be weighted by factors such as the range of the two regions, semantic significance (there are face colors), etc. If necessary, the system can instruct the classifier to choose which of the two areas should be better, for example with a user interface that allows it to increase the number of Nc2 codes (i.e. the local slope) in steps, which then means fewer codes for Nc2*, which the classifier might think is visually acceptable. Of course, some modalities may work fully automatically behind the scenes when selecting the technical mapping curve and the GT_IDR classification, in which case the system may for example simply abandon fine-tuning around a preferred LDR classification curve, and immediately switch to one of the predefined technical curves that works well (e.g. one that has the least strain, compared to the curve that defines the appearance of the LDR (902), as calculated as SUM(wi*[FT(L_HDR)-FL(L_HDR)), where the brackets indicate some function as an absolute or squared value, where FT is the currently selected technical function per L_HDR value, and FL is the LDR preferred appearance classification mapping, and the weights wi may be uniform, but also weigh more in certain areas of L_HDR, for example where faces reside), in which case the appearance is then defined by the additional mapping function Fi(MP_DL). identifies the amount of codes available in the LDR representation versus the codes needed in the HDR region. Note that even changing an interval does in fact distribute an error over the entire range compared to what the classifier would like to see, but this need not necessarily be a large error as it is relatively distributed and seen, as the eye is not really designed to function as an absolute luminance meter anyway, and anyway this can be, in most practical systems, recalculated on the receiving end with Fi(MP_DL). One skilled in the art understands that there may be many other ways to perform similar functions. For example, there may be enough codes (because the M_HDR image does not reach L_HDR_MAX, corresponding to Y_LDR_MAX), and one may have a lot of freedom to re-specify at least some parts of function 903, but still curve 902 was still too quantized in the range [L3,L4], and needed to be corrected. In this type of scenario, you can more freely shift the midpoint luminance of the [L3,L4] range, and the other curve regions. These scenarios correspond to increasing the brightness of some objects, etc. The other scenario is where the system is really critical, and redistributing the quantization outside the [L3,L4] range can lead to inadequate quantization in the system. In that case, mitigation strategies can be used to determine a final curve. An example of a mitigation strategy is to divide the remaining errors over the most critical ranges, such as [L3,L4] and [Lx,Ly], outside of where the largest quantization error occurs for the current classification curve, or whatever curve, trying to maintain a reasonably close approximation of the current human classifier curve. You can also decide to heavily allocate the errors to some regions. For example, one could clip the luminances even to something above L1 for the only luminance value 0, or one could decide to clip on the bright end, even in the technical classification GT_IDR. The REC_HDR image is therefore not perfectly rebuildable, but these scenarios can be used on systems that have an overflow fix. For example, clipped values can be encoded into a second image, separate from GT_IDR, which contains only the data for a bright region and clipped to GT_IDR. By comparing the current mapping curve with one that has good technical properties (characterized by having at least a minimum amount of codes per range) it is clear that the automatic classification unit will check if there are actually any pixel colors in that range, otherwise this can seriously distort the function in that range.

[074] Figure 12 (Figure 12b) shows an example of how a classifier can influence the technical curve that allocates the codes used for the GT_IDR technical classification. As stated, all of this can, in some embodiments, take place behind the scenes without the classifier knowing, but an example is given here of how a classifier can specify or influence the amount of codes allocated to a specific region in the luminance range in HDR 1210 which should be presumed to contain facial colors. Suppose that, in this example, the automatic allocation of code zones was very good (possibly bearing in mind that one only has 8 bits of luminance instead of 10 bits available, or perhaps for 10 bits a less suitable color space, which introduces very severe quantization at least for some colors, for example saturated blue), but the classifier looking at its quality reference display device still wants a little more precision, for example to have a less blurred face. He may then find the local slope in the luminance range 1210 to be too low, and may want to increase it through the user interface slope changing means 1203, which may be for example an arrow that increases when the top arrow is clicked, increases the slope by X%, or a draggable cursor, etc. He can specify range 1210 directly in his curve tool viewport, and drag one or more boundary markers 1202. The user interface can also aid in quick selection, for example by allowing one to draw a squiggle 1215 over the currently sorted representative image from a scene (see Figure 12a).

[075] All this time, the classifier is looking at the reconstructed HDR image rendering. If he now wants to work on the LDR image again, he will switch to that view and further specify his LDR classification, again starting from that technical curve, for an additional curve or mapping strategy. Motion tracking means for tracking the face and fine-tuning its properties if it moves under varying lighting can help in the determination if needed, but in general such complexity is not necessary for the present invention, as the technical curve is only intended to be broadly good, and not ultimately specific. In any case, however, the classifier can be offered to fine-tune any moment in the film that he considers interesting, both the technical curve and the mapping curve to obtain optimal LDR images. The software can now be configured to change the slope compared to the midpoint (curve 1204). However, the classifier may see this as introducing issues regarding color classification, which it may want to resolve now (rather than in the second mapping functions in LDR). For example, when the algorithm or hardware calculates the new curve, it will, in simpler versions, reallocate the error, which can be done, for example, by stretching the remaining shape of the curve to a maximum value of 1.0, starting from the new high point of the locally stretched range. The classifier, however, may consider that this results in overly bright colors in regions 1205. Therefore, the software may have means for position adjustment 1206, which allows the classifier to shift the local curve in range 1210 a little up or down, resulting in the final reasonable curve 1207. set a lower bound to a range of bright colors that can be further quantized when needed. If it is necessary to balance the color properties, given all the technical limitations, this can be a good way to arrive at a reasonable optimum, especially if the source material was not captured perfectly in the bright areas anyway, but e.g. with somewhat pastel colors. This slider then gives eg the reference HDR luminance position above which are eg 20m of codes, distributed eg via gamma 2.2, or psychovisual curve based on JND, etc. In that case, the mathematics of the algorithm can take this into account when redistributing the errors, for example by penalizing a difference of the 20 codes in a weighted form with the remaining codes between the high point of the range 1201 and that low value of the upper set of ranges at 1201. Of course, the classifier, if it considers the problem critical enough to spend more time, can also select one or more of these ranges for fine-tuning and, for example, add a clamping resistance to the curves already determined, indicating that they cannot lose any codes, or, at most 20% of codes, or losing codes at a rate 10x lower than the current interval, etc. This provides some inertia in respecifying another range. In general, however, the classifier will not have to recode many critical regions, otherwise it can simply let the hardware make an automatic proposal.

[076] Figure 3 shows a possible modality of a coding system that follows the principles of the present invention, where the human being derives his classification from a technical classification GT_IDR. The reader will understand that the technical variants disclosed herein (eg regarding proper technical color mappings, etc.) will also be applicable to the class of embodiments of Figure 2 or other embodiments, and vice versa.

[077] A color mapping derivation unit 214 determines a suitable color mapping (for example, the luminance mapping curve, and the corresponding chromatic color coordinate handling strategy) to map M_HDR to GT_IDR. The main purpose of this is to determine a GT_IDR classification that is most suitable from a technical point of view. In particular, one needs to be able to reconstruct a REC_HDR (by applying CMAP_2 which is the inverse color mapping of Fi(MP_T)) which will be an approximation to M_HDR (according to some image deviation criteria), or at least will fit into a second predefined accuracy from M_HDR. One skilled in the art understands that there are several defined ways to measure deviations between images. For example, a widely accepted measure is the PSNR, but this is a rather simple blind measure, which can sometimes make high contributions to differences in noise, which are psychovisually barely visible, while measuring to a lesser extent some deviations from real objects. Thus, it is intended to use measures that more accurately measure what happens to the various objects, especially in a mathematical framework correlated to psychovisual principles. For example, the image analysis unit 213 can do some approximate segmentation of REC_HDR and M_HDR into segments (pseudo-objects). It can, for example, look for relatively smooth segments, and measure an amount of posterization present there. A quantity can be, for example, the amount of colors used versus the area of the smooth gradient region, which will result in a measure of accuracy that is similar to counting the run lengths of rows of pixels with the same quantized color. One can also calculate the functional correlations or cumulative differences between the original M_HDR luminance shape over space, and the ladder function in REC_HDR. One skilled in the art will understand that one can introduce (pre)semantic information into the present accuracy assessment and into the one or more choices resulting from mapping algorithms. For example, if there is only one small object, especially if it is in the background near the side of the image, the object is probably less important, and you can encode it with fewer luminance codes, leaving more codes available for other codes. One skilled in the art will understand that a total accuracy or an error (e.g. as an accuracy image) can be formed from a pre-agreed set (e.g. loaded into the automatic classification unit 303 via a software update) of measurement algorithms, which can take into account geometric properties such as size or position of a segment or object, statistical properties such as what type of texture or color the segment/object has, semantic properties such as looking at a face or the sky (with a detector face or sky), etc. Accuracy mathematics can also have special measures to analyze effects in HDR, for example an explosion can be characterized not as an absolute difference in pixel colors between REC_HDR and M_HDR, but with a relative measure based on parameters such as a difference between the average color in the fireball and the surrounding environment, a color variation in the fireball, etc. The REC_HDR will then be seen as sufficiently accurate if a measurement of it is below or above a threshold, i.e. even if the fireball is slightly less bright or contrasty in the reconstruction, as long as it has enough impact that it is still much brighter than the surrounding environment, the reconstruction is seen as a good HDR reconstruction. These variants are especially useful for systems that are more critical due to physical resource constraints, such as the amount of bits in the GT_IDR encoding. Effects in HDR can be characterized in a complex way, or just selected as regions of high brightness, for example above a relative LT threshold. One skilled in the art will also understand that, for example in a recursive strategy to stepwise arrive at the optimal color mapping Fi(MP_T), the color mapping derivation unit 214 may not simply determine its mapping based on an aggregated total accuracy, but fine-tune based on the partial accuracies. Similar to the present local adjustment example in Figures 9 and 10, unit 214 can heal a face that is represented too coarsely, because it identifies chunks in the face and then allocates more codes upon changing that part of the function. The accuracy criterion need not be met by calculating it per se. Instead, one can use a set of pre-agreed functions or ALG(CRV_i) color mapping algorithms, which are considered to reasonably meet the accuracy criteria for a specific application, from a practical point of view. Even if a selected optimal mapping curve still introduces a slightly more severe error somewhere in some inconvenient image in M_HDR, this is then considered acceptable. The determination in any of those scenarios can either be automatic within unit 303 without any manual intervention to disturb the artistic classifier, or it can be partially guided or fully determined by the classifier, for example letting that classifier select one of a number of possible mapping algorithms or curves. Typically, unit 303 will know, and the classifier will have defined, some generic parameters pertaining to the mapping and encoding situation, for example the dynamic range (e.g. CODR or CDR) of the M_HDR image, and the classifier may have selected from a menu list that it is currently classifying a "night image", etc.

[078] For example, the image analysis unit 213 may examine the M_HDR image, and find that there are two well-separated lobes in the luminance histogram. A mapping function can then be derived (at least initially) that maps those to suitable subranges of the 8-bit luminance code, taking into account that the human visual system is more sensitive to darker parts, which will therefore need a larger subrange. So, on a coarse scale, the mapping could go to eg [0,170] and [180, 255], i.e. any function that does this is a viable candidate. Within these ranges one can produce additional bending of the Fi(MP_T) mapping curves, for example by giving the faces a slightly higher number of codes (which, to be clear, need not by themselves correspond to a higher contrast on the face in the final rendering, as the display device transform can still reduce the contrast over the face, but then at least you have good texture and facial lighting accuracy).

[079] The simplest versions can be, for example, a set of parametric curves similar to gamma Y_LDR=k*L_HDR below L1 and l*power(L_HDR, gamma)+out above L1. In this case, the algorithms that determine the automatic technical mapping can, typically, evaluate what is the structure of the image in the dark regions, and determine a sufficiently well-characterized linear part in them. If there are many objects, especially with a complicated geometric structure (such as a shed containing many objects, such as wooden planks stored for later construction, metal structures, tools, etc. all stacked and mixed together in the dark), then the unit/algorithm may decide to allocate more codes to this by adjusting the first (e.g. linear) part of the gamma-like curve. Similarly, if there are actors in the dark, the system might want to characterize them with enough code, even if they are ultimately rendered too dark, and the viewer can't see much detail in the bodies anyway (but note that a viewer could always, via their remote control, apply a display device transform to add brightness, and good coding should cater for that).

[080] Similarly, the M_HDR of the image can be analyzed and segmented into a central part (for example, additional image analysis algorithms such as a motion analyzer can help determine a main action region), a bright part, and a dark part, and then a sigmoidal or three-segment curve can be determined for this, etc.

[081] Alternatively, the human classifier can be asked to select an optimal technical curve via its user interface 230. For example, it can choose the optimal curve from a number of gamma-like curves, but the skilled person understands that this could be another one among pre-agreed fixed curves, and in a recursive technical optimization the classifier could even start adjusting curves, for example by dragging the CP control points. The curves may have, for example, some internal mechanism of elasticity, preventing the classifier from choosing segments with too low an angular coefficient, or with other inappropriate characteristics, such as inversions, double allocation (which cannot be reversed like a CMAP_2), etc. Typically, the algorithm will reach a steady state (initial or transient) by itself, for example by sounding an alert if the curve becomes deformed to the point of becoming unusable, and then returning it to a similar one with good reconstruction properties. Typically, the system will then generate the REC_HDR, and allow the classifier to switch with M_HDR on its reference HDR display device, to see accuracy or errors. The system will send GT_IDR to a reference LDR display device so the classifier can check that too. This image may already be enough in some scenarios, and then the classifier no longer needs to produce a second Im_LDR, but even if it is a lower quality LDR classification, some receiving systems will still want or need to use it (e.g. due to a backwards compatible implementation in TSIG, for a legacy BD player that ignores the color mapping data and simply plays the GT_IDR; but also, e.g. medium dynamic range display device, etc.). If the classifier is satisfied, it will continue with the next image or take of images, and if not, it will change something of the curve or algorithm even more. The encoder may have ways to help you with this. For example, when scribbling within a region with larger errors, the encoder can fall back on the curve whose luminance range those colors fall within. There may even already be initial analyzes of the artifacts, and suggestions (e.g. "it is suggested to double the amount of codes" and already apply the new curve to get a second REC_HDR_2 (by colormapping unit 215), as the unit/algorithms need to do enough on their own to spare a busy artist as much as possible from this technical grading - although it is important as it is an easy way of rough grading for the final look).

[082] Now regarding the class of modalities in Figure 3, the classifier will continue to fine-tune the GT_IDR image, through the use of the color classification unit 301, to obtain its optimal appearance as the Im_LDR classification. In principle it could now liberally apply any color transformation, as IM_LDR is not used to reconstruct REC_HDR. However, in a practical system, it is useful if a limited set of color mapping algorithms is supported, which allows all or most of the color changes that a classifier might typically want to make, as its Fi(MP_DL) definition data needs to be encoded in the TSIG signal. Some signal patterns could preferably be designed with update capability, so that the latest new color mappings data can be written to the metadata (with a new type of flag, skippable by older systems). This is useful for future decoders that are easily upgradeable, such as software running on a computer to decode movies purchased from an internet movie database. For systems with a shorter turnaround, for which only at certain times (expensive relative to total system cost) will a new processing integrated circuit be designed, it is better to use an a priori agreement on a fixed set of color mappings (e.g. functions realized in the form of LUTs, etc.). Finally, all GT_IDR, Fi(MP_T) and Fi(MP_DL) data (or derivatives thereof) are formatted by formatter 220 to the specifics of one or more chosen signal formats, and sent out via some signal communication medium.

[083] Figure 4 shows one possible embodiment of a receiver system, and the skilled person will understand that there can be many such systems. For example, the picture decoder 401 may be comprised on a separate unit (such as a BD player, or STB), and may be comprised on a display device or an apparatus comprising a display device (e.g., a television, or mobile phone (note that the mobile phone, although it may not have an HDR display device, may still need to read the HDR encoding, and extract the Im_LDR rating therefrom)), a computer, etc. There may be professional systems which also comprise the encoder, for example a transcoder on the premises of a content provider, which for example creates from a first HDR encoding variant, in accordance with the present principles, an image encoding in a second variant, to be distributed for example by a pay-per-view system, etc.

[084] The picture decoder 401 comprises a picture derivation unit 403 which is arranged to build all the necessary pictures. For example, it can extract the Fi(MP_DL) color mapping data and do MPEG_HEVC decoding into GT_IDR. And then it applies the color mapping to derive REC_LDR. There is also, in this embodiment, a system configuration unit 402, which can be arranged to, for example, check what types of display devices are currently connected, or what type of storage devices need specific forms of reconstructed images (e.g., a REC_HDR, or an interpolated REC_MDR classification, etc.), and can appropriately control the image derivation unit 403 to perform the necessary processing. In this example, you could send data (for example, an HDR image already optimized for the connected display device, and/or intermediate data, for example color mapping data, which would allow the TV to further fine-tune the incoming HDR image) via a 410 wired network connection, for example an HDMI interface connection, to a backlit 2D LED (or OLED, etc.) TV. 411. High dynamic range display devices can be produced in several ways. For example, you can intersperse in the material structure of LC, which in RGB blocks much of the light, cells that transmit most of the light if driven to be fully open. Or you can have an LED laser-illuminated display device, for example in a projector where you can locally project more LEDs into a DMD IC region if suddenly excess brightness is needed, or in cinemas you can have an additional projector structure to create highlights, etc. Or, data can be sent wirelessly via an antenna 415 to an LDR display device 416, for example a tablet display device, etc. One could also symbolically show another classified image signal being supplied by formatter 407, for example an optimal average dynamic range image for a display device of, say, 1800 nits white_peak, and send to that display device, or to a storage memory for later use, or over a network to another location of the user, his mobile phone residing somewhere in the outside world, or one of his friends, etc.

[085] Figure 5 shows the components in question inside a camera 501, which obtains an image in RAW format from the image sensor 504, through a lens 502. A knowledge mechanism 520 can be configured in various ways to obtain structural, statistical and/or semantic knowledge by studying captured RAW format images, and guide the derivation of technical mapping by the color mapping derivation unit 214, as exemplified above. The camera may have its own user interface 550 (connected or remote, for example from a display device for the director and/or DOP to follow the capture, and guide via communications antenna 580) to influence technical mapping algorithms, for example changing the local contrast of some luminance range. Mapping to GT_IDR can be used to have a quick preview image, where then an HDR image encoding is sent, e.g. for final or intermediate recording (in some cases the targeted camera may already perform a sufficient classification of the LDR container, i.e. encode M_HDR and Im_LDR, but in other cases a first HDR encoding forms a basis for further fine-tuning of the classification). This exemplary camera can transmit to a receiving end, for example via a satellite communications antenna 599, or by alternative means of communication.

[086] Figure 11 presents some examples of how the various mapping methods can be performed in a three-dimensional (or N-dimensional) color space. Figure 11 schematically shows the gamma of the HDR reference display (for the M_HDR classification) and the LDR reference display (for e.g. GT_IDR or Im_LDR), on a slice with luminance on the y-axis, and one of the chromatic coordinates, specifically an S saturation on the x-axis (these can be defined e.g. as in a CIE definition, and again several options are possible, e.g. CIE_Lab space etc.). It is shown how a color defined in M_HDR, specifically Col_HDR, is mapped to its corresponding color Col_LDR from the LDR container. The upper part in Figure 11a is a gamma format according to the color mapping algorithm as described in EP12187572 (PCT/EP2013/069203) (not yet published). The principle is that first a function is defined to transform luminances, for example along the neutral axis. Then, for each color with chromatic coordinates (eg, hue h and saturation S) take the maximum possible luminance Lmax(h, S) for that chromatic color, and scale the luminance mapping function by that value. This ensures a gamma value within LDR for all colors in HDR. Figure 11b shows another possible category of color mappings. Here, just apply any transformation on Col_HDR, so that it can end up in Col_LDR1, outside the range of LDR. Then, typically, one follows with a gamma mapping algorithm, which brings the color through, for example, desaturation within the range of LDR to Col_LDR2. Instead of a two-step projection, one can also determine for each luminance what is the worst case, that is, which pixel will be farthest from the maximum saturation for that luminance in the LDR range. You can derive a DESATPR desaturation function from it, and remap all colors taking that desaturation into account. There are also other ways to determine a desaturation algorithm. A third category of color mappings will work in RGB space and so applying mapping functions on those means that colors will also remain in both gamuts. Any function can be used for color handling, for example the local functional remapping of a saturation correlate, only along those regions of the luminance axis where it is desirable, especially in a technical classification where the actual values matter less, provided that, for the classifications to be used, reasonable colors can be derived from them by means of suitable additional mapping strategies.

[087] A further elaboration on useful saturation mappings in an HDR structure will now be presented, which can be seen separately from other teachings in the present application. Brightness and luminosity are derived from the responses of human cones, which have an activation state of cone-opsin molecules, and show how much light is arriving from the various colors as a function of both the reflectivity characteristics of the object and its illumination (luminosity being an estimate of grayness compared to a reference white by the brain, analyzing all cone spatial signals from a geometrically extended complex scene image). Hue is a function of the spectral activation ratios (according to monochromatic or polychromatic activation) of the different cones, and can be estimated from the differences in these cone activations. This serves for the determination of dominant colors, for example the broadband nature of various molecules allows the identification of specific chemical content, for example the red ripeness of an apple. Under relatively easy to estimate slowly varying illuminants such as sun+skylight, the various discernible hues can serve well for many visual tasks. Saturation or purity is a measure of how excited the color channels of ganglion cells and additional parts of the visual system are compared to a neutral (gray) stimulus. That is, it is the amount of pure color (for example, a narrowband spectrum color) added to a neutral color, or vice versa. With the hues topologically ordered on a circle in the color space, a radial dimension representing a saturation was required. Painters use the principle by adding a white color to a pure color such as red, producing a sequence of shades. In nature, saturation is determined by two important principles. First, in specular/glossy media the white illuminant is strongly added to the light in the object color from deeper interactions, leading to strong desaturation, but with high saturation in non-specular directions. More importantly, saturation is related to an amount of pigment, and this can be used, for example, by an animal to judge the health of a potential mate. Saturation exists in two “variants”. First, there is brightness-dependent saturation, which can be modeled with hue or chroma, as brighter colors appear more saturated. This can be modeled in cone-shaped color spaces, where a color plane (eg uv) becomes progressively wider along the brightness axis. The human brain can, again, discount lighting and judge how intrinsically saturated an object is, with monochromatic reflections being the theoretically more saturated situation. This can be modeled in cylindrical spaces, where the shape of the color plane remains the same along the brightness axis.

[088] Physically, this cone or cylinder could be extended towards infinity, as it can produce brighter and brighter colors, but technologically this doesn't make much sense, as any real recording or reproduction system has limits. Already the cones of the eye, in a certain state of adaptation (a quantity of cone-opsin prepared in the cone, and intermediate molecules being in a state to multiply any sense of activation of the cone, until finally a "digital" signal is obtained from a number of pulses along the neurons) will eventually discolor, so that so many molecules of cone-opsin have been activated that accurate color detection is no longer possible for some time, which occurs when looking into a bright lamp. Something similar happens, for example, with a photographic record (say, a slide). At a certain point, some maximum white needs to be registered (and then reproduced), and the luminances of scene objects above that will be clipped to the white peak. The same goes for any additive RGB space, whether it's just a coding space that can be referenced to a reference monitor to make it absolute, or a real trigger signal space for a real display device. These spaces can be topologically equated to double-cone spaces. For example, painters know that they can produce shadows with lowered chroma but the same saturation by adding black to pure colors and producing tonalities towards pure white at the top of the upper cone. That is, at the top of this type of space there can only be unsaturated colors (zero chroma), which is inconvenient compared to other colors that may exist in nature, for example those in a wider range of, for example, a display device with a higher dynamic range. For example, what to do with a color that has been dimmed in terms of brightness to the LDR (lower dynamic range) but still resides in the upper cone? Does your saturation change heavily, or maybe even lower? What if that color is only in an in-between space that serves to still map for reinforcement to a larger space, again?

[089] Thus, for these situations, in addition to theoretical saturations of any color, it may be necessary to look at saturation and saturation changes in any limited space of permissible colors. Having any math transforms within this type of space (typically cylinder-shaped), especially useful when mapping between spaces that can at least be widely placed (such as a scaled [0,1] RGB HDR space into an LDR RGB space) has the advantage of producing existing colors, rather than transformations that run out of space and have yet to be translated into realizable colors, but the non-linear nature of the math can distort other correlates of appearance such as lightness or hue. If it is possible to draw gamma/space with beginning and end in 3D in any format, in principle it is not necessary to worry so much about it, since any mapping strategy can be designed.

[090] One can deal with many of these complications by having a color sampler do the desired transformations, provided it has a minimum (while typically simple, one must also consider that these transforms typically indicate mappings needed for sufficiently faithful renderings, or at least improved compared to blind rendering, by determining the corresponding colors dependent on the rendering situation for various display devices, i.e. the hardware ICs or software on those display devices or connected video processing boxes, should preferably only use simple mathematical functions , with the complexity being handled by sampling all possible color rendering scenarios to be related by the classifier, which defines some important classification situations between which it can then be interpolated to other intermediate rendering situations) set of mathematical functions to determine saturation that it can specify.

[091] It is a known fact that mapping between different dynamic ranges can lead to colors that are either too pastel, or too saturated like a comic book, and the situation can be complex with some classifiers potentially having critical desires (for example, one might be critical about faces, but also about the shades of blue in water, or even the color appearance of dark clouds).

[092] The present innovative saturation processing can be used not only in technical classifications, but in fact on any classified image (e.g. prime HDR, or an LDR classification; to obtain any other image classification, with a different or similar dynamic range; i.e., looking great when rendered, for example on a 2000 nit HDR display device), and even on camera captures in raw format, whether inputted into another device such as a classification computer, or even still in the camera. To describe the rendering in principle one does not need the input color space (which can be the same as the output space, or anything else, for example a larger space), so it will be described with the output space of any color encoding (either as an intermediary, or device dependent directly usable for rendering). The principle will be described with a Luv space of the cylindrical type, that is, the plane directions of which only the u axis (red-green) is shown in Figure 15a, form triangles of the same size along the normalized luminance axis L, until the tent starts to shrink towards white. Of course other possibilities can be similarly implemented, and instead of a physical luminance one can use a more psychological quantity, for example a luminosity as the third axis. The gamut of all actually achievable colors is 1501. We can now apply a mathematical transformation that moves colors (whether in or out of gamut) in a direction of increasing or decreasing saturation, which is represented by curve 1503. While this illustrates the mathematical principle, Figure 15a can typically also be the view of the user interface that a color sampler sees in one of its subwindows, a main window obviously showing the effect on the color appearance of transformations on an image to be classified or reclassified. In principle, any mathematics can be used for saturation, but preferably it will be a function that largely decouples the coordinates, that is, it has a major effect on saturation, with little or no change in hue, luminance or lightness. In practice (although space is obviously just a simplistic model of what color actually looks like, in the end there may still be some visible side effects on non-saturation aspects of colors) the math can be orthogonal math, so while a generic variant is shown with a slightly bent saturation change curve (that is, also slightly lightening the colors by saturating them), often this will just be a line in the plane orthogonal to the L-axis. saturations of objects or regions, giving a total appearance to an image, the classifier now has the possibility of not only defining a global saturation multiplier, but also a multiplier that depends on the luminance of the colors to be processed. This a_s=f(L) function can be registered as a parametric function or a lookup table. The relevant luminance Li, which defines which colors should be selected for processing, is determined by the achromatic color in the curve 1503. Now, the only thing needed is some reference saturation level S_ref (1502), which could be equated to a normalized value 1. It is assumed, in this exemplary embodiment, that the saturation is defined as the Euclidean length, that is, sqrt(u*u+v*v) and, for example in a Lab space which would be sqrt(a*a+b*b), but of course other definitions would be possible. A practical choice for this reference level would be to place it at the position (u,v) of the most saturated of the three (R,G,B) or more primary colors that define the color space. Now, a quick and simple way, and in general accurate enough, to define the saturation curve a_s=f(L) would consist in the determination, by the classifier, of sample points for a certain number of luminances (level 1504 etc.) on the luminance axis. He marks them with 1505 points. The position of these points determines the saturation, and whether it consists of a boost or a cut. The Euclidean distance from point 1505 to the L axis is compared to the distance of the reference cylindrical sleeve, S_ref, and is for example 0.3 with S_ref=1, so this means that all colors with that luminance should be faded by multiplying their saturation by 0.3 (note that multiplicative operations should be sufficient for processing the saturation, although other functions can also be similarly used, of course). In the darkest region, point 1513 specifies a saturation boost for those reds.

[093] So let's say, for example that - no matter how the input image was generated, for example by submapping from a main image in HDR - if the classifier considers that the lighter colors have sufficient quality, but the darker colors would benefit from a boost in saturation, it can determine a luminance level (for example, 0.25) and, for that position, specify a point at, say, 1.8. To save time, the algorithm will determine a full curve spanning the entire luminance range from 0.0 to 1.0 from that, for example you can apply a linear interpolation which is 1.8 off for blacks, and apply a multiplier of 1.0 to colors above the L-0.25 level (of course other interpolation strategies can be used by the software, for example streaks, and the classifier can add additional points if you want further fine-tuning of the color appearance). While not necessary, it may be advantageous if the classifier also observes the volume of colors actually occurring in image 1506. In Figure 1506, only the initial situation is shown before saturation processing, but typically the final situation (or a continuously changing volume) will also be shown, which in addition to observing the actual processed image (intermediate or output), gives the classifier an idea of where colors move closer to the gamma limit and clipping or smooth clipping may occur (there may be a built-in strategy to change not linearly the multiplier when within a certain region in relation to the range limit; and these options for defining behavior will typically be turned on or off in the software by the classifier, before starting its classification). While these luminance-only dependencies are sufficient for many situations, it can be advantageous if the classifier can define different behaviors for different hues. For example, he might specify 4 LUTs for 4 hue sectors. As explained above, the same boost or decrease in saturation would apply to the red and green directions from the L-axis but, as seen, the color volume 1506 may be closer to the gamma boundary in the green direction than in the red direction, for example because the current shot of the movie, or the current still image, is of a forest scene (and, in an earlier rank saturation may have been adjusted upwards, to emulate a sunny look in an LDR encode). Then, the classifier can demarcate hue sectors, and specify saturation processing in a similar way as described above. A more complex example is presented, too, where a single multiplicative behavior is not sufficient for at least one luminance level (and perhaps hue sector) in the image. For example, dark reds can be intensified to make a Ferrari parked in a darker part of the image (say, a garage) appear prettier, but when these colors also occur in faces, the faces can become very reddish. To this end, the classifier may define a second saturation reference S_ref2 (1510), which will now typically also serve as a color region demarcation determining which “face” colors will be processed. Compared to this level, pentagon 1511 now shows that the saturations there should be dimmed by, say, 0.75. Figure 15b then shows how this behavior will then modify the saturation of the colors with luminances similar to those of the L level corresponding to pentagon 1511. In many situations, a discontinuous behavior can be sufficient, as the face can occupy a part of the color space and, therefore, there can be no other colors until the Ferrari, but the smoothing of transitions 1520 can, also, be applied, either automatically by the software, or adjusted in this type of graphic in a subwindow , by the classifier. Also in the other directions, at least in luminance and, if necessary, also in hue, the classifier can determine in which range this behavior should be applied, for example a higher luminance level 1512 (and similarly a lower luminance level could be specified). Outside this range, saturation processing can be discontinuously transitioned to the other specified behavior, or it can happen more gradually if necessary.

[094] While this processing can in principle be applied to any situation of processing the saturation of any image, it is particularly useful when switching between classifications for rendering scenarios with different dynamic ranges (i.e., determining an optimally classified encoding suitable for driving a 4000-nit HDR display device in a dim viewing environment, based on an LDR encoding, or vice versa). The HDR space can then be normalized to the same range [0.0, 1.0] as the LDR space, although this is not necessary. If this is done in a tweakable scenario (where the ratings are set to be able to render well under various rendering scenarios, typically the display device's white_peak and the surrounding environment, where those ratings actually constitute a content creator-approved sampling of how the scene should look under various situations, avoiding the problem of modeling the appearance of complex colors and converting the same to simple interpolation between representative ratings), the processing will typically be co-coded in the form of metadata to an encoding of the input image, for any rendering system to apply it accordingly (for example, if a TV has a brightness intermediate to the two ratings, for example the original being an LDR rating of 100 or 500 nits, and the saturation processing being part of a mapping strategy to obtain a rating of 4000 nits, a display device of 2000 nits may decide to apply, for example, half the amount of suggested boosting, or determine a non-linear strategy starting from the co-coded behavior information of saturation).

[095] Figure 16a shows an exemplary embodiment of a classification apparatus 1600, arranged in such a way as to be able to apply saturation processing to an input image Im_i (say, for example, an LDR classification that needs to be converted into an image in average dynamic range, MDR, for a display device of 1,200 nits; where the classifier has (at least) this display device of 1,200 nits 160 2 connected to see the result of your specifications) and further encoding also the specification into an S_o video signal, which typically encodes the video pixels according to a standard, such as an MPEG standard, and the saturation processing works as metadata for it, for example in parts of the signal, or separate transport packets that can be associated with the video by means such as a PMT and a presentation time, or other means to define a specific image number in the video to which the processing corresponds (e.g. all images until the time presentation of the next saturation processing function data). The classification apparatus comprises at least one saturation processing unit 1601 which is arranged to apply the saturation change to an input image according to any of the methods explained above. As output, it can produce an Im_o output image (eg with enhanced saturation), but also a P_s encoding of the processing function, for example a LUT a=ai(Li). An encoder 1610 will format this to the requirements of an agreed (current or future) standardized video signal encoding. It may be advantageous to facilitate user interaction if there is an image analysis unit 1603. This unit will at least examine the definition of how the image is encoded, for example to determine the triangular points of R,G and B of the gamma 1501, but may also generate, for example the volume 1506. for example, defines the hue limits for a specific processing). Thus, this will allow, based on information entered by the usr_inp user (for example, from a keyboard or a special classification keyboard), for example, to place points indicating the amount of reinforcement or decrease of saturation.

[096] Any receiving apparatus, for example a video processing apparatus 1650, can receive this type of S_o encoded signal, and apply the specified saturation processing, either directly, or deriving its own optimal saturation processing based thereon. The video processing apparatus comprises at least one saturation processing unit 1651 arranged to apply such luminance-dependent saturation strategy as described above to the input image Im_i. This input image can be obtained in a number of ways, but typically the video processing apparatus 1650 may comprise a decoder 1653, arranged to perform e.g. video decoding in AVC or HEVC to obtain a pixelated color image Im_i, and decoding metadata from the saturation processing functions, converting it into an internally usable format (for example, this information could be encoded in various ways, such as run-length encoding, or the decoder could want to convert the spec to one with a different precision, etc.). In general, the saturation processing will form part of an overall color processing/mapping performed by a color processing unit 1652, which may also map the luminances of the colors of Im_1 to new values (e.g., if the input image is an HDR image encoded at [0.0-1.0], the darkest parts may be too dark to use for LDR rendering, and may need to be brightened either before (preferably) or after the saturation processing). The video processing apparatus 1650 produces an output image Im_o, which may for example be directly fitted to a specific display device (there may of course be further conversion, such as to take into account aspects of the display device such as its internal EOTF, but this is not important for the present discussion), or the Im_o may be output for another use, for example for storage in a memory such as a blu-ray disk, or on a video server etc. Such video processing apparatus 1650 may, for example, be incorporated into a television, computer or converter, or a professional apparatus, for example a digital cinema video manipulator for use in cinemas, or a computer system of an image analysis department, etc.

[097] For further clarification, two examples of possible use in a tunable scenario are offered in Figure 17. In Figure 17a, we want to derive colors in LDR from a main HDR classification, according to classifier criteria such as good local contrast, simulated appearance of light sources or bright areas, etc. For tone mapping in the luminance direction, it is assumed to use a (u,v) chromaticity preserving mapping, but you do not want to scale everything up to the maximum of the output gamma, as in EP12187572. This presents the risk that some colors will fall outside the G_LDR output range, even if all luminances are placed within the range surmounted by L_LDRm. The classifier can solve this technical problem by artistically optimal balance of brightness versus saturation by, before submapping the luminance, doing a saturation decrease in the HDR input color space (arrow 1701). Figure 17b offers another example, this time with an intermediate encoding. What you see is the input and output space (and range) being defined in a similarly normalized and hence collocated manner. You have an intermediate image encoding of an HDR image (i.e. an image with enough luminance information to be used in HDR rendering), which has however been encoded (tweaked) somewhat to also still be reasonably renderable on a display device with lower dynamic range (either directly or with some final color mapping optimization, typically implemented by a color mapping unit on the display device side, for example inside the display device). This means that, for example, an outdoor bright region has been encoded with luminance values at the point where the luminance mapping arrow TM_L2H starts. The compensation consisted of giving some brightness boost to these regions of the image and their colors (when used, for example directly in a lower dynamic range rendering), and then the saturation needed to be reduced due to the mathematical format of the gamma. For HDR rendering, you want these regions to be bright, but not close to the maximum of gamma, as these luminances are reserved for lamps and explosions, i.e. in the G_HDR range of the normalized HDR output, you need to transform the colors, giving the same lower luminances (in L_os luminance regions). Now those colors look paler than they should (could) be, so the classifier will co-specify a saturation boost to get the final rendering, but for those luminances (at least) as other regions of the color space might be fine.

[098] Typically, the above will be realized in the form of various embodiments of an image color classification apparatus (1600), which comprises: - an input (240) for a color input image (Im_i), and - user interaction unit (1605) arranged to allow a color classifier to specify a saturation processing strategy comprising at least a first saturation change factor for a first range of color luminances to be processed, and a second different saturation change factor for a second luminance range of the colors to be processed, wherein the first and second saturation change factors are preferably multiplicative.

[099] The at least one characterizing factor for the luminance-dependent saturation change could consist of several, for example a coefficient could specify a parabolic or sigmoidal saturation change behavior along at least one line of constant (or nearly constant) luminance (e.g. sigmoidal behavior in a plot as in Figure 15b might start with a small decrease in saturation and then a sigmoidal increase to a large boost to higher values, up to some maximum which would start by clipping a considerable amount of highly saturated colors in the input image, but of course an additional parameter could be coded for that luminance level, to again decrease the saturation boost in those areas to 1.0 or even below, to make those colors fit better in the available gamut), but in many situations a multiplicative factor changing an input saturation s_in into an output saturation s_out=a*s_in will have enough control complexity and visual precision .

[0100] Although some modalities may only specify for one or a small region of luminances a characterizing factor of saturation processing (the rest of the colors, for example staying in the default of remaining the same, which would be identical to multiplying by a factor of 1.0), it may be advantageous to specify factors for the entire range of possible color luminance in the input image (for example, 0.0-1.0) or some other range of luminance, from which some colors may have defined saturation processing , even if they do not occur in the input image. This can be done either by actually specifying them (e.g. the algorithm creating a continuous interpolation and the classifier accepting or correcting this), which can be co-coded into the image signal S_o, e.g. in the form of a LUT of sufficient precision (which could still be further interpolated on a receiving side), but it is sufficient if the processing required for each color possible with the luminance Li is derivable, i.e. the metadata specifying the saturation processing strategy for a receiver it can only understand the functional parameters, or the positions of the points like 1505 etc.

[0101] If more precision is needed, it may be advantageous if that user interaction unit (1605) allows specification of saturation processing based on additional properties of colors in the color space, for example a hue dependency, for example s_out=fi(L, h_i), where there is a set of h_i's being centroid hues for the hue sectors, and a color (u,v) is processed by a saturation mapping based on the closest proximity to all those centroid hues, or another hue-dependent definition s_out=fi(L, f_hi()), where f_hi() is some algorithmic function or strategy mapping the hue of an input color to some coefficient i, which defines a specific saturation processing strategy. Similarly, there can be multiple strategies for different saturation subranges of at least one luminance range (at or around Li), and one can treat complementary hues as if they had negative saturation. This can be mathematically defined, for example s_out=fi(L, f_si()), where there is now a categorical allocation (eg boolean if two regions are involved) based on the color saturation (u,v) of the input image to be processed. While this accuracy is generally sufficient, you can generally define strategies that differ based on both the hue and saturation of colors in selected luminance subbands.

[0102] Thus, a method has been described for specifying a saturation processing strategy for an input image (Im_i), which comprises specifying at least a first saturation change factor for a first luminance range of colors of the input image to be processed, and a second different saturation change factor for a second luminance range of other colors of the input image to be processed, and preferably comprising an encoding of this strategy in the form of metadata associated with the input image, and variants thereof.

[0103] An apparatus complementary thereto will be a video processing apparatus (1650), comprising; - an input for an input image (Im_i), and - a saturation processing unit (1651), arranged so as to apply a first saturation change to the input image colors falling within a first luminance range, and a second different saturation change to the input image colors falling into a second luminance range, wherein the video processing apparatus comprises means for obtaining first and second saturation change factors characterizing the first and respectively the second saturation change factors, wherein said means preferably comprise a decoder for decoding the first and a second saturation change factors from the metadata in an image signal (S_o). While this apparatus may form part of a single location or single use system, typically a classifier or reclassifier for existing content will specify the classifications once and then, on a later and different occasion, the use of those classifications will be by the video processing apparatus. This may for example typically be a consumer appliance. The consumer may have purchased a movie over the internet, which he watched for example 5 years ago on his LDR display device. Now, still having the rights to watch the content, he indicates to the management module on the server that he has purchased an HDR display device, and that he wants to receive the metadata for the video program images, specifying among other things this saturation. Of course, the user can also buy the video encoding processing metadata (pixelized image colors) + the colors (saturation) on a single memory product, e.g. a blu-ray disc, a solid state memory stick or pre-installed, e.g. on a video player device such as a portable player, etc.

[0104] This corresponds to a video processing method that comprises applying a first saturation change to colors of an input image that fall within a first luminance range, and a second different saturation change to colors of the input image that fall into a second luminance range, and the various modalities thereof in accordance with the clarifying principles explained above.

[0105] Figure 13 shows an example of how different LDR images can be obtained for rendering. In this example, a smooth curve 1301 was chosen for technical classification, which allows recovering all luminance ranges of the original main HDR image to be encoded (whatever range it may have had) with reasonable accuracy. When this is saved to the technically classified GT_IDR LDR image, a “dumb” legacy system will render, although the image is recognizable, a rather soft image on an LDR display device, with contrast not preferred in key regions such as the actor. Any such system could use automatic processing to increase this contrast, or otherwise try to optimize the image, but it would have to do so blindly. It would be much better if the content provider could encode what the receiving side can do to get a better LDR rating than the technical rating. The data needed to specify this second tone mapping from the technically classified GT_IDR could be as simple as specifying two thresholds, gt_Mh and gt_Ml, which indicate where key information resides in the code space, and which other colors can be (seriously) spoiled at the expense of others. The receive-side system then only needs to stretch the luminances taking these important values into account. This has been shown in a graph 1302 which, when applied directly to the display device (with known calibrated properties, for example default gamma and viewing environment behavior) results in luminances rendered in x as per that graph. In this example, the colormapper on the receiving end decided to mostly stretch the blacks, still retaining some of the posterized HDR information however this can be rendered under the given surrounding environment, and decided to use a hard clipping strategy, mapping gt_Mh to white (i.e. setting the same as the white in LDR in the full color space in HDR). All of the above colors may therefore not render on this display device with eg 700 nits. Of course, more complex specifications can be co-coded on what a receive-side color mapping should do with the received GT_IDR encoded image, to achieve optimal visual appearance on one or more intended display devices (e.g. 700 nits does X processing, 1500 nits does Y processing), and all of this can be defined in color mapping functions and applied based on the received GT_IDR image (e.g. additional characteristic gray values can help with further parametrically specified enhancement of the LDR rating to be obtained, or 1 explicit mapping strategy can be specified for the whole range, per receiving display device category, and this can be done, for example with a LUT of boosting factors [between 1/X and Y] per luminance value). Thus, simple decoding systems will render a reasonable LDR image, and decoders capable of handling all present possibilities will produce great images in LDR or HDR, or any MDR (medium dynamic range), or ODR (typical external dynamic range, such as subLDR with extremely low contrast). While the present framework allows for exact ratings to be specified for N LDR (and other) viewing scenarios (e.g. 100 nits TV and 500 nits under dark, dim, and bright viewing scenarios = 6 ratings), it is clear that it is not always necessary to render an optimal rating, but a good quality picture will do in some scenarios as well. This has been illustrated with Figure 13, as a mere example. Suppose you have a news program with HDR lighting that is supposed to look pretty good, but since LDR is an approximation it will only look good, and the classifier needs to be able to set your system up in a couple of seconds before starting the news program in the studio. To do this, you can define two additional demarcation thresholds, gt_H2 and gt_L2, so that the receiving side can decide how to color-map the GT_IDR to get its trigger image from the display device. For example, it can be defined (by co-coding these values into specifically reserved codes such as MINIMAL_LDR_Low and MINIMAL_LDR_High, or BROAD_LDR_Low and BROAD_LDR_High, or even more LDR subrange delimiters) that gt_ML and gt_Mh are the “definitive” LDR subrange delimiters of the HDR scene's main action, which still contains some of the HDR information (like some already brighter parts in the studio), and gt_L2 and gt_L2 contain the “absolute minimum” required for LDR rendering (e.g. no (hard) clipping of highlights on faces). Color mapping on the receiving side can then select its strategy for producing an LDR image. For example, it can define a proprietary strategy of smooth clipping in the regions between gt_ML and gt_L2, and gt_H2 and gt_H2, after having defined a stretching strategy for the middle range of colors with good rendering absolutely necessary between gt_L2 and gt_H2 (for example, mapping these to values 20 and 220). But if the receiving system decides to do a hard stretch mapping of the range [gt_L2, gt_H2] to [0.255] and outer clipping, the LDR rendering will also look reasonable. The receiving side could decide to choose an option, for example based on the amount of surrounding lighting available. Thus, it is observed that the system allows many possibilities, from rigorously controlled complex classification definition systems, to really simple systems that have only a few co-coded guide parameters. The dynamic range appearance of a technical rating can for example be LDR or MDR (ie look good on a reference display device of eg 1200 nits). But the principle is always to decouple technical requirements (such as reversibility, which is handled in the HDR-GT_IDR relation) from artistic freedom (doing arbitrary recoloring of all image objects in LDR as much as desired by the classifier from GT-IDR, and a mapping function as complex as necessary, though typically with a certain number of supported base functions (which the decoder needs to support), e.g. luminance mapping and multi-subfunction colors (e.g. with LUTs), definition of local object segments and mapping functions for them, etc.). The user interfaces can be too simple for the classifier, for example since for many systems the precise and finely tuned position of gt_Mh, gt_H2 etc. may not be of critical importance, he can define them by quickly marking a couple of regions of the currently captured image of a scene, for example the journalist's face, the table behind which she is sitting and, if necessary, with another pen (defining the external regions, like the glows above gt_Mh) the luminous screen behind her back. Of course more information can be provided - for example with more gt feature points - for example shadows or highlights in the HDR lighting of your face, or the table, and this can all be used for more complex color mapping strategies. Furthermore, additional specifications of these regions can be made, for example a geometric function across its face, defining a contrast trajectory, and functions to redefine/remap those under various conditions (e.g. leave the end of the curve referring to the highlights, but lighten the dark parts a little), etc. All of this can be added to the metadata if needed, but in general simple systems with the minimum amount of data needed are preferred, and at least one strip of LDR can be useful (but a second one around a person's colors can also be useful in some scenarios).

[0106] Figure 14 shows an example of how technical classifications can also work with colorimetric principles. Suppose you have a mathematical color space 1401 with defined primaries, so that you can make less saturated colors than might be necessary for some anticipated (perhaps future) display devices, with physical gamma 1402. This may not be much of an issue for darker colors, as the display device can do some saturation boosting, and there may be enough information logged for this to work well (perhaps by applying a post-banding filter, if necessary). In the breadth of the gamut though, there could be a problem, and this is where one might want to have some high-gloss saturated colors rather than more pastel colors. If this is a problem, the classifier may decide to set its classification down to a new white point W* (by filling the tent, leaving a possibility to define more saturated colors close to the maximally bright colors), but then, to avoid confusion, this white point W* (being the brightest possible color according to this code definition) can be cocoded (to declare that it is not just a “no white” scene). Of course, the receiving side could also just consider what is the brightest encoding in the input image, and render with that, as anyway the visual system adapts with a gray appearing white to glossy display devices, but then an LDR system can use the same to boost some parts of the image to its maximum brightness.

[0107] The skilled person will understand that many variants are possible for the above concepts. For example, although in the specific illuminating examples in the Figures it was assumed that the colormapping data was co-coded with the image pixel data (GT_IDR), for example as metadata within reserved places defined in the image encoding standard, for example SEI messages or similar, or within a reserved section of memory, for example a DB section, it is clear that other examples may transmit the colormapping data via a communication channel other than the GT_IDR. For example, the content creator can place additional restrictions on the properties of the color mappings or the resulting GT_IDR, for example he can give it a totally different appearance than that of M_HDR and Im-LDR, or even an ugly image, and deliver the color mapping data through a secure channel upon receiver verification, or payment, etc.

[0108] The algorithmic components disclosed in this text can (wholly or in part) be realized in practice in the form of hardware (for example, parts of an application-specific integrated circuit (IC)) or in the form of software running on a special digital signal processor, or a generic processor, etc. They can be semi-automatic in the sense that at least some user-entered information may be/were present (eg factory-entered, or consumer-entered, or other human-entered information).

[0109] It should be understandable to the person skilled in the art, from the present presentation, which components can be optional enhancements and can be performed in combination with other components, and how (optional) steps of methods correspond to the respective apparatus means, and vice versa. The fact that some components are disclosed in the invention in a certain relationship (for example, in a single figure in a certain configuration) does not mean that other configurations are not possible as modalities under the same thought of the invention, as disclosed for patent, in the present document. Furthermore, the fact that, for pragmatic reasons, only a limited spectrum of examples has been described does not mean that other variants cannot fall within the scope of the claims. Indeed, the components of the present invention can be incorporated in different variants along any chain of use, for example all variants of a creation side, such as an encoder, can be similar to, correspond to, corresponding appliances on a consumption side of a decomposed system, for example a decoder, and vice versa. Various components of the embodiments can be encoded as specific signal data into one signal for transmission, or for further use as coordination, in any transmission technology between encoder and decoder, etc. The word "apparatus", in this application, is used in its broadest sense, specifically a group of means that enable the achievement of a specific objective, and consequently may, for example, be a (or a small part of an) IC, or a dedicated apparatus (such as an apparatus with a display device), or part of a networked system, etc. The terms "arrangement" or "system" are also intended to be used in their broadest sense, so they can include, among other things, a single physical device that can be purchased, a part of a device, a collection of (parts of) devices in cooperation, etc.

[0110] The computer program product denotation needs to be understood as encompassing any physical realization of a collection of commands that allow a generic or special purpose processor, after a series of loading steps (which may include intermediate conversion steps, such as translation into an intermediate language, and a final processor language) to send commands to the processor, to perform any of the characteristic functions of an invention. In particular, the computer program product can be realized in the form of data on a medium, for example a disk or tape, data present in a memory, data traveling through a network connection - wired or wireless - or a program code on paper. Except for the program code, the characteristic data needed for the program can also be incorporated in the form of a computer program product. This data can be (partially) provided in any way.

[0111] The invention or any usable data, according to any philosophy of the present modalities, such as video data, can also be incorporated in the form of signals on data carriers, which can be removable memories such as optical discs, flash memories, removable hard disks, portable devices recordable by wireless means, etc.

[0112] Some of the steps necessary for the operation of any presented method may already be present in the functionality of the processor or any apparatus embodiments of the invention, rather than described in the computer program product or in any unit, apparatus or method described herein (with details of the embodiments of the invention), such as data input and output steps, processing steps well known and typically incorporated as triggering a conventional display device, etc. The inventors also want protection for resultant and similar resultant products, for example the specific innovative signals involved in any step of the methods or any subpart of the instruments, as well as any new uses of those signals, or any related methods.

[0113] The term image signal typically means, herein, any of the existing or similar ways to compress image data. Except for a pixelated structure of color tuples, which is called an image (or picture), this type of signal can contain metadata such as descriptors for the meaning of the data, for example the aspect ratio of the image, and additional metadata containing useful information related to the encoded image, such as modifying it on a receiving side, etc. Signals can have various physical/technical forms of modalities, for example they can be defined as electrical modulations of a carrier wave, or bits represented in the form of mechanical depressions, or material changes, for example a local state of magnetization, etc.

[0114] It should be noted that the aforementioned embodiments illustrate the invention, rather than limiting it. Where the person skilled in the art can easily map the presented examples to other regions of the claims, for the sake of brevity all these options have not been mentioned in depth. Except for the combinations of elements of the present invention as combined in the claims, other combinations of the elements are possible. Any combination of elements can be realized in a single dedicated element.

[0115] Any reference sign in parentheses in the claim is not intended to limit the claim, nor is any specific symbol in the drawings. The word “comprises” does not exclude the presence of elements or aspects not mentioned in a claim. The word "a" or "an" before an element does not exclude the presence of a plurality of such elements.

Claims (15)

1. IMAGE ENCODER (202), characterized in that it comprises: - an input (240) for an input image with high dynamic range (M_HDR); - an image classification unit (201, 301) arranged to allow a human color classifier to specify a color mapping from a representation (HDR_REP) of the input image with high dynamic range, to an image with low dynamic range (Im_LDR), by means of a color mapping algorithm determined by a human being, and arranged to produce data specifying the color mapping (Fi(MP_DH), Fi(MP_DL)); and - an automatic classification unit (203, 303) arranged to derive a second low dynamic range (GT_IDR) image by applying an automatic color mapping algorithm Fi(MP_T) to the input high dynamic range (M_HDR) image, with a color mapping algorithm meeting a condition that a reconstructed HDR image (REC_HDR) that fits a second predefined accuracy from the input high dynamic range (M_HDR) image ) can be calculated by applying a second color mapping algorithm (CMAP_2), which is the inverse of the automatic color mapping algorithm Fi(MP_T), to the second image with low dynamic range (GT_IDR). 2. IMAGE ENCODER (202), according to claim 1, characterized in that it is arranged to use as the representation (HDR_REP) the input image with high dynamic range (M_HDR) or the second image with low dynamic range (GT_IDR). 3. IMAGE ENCODER (202), according to claim 1, characterized in that the image classification unit (201) and the automatic classification unit (203) are arranged to apply a monotonic mapping function on a luminance correlate of pixels in their respective input image, in at least one geometric region of the respective input image corresponding to the same geometric region of the input image with high dynamic range (M_HDR). 4. IMAGE ENCODER (202), according to claim 3, characterized in that the automatic classification unit (203) is arranged to determine the monotonic mapping function (Fi(MP_T)) of the luminance correlates of pixels in the input image with high dynamic range (M_HDR) to the luminance correlates of pixels in the second image with low dynamic range (GT_IDR), according to a criterion that determines the respective ranges of luminance correlates of pixels in the image high dynamic range (M_HDR) input images allocated to the respective unique values of a pixel luminance correlate of the second low dynamic range (GT_IDR) image, where the respective ranges form a set of luminance correlate ranges covering the entire range of possible luminance correlate values for the high dynamic range (M_HDR) input image. 5. IMAGE ENCODER (202), according to any one of claims 1 to 4, characterized in that it comprises a data formatter (220) arranged so as to produce, in an image signal (TSIG), the second image with low dynamic range (GT_IDR) and at least one of, or both of, data describing the color mapping (Fi(MP_T)) between the input image with high dynamic range (M_HDR) and the second image with low dynamic range (GT) _IDR), and data describing the color mapping (Fi(MP_DL)) between the image with low dynamic range (Im_LDR) and the second image with low dynamic range (GT_IDR). 6. IMAGE DECODER (401), characterized in that it is arranged to receive, via an image signal input (405), an image signal comprising a second image with low dynamic range (GT_IDR), and data describing a first color mapping (Fi(MP_T)) allowing the reconstruction of a reconstruction (REC_HDR) of an image with high dynamic range (M_HDR) based on the second image with low dynamic range (GT_IDR), and data describing a second color mapping (Fi(MP_DL)) that allows the calculation of a low dynamic range image (Im_LDR) based on the second low dynamic range image (GT_IDR), wherein the image decoder comprises an image derivation unit (403), arranged to derive at least the low dynamic range image (Im_LDR) based on the data describing the second color mapping (Fi(MP_DL)) and the pixel colors encoded in the second image with low dynamic range (GT_IDR). 7. IMAGE DECODER (401), according to claim 6, characterized in that it comprises a system configuration unit (402) arranged to determine whether the decoder is connected to at least one of a high dynamic range display device (411) and a low dynamic range display device (416), and wherein the system configuration unit (402) is arranged to configure the image derivation unit (403) to determine at least the reconstruction (RE C_HDR), in case of a connection to the display device with high dynamic range (411), and arranged to configure the image derivation unit (403) to determine at least the image with low dynamic range (Im_LDR), in case of a connection to the display device with low dynamic range (416). 8. IMAGE DECODER (401), according to claims 6 or 7, characterized in that it has as an output a wired connection (410) or a wireless connection (415) to any connectable display device, and a signal formatter (407) arranged to transmit at least one or both of the reconstruction (REC_HDR) and the image with low dynamic range (Im_LDR) to any connected display device. 9. IMAGE DECODER (401), according to claims 6 or 7, characterized in that the image derivation unit (403) is arranged in such a way as to determine an additional image (FURTHGR), based on the reconstruction (REC_HDR) and on the image with low dynamic range (Im_LDR), or on the second image with low dynamic range (GT_IDR) and on the data describing the first color mapping (Fi(MP_T)) and on the data describing the second color mapping (Fi(MP_DL)). 10. IMAGE DECODER (401), according to claims 6 or 7, characterized in that the image signal input (405) is connected to a reading unit (409) arranged to read the image signal from a memory object (102) such as, for example, a blu-ray disk. 11. IMAGE DECODER (401), according to claims 6 or 7, characterized in that the image signal input (405) is connectable to a network connection (408) for a source of an image signal. 12. IMAGE CODING METHOD, characterized by comprising: - accessing an input image with high dynamic range (M_HDR); - specifying, by a human color classifier, a color mapping from a representation (HDR_REP) of the input image with high dynamic range to an image with low dynamic range (Im_LDR), as a color mapping algorithm determined by a human, and producing data specifying the color mapping determined by a human (Fi(MP_DH)); and - automatically derive, by means of image processing software and/or hardware, a second low dynamic range (GT_IDR) image by applying an automatic color mapping algorithm to the input high dynamic range (M_HDR) image, with a color mapping algorithm meeting a condition that a reconstructed HDR image (REC_HDR) that falls within a second predefined accuracy from the input high dynamic range (M_HDR) image can be calculated by applying a second color mapping algorithm (CMAP_2), which is the inverse of the automatic Fi(MP_T) color mapping algorithm, to the second image with low dynamic range (GT_IDR). 13. IMAGE DECODING METHOD, characterized by comprising: - receiving an image signal comprising a second image with low dynamic range (GT_IDR), and data describing a first color mapping (Fi(MP_T)) that allows the reconstruction of a reconstruction (REC_HDR) of a high dynamic range (M_HDR) based on the second image with low dynamic range (GT_IDR), and data describing a second color mapping (Fi(MP_DL)) that allows the calculation of an image with low dynamic range (Im_LDR) based on the second image with low dynamic range (GT_IDR), and - deriving at least the image with low dynamic range (Im_LDR) based on the data describing the second color mapping (Fi(MP_DL)) and the encoded pixel colors in the second image with low dynamic range (GT_IDR) and optionally also the reconstruction (REC_HDR) based on the data describing the first color mapping (Fi( MP_T)) and the pixel colors encoded in the second image with low dynamic range (GT_IDR). 14. IMAGE SIGNAL, characterized by comprising a second image with low dynamic range (GT_IDR), and data describing a first color mapping (Fi(MP_T)) that allows the reconstruction of a reconstruction (REC_HDR) of a high dynamic range (M_HDR) based on the second image with low dynamic range (GT_IDR), and data describing a second color mapping (Fi(MP_DL)) that allows the calculation of an image with low dynamic range (Im_LDR) ) based on the second image with low dynamic range (GT_IDR). 15. MEMORY OBJECT, for example a Blu-ray disc, characterized by storing an image signal as defined in claim 14.