EP4315232A1 - Réglage de luminance basé sur un état d'adaptation de spectateur - Google Patents

Réglage de luminance basé sur un état d'adaptation de spectateur

Info

Publication number
EP4315232A1
EP4315232A1 EP22710270.4A EP22710270A EP4315232A1 EP 4315232 A1 EP4315232 A1 EP 4315232A1 EP 22710270 A EP22710270 A EP 22710270A EP 4315232 A1 EP4315232 A1 EP 4315232A1
Authority
EP
European Patent Office
Prior art keywords
pupil size
image frame
luminance value
current
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22710270.4A
Other languages
German (de)
English (en)
Inventor
Jaclyn A. PYTLARZ
Jake W. ZUENA
Per Jonas A. KLITTMARK
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dolby Laboratories Licensing Corp
Original Assignee
Dolby Laboratories Licensing Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dolby Laboratories Licensing Corp filed Critical Dolby Laboratories Licensing Corp
Publication of EP4315232A1 publication Critical patent/EP4315232A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/98—Adaptive-dynamic-range coding [ADRC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/60—Colour correction or control
    • H04N1/6083—Colour correction or control controlled by factors external to the apparatus
    • H04N1/6086—Colour correction or control controlled by factors external to the apparatus by scene illuminant, i.e. conditions at the time of picture capture, e.g. flash, optical filter used, evening, cloud, daylight, artificial lighting, white point measurement, colour temperature
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/60—Colour correction or control
    • H04N1/6083—Colour correction or control controlled by factors external to the apparatus
    • H04N1/6088—Colour correction or control controlled by factors external to the apparatus by viewing conditions, i.e. conditions at picture output
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/485—End-user interface for client configuration
    • H04N21/4854—End-user interface for client configuration for modifying image parameters, e.g. image brightness, contrast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/40—Picture signal circuits
    • H04N1/407—Control or modification of tonal gradation or of extreme levels, e.g. background level
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/60—Colour correction or control
    • H04N1/6027—Correction or control of colour gradation or colour contrast

Definitions

  • This application relates generally to systems and methods of adjusting luminance based on an adaptation state of a viewer.
  • DR dynamic range
  • HVS human visual system
  • DR may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (blacks) to brightest whites (highlights).
  • DR relates to a ‘scene -referred’ intensity.
  • DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth.
  • DR relates to a ‘display -referred’ intensity.
  • a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably.
  • high dynamic range relates to a DR breadth that spans some 14-15 orders of magnitude of the human visual system (HVS).
  • HVS human visual system
  • EDR enhanced dynamic range
  • VDR visual dynamic range
  • n ⁇ 8 e.g., color 24-bit JPEG images
  • images where n>8 may be considered images of enhanced dynamic range.
  • EDR and HDR images may also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.
  • Metadata relates to any auxiliary information that is transmitted as part of the coded bitstream and assists a decoder to render a decoded image.
  • metadata may include, but are not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as those described herein.
  • HDR lower dynamic range
  • SDR standard dynamic range
  • HDR content may be color graded and displayed on HDR displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,000 nits or more).
  • High Dynamic Range (HDR) content authoring is now becoming widespread as this technology offers more realistic and lifelike images than earlier formats.
  • many display systems including hundreds of millions of consumer television displays, are not capable of reproducing HDR images.
  • HDR content optimized on one HDR display may not be suitable for direct playback on another HDR display.
  • One approach being used to serve the overall market is to create multiple versions of new video content; say, one using HDR images, and another using SDR (standard dynamic range) images.
  • SDR standard dynamic range
  • HDR technology allows for content to be much brighter than previously provided. Brightness jumps in content, from dark to bright and bright to dark, can be an uncomfortable experience for a viewer of the content. Such brightness jumps may occur at image junctions such as channel changes or advertisement insertion, as well as used for creative effect. Accordingly, techniques for reducing such strain while maintaining an intended viewing experience of the content author have been developed. Techniques may further account for output device characteristics while maintaining the intended viewing experience.
  • Various aspects of the present disclosure relate to devices, systems, and methods for adjusting luminance based on a state of a viewer.
  • a video delivery system for luminance adjustment based upon a viewer adaptation state.
  • the video delivery system comprises a processor to perform post-production editing of video data.
  • the processor is configured to: receive a source image including a current image frame including metadata corresponding to a mean luminance value of the current image frame, and the source image including an upcoming image frame including metadata corresponding to a mean luminance value of the upcoming image frame.
  • the processor is configured to determine, for the current image frame and the upcoming image frame, an ambient luminance value based on an ambient luminance, and to determine, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the mean luminance value.
  • the processor is further configured to determine, using a model that estimates pupil size as a function of incident luminance, a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance value of the upcoming image frame, and wherein the current pupil size is determined based on the incident luminance value of the current image frame and one or more previous image frames.
  • the processor is further configured to determine a difference between the current pupil size and the target pupil size and to generate an output image by including in the source image metadata indicative of an expected change in pupil size between the current image frame and the upcoming image frame, wherein said metadata indicative of an expected change in pupil size is determined as a function of the difference between the current pupil size and the target pupil size.
  • a method for luminance adjustment based upon a viewer adaptation state comprising receiving a source image including a current image frame including metadata corresponding to a mean luminance value of the current image frame, and the source image including an upcoming image frame including metadata corresponding to a mean luminance value of the upcoming image frame, determining, for the current image frame and the upcoming image frame, an ambient luminance value based on an ambient luminance, determining, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the mean luminance value, determining, using a model that estimates a pupil size as a function of incident luminance, a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance value of the upcoming image frame, and wherein the current pupil size is determined based on the incident luminance value of the current image frame and one or more previous image frames, determining a difference between the current pupil size and the
  • a video delivery system for luminance adjustment based upon a viewer adaptation state.
  • the delivery system comprises a processor to decode a received coded bit stream.
  • the processor is configured to: receive an input image including a current image frame, and upcoming image frame, and metadata indicative of an expected change in pupil size between the current image frame and the upcoming image frame.
  • the processor is further configured to determine, for the current image frame and the upcoming image frame, a target luminance value, to determine an ambient luminance value based on an ambient luminance, and to determine, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the target luminance value.
  • the processor is further configured to select a tone mapping curve based on a characteristic of a device configured to provide the image and to determine, using a model that estimates pupil size as a function of incident luminance, a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance of the upcoming image frame, and wherein the current pupil size is determined based on the incident luminance of the current image frame and one or more previous image frames.
  • the processor is further configured to determine a difference between the current pupil size and the target pupil size, alter the tone mapping curve based on the expected change in pupil size and the difference between the current pupil size and the target pupil size, and apply the altered tone mapping curve to the input image to generate an output image.
  • a method for luminance adjustment based upon a viewer adaptation state comprises: receiving an input image including a current image frame, an upcoming image frame, and metadata indicative of an expected change in pupil size between the current image frame and the upcoming image frame and determining, for the current image frame and the upcoming image frame, a target luminance value.
  • the method further comprises determining an ambient luminance value based on an ambient luminance, determining, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the target luminance value, and selecting a tone mapping curve based on a characteristic of a device configured to provide the image.
  • the method further comprises determining, using a model that estimates pupil size as a function of incident luminance, a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance of an upcoming image frame, and wherein the current pupil size is determined based on the incident luminance of a current image frame and one or more previous image frames, altering the tone mapping curve based on the expected change in pupil size and the difference between the current pupil size and the target pupil size, and applying the altered tone mapping curve to the input image to generate an output image.
  • a non- transitory computer-readable medium storing instructions that, when executed by a processor of a video delivery system, cause the video delivery system to perform a method of the present disclosure.
  • various aspects of the present disclosure provide for the display of images having a high dynamic range and high resolution, and effect improvements in at least the technical fields of image projection, holography, signal processing, and the like.
  • FIG. 1 depicts an example process for a video delivery pipeline.
  • FIG. 2 depicts an example process for luminance adjustment based on a viewer adaptation state.
  • FIG. 3 depicts an example two-dimensional display environment.
  • FIG. 4 depicts an example one-dimensional cross-section of the display environment of
  • FIG. 5 depicts an example model of a steady-state pupil.
  • FIG. 6 depicts an example model of a pupil response to change in experienced luminance.
  • FIG. 7 depicts an example process for luminance adjustment based on received metadata.
  • FIG. 8 depicts an example graph for determining a slider value.
  • FIGS. 9A-9B depict example graphs illustrating a user preference setting.
  • This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like.
  • ASICs application specific integrated circuits
  • FPGAs field programmable gate arrays
  • Disclosed systems and methods can be used in any device in which there is a need to project light; for example, cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, and the like.
  • Disclosed systems and methods may be implemented in additional display devices, such as with an OLED display, an LCD display, a quantum dot display, or the like.
  • FIG. 1 depicts an example process of a video delivery pipeline (100) showing various stages from video capture to video content display.
  • a sequence of video frames (102) is captured or generated using image generation block (105).
  • Video frames (102) may be digitally captured (e.g. by a digital camera) or generated by a computer (e.g. using computer animation) to provide video data (107).
  • video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide video data (107).
  • a production phase (110) video data (107) is edited to provide a video production stream (112).
  • the video data of production stream (112) is then provided to a processor (or one or more processors such as a central processing unit (CPU)) at block (115) for post-production editing.
  • Block (115) post-production editing may include adjusting or modifying colors or brightness in particular areas of an image to enhance the image quality or achieve a particular appearance for the image in accordance with the video creator’s creative intent. This is sometimes called “color timing” or “color grading.” Methods described herein may be performed by the processor at block (115).
  • coding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate coded bit stream (122).
  • the coded bit stream (122) is decoded by decoding unit (130) to generate a decoded signal (132) representing an identical or close approximation of signal (117).
  • the receiver may be attached to a target display (140) which may have completely different characteristics than the reference display (125).
  • a display management block (135) may be used to map the dynamic range of decoded signal (132) to the characteristics of the target display (140) by generating display-mapped signal (137). Additional methods described herein may be performed by the decoding unit (130) or the display management block (135). Both the decoding unit (130) and the display management block (135) may include their own processor, or may be integrated into a single processing unit.
  • the viewer state of adaptation may be, for example, the speed at which the pupils of the viewer react to the change in brightness, as described in more detail below. Maintaining the creative intent in such a manner is accomplished by modeling both the content creator’s state of adaptation and the observer’s state of adaptation at any given point while watching a sequence of changing frames. Specifically, a model estimates a change in pupil diameter of a viewer based on the output luminance of the device as the video content is provided. Additional information may be further accounted for, such as ambient light, screen reflection, and chromatic adaptation.
  • FIG. 2 provides a method (200) for luminance adjustment of content based on a reference adaptation state.
  • the method (200) includes, at step (205), receiving a source image, such as the video data (107).
  • the source image may include a current image frame (e.g., the frame of the video data (107) currently being viewed).
  • the source image may also include metadata associated with the current image frame. For example, an image “Smid.” which represent the luminance level proportional to a perceived brightness of the source image, may be stored in LI metadata that describes the mean image luminance of the source image.
  • the Smid value is a measurement of the average (e.g., arithmetic, median, geometric) luminance provided by the source image when the source image is displayed on the reference display (125).
  • the Smid value may be estimated to be equivalent to the average of the Perceptual-Quantizer (PQ)-encoded maximum color component (RGB) values within the source image.
  • the Smid value may denote the average or median of a selected region (for example, a face). In instances where the LI metadata is not available, the Smid may be calculated, or may be assumed (such as at a value of 0.36).
  • the source image may further include an upcoming (e.g., future) image frame.
  • the upcoming image frame may be the frame directly subsequent to the current image frame, or an image frame several frames behind the current image frame. Accordingly, the production phase (110) may receive a current image frame, an upcoming image frame, and/or both.
  • the method (200) includes, at step (210), determining an ambient luminance value based on an ambient luminance. For example, an ambient luminance of an area surrounding the reference display (125) is determined. This may be accomplished using ambient light sensors that detect luminance values of light, color of light, and the like.
  • the ambient luminance value may be determined by communication with smart devices, such as smart light bulbs that communicate the color of light they provide to an external device.
  • the time of day may be accounted for when determining the ambient luminance value, as the time of day may be associated with a typical illumination (for example, brighter at daytime, darker at nighttime).
  • a value of 5cd/m 2 may be used as a default value when no ambient luminance value is known and cannot be determined.
  • the method (200) includes, at step (215), computing an incident luminance.
  • the incident luminance provides an estimation of light falling on the eyes of a viewer, and may be based on both the ambient luminance value from step (210) and the mean luminance value from step (205).
  • FIG. 3 provides a display (300) (e.g., the reference display (125)) within a surrounding ambience (305).
  • the display (300) has a first luminance value, such as the mean luminance value from step (205).
  • the surrounding ambience (305) has a second luminance value, such as the ambient luminance value from step (210).
  • the size of the display (300) relative to surrounding ambience (305) is dependent on both the size of the screen of the display (300) and the viewing distance (e.g., how far the display (300) is from the eyes of the viewer).
  • Cross-section line (310) will be discussed in conjunction with FIG. 4 below.
  • the incident luminance may be calculated using a cosine cubed function.
  • FIG. 4 provides a graph (400) illustrating a cosine cubed falloff function (415).
  • the graph (400) is a one-dimensional cross-section along line (310) of FIG. 3.
  • the graph (400) includes mean luminance value (405) corresponding to the luminance of the display (300), and an ambient luminance value (410) corresponding to the surrounding ambience (305).
  • the cosine cubed falloff function (415) is multiplied by the mean luminance value (405) and the ambient luminance value (410) to scale the cosine cubed falloff function (415) to a given scenario.
  • the Y- coordinate provides luminance values
  • the X-coordinate provides values of visual degrees.
  • the cosine cubed falloff function (415) is scaled such that the area under the curve is 1 and the maximum degrees is 45. While the cosine cubed falloff function (415) provides an estimation of experienced luminance based on gaze position, other embodiments may be contemplated based on the visual influence of the ambient luminance. For example, the same method can be adjusted for two-dimensional and three-dimensional scenarios.
  • step (215) pseudocode for calculating the incidence luminance using the mean luminance value (405), ambient luminance value (410), and the cosine cubed falloff function (415) is provided below:
  • the incident luminance may be determined for both the current image frame and the upcoming image frame.
  • the mean luminance value (405) varies based on the image provided on the display (300).
  • a difference between the incident luminance for the current image frame and the incident luminance for the upcoming image frame may be determined, as discussed further below.
  • incident_Lum provides an estimate of the incident luminance falling on the eyes of the viewer, which may influence or alter the pupil diameter of the eyes (e.g., spatially compensated corneal flux in cd/m 2 )
  • display _Y represents the mean luminance value
  • surround_Y represents the ambient luminance value.
  • the method (200) includes determining a difference between a current pupil size and a target pupil size at step (220).
  • the current pupil size is a function of the incident luminance and correlates with how the pupil reacts to the upcoming image frame the instant it is shown (i.e., as soon as the switch occurs between the current image frame and the upcoming image frame).
  • the current pupil size is a continuous filtered result to represent the adaptation of the pupil over time (e.g., from frame to frame). In this manner, the current pupil size may be a function of all previous frames, and how the pupil size changes from frame to frame may also be referred to as an adapted pupil size.
  • the current pupil size may be determined using a sensor, such as an eye tracking sensor.
  • the sensor observes the pupil size and provides a measurement of the pupil size.
  • the target pupil size correlates with the instantaneous desired pupil diameter for the upcoming image frame. In this manner, the target pupil size may not account for previous frames, and represents the pupil diameter that would be achieved by staring at the upcoming image frame for a long period of time.
  • the difference between the current pupil size and the target pupil size may be defined as a delta pupil response, provided in Equation 1:
  • FIG. 5 provides a steady-state pupil model to estimate and/or predict pupil diameter with respect to incident luminance.
  • the steady-state pupil model presented in FIG. 5 is directly adapted from that presented in the paper “A Unified Formula for Light- Adapted Pupil Size” by A. B. Watson and J. I. Yellot, which is hereby incorporated by reference in its entirety. It is possible to utilize alternative models and methodologies to determine the steady-state pupil size.
  • the constant rate of constriction of the pupil is 3 mm/s and the dilation of the pupil is 0.5 mm/s.
  • other constant rates of constriction and dilations may be utilized.
  • the color of an image may also influence the rate of constriction.
  • Pupil diameter may further vary based on degree area, number of eyes, and age.
  • the pupil works to achieve a steady- state pupil size based on the incident luminance, and either constricts or dilates depending on whether the target pupil size is larger or smaller than the current pupil size.
  • both the constriction and dilation have estimated velocities for the sampling rate of the image frames (represented mathematically by the reciprocal of the image frame rate).
  • pseudocode for determining the difference between the current pupil size and the target pupil size e.g., a delta pupil response
  • the exact method of estimation of the steady-state pupil size is left ambiguous.
  • the current pupil is the diameter of the pupil given the frame that is going to be shown, along with the duration of that frame.
  • the method (200) includes, at step (225), generating an output image.
  • the output image may be generated by modifying the source image (e.g., video data (107)) based on a determined luminance adjustment factor.
  • the luminance adjustment factor may be based on, for example, the delta pupil response.
  • a relationship is derived between the delta pupil response and experienced discomfort. For example, as provided in Equation 2:
  • Equation 2 Delta Pupil Response .
  • Equation 2 provides an exponential function, other functions may be used to determine the perceived discomfort, such as a cubic roll-off function. The perceived discomfort is accounted for when generating the output image at step (225).
  • pseudocode is provided below for converting the perceived discomfort to a “creative experience” (CE) value, which is a value that indicates the change in size of the pupils.
  • CE value is used to describe how an observer reacts to changing luminance levels across the duration of presented content.
  • functions to determine the output image luminance are selected such that the CE value is negative when the pupil dilates, is positive when the pupil constricts, and is zero when the pupil diameter is constant (e.g., does not change).
  • FIG. 6 provides one example of the CE value determined over a plurality of image frames.
  • FIG. 6 contains several instances of changes between bright and dark luminance values. For example, the pupil dilates during dip (600) and dip (602), indicating a change to a dark luminance value at dip (600) and dip (602).
  • the CE value is determined for each frame included in the video content, and is transformed into metadata included in the coded bit stream (122).
  • FIG. 7 provides a method (700) for decoding the coded bit stream (122).
  • the method (700) includes, at step (705), receiving an input image, such as data included in the coded bit stream (122).
  • the method includes, at step (705), receiving an input image, such as data included in the coded bit stream (122).
  • step (700) includes, at step (710), determining an ambient luminance value based on an ambient luminance. For example, an ambient luminance of an area surrounding the target display (140) is determined, as described above with respect to step (210).
  • a “Tmid” value is calculated, providing an estimate of the average luminance for the image described by the coded bit stream (122) (e.g., a target luminance value), when the image is displayed on the target device (140).
  • Tone Curve Mapping for High Dynamic Range Images by J. Pytlarz and R.Atkins, which is incorporated herein by reference in its entirety, the inventors propose methods for determining a tone curve for display mapping of high dynamic range (HDR) images.
  • the tone mapping curve may further be adjusted using functions to make the input image brighter (resulting in a higher CE value) or darker (resulting in a lower CE value).
  • pseudocode is provided below for calculating the Tmid value.
  • a predicted CE value for the given target device (140) may be determined by using a dynamic look-up-table that compares the CE value with the input tone curve parameters. This may be used to ensure the selected tone mapping curve is appropriate for the target device (140).
  • a predicted CE value may be determined using the tone mapping curve. For example, using the tone mapping curve values, the Tmid value, incident luminance, and current and target pupil sizes of a viewer are determined, as described above with respect to method (200), for each frame included in the coded bit stream (122). These are used to calculate a predicted CE value for each frame. The predicted CE value may then be compared to the received CE value to determine how to adjust the luminance of each image to achieve the desired output. As one particular example, pseudocode is provided below to determine an intersection between the predicted CE value and the actual provided CE value, as received in the metadata of the coded bit stream (122).
  • FIG. 8 illustrates a graph providing this intersection between the predicted CE value and the provided CE value (e.g., reference CE value).
  • the provided CE value indicated by the dashed line, is - 1.
  • the solid line provides the calculated CE value, as determined by the calculated Tmid value of the given device at each slider value.
  • the intersection provides the slider value the target display (140) must use to output the desired CE value.
  • the slider value is approximately -0.15. Accordingly, the luminance of the image will be lowered to output the desired CE value.
  • the method (700) displays an output image using the slider value determined at step (715). Accordingly, the output image has a creative experience that corresponds to the luminance indicated by the CE value included in the metadata of the coded bit stream (122). In other words, the output image is brightened or darkened in order to achieve the luminance indicated by the CE value.
  • Method (700) is performed for each frame provided in the video data, such that all frames are output with a creative experience value intended by the content creator.
  • the creative experience desired by the content creator of the video data can be achieved regardless of the capabilities of the user device, closing the gap in viewing differences between devices such as home theaters compared to mobile phones.
  • the methods described herein may be used to reduce discomfort in changing luminance for a variety of cases.
  • One such embodiment includes reducing viewer discomfort during fast-forward functions. When fast-forward is initiated, frames are rapidly skipped and displayed. When frames change from dark to bright and bright to dark, a strobing effect may occur. To combat this, the sampling rate over which the pupil calculations occur at steps (220) and (715) may be increased based on the rate of fast-forward.
  • one video includes a person exiting a cave into bright sunlight, resulting in a luminance change from dark to bright. This may occur over 10 seconds. However, if fast-forward is initiated such that the scene occurs over 5 seconds, the pupil adaptation rate changes substantially.
  • the mapping of the CE value to the tone mapping curve may be adjusted in this case by reducing the adaptation duration [0043] by a 50% factor to account for this change in time. The speed of constriction and dilation will be increased proportionally based on the fast-forward speed due.
  • zoom function Another embodiment in which methods disclosed herein may be implemented is a zoom function.
  • zoom When zoom is activated, focus turns to a portion of an image, which may significantly change the average luminance of the image.
  • a user may move between portions of the image themselves, switching between dark areas of interest and bright areas of interest, causing discomfort.
  • Knowledge of the luminance attributes of the zoomed-in regions may provide for dampening or brightening to more closely achieve the desired creative experience.
  • Methods described herein may be used for volumetric experiences, such as virtual reality gaming.
  • Observers develop a level of adaptation to a computer-generated landscape within the virtual reality experience. As observers move and look around, the sudden reveal of bright objects or reflections may be jarring to the observer. Similar adjustments to images or objects being viewed may be applied to limit viewing discomfort from changes in luminance.
  • multiple users may be in the same virtual reality experience at the same time. Each user may be looking at different objects, resulting in different experiences that may give one gamer an advantage. Luminance may be adjusted to further balance the experiences and even the field for each.
  • Advertisements may be inserted into videos being viewed by a user, which may drastically change the average luminance value being displayed, regardless of device type.
  • the CE value may be used to apply a smooth comfortable transition. For example, during a dark scene in a movie, an advertisement is provided with a bright luminance value. Using the CE value of both the movie and the advertisement, the provided image frames compensate for the sudden change by lowering the brightness of the advertisement and slowly increasing the brightness over time, until the advert is complete and then fading back to match the CE value of the movie.
  • FIG. 9A provides an example of luminance adjustment to limit experienced discomfort while preserving intended luminance changes. The overall intended viewing experience is maintained, but the extremes in luminance are limited to minimize change in pupil response.
  • a hard cutoff may be used instead of a general scalar.
  • FIG. 9B provides an example of luminance adjustment with a strict threshold.
  • User preference may further be split based on constriction and dilation information. For example, only constriction may have a set threshold, or only dilation may have a set threshold. Both constriction and dilation may have unique thresholds different from the other.
  • methods described herein may be implemented to provide a viewing experience similar to how it was originally captured.
  • users may capture images using a camera.
  • the image is processed such that it includes CE metadata indicating a luminance value of the original scene. Accordingly, when the picture is viewed on other devices, the image is provided in a way similar to how it was originally captured.
  • the above video delivery systems and methods may provide for luminance adjustment based upon a viewer adaptation state.
  • Systems, methods, and devices in accordance with the present disclosure may take any one or more of the following configurations.
  • a video delivery system for luminance adjustment based upon a viewer adaptation state comprises a processor to perform post-production editing of video data.
  • the processor is configured to: receive a source image including a current image frame including metadata corresponding to a mean luminance value of the current image frame, and the source image including an upcoming image frame including metadata corresponding to a mean luminance value of the upcoming image frame, determine an ambient luminance value based on an ambient luminance, determine, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the mean luminance value, determine a difference between a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance value of the upcoming image frame, and wherein the current pupil size is determined based on the incident luminance value of the current image frame and one or more previous image frames, and generate an output image by modifying the source image based on a luminance adjustment factor, the luminance adjustment factor being a function of the difference between
  • determining the ambient luminance value includes at least one of receiving the ambient luminance value from one or more ambient light sensors, receiving the ambient luminance value from one or more smart light devices, or determining the ambient luminance value based on a time of day.
  • determining the incident luminance value includes applying a cosine cubed function to the mean luminance value and the ambient luminance value to obtain an average adaptation state.
  • a method for luminance adjustment based upon a viewer adaptation state comprising: receiving a source image including a current image frame including metadata corresponding to a mean luminance value of the current image frame, and the source image including an upcoming image frame including metadata corresponding to a mean luminance value of the upcoming image frame, determining an ambient luminance value based on an ambient luminance, determining, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the mean luminance value, determining a difference between a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance value of the upcoming image frame, and wherein the current pupil size is determined based on the incident luminance value of the current image frame and one or more previous image frames, and generating an output image by modifying the source image based on a luminance adjustment factor, the luminance adjustment factor being a function of the difference between the current pupil size and the target pupil size.
  • determining the ambient luminance value includes at least one of receiving the ambient luminance value from one or more ambient light sensors, receiving the ambient luminance value from one or more smart light devices, or determining the ambient luminance value based on a time of day.
  • determining the incident luminance value includes applying a cosine cubed function to the mean luminance value and the ambient luminance value to obtain an average adaptation state.
  • a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any one of (11) to (19).
  • (21) A video delivery system for luminance adjustment based upon a viewer adaptation state, the delivery system comprising a processor to decode a received coded bit stream, the processor configured to: receive an input image including a current image frame, an upcoming image frame, and metadata corresponding to an expected change in pupil size, determine, for the current image frame and the upcoming image frame, a target luminance value, determine an ambient luminance value based on an ambient luminance, determine, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the target luminance value, select a tone mapping curve based on a characteristic of a device configured to provide the image, determine a difference between a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance of the upcoming image frame, and
  • determining the tone mapping curve includes determining an intersection between the target luminance value of the current image frame and the input tone curve parameters.
  • a method for luminance adjustment based upon a viewer adaptation state comprising: receiving an input image including a current image frame, an upcoming image frame, and metadata corresponding to an expected change in pupil size, determining, for the current image frame and the upcoming image frame, a target luminance value, determining an ambient luminance value based on an ambient luminance, determining, for the current image frame and the upcoming image frame, an incident luminance value based on the ambient luminance value and the target luminance value, selecting a tone mapping curve based on a characteristic of a device configured to provide the image, determining a difference between a current pupil size and a target pupil size, wherein the target pupil size is determined based on the incident luminance of an upcoming image frame, and wherein the current pupil size is determined based on the incident luminance of a current
  • determining the tone mapping curve includes determining an intersection between the target luminance value of the current image frame and the input tone curve parameters.
  • determining the ambient luminance value includes at least one of receiving the ambient luminance value from one or more ambient light sensors, receiving the ambient luminance value from one or more smart light devices, or determining the ambient luminance value based on a time of day.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Social Psychology (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Image Processing (AREA)
  • Transforming Electric Information Into Light Information (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

Un système de distribution de vidéo pour un réglage de luminance sur la base d'un état d'adaptation de spectateur comprend un processeur configuré pour : recevoir une image source comprenant une trame d'image actuelle comprenant des métadonnées correspondant à une valeur de luminance moyenne de la trame d'image actuelle, et l'image source comprenant une trame d'image à venir comprenant des métadonnées correspondant à une valeur de luminance moyenne de la trame d'image à venir. Le processeur est configuré pour déterminer une valeur de luminance ambiante sur la base d'une luminance ambiante, déterminer une valeur de luminance incidente sur la base de la valeur de luminance ambiante et de la valeur de luminance moyenne, déterminer une différence entre une taille de pupille actuelle et une taille de pupille cible, et générer une image de sortie par modification de l'image source sur la base d'un facteur de réglage de luminance, le facteur de réglage de luminance étant une fonction de la différence entre la taille de pupille actuelle et la taille de pupille cible.
EP22710270.4A 2021-03-22 2022-03-02 Réglage de luminance basé sur un état d'adaptation de spectateur Pending EP4315232A1 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US202163164165P 2021-03-22 2021-03-22
EP21163880 2021-03-22
PCT/US2022/018547 WO2022203826A1 (fr) 2021-03-22 2022-03-02 Réglage de luminance basé sur un état d'adaptation de spectateur

Publications (1)

Publication Number Publication Date
EP4315232A1 true EP4315232A1 (fr) 2024-02-07

Family

ID=80738968

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22710270.4A Pending EP4315232A1 (fr) 2021-03-22 2022-03-02 Réglage de luminance basé sur un état d'adaptation de spectateur

Country Status (4)

Country Link
US (1) US20240169504A1 (fr)
EP (1) EP4315232A1 (fr)
JP (1) JP2024516080A (fr)
WO (1) WO2022203826A1 (fr)

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120182278A1 (en) * 2011-01-17 2012-07-19 Dolby Laboratories Licensing Corporation Methods and Apparatus for Estimating Light Adaptation Levels of Persons Viewing Displays
TWI538474B (zh) * 2011-03-15 2016-06-11 杜比實驗室特許公司 影像資料轉換的方法與設備
WO2018119161A1 (fr) * 2016-12-22 2018-06-28 Dolby Laboratories Licensing Corporation Gestion d'affichage s'adaptant à la lumière ambiante
PL3559901T3 (pl) 2017-02-15 2020-11-16 Dolby Laboratories Licensing Corporation Odwzorowywanie krzywej tonalnej dla obrazów o wysokim zakresie dynamiki
US10983349B2 (en) * 2018-06-14 2021-04-20 Google Llc Method of dynamically adjusting display luminance flux in wearable heads-up displays

Also Published As

Publication number Publication date
US20240169504A1 (en) 2024-05-23
JP2024516080A (ja) 2024-04-12
WO2022203826A1 (fr) 2022-09-29

Similar Documents

Publication Publication Date Title
RU2609760C2 (ru) Устройства и способы усовершенствованного кодирования изображений
CN110337667B (zh) 高动态范围图像的色调曲线映射
CN109219844B (zh) 在视频优先级与图形优先级之间转换
EP3566203B1 (fr) Préservation perceptuelle de chromaticités et de contrastes relatifs à une scène
US11587526B2 (en) Luminance adaption to minimize discomfort and improve visibility
EP3552178B1 (fr) Systèmes et procédés d'ajustement des courbes de traitement vidéo pour des images à grande gamme dynamique
KR102130667B1 (ko) 위치 기반 디스플레이 특성의 환경 적응을 갖춘 전자 디스플레이
US20130162766A1 (en) Overlaying frames of a modified video stream produced from a source video stream onto the source video stream in a first output type format to generate a supplemental video stream used to produce an output video stream in a second output type format
WO2018152063A1 (fr) Mappage de courbe de tonalité pour des images à plage dynamique élevée
CN110192223B (zh) 高动态范围图像的显示映射
CN107743637B (zh) 用于处理外围图像的方法和设备
WO2018111682A1 (fr) Systèmes et procédés permettant de régler des courbes de traitement vidéo pour des images à plage dynamique élevée
WO2018175337A1 (fr) Préservation perceptive de contrastes et de chromaticités de scène
US20240169504A1 (en) Luminance adjustment based on viewer adaptation state
US10114447B2 (en) Image processing method and apparatus for operating in low-power mode
CN115293994A (zh) 图像处理方法、装置、计算机设备和存储介质
CN117043812A (zh) 基于观看者适应状态的光亮度调整
US7961939B2 (en) Color transforming method
US20130162765A1 (en) Modifying luminance of images in a source video stream in a first output type format to affect generation of supplemental video stream used to produce an output video stream in a second output type format
JP2023529917A (ja) 異なるアスペクト比を有するディスプレイ上で提示するためのビデオ画像の製作と適応
Goudé et al. A perceptually coherent TMO for visualization of 360 HDR images on HMD
Goudé et al. HMD-TMO: a tone mapping operator for 360 HDR images visualization for head mounted displays
CN111583163A (zh) 基于ar的人脸图像处理方法、装置、设备及存储介质
WO2023064105A1 (fr) Mappage de couleurs dépendant du contexte d'images et de données vidéo
KR20240065212A (ko) 낮은 동적 범위 호환성을 갖는 높은 동적 범위 이미지 포맷

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230906

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20240319