US20240013354A1 - Deep SDR-HDR Conversion - Google Patents
Deep SDR-HDR Conversion Download PDFInfo
- Publication number
- US20240013354A1 US20240013354A1 US18/372,510 US202318372510A US2024013354A1 US 20240013354 A1 US20240013354 A1 US 20240013354A1 US 202318372510 A US202318372510 A US 202318372510A US 2024013354 A1 US2024013354 A1 US 2024013354A1
- Authority
- US
- United States
- Prior art keywords
- image
- sdr
- hdr
- training
- actions
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000006243 chemical reaction Methods 0.000 title description 21
- 238000012549 training Methods 0.000 claims abstract description 71
- 238000013528 artificial neural network Methods 0.000 claims abstract description 57
- 230000009471 action Effects 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims description 44
- 238000013507 mapping Methods 0.000 claims description 6
- 238000010801 machine learning Methods 0.000 abstract description 9
- 238000013459 approach Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 230000007704 transition Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 208000004547 Hallucinations Diseases 0.000 description 1
- 230000002730 additional effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G06T5/007—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20208—High dynamic range [HDR] image processing
Definitions
- a display device may support Standard Dynamic Range (SDR) content and High Dynamic Range (HDR) content.
- SDR Standard Dynamic Range
- HDR High Dynamic Range
- HDR content may support a greater dynamic range of luminosity, more contrast and a wider range of colors. Therefore, HDR content may provide a better viewing experience.
- HDR color grading and HDR remastering require a significant amount of manual input. These processes may include multiple iterations and quality control checks that may take highly experienced technicians anywhere from a few days to a few weeks to finish. As a result, HDR color grading and HDR remastering may require a significant amount of resources. Due to this cost, a large amount of available content is not configured in HDR format. Accordingly, there is a need to reduce the complexity, time, money and other resources involved in HDR color grading and HDR remastering.
- FIG. 1 shows a method for training a neural network to convert standard dynamic range (SDR) content to high dynamic range (HDR) content according to various exemplary embodiments.
- FIG. 2 illustrates an autoencoder network architecture with fully convolutional layers and skip-links between encoder and decoder.
- FIG. 3 shows a method for using the neural network to convert SDR content to HDR content according to various exemplary embodiments.
- FIG. 4 shows a method for training a neural network to convert SDR content to HDR content according to various exemplary embodiments.
- FIG. 5 illustrates an exemplary fully-convolutional asynchronous advantage actor-critic (A3C) network architecture.
- FIG. 6 shows a method for using the neural network to convert SDR content to HDR content according to various exemplary embodiments.
- FIG. 7 shows an HDR remastering device according to various exemplary embodiments.
- the exemplary embodiments may be further understood with reference to the following description and the related appended drawings, wherein like elements are provided with the same reference numerals.
- the exemplary embodiments relate to converting Standard Dynamic Range (SDR) content to High Dynamic Range (HDR) content using a novel machine learning system.
- SDR Standard Dynamic Range
- HDR High Dynamic Range
- the quality of results produced by prior machine learning systems proved inconsistent and tended to be too blurry to be used in practice.
- previous machine learning systems restricted problem domains (e.g. focusing solely on reconstructing details of over exposed sky images), thus being restricted to specific and simple problem domains and only capable of handling very low resolution images. Accordingly, there is a need for a machine learning system that may be used for converting SDR content to HDR content that adequately addresses the fundamental issues mentioned above without being restricted to a particular problem domain or resolution and without producing excessive blur.
- SDR and HDR refer to an image/video format that is capable of a particular dynamic range of luminosity.
- SDR content may be configured with a dynamic range of luminosity that is based on a gamma curve or any other standard range.
- HDR content may be configured with a dynamic range of luminosity that exceeds a standard dynamic range of luminosity.
- HDR content may also support a wider color gamut compared to SDR content.
- SDR and HDR There are multiple variations of SDR and HDR respectively.
- the exemplary embodiments are not limited to any particular type of SDR or HDR. Those skilled in the art will understand the scope of the terms SDR and HDR and how they are distinct from one another. Accordingly, throughout this description, SDR content may refer to image or video data configured with certain display related parameters that are restricted to a standard range and HDR content may refer to image or video data configured with one or more display parameters that have a range that exceeds what is considered standard.
- the exemplary embodiments are also described with regard to a machine learning system configured to perform SDR to HDR conversation.
- the exemplary embodiments relate to a neural network configured to automatically infer HDR content from input SDR content using a high dimensional mapping.
- a supervised learning neural network may be trained to generate a model for mapping an input SDR image to an HDR image.
- this approach may be described with regard to an autoencoder architecture with fully convolutional layers and skip-links between the encoder and decoder. Specific examples of training this type of neural network and how it may be used for SDR to HDR conversion will be explained in more detail below.
- any reference to a supervised learning neural network or an autoencoder architecture is merely provided for illustrative purposes.
- the exemplary embodiments may apply to any appropriate machine learning approach that may be trained to learn an implicit mapping from SDR to HDR.
- the exemplary embodiments relate to a neural network configured to generate an HDR image by applying one or more user defined actions to an input SDR image.
- DRL deep reinforced learning
- A3C asynchronous advantage actor-critic network architecture that includes a policy network and a value network. Specific examples of training this type of neural network and how it may be used for SDR to HDR conversion will be explained in more detail below.
- any reference to DRL or a A3C network is merely provided for illustrative purposes.
- the exemplary embodiments may apply to any appropriate machine learning approach that may be trained to perform SDR to HDR conversion using explicit user defined actions. Further, the exemplary embodiments described herein may be used in conjunction with other currently implemented SDR to HDR conversion techniques, future implementations of SDR to HDR conversion techniques or independently from other SDR to HDR conversion techniques.
- a supervised learning neural network performs the task of learning a function that maps an input to an output using training input-output pairs.
- the input-output pairs may be an SDR image and a corresponding HDR image.
- the neural network may then be used to infer HDR content from SDR input.
- FIG. 1 shows a method 100 for training a neural network to convert SDR content to HDR content according to various exemplary embodiments.
- FIG. 1 shows a method 100 for training a neural network to convert SDR content to HDR content according to various exemplary embodiments.
- each input-output pair may include an SDR image and a corresponding HDR image.
- reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data. For example, video data including multiple frames may be used for training.
- an image may be characterized as a “training image” when the image is to be used to train a neural network.
- a training SDR image and a training HDR image may form a pair of training images.
- the method 100 will be described with regard to a single pair of training images. However, in an actual training process, a training dataset of multiple pairs of training images may be utilized to train the neural network.
- the neural network is configured to generate an HDR image from SDR input. This may include estimating portions of the SDR input image that may be Lost due to its dynamic range of luminosity. For example, certain features of the SDR image may be lost in areas of pure white (over-exposed) or pure black (under-exposed) because the dynamic range of luminosity may not be capable of showing variations between shades that exceed a maximal or minimal value of the dynamic luminosity range.
- the dynamic range of luminosity for HDR exceeds the dynamic range of luminosity for SDR.
- the portions of the SDR image that were lost due to its limited dynamic range of luminosity may be reconstructed in the HDR image.
- “Highlight roll off” may refer to the transition between a clipped light level (e.g., a maximum threshold of the dynamic range of luminosity) and the surrounding highlights.
- the neural network may reconstruct rolled off highlights when generating the HDR image from SDR input.
- the neural network may reconstruct transitions between shadows and a clipped light level (e.g., a minimum threshold of the dynamic range of luminosity).
- the goal is to provide a smooth transition or roll off.
- the neural network may be configured to provide a smooth transition, an abrupt transition, or any other appropriate type of transition.
- FIG. 2 illustrates an autoencoder network architecture with fully convolutional layers and skip-links between encoder and decoder.
- this neural network architecture may be used for mapping the dynamic range of luminosity and color gamut from SDR images to HDR images.
- an autoencoder is a type of neural network that includes an encoder that is configured to transform data into a latent representation and a decoder that is configured to transform the latent representation into another type of data.
- the neural network may encode it into a low dimensional latent representation.
- the input SDR image 205 is encoded to the latent representation 210 .
- the latent representation may then be decoded to generate an HDR output image 215 .
- the skip-connections 202 from each encoder layer to its corresponding decoder layer enables local detail information reconstruction during decoding.
- concatenation of the convolutional layers in the encoder and corresponding decoder layers may be applied.
- 3 ⁇ 3 kernel sizes may be used to generate feature maps for all of the convolutional layers.
- max-pooling may be employed in the encoder and up-sampling may be employed in the decoder.
- this neural network may be trained using exponential linear unit (ELU) activation for all the convolutional layers.
- ELU exponential linear unit
- a low dimensional latent representation of the training SDR image is generated.
- the SDR image may be encoded by the autoencoder architecture.
- an HDR image is generated by decoding the low latent representation of the SDR image.
- the neural network attempts to generate an HDR image that is at least similar to the training HDR image corresponding to the training SDR image.
- an SDR to HDR conversion model is generated based on processing the training dataset.
- the SDR to HDR model is configured to reconstruct rolled off highlights and map the dynamic range of luminosity and color gamut from SDR to HDR.
- the neural network may learn an SDR to HDR conversion model, , using the following formula:
- S SDR represents the input SDR image
- ⁇ represents model parameters.
- the above formula may be configured as an optimization task where the neural network attempts to predict as accurate as possible to the training HDR image, HDR , by minimizing a loss function, using the following formula:
- FIG. 3 shows a method 300 for using the neural network to convert SDR content to HDR content according to various exemplary embodiments.
- the method 300 provides an example of how a user may perform SDR to HDR conversion using the model learned by the neural network in accordance with the method 100 .
- Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard to FIG. 7 .
- an input SDR image is received.
- reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data.
- a low dimensional latent representation of the SDR image is generated.
- the SDR image may be encoded by the autoencoder architecture.
- an HDR image is generated by decoding the low latent representation of the SDR image. For example, using the learned SDR to HDR conversion model, , an HDR image may be predicted in the decoder from the encoded SDR image.
- HDR color grading there is an artistic aspect to HDR color grading. Thus, it may be beneficial to provide the user with the option to edit the HDR image.
- a set of tonal curves is generated based on the HDR image. For example, a regression may be performed on the HDR image to generate a set of tonal curves that approximate the HDR image inferred by the neural network.
- the tonal curves may be displayed on a graphical user interface (GUI).
- GUI graphical user interface
- a colorist may edit the tonal curves displayed on the GUI to tune the HDR image in accordance with their artistic vision.
- a spline based regression may be used to generate the tonal curve.
- the GUI may display a tonal curve configured with multiple control points.
- the user input may manipulate the control points to alter the HDR image. There may be multiple tone curves for different color channels.
- a modified HDR image is generated based on the user input.
- the user may retain artistic control over the conversion process by manually editing the tonal curves.
- DRL is a machine learning technique where a neural network learns from its actions using a reward system.
- this neural network may perform dynamic range and color gamut adjustments on an input SDR image to generate an HDR image. These adjustments may be modeled after the actions a colorist may perform during the color grading process.
- a method for training a neural network using the DRL approach will be described below with regard to FIGS. 4 - 5 .
- a method for using the neural network for SDR to HDR conversion will be described below with regard to the method 600 .
- FIG. 4 shows a method 400 for training a neural network to convert SDR content to HDR content according to various exemplary embodiments.
- FIG. 4 shows a method 400 for training a neural network to convert SDR content to HDR content according to various exemplary embodiments.
- Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard to FIG. 7 .
- each input-output pair may include an SDR image and a corresponding HDR image.
- reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data.
- multiple color grading actions are defined.
- a set of possible actions that may be applied to an SDR image to generate an HDR image may be generated based on an explicit action-wise modeling of the color grading process.
- the color grading actions may include, but are not limited to, adjusting the brightness, adjusting the contrast, adjusting the color saturation, adjusting the exposure, etc.
- the adjustments may be applied to bright regions or shadows and on different color channels.
- contextual and color features are extracted from the training SDR image.
- a convolution neural network may be employed to perform the contextual and color feature extraction.
- one or more color grading actions are applied to the contextual and color features extracted from the training SDR image.
- an intermediate representation of the HDR image is generated. If the intermediate representation satisfies a threshold value, this may indicate that the action applied to the training SDR image has generated an HDR image that is at least substantially similar to the training HDR image. If the grade does not satisfy the threshold value, this may indicate that additional actions may need to be applied to the intermediate representation for a satisfactory result. This process is repeated until a convergence result between the training SDR image and the training HDR image is achieved.
- an SDR to HDR conversion model is generated based on processing the training dataset.
- a more detailed description of the actions performed at the neural network level is described in more detail below with regard to FIG. 5 .
- FIG. 5 illustrates an exemplary fully-convolutional asynchronous advantage actor-critic (A3C) network architecture 500 .
- A3C network includes one or more agents (not pictured), a feature network 505 , a policy network 510 and a value network 515 .
- a training dataset 502 is generated.
- Thetraining dataset 502 may include multiple training image pairs.
- the fully-convolutional A3C network 500 may extract contextual and color features using the feature network 505 .
- the feature network 505 may be a convolution neural network configured to perform the contextual and color feature extraction.
- the policy network 510 may determine one or more actions the agent may apply to the contextual and color features of the training SDR images. This generates an intermediate representation 520 of the HDR image.
- the value network 515 may measure how close the intermediate representation 520 is to the corresponding training HDR image. Those skilled in the art will understand that this may include the value network 515 rewarding the agent. A reward may serve as positive or negative reinforcement for the one or more actions used to generate the intermediate representation 520 .
- the value network 515 may then forward this information to the policy network 510 to improve the actions taken by the policy network 510 .
- the feedback from the value network 515 helps the policy network 510 learn which actions to take during SDR to HDR conversion.
- the fully-convolution A3C network 500 is trained to maximize a reward defined as a pixel level distance between the training SDR image and the target training HDR image.
- the reward is defined by an objective function that measures the similarity between the intermediate representation 520 and the corresponding training HDR image.
- FIG. 6 shows a method 600 for using the neural network to convert SDR content to HDR content according to various exemplary embodiments.
- the method 600 provides an example of how a user may perform SDR to HDR conversion using the model learned by the neural network in accordance with the method 400 .
- Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard to FIG. 7 .
- an input SDR image is received.
- reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data.
- an output HDR Image is generated. As indicated above, this may include applying one or more predefined actions to the input SDR image. The actions applied by the neural network are based on the training performed using the training dataset.
- the one or more actions applied by the neural network are indicated to the user.
- a GUI may display which user defined actions were applied to the input SDR image.
- a colorist may edit the actions displayed on the GUI to tune the HDR image in accordance with their artistic vision. Like the method 300 , in some embodiments, this may include editing a set of tonal curves displayed on the GUI.
- a modified HDR image is generated based on the user input. Thus, the user may retain artistic control over the conversion process by manually editing the tonal curves.
- FIG. 7 shows an HDR remastering device 700 according to various exemplary embodiments.
- the HDR remastering device 700 may represent any electronic device that is configured with the hardware, software or firmware to generate HDR content from SDR content.
- the HDR remastering device 700 may be a server, a desktop computer, a mobile phone, a tablet computer, a smartphone, an of Internet of Things (IoT) devices, etc.
- IoT Internet of Things
- the HDR remastering device 700 may include a processor 705 , a display device 710 , an input/output (I/O) device 715 , a memory arrangement 720 , a communication interface 725 , and other components 730 .
- the display device 710 may be a hardware component configured to present a graphical display.
- the I/O device 715 may be a hardware component that enables the user to enter inputs.
- the display device 710 and the I/O device 715 may be integrated together such as a touchscreen.
- the I/O device 715 may be represented as one or more buttons.
- the communication interface 725 may be a hardware component configured to communicate with a network over a wireless or wired connection.
- the communication interface 725 may also be configured to communicate with other devices using a wired or wireless connection.
- the processor 705 may be configured to execute a plurality of engines for the HDR remastering device 700 .
- the engines may include a training engine 735 , a conversion engine 740 and an artistic control engine 745 .
- the training engine 735 may perform a variety of different operations related to training a neural network. As indicated above, this may include training a neural network configured to automatically infer HDR content from input SDR content and training a neural network configured to generate an HDR image by applying one or more user defined actions to an input SDR image.
- the conversion engine 740 may perform SDR to HDR conversions using the exemplary technique describe herein.
- the artistic control engine 740 may generate for display a GUI that the user may interact with to modify an HDR image.
- the GUI may display a tonal curve configured with multiple control points.
- the user input may manipulate the control points to alter the HDR image. There may be multiple tone curves for different color channels.
- the above referenced engines each being an application (e.g., a program) executed by the processor 705 is only exemplary.
- the functionality associated with the engines may also be represented as a separate incorporated component of the HDR remastering device 700 or may be a modular component coupled to the HDR remastering device 700 , e.g., an integrated circuit with or without firmware.
- the integrated circuit may include input circuitry to receive signals and processing circuitry to process the signals and other information.
- the engines may also be embodied as one application or separate applications.
- the functionality described for the processor 705 may be split among two or more processors.
- the exemplary embodiments may be implemented in any of these or other configurations of a monitoring device.
- the memory arrangement 730 may be a hardware component configured to store data related to operations performed by the HDR remastering device 700 .
- the other components 730 may include, for example, ports to electronically connect to other devices.
- An exemplary hardware platform for implementing the exemplary embodiments may include, for example, an Intel x86 based platform with compatible operating system, a Windows platform, a Mac platform and MAC OS, a Linux based OS, a mobile device having an operating system such as iOS, Android, etc.
- the exemplary embodiments of the above described method may be embodied as a computer program product containing lines of code stored on a computer readable storage medium that may be executed on a processor or microprocessor.
- the storage medium may be, for example, a local or remote data repository compatible or formatted for use with the above noted operating systems using any storage operation.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Algebra (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The exemplary embodiments relate to converting Standard Dynamic Range (SDR) content to High Dynamic Range (HDR) content using a machine learning system. In some embodiments, the neural network is trained to convert an input SDR image into an HDR image using the encoded representation of a training SDR image and a training HDR image. In other embodiments, the neural network is trained to convert an input SDR image into an HDR image using a predefined set of color grading actions and the training images.
Description
- A display device may support Standard Dynamic Range (SDR) content and High Dynamic Range (HDR) content. Compared to SDR content, HDR content may support a greater dynamic range of luminosity, more contrast and a wider range of colors. Therefore, HDR content may provide a better viewing experience.
- Workflows for HDR color grading and HDR remastering require a significant amount of manual input. These processes may include multiple iterations and quality control checks that may take highly experienced technicians anywhere from a few days to a few weeks to finish. As a result, HDR color grading and HDR remastering may require a significant amount of resources. Due to this cost, a large amount of available content is not configured in HDR format. Accordingly, there is a need to reduce the complexity, time, money and other resources involved in HDR color grading and HDR remastering.
- Techniques for converting SDR content to HDR content (e.g., HDR color grading, HDR remastering, etc.) often encounter two fundamental issues. One issue is that quantization effects and banding artifacts may become visible due to the dynamic range extension from SDR format to HDR format. Under most circumstances, it may take an experienced technician several iterations to adequately blend the different shades of similar colors. Another issue is the hallucination of clipped details that may appear in the under-exposed or over-exposed regions in the SDR image. For example, due to its limited range, a pixel value may be clipped at a maximal or minimal value and the variation in bright or dark regions may be lost in the SDR image. Under most circumstances, it may take an experienced technician several iterations to render an HDR image that adequately portrays the details of the under-exposed or over-exposed regions of the SDR image.
- Accordingly, there is a need for a system that may be used for converting SDR content to HDR content that adequately addresses the fundamental issues mentioned above without being restricted to a particular problem domain or resolution and without producing excessive blur.
-
FIG. 1 shows a method for training a neural network to convert standard dynamic range (SDR) content to high dynamic range (HDR) content according to various exemplary embodiments. -
FIG. 2 illustrates an autoencoder network architecture with fully convolutional layers and skip-links between encoder and decoder. -
FIG. 3 shows a method for using the neural network to convert SDR content to HDR content according to various exemplary embodiments. -
FIG. 4 shows a method for training a neural network to convert SDR content to HDR content according to various exemplary embodiments. -
FIG. 5 illustrates an exemplary fully-convolutional asynchronous advantage actor-critic (A3C) network architecture. -
FIG. 6 shows a method for using the neural network to convert SDR content to HDR content according to various exemplary embodiments. -
FIG. 7 shows an HDR remastering device according to various exemplary embodiments. - The exemplary embodiments may be further understood with reference to the following description and the related appended drawings, wherein like elements are provided with the same reference numerals. The exemplary embodiments relate to converting Standard Dynamic Range (SDR) content to High Dynamic Range (HDR) content using a novel machine learning system. The quality of results produced by prior machine learning systems proved inconsistent and tended to be too blurry to be used in practice. Furthermore, previous machine learning systems restricted problem domains (e.g. focusing solely on reconstructing details of over exposed sky images), thus being restricted to specific and simple problem domains and only capable of handling very low resolution images. Accordingly, there is a need for a machine learning system that may be used for converting SDR content to HDR content that adequately addresses the fundamental issues mentioned above without being restricted to a particular problem domain or resolution and without producing excessive blur.
- The exemplary embodiments are described with regard to SDR and HDR. Generally, these terms refer to an image/video format that is capable of a particular dynamic range of luminosity. For example, SDR content may be configured with a dynamic range of luminosity that is based on a gamma curve or any other standard range. In contrast, HDR content may be configured with a dynamic range of luminosity that exceeds a standard dynamic range of luminosity. Further, HDR content may also support a wider color gamut compared to SDR content.
- There are multiple variations of SDR and HDR respectively. The exemplary embodiments are not limited to any particular type of SDR or HDR. Those skilled in the art will understand the scope of the terms SDR and HDR and how they are distinct from one another. Accordingly, throughout this description, SDR content may refer to image or video data configured with certain display related parameters that are restricted to a standard range and HDR content may refer to image or video data configured with one or more display parameters that have a range that exceeds what is considered standard.
- The exemplary embodiments are also described with regard to a machine learning system configured to perform SDR to HDR conversation. In one approach, the exemplary embodiments relate to a neural network configured to automatically infer HDR content from input SDR content using a high dimensional mapping. For example, a supervised learning neural network may be trained to generate a model for mapping an input SDR image to an HDR image. Throughout this description, this approach may be described with regard to an autoencoder architecture with fully convolutional layers and skip-links between the encoder and decoder. Specific examples of training this type of neural network and how it may be used for SDR to HDR conversion will be explained in more detail below. However, any reference to a supervised learning neural network or an autoencoder architecture is merely provided for illustrative purposes. The exemplary embodiments may apply to any appropriate machine learning approach that may be trained to learn an implicit mapping from SDR to HDR.
- In another approach, the exemplary embodiments relate to a neural network configured to generate an HDR image by applying one or more user defined actions to an input SDR image. For example, deep reinforced learning (DRL) may be used to generate a model for applying user defined actions to an input SDR image to generate an HDR image. Throughout this description, this approach may be described with regard to an asynchronous advantage actor-critic (A3C) network architecture that includes a policy network and a value network. Specific examples of training this type of neural network and how it may be used for SDR to HDR conversion will be explained in more detail below. However, any reference to DRL or a A3C network is merely provided for illustrative purposes. The exemplary embodiments may apply to any appropriate machine learning approach that may be trained to perform SDR to HDR conversion using explicit user defined actions. Further, the exemplary embodiments described herein may be used in conjunction with other currently implemented SDR to HDR conversion techniques, future implementations of SDR to HDR conversion techniques or independently from other SDR to HDR conversion techniques.
- As mentioned above, one of the exemplary approaches relates to utilizing a supervised learning neural network. Those skilled in the art will understand that a supervised learning neural network performs the task of learning a function that maps an input to an output using training input-output pairs. As will be described in more detail below, in this context, the input-output pairs may be an SDR image and a corresponding HDR image. Once trained, the neural network may then be used to infer HDR content from SDR input.
-
FIG. 1 shows amethod 100 for training a neural network to convert SDR content to HDR content according to various exemplary embodiments. Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard toFIG. 7 . - In 105, one or more training input-output pairs are collected. In this example, each input-output pair may include an SDR image and a corresponding HDR image. However, reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data. For example, video data including multiple frames may be used for training.
- Throughout this description, an image may be characterized as a “training image” when the image is to be used to train a neural network. For example, a training SDR image and a training HDR image may form a pair of training images. The
method 100 will be described with regard to a single pair of training images. However, in an actual training process, a training dataset of multiple pairs of training images may be utilized to train the neural network. - The neural network is configured to generate an HDR image from SDR input. This may include estimating portions of the SDR input image that may be Lost due to its dynamic range of luminosity. For example, certain features of the SDR image may be lost in areas of pure white (over-exposed) or pure black (under-exposed) because the dynamic range of luminosity may not be capable of showing variations between shades that exceed a maximal or minimal value of the dynamic luminosity range. The dynamic range of luminosity for HDR exceeds the dynamic range of luminosity for SDR. Thus, when generating the HDR image from SDR input, the portions of the SDR image that were lost due to its limited dynamic range of luminosity may be reconstructed in the HDR image.
- “Highlight roll off” may refer to the transition between a clipped light level (e.g., a maximum threshold of the dynamic range of luminosity) and the surrounding highlights. The neural network may reconstruct rolled off highlights when generating the HDR image from SDR input. Similarly, the neural network may reconstruct transitions between shadows and a clipped light level (e.g., a minimum threshold of the dynamic range of luminosity). Generally, the goal is to provide a smooth transition or roll off. However, there is an artistic aspect to color grading and thus, the neural network may be configured to provide a smooth transition, an abrupt transition, or any other appropriate type of transition.
- Prior to discussing how to train the neural network, a description of the neural network architecture is provided.
FIG. 2 illustrates an autoencoder network architecture with fully convolutional layers and skip-links between encoder and decoder. In addition to reconstructing rolled off highlights, this neural network architecture may be used for mapping the dynamic range of luminosity and color gamut from SDR images to HDR images. - Those skilled in the art will understand that an autoencoder is a type of neural network that includes an encoder that is configured to transform data into a latent representation and a decoder that is configured to transform the latent representation into another type of data. For example, given an input SDR image, the neural network may encode it into a low dimensional latent representation. With regard to
FIG. 2 , the input SDR image 205 is encoded to thelatent representation 210. The latent representation may then be decoded to generate an HDR output image 215. - The skip-
connections 202 from each encoder layer to its corresponding decoder layer enables local detail information reconstruction during decoding. In the decoder, concatenation of the convolutional layers in the encoder and corresponding decoder layers may be applied. In some embodiments, 3×3 kernel sizes may be used to generate feature maps for all of the convolutional layers. Further, max-pooling may be employed in the encoder and up-sampling may be employed in the decoder. In some embodiments, this neural network may be trained using exponential linear unit (ELU) activation for all the convolutional layers. Those skilled in the art will understand the ELUs code the degree of presence of a particular phenomena in the input without quantitatively modeling the degree of their absence. In addition, ELUs saturate a negative value when the argument gets smaller. Therefore, the representation is both noise-robust and low-complex. These features speed up learning in deep neural networks and provide higher classification accuracies compared to the units of other activation functions. (See, e.g., D.-A. Clevert, T. Unterthiner, S. Hochreiter, Fast and accurate deep network learning by exponential linear units (elus), arXiv preprint arXiv:1511.07289 (2015).) - Returning to the
method 100, in 110, a low dimensional latent representation of the training SDR image is generated. For example, as discussed above with regard toFIG. 2 , the SDR image may be encoded by the autoencoder architecture. In 115, an HDR image is generated by decoding the low latent representation of the SDR image. During training, the neural network attempts to generate an HDR image that is at least similar to the training HDR image corresponding to the training SDR image. - In 120, an SDR to HDR conversion model is generated based on processing the training dataset. As mentioned above, the SDR to HDR model is configured to reconstruct rolled off highlights and map the dynamic range of luminosity and color gamut from SDR to HDR. For example, the neural network may learn an SDR to HDR conversion model, , using the following formula:
- Here, SSDR represents the input SDR image, represents the output HDR image and θ represents model parameters. For training, the above formula may be configured as an optimization task where the neural network attempts to predict as accurate as possible to the training HDR image, HDR, by minimizing a loss function, using the following formula:
-
-
-
FIG. 3 shows amethod 300 for using the neural network to convert SDR content to HDR content according to various exemplary embodiments. Themethod 300 provides an example of how a user may perform SDR to HDR conversion using the model learned by the neural network in accordance with themethod 100. Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard toFIG. 7 . - In 305, an input SDR image is received. As mentioned above, reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data.
- In 310, a low dimensional latent representation of the SDR image is generated. For example, as discussed above with regard to
FIG. 2 , the SDR image may be encoded by the autoencoder architecture. In 315, an HDR image is generated by decoding the low latent representation of the SDR image. For example, using the learned SDR to HDR conversion model, , an HDR image may be predicted in the decoder from the encoded SDR image. - As mentioned above, there is an artistic aspect to HDR color grading. Thus, it may be beneficial to provide the user with the option to edit the HDR image.
- In 320, a set of tonal curves is generated based on the HDR image. For example, a regression may be performed on the HDR image to generate a set of tonal curves that approximate the HDR image inferred by the neural network. The tonal curves may be displayed on a graphical user interface (GUI).
- In 325, user input is received. For example, a colorist may edit the tonal curves displayed on the GUI to tune the HDR image in accordance with their artistic vision. For example, a spline based regression may be used to generate the tonal curve. The GUI may display a tonal curve configured with multiple control points. The user input may manipulate the control points to alter the HDR image. There may be multiple tone curves for different color channels.
- In 330, a modified HDR image is generated based on the user input. Thus, the user may retain artistic control over the conversion process by manually editing the tonal curves.
- As mentioned above, one exemplary approach relates to DRL. Those skilled in the art will understand that DRL is a machine learning technique where a neural network learns from its actions using a reward system. As will be described below, this neural network may perform dynamic range and color gamut adjustments on an input SDR image to generate an HDR image. These adjustments may be modeled after the actions a colorist may perform during the color grading process. A method for training a neural network using the DRL approach will be described below with regard to
FIGS. 4-5 . A method for using the neural network for SDR to HDR conversion will be described below with regard to themethod 600. -
FIG. 4 shows amethod 400 for training a neural network to convert SDR content to HDR content according to various exemplary embodiments. Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard toFIG. 7 . - In 405, one or more training input-output pairs are collected. In this example, each input-output pair may include an SDR image and a corresponding HDR image. However, reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data.
- In 410, multiple color grading actions are defined. A set of possible actions that may be applied to an SDR image to generate an HDR image may be generated based on an explicit action-wise modeling of the color grading process. For example, the color grading actions may include, but are not limited to, adjusting the brightness, adjusting the contrast, adjusting the color saturation, adjusting the exposure, etc. The adjustments may be applied to bright regions or shadows and on different color channels.
- In 415, contextual and color features are extracted from the training SDR image. For example, a convolution neural network may be employed to perform the contextual and color feature extraction.
- In 420, one or more color grading actions are applied to the contextual and color features extracted from the training SDR image. In 425, an intermediate representation of the HDR image is generated. If the intermediate representation satisfies a threshold value, this may indicate that the action applied to the training SDR image has generated an HDR image that is at least substantially similar to the training HDR image. If the grade does not satisfy the threshold value, this may indicate that additional actions may need to be applied to the intermediate representation for a satisfactory result. This process is repeated until a convergence result between the training SDR image and the training HDR image is achieved.
- In 430, an SDR to HDR conversion model is generated based on processing the training dataset. A more detailed description of the actions performed at the neural network level is described in more detail below with regard to
FIG. 5 . -
FIG. 5 illustrates an exemplary fully-convolutional asynchronous advantage actor-critic (A3C)network architecture 500. Those skilled in the art will understand that an A3C network includes one or more agents (not pictured), afeature network 505, a policy network 510 and avalue network 515. - As indicated above, initially, a
training dataset 502 is generated.Thetraining dataset 502 may include multiple training image pairs. The fully-convolutional A3C network 500 may extract contextual and color features using thefeature network 505. As indicated above, thefeature network 505 may be a convolution neural network configured to perform the contextual and color feature extraction. - The policy network 510 may determine one or more actions the agent may apply to the contextual and color features of the training SDR images. This generates an intermediate representation 520 of the HDR image. The
value network 515 may measure how close the intermediate representation 520 is to the corresponding training HDR image. Those skilled in the art will understand that this may include thevalue network 515 rewarding the agent. A reward may serve as positive or negative reinforcement for the one or more actions used to generate the intermediate representation 520. Thevalue network 515 may then forward this information to the policy network 510 to improve the actions taken by the policy network 510. Thus, the feedback from thevalue network 515 helps the policy network 510 learn which actions to take during SDR to HDR conversion. - The fully-
convolution A3C network 500 is trained to maximize a reward defined as a pixel level distance between the training SDR image and the target training HDR image. The reward is defined by an objective function that measures the similarity between the intermediate representation 520 and the corresponding training HDR image. -
FIG. 6 shows amethod 600 for using the neural network to convert SDR content to HDR content according to various exemplary embodiments. Themethod 600 provides an example of how a user may perform SDR to HDR conversion using the model learned by the neural network in accordance with themethod 400. Those skilled in the art will understand that the operations described below may be performed by any appropriate combination of hardware, software or firmware configured to implement the exemplary concepts described herein. Specific examples of the type of hardware, software or firmware that may be utilized will be described below with regard toFIG. 7 . - In 605, an input SDR image is received. As mentioned above, reference to an image is merely provided for illustrative purposes. Those skilled in the art will understand that the exemplary concepts described herein may be applicable to any appropriate type of image or video data.
- In 610, an output HDR Image is generated. As indicated above, this may include applying one or more predefined actions to the input SDR image. The actions applied by the neural network are based on the training performed using the training dataset.
- In 615, the one or more actions applied by the neural network are indicated to the user. For example, a GUI may display which user defined actions were applied to the input SDR image.
- In 620, user input is received. For example, a colorist may edit the actions displayed on the GUI to tune the HDR image in accordance with their artistic vision. Like the
method 300, in some embodiments, this may include editing a set of tonal curves displayed on the GUI. In 625, a modified HDR image is generated based on the user input. Thus, the user may retain artistic control over the conversion process by manually editing the tonal curves. -
FIG. 7 shows anHDR remastering device 700 according to various exemplary embodiments. TheHDR remastering device 700 may represent any electronic device that is configured with the hardware, software or firmware to generate HDR content from SDR content. For example, theHDR remastering device 700 may be a server, a desktop computer, a mobile phone, a tablet computer, a smartphone, an of Internet of Things (IoT) devices, etc. - The
HDR remastering device 700 may include aprocessor 705, adisplay device 710, an input/output (I/O)device 715, amemory arrangement 720, a communication interface 725, andother components 730. - The
display device 710 may be a hardware component configured to present a graphical display. The I/O device 715 may be a hardware component that enables the user to enter inputs. In one example, thedisplay device 710 and the I/O device 715 may be integrated together such as a touchscreen. In another example, the I/O device 715 may be represented as one or more buttons. The communication interface 725 may be a hardware component configured to communicate with a network over a wireless or wired connection. The communication interface 725 may also be configured to communicate with other devices using a wired or wireless connection. - The
processor 705 may be configured to execute a plurality of engines for theHDR remastering device 700. For example, the engines may include atraining engine 735, a conversion engine 740 and anartistic control engine 745. Thetraining engine 735 may perform a variety of different operations related to training a neural network. As indicated above, this may include training a neural network configured to automatically infer HDR content from input SDR content and training a neural network configured to generate an HDR image by applying one or more user defined actions to an input SDR image. The conversion engine 740 may perform SDR to HDR conversions using the exemplary technique describe herein. The artistic control engine 740 may generate for display a GUI that the user may interact with to modify an HDR image. For example, the GUI may display a tonal curve configured with multiple control points. The user input may manipulate the control points to alter the HDR image. There may be multiple tone curves for different color channels. - The above referenced engines each being an application (e.g., a program) executed by the
processor 705 is only exemplary. The functionality associated with the engines may also be represented as a separate incorporated component of theHDR remastering device 700 or may be a modular component coupled to theHDR remastering device 700, e.g., an integrated circuit with or without firmware. For example, the integrated circuit may include input circuitry to receive signals and processing circuitry to process the signals and other information. The engines may also be embodied as one application or separate applications. In addition, the functionality described for theprocessor 705 may be split among two or more processors. The exemplary embodiments may be implemented in any of these or other configurations of a monitoring device. - The
memory arrangement 730 may be a hardware component configured to store data related to operations performed by theHDR remastering device 700. Theother components 730 may include, for example, ports to electronically connect to other devices. - Those skilled in the art will understand that the above-described exemplary embodiments may be implemented in any suitable software or hardware configuration or combination thereof. An exemplary hardware platform for implementing the exemplary embodiments may include, for example, an Intel x86 based platform with compatible operating system, a Windows platform, a Mac platform and MAC OS, a Linux based OS, a mobile device having an operating system such as iOS, Android, etc. In a further example, the exemplary embodiments of the above described method may be embodied as a computer program product containing lines of code stored on a computer readable storage medium that may be executed on a processor or microprocessor. The storage medium may be, for example, a local or remote data repository compatible or formatted for use with the above noted operating systems using any storage operation.
- Although this application described various embodiments each having different features in various combinations, those skilled in the art will understand that any of the features of one embodiment may be combined with the features of the other embodiments in any manner not specifically disclaimed or which is not functionally or logically inconsistent with the operation of the device or the stated functions of the disclosed embodiments.
- It will be apparent to those skilled in the art that various modifications may be made in the present disclosure, without departing from the spirit or the scope of the disclosure. Thus, it is intended that the present disclosure cover modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalent.
Claims (17)
1-20. (canceled)
21: A method comprising:
obtaining multiple training images, wherein the training images include a training standard dynamic range (SDR) image and a training high dynamic range (HDR) image;
training a neural network, using the training images and a set of color grading actions, for converting SDR images into HDR images;
receiving an input SDR image; and
converting the input SDR image into the HDR image using the training images and one or more color grading actions from the set of color grading actions.
22: The method of claim 21 , further comprising:
receiving a user input to modify the one or more color grading actions; and
modifying the HDR image based on the user input.
23: The method of claim 21 , wherein training the neural network comprises:
applying a first color grading action from the set of color grading actions to the training SDR image;
wherein the first color grading action is selected based on the training HDR image.
24: The method of claim 21 , wherein the neural network is configured to extract contextual features or color features from the training SDR image.
25: The method of claim 21 , wherein the set of color grading actions includes at least one of adjusting brightness, adjusting contrast, adjusting color saturation or adjusting exposure.
26: A system comprising:
one or more processors configured to:
obtain multiple training images, wherein the training images include a training standard dynamic range (SDR) image and a training high dynamic range (HDR) image;
train a neural network, using the training images and a set of color grading actions, for converting SDR images into HDR images;
receive an input SDR image; and
convert the input SDR image into the HDR image using the training images and one or more color grading actions from the set of color grading actions.
27: The system of claim 26 , the one or more processors are further configured to:
receive a user input to modify the one or more color grading actions; and
modify the HDR image based on the user input.
28: The system of claim 26 , wherein training the neural network comprises:
applying a first color grading action from the set of color grading actions to the training SDR image;
wherein the first color grading action is selected based on the training HDR image.
29: The system of claim 26 , wherein the neural network is configured to extract contextual features or color features from the training SDR image.
30: The system of claim 26 , wherein the set of color grading actions includes at least one of adjusting brightness, adjusting contrast, adjusting color saturation or adjusting exposure.
31: A method comprising:
receiving a standard dynamic range (SDR) image;
converting the SDR image into a high dynamic range (HDR) image using a neural network trained to reconstruct a rolled off highlight in the HDR image that is not visible in the SDR image;
generating a set of tonal curves based on performing a regression on the HDR image;
receiving a user input to modify the set of tonal curves; and
modifying the HDR image based on the modified set of tonal curves.
32: The method of claim 31 , further comprising:
displaying a representation of the tonal curves that includes a control point, wherein the user input includes manipulating the control point.
33: The method of claim 31 , wherein the rolled off highlight is reconstructed in a region of the HDR image that corresponds to a region of the SDR image that includes a clipped light level caused by a dynamic range of luminosity of the SDR image.
34: The method of claim 31 , wherein the SDR image uses a first dynamic range of luminosity, and the HDR image uses a second dynamic range of luminosity, and wherein converting the SDR image into to the HDR image includes mapping a first value defined relative to the first dynamic range of luminosity to a second value defined relative to the second dynamic range of luminosity.
35: The method of claim 31 , wherein the SDR image uses a first color gamut and the HDR image uses a second color gamut, and wherein converting the SDR image into to the HDR image includes mapping a first value defined relative to the first color gamut to a second value defined relative to the second color gamut.
36: The method of claim 31 , wherein converting the SDR image into the HDR image includes:
generating a low dimensional latent representation of the SDR image; and
decoding the low dimensional latent representation of the SDR image into the HDR image.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/372,510 US20240013354A1 (en) | 2020-09-14 | 2023-09-25 | Deep SDR-HDR Conversion |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/948,336 US11803946B2 (en) | 2020-09-14 | 2020-09-14 | Deep SDR-HDR conversion |
US18/372,510 US20240013354A1 (en) | 2020-09-14 | 2023-09-25 | Deep SDR-HDR Conversion |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/948,336 Continuation US11803946B2 (en) | 2020-09-14 | 2020-09-14 | Deep SDR-HDR conversion |
Publications (1)
Publication Number | Publication Date |
---|---|
US20240013354A1 true US20240013354A1 (en) | 2024-01-11 |
Family
ID=80626878
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/948,336 Active 2040-11-12 US11803946B2 (en) | 2020-09-14 | 2020-09-14 | Deep SDR-HDR conversion |
US18/372,510 Pending US20240013354A1 (en) | 2020-09-14 | 2023-09-25 | Deep SDR-HDR Conversion |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/948,336 Active 2040-11-12 US11803946B2 (en) | 2020-09-14 | 2020-09-14 | Deep SDR-HDR conversion |
Country Status (1)
Country | Link |
---|---|
US (2) | US11803946B2 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020132371A1 (en) * | 2018-12-20 | 2020-06-25 | Warner Bros. Entertainment Inc. | Profile-based standard dynamic range and high dynamic range content generation |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7009733B2 (en) * | 2001-07-02 | 2006-03-07 | Coral Corporation | Manual correction of an image color |
KR20130031574A (en) | 2011-09-21 | 2013-03-29 | 삼성전자주식회사 | Image processing method and image processing apparatus |
US8965141B2 (en) | 2012-08-17 | 2015-02-24 | Thomson Licensing | Image filtering based on structural information |
BR112015011219A2 (en) | 2012-11-16 | 2017-07-11 | Thomson Licensing | high dynamic range image processing |
US9754356B2 (en) | 2013-04-12 | 2017-09-05 | Agency For Science, Technology And Research | Method and system for processing an input image based on a guidance image and weights determined thereform |
US20190080440A1 (en) | 2017-09-08 | 2019-03-14 | Interdigital Vc Holdings, Inc. | Apparatus and method to convert image data |
US11094043B2 (en) * | 2017-09-25 | 2021-08-17 | The Regents Of The University Of California | Generation of high dynamic range visual media |
US20210166360A1 (en) | 2017-12-06 | 2021-06-03 | Korea Advanced Institute Of Science And Technology | Method and apparatus for inverse tone mapping |
WO2020132371A1 (en) * | 2018-12-20 | 2020-06-25 | Warner Bros. Entertainment Inc. | Profile-based standard dynamic range and high dynamic range content generation |
JP2020115599A (en) * | 2019-01-17 | 2020-07-30 | ソニー株式会社 | Image processing control apparatus and image processing control method |
CN111754412B (en) * | 2019-03-29 | 2024-04-19 | Tcl科技集团股份有限公司 | Method and device for constructing data pair and terminal equipment |
WO2021247435A1 (en) * | 2020-06-05 | 2021-12-09 | Magic Leap, Inc. | Enhanced eye tracking techniques based on neural network analysis of images |
-
2020
- 2020-09-14 US US16/948,336 patent/US11803946B2/en active Active
-
2023
- 2023-09-25 US US18/372,510 patent/US20240013354A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US11803946B2 (en) | 2023-10-31 |
US20220084170A1 (en) | 2022-03-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Lv et al. | Attention guided low-light image enhancement with a large scale low-light simulation dataset | |
US10861133B1 (en) | Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium | |
Kim et al. | Global and local enhancement networks for paired and unpaired image enhancement | |
US9275445B2 (en) | High dynamic range and tone mapping imaging techniques | |
CN108335306B (en) | Image processing method and device, electronic equipment and storage medium | |
WO2021218119A1 (en) | Image toning enhancement method and method for training image toning enhancement neural network | |
CN113379775A (en) | Generating a colorized image based on interactive color edges using a colorized neural network | |
CN105850129B (en) | High dynamic range images are carried out with the method and apparatus of tone mapping | |
CN112200736B (en) | Image processing method based on reinforcement learning and model training method and device | |
US20240013354A1 (en) | Deep SDR-HDR Conversion | |
US11727543B2 (en) | Systems and methods for content-aware enhancement of images | |
Smelyakov et al. | Comparative efficiency analysis of gradational correction models of highly lighted image | |
US20220237905A1 (en) | Method and system for training a model for image generation | |
US11138693B2 (en) | Attention-driven image manipulation | |
KR101437626B1 (en) | System and method for region-of-interest-based artifact reduction in image sequences | |
CN111382647A (en) | Picture processing method, device, equipment and storage medium | |
Al Sobbahi et al. | Low-light image enhancement using image-to-frequency filter learning | |
Yadav et al. | Frequency-domain loss function for deep exposure correction of dark images | |
Kumar et al. | Dynamic stochastic resonance and image fusion based model for quality enhancement of dark and hazy images | |
US11468548B2 (en) | Detail reconstruction for SDR-HDR conversion | |
CN111861940A (en) | Image toning enhancement method based on condition continuous adjustment | |
Moran et al. | MTNet: a multi-task cascaded network for underwater image enhancement | |
Zhou et al. | An improved algorithm using weighted guided coefficient and union self‐adaptive image enhancement for single image haze removal | |
US20220398704A1 (en) | Intelligent Portrait Photography Enhancement System | |
CN110555799A (en) | Method and apparatus for processing video |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |