EP4677535A1

EP4677535A1 - Method and device for image energy reduction based on an invertible neural network

Info

Publication number: EP4677535A1
Application number: EP24708780.2A
Authority: EP
Inventors: Olivier Le Meur; Franck Aumont; Laurent Blonde; Claire-Helene Demarty; Erik Reinhard
Original assignee: InterDigital CE Patent Holdings SAS
Current assignee: InterDigital CE Patent Holdings SAS
Priority date: 2023-03-06
Filing date: 2024-02-29
Publication date: 2026-01-14
Also published as: CN120752665A; WO2024184196A1

Abstract

A method and device allow to derive an energy-reduced image from an original image, the energy-reduced image being self-contained and invertible, meaning that the original image can be reconstructed without any side information but only based on the energy-reduced image. The difference between an original image and an energy-aware image is that the values of the luminance or of the color components of the energy-aware image are globally lower than the values of the luminance or of the color components of the input image. The proposed solution is based on an invertible energy-aware network InvEAN. Forward mapping the InvEAN to an input original image determines an energy-aware image allowing to reduce the energy consumption at the display device while maximizing the QoE. Backward mapping the InvEAN to an energy-aware image will allow to reconstruct the original image. Both operations rely on the same InvEAN Invertible Neural Network trained on an image dataset according to a set of losses.

Description

METHOD AND DEVICE FOR IMAGE ENERGY REDUCTION BASED ON AN INVERTIBLE NEURAL NETWORK TECHNICAL FIELD At least one of the present embodiments generally relates to reducing energy consumption when displaying an image on a display device, and more particularly to a method and a device based on an invertible neural network allowing to generate a self-contained invertible energy-aware image from an original image and allowing to reconstruct the original image back from such self-contained invertible energy-aware image. BACKGROUND Reducing energy consumption of electronic devices has become a requirement not only for manufacturers of electronic devices but also to limit, as much as possible, the environmental impact and to contribute to the emergence of a sustainable display industry. The increase in display resolution from SD to HD, then to 4K and in the near future to 8K and beyond, as well as the introduction of high dynamic range imaging, has brought about a corresponding increase in energy requirements of display devices. This is not consistent with the global need to reduce energy consumption knowing that a huge number of devices has a display (i.e., TV, Mobile phones, tablets, etc.). Indeed, displays are the most important source of energy consumption, for consumer electronic devices, either battery-powered (e.g., smartphones, tablets, head- mounted displays, car display screens) or not (e.g., television sets, advertisement display panels). Different display technologies have been developed in the recent years. Although modern displays consume energy in a more controllable and efficient manner than older displays, they remain the most important source of energy consumption in a video chain. Organic Light Emitting Diode (OLED) is one example of display technology that is getting more and more popular because of numerous advantages compared to former technologies such as Thin-Film Transistor Liquid Crystal Displays (TFT-LCDs). Rather than using a uniform backlight, OLED displays are composed of individual LEDs as image pixels. OLEDs power consumption is therefore highly correlated to the image content and the power consumption for a given input image can be estimated by considering the values of the displayed image pixels. Since the display of images is the most important source of energy consumption of a display device, reducing overall energy consumption of such device can be obtained by displaying images that need less energy to be displayed, in other words, displaying images that have reduced luminance values or reduced color component values, in other words, images that are dimmed (i.e., darkened) compared to the original images. Such image is hereafter named as “energy-aware image”. This transformation into an energy-aware image should be achieved while preserving as much as possible the visual similarity with the original image to avoid hindering the user experience. Different methods may be used to produce an energy-aware image. These methods can be grouped into four categories. A first category of methods is based on histogram or Look- Up-Tables which reduce the energy by, for example, manipulating the histogram. It could be a decrease or a modification of the number of empty bins in an energy saving manner. A second category is relying on luminance-based or color-based transformation of original images. The resulting colors/luminance values are chosen so that they consume less energy when displayed, and according to another criterion, for example according to luminance and/or contrast-based Just-Noticeable-Difference (JND) models. A third category of methods requires side information, such as saliency maps, objective quality metrics, depth information or gaze tracking, for example to apply region-based image modifications that reduce the energy requirements. A fourth category of more recent methods leverages the power of deep networks. For this specific task, networks are trained in a fully unsupervised manner, as there is no ground truth. Deep networks are then used to maximize the quality of the output image under the constraint of the prior targeted power saving rate R. The reduction of energy consumption of a display device is often solved using a dimming map indicating in a pixel wise manner how each pixel luminance should be modified. Unfortunately, this requires transmitting such a dimming map all along the video chain as side- information using metadata for example. This can be costly in terms of bandwidth and convenience, can be difficult in particular use-cases, and the side-information can be also corrupted. In general, it is more efficient and practical not having to transmit any side information. SUMMARY Embodiments described hereafter have been designed with the foregoing in mind and introduce the notion of a self-contained invertible energy-aware image. This energy-aware image is obtained using an invertible energy-aware network in a forward manner to map an original image into an energy-aware image. The proposed solution also allows to reconstruct the exact original image by using the same invertible energy-aware network in a backward manner on an energy-aware image and without significant losses. A first aspect of at least one embodiment is directed to a method comprising obtaining an input image, determining a modified image of the input image by using an invertible neural network in a forward manner to map the input image into a modified image by modifying the values of the luminance or of the color components of the input image, wherein the modified image is self-contained and invertible, and providing the modified image. In a variant of the first aspect, the modification of the values of the luminance or of the color components is a global reduction of the sum of the values of the luminance or of the color components of the overall image. A variant of the first aspect further comprises obtaining the modified image, reconstructing the input image by using the same invertible neural network in a backward manner to map the modified image into a reconstructed image, and providing the reconstructed image. A second aspect of at least one embodiment is directed to an apparatus comprising a processor configured to obtain an input image, determine a modified image of the input image by using an invertible neural network in a forward manner to map the input image into a modified image by modifying the values of the luminance or of the color components of the input image, wherein the modified image is self-contained and invertible, and provide the modified image. In a variant of the second aspect, the modification of the values of the luminance or of the color components is a global reduction of the sum of the values of the luminance or of the color components of the overall image. A variant of the second aspect further comprises obtaining the modified image, reconstructing the input image by using the same invertible neural network in a backward manner to map the modified image into a reconstructed image, and providing the reconstructed image. In a variant of the first or the second aspects, the invertible neural network is trained according to a loss function that is a linear combination of a set of loss functions comprising: a mean absolute error loss function that characterizes the difference of luminance between an input image and a modified image, a mean absolute error loss function that characterizes the difference of luminance between an input image and a reconstructed image, a power loss function that characterizes the difference of power between an input image and a modified image, a structural similarity index measure loss function that characterizes the difference between an input image and a modified image; and a structural similarity index measure loss function that characterizes the difference between an input image and a reconstructed image. A third aspect of at least one embodiment is directed to a method comprising obtaining a first image, generating a second image by using an invertible neural network in a backward manner to map the first image into the second image, and providing the second image, wherein the invertible neural network has been trained according to a loss function and wherein the sum of the values of the luminance or of the color components of the second image is higher than the sum of the values of the luminance or of the color components of the first image. A fourth aspect of at least one embodiment is directed to an apparatus comprising a processor configured to obtain a first image, generate a second image by using an invertible neural network in a backward manner to map the first image into the second image, and provide the second image, wherein the invertible neural network has been trained according to a loss function and wherein the sum of the values of the luminance or of the color components of the second image is higher than the sum of the values of the luminance or of the color components of the first image. A fifth aspect of at least one embodiment is directed to a computer program comprising program code instructions executable by a processor, the computer program implementing at least the steps of a method according to the first aspect or one of its variants or to the third aspect. A sixth aspect of at least one embodiment is directed to a non-transitory computer readable medium comprising program code instructions executable by a processor, the computer program product implementing at least the steps of a method according to the first aspect or one of its variants or to the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 illustrates a block diagram of an example of an environment including a display device in which various aspects and embodiments are implemented. Figure 2 illustrates an example of distribution network based on an invertible energy- aware network for delivering images according to at least one embodiment. Figure 3 illustrates an example of overall system architecture using an invertible energy-aware network according to at least one embodiment. Figure 4 illustrates an example of internal architecture of an invertible energy-aware network according to at least one embodiment. Figure 5 illustrates an example process for training an invertible energy-aware network model according to at least one embodiment. Figure 6 illustrates an example process for generating an energy-aware image from an original image through a forward mapping of an invertible energy-aware network according to at least one embodiment. Figure 7 illustrates an example process for reconstructing an original image from an energy-aware image through a backward mapping of an invertible energy-aware network according to at least one embodiment. Figure 8 illustrates the objective performances of different versions of invertible energy-aware networks according to at least one embodiment. Figure 9 illustrates different versions of invertible energy-aware networks according to embodiments. Figure 10 illustrates the objective performance of a selected version of invertible energy-aware network according to at least one embodiment, in function of the energy reduction rate for the no-reference quality metric NIQE over the BSDS dataset. Figure 11 illustrates the impact in terms of PSNR and SSIM metrics of an iterative application of the invertible energy-aware networks according to at least one embodiment. Figure 12 illustrates an example of the influence of the JPEG compression on the invertible energy-aware network according to at least one embodiment. Figure 13 illustrates an example of observed actual energy reduction rate when using the invertible energy-aware network according to at least one embodiment. DETAILED DESCRIPTION Figure 1 illustrates a block diagram of an example of an environment including a display device in which various aspects and embodiments are implemented. In the depicted environment, a user interacts with the display device 100 that is connected to a data provider 180 through a communication network 150. The display device 100 comprises a processor 101. The processor 101 may be a general- purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor may perform data processing such as the pixel value reduction process 600 of figure 6. The processor 101 may be coupled to an input unit 102 configured to convey user interactions. Multiple types of inputs and modalities can be used for that purpose. Physical keypad or a touch sensitive surface are typical examples of input adapted to this usage although voice control could also be used. In addition, the input unit may also comprise a digital camera able to capture still pictures or video in two dimensions or a more complex sensor able to determine the depth information in addition to the picture or video and thus able to capture a complete 3D representation. The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Multiple types of displays can be used for that purpose such as a liquid crystal display (LCD) or organic light-emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to render sound data to be converted into audio waves through an adapted transducer such as a loudspeaker for example. The processor 101 may be coupled to a communication interface 105 configured to exchange data with external devices. The communication preferably uses a wireless communication standard to provide mobility of the display device, such as cellular (e.g., LTE) communications, Wi-Fi communications, and the like. The processor 101 may access information from, and store data in, the memory 106, that may comprise multiple types of memory including random access memory (RAM), read- only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, any other type of memory storage device. In embodiments, the processor 101 may access information from, and store data in, memory that is not physically located on the device, such as on a server, a home computer, or another device. The processor 101 may receive power from the power source 108 and may be configured to distribute and/or control the power to the other components in the device 100. The power source may be any suitable device for powering the device. As examples, the power source may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), and the like), solar cells, fuel cells, and the like. While the figure depicts the processor 101 and the other elements 102 to 108 as separate components, it will be appreciated that these elements may be integrated together in an electronic package or chip. It will be appreciated that the display device 100 may include any sub-combination of the elements described herein while remaining consistent with the embodiments described hereafter. The processor 101 may further be coupled to other peripherals or units not depicted in figure 1 which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals may include a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like. For example, the processor 101 may be coupled to a localization unit configured to localize the display device within its environment. The localization unit may integrate a GPS chipset providing longitude and latitude position regarding the current location of the display device but also other motion sensors such as an accelerometer and/or an e-compass that provide localization services. In at least one embodiment, the processor 101 of the display device 100 is configured to display on the display unit 103 an image according to embodiments described further below. In a first variant embodiment, the image 190 is obtained from the data provider 180 through the communication network 150. In a second variant embodiment, the image is obtained from the memory 106, stored for example after being captured by the input unit 102. Typical examples of device 100 are smartphones, tablets, laptops, external monitors, head-mounted displays, television set, video projectors, computer screens, vehicles (e.g., control and/or entertainment systems for cars, planes, boats, etc.), advertisement display panels, medical monitors, etc. However, any device or composition of devices that provides similar functionalities can be used as display device 100 while still conforming with the principles of the disclosure. In at least one embodiment, the device does not include a display unit but prepares data for display so that another device, such as a screen, can perform the display. Example of such devices are set top boxes, media players, desktop computers, encoders, decoders, servers, computing grids, cloud computers, etc. The communication network 150 preferably uses a communication standard to provide interoperability between content provider and display devices. Such communication standard may be wireless, such as cellular (e.g. LTE) communications, Wi-Fi communications, and the like, to ensure the mobility of the display device. Cable, satellite or terrestrial digital television broadcast communication may also be used for the communication network 150 as well as broadband television communications. Such digital television standards may on based on well- established standards like DVB, ATSC, or the like. General purpose network standards may also be used, for example based on Ethernet. The design of the proposed solution has been driven by several requirements and constraints. First, the solution should preferably preserve as much as possible the visual quality of the energy-aware image while reducing its energy consumption on displays. Second, the solution should be invertible so that the original image may be reconstructed from the energy- aware image. Thirdly, the reconstruction process should not require any side information such as metadata associated to the energy-aware image. Embodiments below describe a new method for deriving an energy-reduced image from an original image, the energy-reduced image being self-contained and invertible, meaning that the original image can be reconstructed without any side information but only based on the energy-reduced image. The difference between an original image and an energy-aware image is that the values of the luminance or of the color components of the energy-aware image are globally lower than the values of the luminance or of the color components of the input image. In other words, the energy-aware image is for example globally dimmed in comparison to the original image and as a result of that dimming, it will need less energy for its display on a screen. Other techniques may be used for modifying the image while providing the same results. However, despite the modifications, the visual similarity between the images is kept very high to ensure a satisfying quality of experience for the viewer. For that purpose, an invertible energy-aware network (InvEAN) that is based on an Invertible Neural Network (INN) framework has been designed. Performing a forward mapping of the original image using the invertible energy-aware network determines an energy-aware image allowing to reduce the energy consumption at the display device while maximizing the visual similarity. Embodiments below also describe a new method for reconstructing the original image from an energy-reduced version of the original image. Indeed, performing a backward mapping of the energy-aware image using the invertible energy-aware network (also known as reverse mapping) will allow to reconstruct the original image with maximal similarity. Both operations rely on the same InvEAN Invertible Neural Network but are used either in forward mapping or in backward mapping modes. Compared to the conventional methods for reducing the luminance of an image, the embodiments present several advantages. The proposed approach uses invertible neural networks to propose a full invertible approach. It may rely on a deep-learning network for example. The invertible neural network targets a specific energy reduction rate. However, no side-information is required to retrieve the original image. In addition, it can be applied in an iterative manner to target higher reduction rates. Embodiments are described herein as an image-based solution. However, the same principles can be applied to other type of media such as videos, immersive 3D scenes, 360° image or video, point clouds or other image-based media. In the case of video for example, a simple frame by frame processing can be envisioned, enhanced with further temporal filtering of the frame-by-frame energy-aware images generated by an InvEAN. Figure 2 illustrates an example of distribution network based on an invertible energy- aware network for delivering images according to at least one embodiment. In such a distribution network 200, different usages of an InvEAN exist, involving different types of devices and configurations. An encoding device 220 transforms an original image 201 into an energy-aware image 202 by performing a forward-mapping operation 221 of the original image 201 using the invertible energy-aware network 290. Such a transformation is for example operated by content provider 180 of figure 1. The energy-aware image 202 is then distributed conventionally, for example through communication network 150 of figure 1 and does not require any side information or metadata for guiding the reconstruction. A first display device 230 receives the energy-aware image 202 and displays it 203 directly. A second display device 240 receives the same energy-aware image 202, reconstructs the original image by performing a backward mapping 241 of the energy-aware image 202 using the invertible energy-aware network 290 and displays the reconstructed original image 204. As a result, the energy consumption of the second identical device 240 will be higher than the energy consumption of the first device: firstly, because the image displayed is itself less energy-aware and secondly because the reconstruction process consumes energy to recover the original image. A third display device 250 receives the original image 201 and transforms it into an energy-aware image 205 by performing a forward mapping operation 251 of the original image 201 using the invertible energy-aware network 290. The energy-aware image 205 is generated and may be displayed. A fourth display device 260 receives the original image 201 and transforms it into an enhanced image 206 by performing a backward mapping operation 261 of the original image 201 using the invertible energy-aware network 290. The enhanced image 206 is generated and may be displayed. Unlike the energy-aware images generated by devices 220 and 250, the enhanced image 206 may require more energy to be displayed. Indeed, for such image, the sum of the values of the luminance or of the color components of the enhanced image 206 is higher than the sum of the values of the luminance or of the color components of the original image 201. Figure 3 illustrates an example of overall system architecture using an invertible energy-aware network according to at least one embodiment. The problem to solve is the following. From an original color image ^^^^ ∈ ℝ ^{^^^^× ^^^^× 3} (W being the width of the image and H being the height of the image), the objective is to determine an energy-aware image ^^^^^∗ ∈ ℝ ^{^^^^× ^^^^× 3}, such that the estimated power consumption of ^^^^^∗ is less than the original one given a reduction ratio called R, while having the highest possible quality. For this purpose, an invertible (energy-aware) neural network 310 is defined, allowing to learn a bijective mapping as ^^^^^∗ = ^^^_Θ^ ( ^^^^) , where Θ is the set of trainable parameters of the network. This approach provides a reversible process allowing to generate an energy-aware image 302 from an original image 301 but also to reconstruct a pristine original image 303 from an energy-aware image 302. The process of generating the energy-aware image uses a forward mapping 330 while the reconstruction from the energy-aware image uses a backward mapping 360. Both operations use the same invertible energy-aware network 310. The proposed InvEAN network 310 is operating on the luminance component and is conditioned by chromatic components of the image, extracted using conventional RGB to YUV functions 320, 350. The output is a modified version of such luminance so that the RGB color image is reconstructed thanks to a luminance ratio as described below: where, ^^^^ _^^^^ represent input pixels ( ^^^^ _^^^^, ^^^^ _^^^^, ^^^^ _^^^^) at spatial location i. ^^^^ _^^^^ and ^^^^ _^^^^ ^∗ represent their corresponding original luminance and the modified luminance, respectively. These operations are depicted in the figure by the elements 340 and 370. The model of the InvEAN is trained according to loss functions 380 based on an original image 301 hereafter noted ^^^^, the corresponding energy-aware image 302 hereafter noted ^^^ _^^_{^^^ ^^^^ ^^^^} and the corresponding reconstructed image 303 hereafter noted ^^^^ _{^^^^ ^^^^ ^^^^}. The loss functions are fundamental to optimize the training according to two main goals. First, running forwardly (performing a forward mapping) the InvEAN network, hereafter noted as ^^^_Θ^ , should allow to obtain an energy-aware image and second, running backwardly (performing a backward mapping) the InvEAN network, hereafter noted as ^^^_Θ^⁻¹, should allow to retrieve the original image. This can be expressed as: ^^^^ _{^^^^ ^^^^ ^^^^} = ^^^_Θ^⁻¹( ^^^ _^^_{^^^ ^^^^ ^^^^}) The learning may be constrained such that the image ^^^^ _{^^^^ ^^^^ ^^^^} stays as close as possible to the input image ^^^^ , but with a power constraint. In other words, the forward loss can be expressed as: For the backward pass, the goal is to retrieve the original image with the maximal similarity. In other words, the backward loss can be expressed as: The forward loss and the backward loss both use a Mean Absolute Error (MAE) that characterizes the difference of luminance between an original image and the reconstructed image for all the pixels of the images. This loss can be expressed as: w_{here ^^^^ is the original image and ^} ^� _{^^^ is the reconstructed image, ^^^^ the total number} of pixels in the image, and i is the spatial coordinate of the pixel. The power loss characterizes the difference of power between an original image and the corresponding modified image for all the pixels of the images. This loss can be expressed as: with: where N is the number of pixels in the image. The structural similarity index measure (SSIM) loss characterizes the difference between an input image and the corresponding modified image. This measure is based on three comparison measurements (i.e., luminance, contrast and structure) and relies on local average, local variance and local covariance. The loss is given by one minus the SSIM value and can be expressed as: ℒ_{^^^^ ^^^^ ^^^^ ^^^^ = 1 − ^^^^ ^^^^ ^^^^ ^^^^( ^^^^, ^} ^� _^^^) where SSIM is the well-known full-reference quality metric proposed in Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." IEEE transactions on image processing 13.4 (2004): 600-612. SSIM is in the range [0,1], where 1 indicates the maximum value. Finally, the total loss minimized during the training is expressed as: The losses are described above as based on luminance but may also be based on color components. The proposed architecture for the InvEAN 310 is inspired by the invertible U-shaped network proposed in Zhao et al., Invertible image decolorization. IEEE Transactions on Image Processing, 30:6081–6095, 2021 and further described in figure 4. Figure 4 illustrates an example of internal architecture of an invertible energy-aware network according to at least one embodiment. This architecture 400 is based on successive layers of Haar wavelet transformations (downscaling 401, 403 and upscaling 405, 407) and conditional double-side affine coupling layers 402, 404, and 406. The same architecture and model are used for forward mapping and for reverse mapping. The difference between forward and reverse mappings are in the formulation of the double-side affine coupling layers: the layer 440 is used for the forward mapping while the layer 441 is used for the backward mapping. The dyadic Haar wavelet transformation is first used in downscaling manner to split (in the left part 401, 403) the image into low and high-frequency wavelet sub-bands and second in upscaling manner, thus concatenating the sub-bands together in a single channel (in the right part 405, 407). In the proposed architecture, one level of wavelet transformation provides one low-frequency sub-band and three high-frequency sub-bands. Considering an input feature ^^^^ _{^^^^ ^^^^} the dyadic Haar transformation provides ^^^ _^^_{^^^ ^^^^ ^^^^} ∈ with H the height in pixels of the image, W the width in pixels of the image, and C the number of color components (typically 3 color components) of the image. In other words, a ‘down’ version of the Haar transform is used to transform N input channels into 4*N output channels with resolutions four times smaller (total number of pixels is unchanged) and an ‘up’ version of the Haar transform is used to transform 4*N input channels into N channels with a resolution four times bigger. There are several benefits to use such transformation. First, the Haar transformation is simple, efficient and fully invertible. Second, the use of Haar transformation is an elegant way to reduce the computational complexity and is less demanding on memory consumption. The affine coupling layer is the key ingredient of InvEAN since it defines a bijective mapping between its input and its output distributions, thus allowing the reconstruction. In order to facilitate both the structure preservation and the energy reduction, the four sub-band channels from a Haar transformation are split in the affine coupling layers into one low- frequency channel, and the three high-frequency channels are combined in a single layer channel. In the forward affine coupling 440, the input ^^^^ is split into two parts [ ^^^^_LF, ^^^^_HF] as described above and an affine transformation conditioned by ^^^^ is applied as: ^_{^^^HF = ^^^^HF ⊙ exp� ^^^^2} ⁽ _{^^^^LF1, ^^^^} ⁾ _{� + ^^^^2( ^^^^LF, ^^^^)} where, ⊙ represents the element-wise multiplication. ^^^ _^^_^^^ and ^^^^ _^^^^ are corresponding respectively to the shift (transform) and scale coefficients of an arbitrary network. A five-layer densely connected convolutional block is used to learn the two coefficients independently. The conditional version of the coupling block is obtained by concatenating data ^^^^ to the subnetworks ^^^ _^^_^^^ and ^^^^ _^^^^. In the backward affine coupling 441, the backward path is simply computed as: ^^^^_HF = ( ^^^^_HF − ^^^^₂( ^^^^ _{^^^^ ^^^^} , ^^^^)) ⊘ exp ( ^^^^₂( ^^^^ _{^^^^ ^^^^}, ^^^^)) ^^^^_LF = ( ^^^^_LF − ^^^^₁( ^^^^ _{^^^^ ^^^^} , ^^^^)) ⊘ exp ( ^^^^₁( ^^^^_HF, ^^^^)) where, ⊘ represents the element-wise division. ^^^ _^^_^^^ and ^^^^ _^^^^ are the same scale and shift coefficients as used for the forward mapping. The equations above for the affine couplings 440 and 441 are using the conventional notation where ^^^^ is an input and ^^^^ an output. These letters are not related to the chrominance elements of the image. The affine coupling being used in both forward and backward mapping is thus a double- side affine coupling. In addition, the affine coupling is conditioned by the color components, thus making it a conditioned double-side affine coupling. In another embodiment, the affine coupling is not conditioned by the color components. This conditioning is the simple concatenation of conditioning data ^^^^ to input of subnetworks ^^^ _^^_^^^ and ^^^^ _^^^^ does not prevent the proposed architecture from being invertible. Indeed, the key point of affine coupling layer is that the subnetworks ^^^ _^^_^^^ and ^^^^ _^^^^ are never inverted. Conditioning data ^^^^ is for example directly based on the chrominance components U and V or based on characteristics extracted from the chrominance components. As introduced above, the architecture is based on a succession of layers. Different cardinalities of these layers have been experimented, as shown in figures 8 and 9. The only differences between a forward and a reverse mapping of the InvEAN is the use of a forward affine coupling 440 or of a backward affine coupling 441 and the use of an energy-aware image for the backward mapping instead of using an original image for the forward mapping. The learned parameters of the model are the same. Figure 5 illustrates an example process for training an invertible energy-aware network model according to at least one embodiment. Such process 500 is for example implemented by a processor 101 of device 100. In step 510, the processor obtains a set of images or patches to be used as a training data set. In step 520, parameters of the lightweight deep learning model are learned by iterating (step 521) on the images or patches of the data set, while minimizing the loss functions described above. This results into a trained model for the InvEAN that can be further provided to another device or unit, in step 530, to be used for generating an energy- aware image from an original image, as described in figure 6, or for reconstructing an original image from an energy-aware image, as described in figure 7. Figure 6 illustrates an example process for generating an energy-aware image from an original image through a forward mapping of an invertible energy-aware network according to at least one embodiment. Such process 600 is for example implemented by a processor 101 of device 100 of figure 1, or by a device 220 or 250 of figure 2. In step 610, an original input image is obtained. In step 620, an energy-aware image is generated from the original input image by performing a forward mapping of the InvEAN. In step 630, the energy-aware image is provided, for example to be transmitted (device 220 of figure 2) over a communication network for further display or to be displayed directly on a screen (device 250 of figure 2). Compared to displaying the original input image, the display of the energy-aware image allows to reduce the energy consumption of the display device, while keeping a satisfying quality of experience. Figure 7 illustrates an example process for reconstructing an original image from an energy-aware image through a backward mapping of an invertible energy-aware network according to at least one embodiment. Such process 700 is for example implemented by a processor 101 of device 100 of figure 1, or by a device 240 of figure 2. In step 710, an energy- aware image is obtained. In step 720, an image is reconstructed from the energy-aware image by performing a backward mapping of the InvEAN. In step 730, the energy-aware image is provided, for example to be displayed directly on a screen. Compared to displaying the energy- aware image, the display of the reconstructed image increases the energy consumption of the display device but restores back the original image with as much fidelity as possible. Figure 8 illustrates the objective performances of different versions of invertible energy-aware networks according to at least one embodiment. The InvEAN architecture and process has been assessed on the BSDS dataset that is composed of 300 images, 200 training images, 40 validation images and 60 for testing. Images of this dataset have a resolution of 481 by 321 pixels, in a landscape or portrait format. These images have been cropped into patches of 128 by 128 pixels which undergo random data augmentation (i.e., horizontal flip, vertical flip and rotation of 90 degrees), leading to a total number of images of 8000 patches (40 epochs with 200 patches by epoch). The network is trained using the following parameters: ADAM solver, learning rate of 0.001, weight decay of 0.0001, batch size of 8. The coefficients of the ^{linear combination of the loss functions were empirically set to} ^{= 1, ^^^^} ₂ ^{= 1, ^^^^} ₃ ^{= 4, ^^^^} ₄ ⁼ _{1, ^^^^5 = 1}.} The proposed InvEAN approach is implemented in PyTorch. To evaluate different architecture variants for the InvEAN, different numbers of downscaling and upscaling invertible Haar wavelet transforms and conditional double-side affine coupling layers are used. For each of these layers, the conditioning chromatic data ^^^^ needs to be rescaled in order to fit the feature map resolution. The rescaling is performed by a bicubic interpolation. The scale and shift coefficients in each coupling layer are learned thanks to a five-layer densely connected convolutional block. Each convolutional filter is of size 3 × 3, with padding 1, followed by a leaky ReLU activation layer with negative slope equal to 0.2. The intermediate channel number of these convolutional blocks is fixed to 32. During the training, to account for quantization effect, a quantization operation is performed after the forward pass or before proceeding to the backward pass. This simulates the real application in which the energy-aware image is stored in an 8-bit unsigned integer format. In figure 8, the following three full-reference conventional quality metrics are used: PSNR, SSIM and VIF. The table shows the average quality scores for these three metrics, computed over three different datasets: the BSDS dataset (top section of the table) comprising 60 images, the Kodak24 dataset (middle section of the table) comprising 24 images, and the VOC2021 dataset (lower section of the table) comprising 400 images. The evaluation is done for different variants of the InvEAN in order to highlight the influence of the number of downsampling operations, of the number of invertible blocks between two downsampling operations and of the conditioning. D[x] I[a,b] stand for D=Number of Down(up)sampling; I=Number of invertible blocks per down(up)sampling. For instance, D[2] I[1,1] describes an architecture with two down(up)sampling operations, and one invertible block per down(up)sampling level. C indicates that conditional double-side affine coupling layer is used. The bold line indicates the best compromise when regarding quality and architecture complexity. The evaluation is also done for different reduction rates varying between 10%, 20%, 40% and 60%. The best tradeoff between objective quality and architecture complexity is the configuration D[2] I[1,1]+C (highlighted in bold text), which corresponds to the architecture illustrated in Figure 4, with two down(up) sampling operations and one conditional double- side affine coupling layer per level of spatial resolution. The conditioning of the double-side affine coupling layer allows to improve globally the performance. This is especially noticeable for the VIF metric. In summary, the D[2] I[1,1]+C architecture provides in average the best compromise even if better results can be observed for specific cases and reduction rates. For the BSDS dataset, the table also presents the performance of the proposed method when performing the reverse mapping (i.e., the backward pass). Results indicate that the proposed InvEAN method succeeds in recovering the original image with a high quality, whatever the reduction rate is. The average PSNR value is equal to 38dB and the average SSIM is equal to 0.98. The average VIF scores vary a bit more going from 0.94 to 0.9 for the lowest to the highest reduction rate respectively. In addition, the proposed method outperforms a method that would be based on linear scaling for the three full-reference objective quality metrics. When considering the R-ACE network as proposed in “R-ACE network for OLED image power saving” by Nugroho and Ruan, in 2022 IEEE 4th Global Conference on Life Sciences and Technologies (LifeTech), pages 284–285 IEEE, 2022, a similar trend is observed, although less important than the previous one and except for the scores obtained with R=10%. The performance of the proposed method is compared to the linear scaling and R-ACE network, respectively with regards to the BSDS dataset, the Kodak24 dataset and the VOC2012 dataset. The numbers suggest that the proposed method provides much better results than the linear scaling and R-ACE network. Beyond this point, it demonstrates the good generalization ability of the proposed method. Figure 9 illustrates different versions of invertible energy-aware networks according to embodiments. In this figure, a rectangle marked with ‘H’ represents a Haar wavelet function while a rectangle marked with ‘A’ represents a double-side affine coupling. The InvEAN 901 is a simple one with a single Haar wavelet function and a single double-side affine coupling. It corresponds to the D[1] I[1] used in the first line of figure 8. The InvEAN 902 is corresponds to the D[2] I[1,1] used in the third line of figure 8. The InvEAN 903 is corresponds to the D[2] I[2,2] used in the fifth line of figure 8. Figure 10 illustrates the objective performance of a selected version of invertible energy-aware network according to at least one embodiment, in function of the energy reduction rate for the no-reference quality metric NIQE over the BSDS dataset. With this quality metric, the higher the better. The proposed method when using D[2] I[1,1]+C architecture provides better results than the linear scaling and the R-ACE method. It is also interesting to note that the average NIQE scores stay also constant for the different rate reductions. In summary, the proposed method outperforms the simple linear scaling and provides competitive performances compared to the R-ACE method. Figure 11 illustrates the impact in terms of PSNR and SSIM metrics of an iterative application of the invertible energy-aware networks according to at least one embodiment. Indeed, performing the InvEAN forward mapping operation iteratively is one solution to generate an energy-aware image with lower energy. Instead of using different networks trained for different energy reduction rate, the network trained for a 10% energy reduction may be applied several times iteratively to obtain an energy-aware image with a reduction greater than 10%. Formally, the iterative process, for the forward pass and the backward pass, respectively correspond to: The figure presents the distortion with regards to energy reduction curves (PSNR for curve 1101 and SSIM for curve 1102) when considering 1, 2, 5 and 10 iterations of a network trained for an energy reduction rate of 10% and iteratively run several times. One iteration corresponds to the baseline method. The diagram also shows the four control points obtained by InvEAN networks trained for energy reduction rates of respectively 10%, 20%, 40% and 60%. The two energy reduction / PSNR curves 1101, i.e., one for independent inferences of energy-aware images and the other for the iterative process, are very similar, suggesting that the two approaches behave similarly. No similar trend is observed when considering the average SSIM scores in curve 1102. Indeed, there is an expected advantage when inferring the energy-aware images with dedicated networks. However, the average SSIM scores do not significantly drop down, indicating that the iterative approach is a good candidate to adjust the target of energy reduction. Therefore, one network trained for the energy reduction rate of 10% could be used in an iterative manner to target energy reduction rates for example between 10% and around 35% with 2 to 5 iterations. Figure 12 illustrates an example of the influence of the JPEG compression on the invertible energy-aware network according to at least one embodiment. The impact is evaluated in both directions: forward on the first line and backward on the second line. The first line is obtained by computing the PSNR/SSIM metrics between the original image and compressed versions of the energy-aware image into JPEG with different quality factors. The second line is obtained computing the PSNR/SSIM metrics between the original image and the reconstructed images obtained by applying the reverse InvEAN network on the compressed energy-aware images to recover the input image and determining the PSNR and SSIM scores. For these measures, the reduction rate is set to 20% and the Kodak24 dataset is used. As expected, the quality of both forward and backward images decreases with the JPEG compression. It is interesting to note that, although not perfectly robust to JPEG compression, the performance of the InvEAN network stays consistent with the amount of compression. More importantly, the quality of backward images does not collapse at all. Figure 13 illustrates an example of observed actual energy reduction rate when using the invertible energy-aware network according to at least one embodiment. The energy consumption of the test images and their corresponding processed versions by the proposed method on a SONY OLED KD – 55AF9 display (55’’, HD) is measured. Figure 13 presents the observed actual energy reduction rate for different targets of energy reduction and for the linear scaling (left side), R-ACE network (center) and the proposed method (right side). As expected, the actual energy rate reduction increases with the target of energy reduction. The three tested methods behave similarly. It is however noticed that the actual energy consumption reduction is much smaller than the target energy consumption. This discrepancy is likely due to the energy model we use. This model assumes that there is a linear relationship between the luminance of the image and the power consumption. This assumption holds for RGB OLED screens, but not for other technologies such as RGBW OLED screens, to which our screen used for testing belongs to. It would therefore be necessary to revisit the energy model to control the final achieved energy consumption in a finer way. In terms of complexity, the number of trainable parameters is 806184. For an input image of size 128 × 128 and without any optimization, the number of FLOPS is equal to 5.297GFlops. Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.1 kgCO2eq/kWh. A cumulative of 50 hours of computation was performed on hardware of type A100 PCIe 40/80GB (TDP of 250W). Total emissions are estimated to be 1.25 kgCO2eq of which 0 percents were directly offset. Although different embodiments have been described separately, any combination of the embodiments together can be done while respecting the principles of the disclosure. The terms “method” and “process” are used interchangeably. Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, mean that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment. Additionally, this application or its claims may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Additionally, this application or its claims may refer to “obtaining” various pieces of information. Obtaining is, as with “accessing”, intended to be a broad term. Obtaining the information may include one or more of, for example, receiving the information, accessing the information, or retrieving the information (for example, from memory or optical media storage). Further, “obtaining” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information. It is to be appreciated that the use of any of the following “/”, “and/or”, and “at least one of”, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and/or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.

Claims

CLAIMS 1. A method comprising: obtaining an input image; determining a modified image of the input image by using an invertible neural network in a forward manner to map the input image into a modified image by modifying the values of the luminance or of the color components of the input image, wherein the modified image is self-contained and invertible; and providing the modified image.

2. The method of claim 1, wherein the modification of the values of the luminance or of the color components is a global reduction of the sum of the values of the luminance or of the color components of the overall image.

3. A method comprising obtaining a first image; generating a second image by using an invertible neural network in a backward manner to map the first image into the second image; and providing the second image, wherein the modification of the values of the luminance or of the color components is a global increase of the sum of the values of the luminance or of the color components of the overall image.

4. The method of any of the preceding claims, wherein the invertible neural network has been trained according to a combined loss function that is a linear combination of a set of loss functions comprising: a mean absolute error loss function that characterizes the difference of luminance between an input image and a modified image; a mean absolute error loss function that characterizes the difference of luminance between an input image and a reconstructed image; a power loss function that characterizes the difference of power between an input image and a modified image; a structural similarity index measure loss function that characterizes the difference between an input image and a modified image; and a structural similarity index measure loss function that characterizes the difference between an input image and a reconstructed image.

5. The method of any of the preceding claims, wherein the architecture of the invertible neural network comprises a succession of layers comprising at least: a Haar wavelet downscaling layer; a double-side affine coupling; and a Haar wavelet upscaling layer.

6. The method of claim 5, wherein the Haar wavelet downscaling layer transforms an input into one low frequency sub-band and three high frequency sub-bands.

7. The method of any of claim 5 or 6, wherein the double-side affine coupling combines the three high frequency sub-bands together.

8. The method of any of claims 5 to 7, wherein the double-side affine coupling performs operations on the low frequency sub-band and high frequency combined sub-band based on scale and shift operations of a deep learning network.

9. The method of any of claims 5 to 8, wherein the double-side affine coupling is conditioned by color components.

10. An apparatus comprising a processor configured to: obtain an input image; determine a modified image of the input image by using an invertible neural network in a forward manner to map the input image into a modified image by modifying the values of the luminance or of the color components of the input image, wherein the modified image is self-contained and invertible; and provide the modified image.

11. The apparatus of claim 10, wherein the modification of the values of the luminance or of the color components is a global reduction of the sum of the values of the luminance or of the color components of the overall image.

12. An apparatus comprising a processor configured to: obtain a first image; generate a second image by using an invertible neural network in a backward manner to map the first image into the second image; and provide the second image, wherein the modification of the values of the luminance or of the color components is a global increase of the sum of the values of the luminance or of the color components of the overall image.

13. The apparatus of any one of claims 10 to 12, wherein the invertible neural network has been trained according to a loss function that is a linear combination of a set of loss functions comprising: a mean absolute error loss function that characterizes the difference of luminance between an input image and a modified image; a mean absolute error loss function that characterizes the difference of luminance between an input image and a reconstructed image; a power loss function that characterizes the difference of power between an input image and a modified image; a structural similarity index measure loss function that characterizes the difference between an input image and a modified image; and a structural similarity index measure loss function that characterizes the difference between an input image and a reconstructed image.

14. The apparatus of any one of claims 10 to 13, wherein the architecture of the invertible neural network comprises a succession of layers comprising at least: a Haar wavelet downscaling layer; a double-side affine coupling; and a Haar wavelet upscaling layer.

15. The apparatus of claim 14, wherein the Haar wavelet downscaling layer transforms an input into one low frequency sub-band and three high frequency sub-bands.

16. The apparatus of any one of claims 14 or 15, wherein the double-side affine coupling combines the three high frequency sub-bands together.

17. The apparatus of any one of claims 14 to 16, wherein the double-side affine coupling performs some operations on the low frequency sub-band and high frequency combined sub- band based on scale and shift operations of a deep learning network.

18. The apparatus of any one of claims 14 to 17, wherein the double-side affine coupling is conditioned by color components.

19. A computer program comprising program code instructions for implementing the method according to any of claims 1 to 9 when executed by a processor.

20. A non-transitory computer readable medium comprising program code instructions for implementing the method according to any of claims 1 to 9 when executed by a processor.