WO2024003589A1 - Method for processing an image and corresponding image processing device and systems - Google Patents

Method for processing an image and corresponding image processing device and systems Download PDF

Info

Publication number
WO2024003589A1
WO2024003589A1 PCT/IB2022/000382 IB2022000382W WO2024003589A1 WO 2024003589 A1 WO2024003589 A1 WO 2024003589A1 IB 2022000382 W IB2022000382 W IB 2022000382W WO 2024003589 A1 WO2024003589 A1 WO 2024003589A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel values
map
image
banding
pixel
Prior art date
Application number
PCT/IB2022/000382
Other languages
French (fr)
Inventor
Foteini Tania Pouli
Charles BONNINEAU
Edern LE BOT
Original Assignee
Fondation B-Com
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fondation B-Com filed Critical Fondation B-Com
Priority to PCT/IB2022/000382 priority Critical patent/WO2024003589A1/en
Publication of WO2024003589A1 publication Critical patent/WO2024003589A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/90Dynamic range modification of images or parts thereof
    • G06T5/92Dynamic range modification of images or parts thereof based on global image properties
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/20Image enhancement or restoration using local operators
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/60Image enhancement or restoration using machine learning, e.g. neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10024Color image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20016Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20172Image enhancement details
    • G06T2207/20208High dynamic range [HDR] image processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging

Definitions

  • the invention relates to the field of image processing.
  • the invention relates to a method for processing an image, to a corresponding image processing device and to corresponding systems.
  • This conversion may be performed for each pixel of the image by transforming the luminance value for this pixel using a function that maps the first dynamic range into the second dynamic range.
  • This function is generally continuous and increasing in order not to alter the appearance of the image; the function may however be nonlinear to better take advantage of the spread of the second dynamic range.
  • the invention provides a method for processing an image represented by first pixel values having a first dynamic range and displayable based on second pixel values respectively corresponding to the first pixel values and having a second dynamic range larger than the first dynamic range, said method including producing, based at least on the first pixel values, a banding map identifying areas where at least a false contour appears when the image is displayed based on the second pixel values but does not appear when the image is displayed based on the first pixel values.
  • banding map interestingly identifies regions where a false contour or banding is likely occur, owing to the dynamic range expansion, when displaying the image using the second pixel values.
  • This banding map may be used for instance to pre-process the image to avoid these banding artifacts upon display, as further explained below.
  • the production of said banding map comprises the following steps:
  • the step of producing a first map, or respectively a second map may for instance comprise the following sub-steps:
  • - identify a pixel as an area of visible contrast if a luminance gradient magnitude at the concerned pixel exceeds the minimum visible contrast determined for the concerned pixel.
  • Each second pixel value may for instance be computed by applying a non-linear, continuous and increasing function to a corresponding first pixel value, thereby obtaining the above-mentioned dynamic range expansion.
  • the method may include a step of training an artificial neural network such as to reduce a distance between said banding map and a third map produced at an output of the artificial neural network when the first pixel values are applied at an input of the artificial neural network.
  • the produced banding map aims at training an artificial neural network.
  • said banding map is produced at an output of an artificial neural network receiving said first pixel values at an input of the artificial neural network.
  • This artificial neural network may for instance be the artificial neural network trained as mentioned above.
  • the method may comprise a step of processing first pixel values respectively associated with pixels lying in areas identified in said banding map.
  • the banding map may thus be used in this case to select pixels which should be (pre-)processed to avoid the production of false contours or banding artifacts when the image is displayed using the second pixel values, i.e. with an expanded dynamic range.
  • the method may comprise a step of sending the processed first pixel values to a client device.
  • the method may comprise a step of sending the first pixel values and data representative of said banding map to a client device.
  • the step of processing said first pixel values respectively associated with pixels lying in areas identified in said banding map may for instance be implemented by the client device.
  • the client device may then convert the processed first pixel values into third pixel values having the second dynamic range.
  • the method may also comprise a step of displaying an image using the third pixel values, e.g. on a high dynamic range display apparatus..
  • the invention also provides an image processing device configured or programmed to implement a method as presented above.
  • the invention provides a system comprising:
  • a server configured to implement a method as presented above and including a step of processing first pixel values respectively associated with pixels lying in areas identified in said banding map, the server including a communication circuit configured to send the processed first pixel values to a client device;
  • the client device including a communication circuit to receive the processed first pixel values and a processor configured to convert the processed first pixel values into third pixel values having the second dynamic range.
  • the invention provides a system comprising:
  • a server configured to implement a method as presented above, the server including a communication circuit configured to send the first pixel values and data representative of the banding map to a client device;
  • the client device including a communication circuit to receive the first pixel values and said data representative of the banding map, the client device being configured to process first pixel values respectively associated with pixels lying in areas identified in the banding map and to convert the processed first pixel values into third pixel values having the second dynamic range.
  • Both of these systems may further include a display apparatus for displaying an image based on the third pixel values.
  • the invention also provides a computer program comprising instructions executable by a processor and designed such that the processor carries out a method as presented above when these instructions are executed by the processor.
  • the invention provides a (non-transitory) computer readable medium storing such a computer program.
  • FIG. 1 shows elements of a first possible architecture for producing a banding map
  • FIG. 2 is a diagram showing steps of a possible method for identifying areas of visible contrast in an image
  • Figure 3 illustrates the use of the architecture of Figure 1 to train an artificial neural network
  • FIG. 4 shows a second possible architecture for producing a banding map
  • FIG. 5 is a diagram showing steps of a possible method for processing an image
  • FIG. 6 shows a first context in which the steps of Figure 4 may be implemented
  • FIG. 7 shows a second context in which the steps of Figure 4 may be implemented.
  • FIG. 8 show elements of a system where the method of Figure 6 or Figure 7 may be implemented.
  • FIG. 1 A first possible architecture of an electronic device for producing a banding map BHDR from an image ISDR is shown in Figure 1 .
  • Such an electronic device includes a first contrast visibility analysis module 2, a range expansion module 4, a second contrast visibility analysis module 6, a comparison module 8 and, possibly, a continuity analysis module 10.
  • This electronic device is thus an image processing device.
  • modules are for instance implemented using a processor executing software instructions corresponding to the module concerned and stored in a memory associated with the processor.
  • modules may in this context be implemented by a same processor, which may execute different software portions respectively corresponding to the various modules. According to a possible variation, some of the modules may each be implemented by an Application Specific Integrated Circuit.
  • the image ISDR is represented by first pixel values respectively associated with a set of pixels (generally a matrix of pixels) of the image ISDR.
  • the first pixel values include three colour components R P , G P , B P for each pixel p (namely a red component R P , a green component G P and a blue component B P ).
  • Another representation may however be used for the image ISDR, such as for instance using a luminance component Y P and two chrominance components Up, V P for each pixel p.
  • the first pixel values have a first dynamic range, here a Standard Dynamic Range.
  • a Standard Dynamic Range extends between 0 and 100 nits (i.e. 100 cd/m 2 ), for instance.
  • the first dynamic range may extend between 0 and a first maximum having a value between 50 nits and 200 nits.
  • the first contrast visibility analysis module 2 receives the first pixel values representing the image ISDR as an input, and is configured to produce a first map CSDR identifying areas of visible contrast in the image ISDR based on these first pixel values.
  • the first map CSDR has the same resolution (i.e. the same number of pixels) as the image ISDR, and/or is defined by binary values (i.e. the first map CSDR is defined by a binary value for each pixel of the image ISDR).
  • the range expansion module 4 also receives the first pixel values representing the image ISDR as an input, and is configured to produce second pixel values respectively corresponding to the first pixel values and having a second dynamic range larger than the first dynamic range.
  • the second dynamic range is for instance a High Dynamic Range.
  • Such a High Dynamic Range extends between 0 and 1000 nits (i.e. 1000 cd/m 2 ), for instance.
  • the second dynamic range may extend between 0 and a second maximum having a value between 500 nits and 2000 nits.
  • the range expansion module 4 may use for instance a non-linear, continuous and increasing function which maps the first dynamic range into the second dynamic range (which is larger as the first dynamic range, as already indicated).
  • the range expansion module 4 is then for instance adapted to compute each second pixel value (relating to a given pixel) by applying the non-linear, continuous and increasing function to the corresponding first pixel value (relating to the same pixel).
  • the range expansion module 4 is for instance designed in accordance with the solutions provided in document EP 3 839 876.
  • the second pixel values thus represent an image I HDR having the same content (from a viewer’s perspective) as the image ISDR represented by the first pixel values, but with a larger dynamic range.
  • the image I HDR is intended for display on a dedicated HDR display apparatus.
  • the image I HDR has the same resolution (i.e. the same number of pixels) as the image ISDR.
  • the second contrast visibility analysis module 6 receives the second pixel values (representing the image I HDR) as an input, and is configured to produce a second map CHDR identifying areas of visible contrast in the image IHDR based on these second pixel values.
  • the second map CHDR has the same resolution (i.e. the same number of pixels) as the image I HDR and hence as the image ISDR.
  • the second map CHDR is here defined by binary values (i.e. the second map CHDR is defined by a binary value for each pixel of the image ISDR or IHDR).
  • the comparison module 8 receives the first map CSDR and the second map CHDR as inputs, and is configured to produce a banding map BHDR by detecting areas of visible contrast identified by the second map CHDR and not identified by the first map CSDR (based on the comparison of the first map CSDR with the second map CHDR), and by indicating these detected areas of visible contrast in the banding map BHDR.
  • the banding map BHDR thus identifies areas where at least a false contour is likely to appear when the image is displayed based on the second pixel values but is unlikely to appear when the image is displayed based on the first pixel values.
  • the banding map BHDR thus has the same resolution (i.e. the same number of pixels) as the image ISDR, and/or is defined by binary values (i.e. the banding map BHDR is defined by a binary value for each pixel).
  • the banding map BHDR may be further processed by the continuity analysis module 10.
  • the continuity analysis module 10 is configured to analyse the banding map BHDR in order to remove isolated pixels and pixels with texture in their neighbourhood.
  • the image I is represented by pixel values (corresponding to the first pixel values for the first contrast visibility analysis module 2 and to the second pixel values for the second contrast visibility analyse module 6).
  • these pixel values include three component values R(i,j), G(i,j), B(i,j) for each pixel (i,j) of the image I (i denoting the line of the concerned pixel, j denoting the column of the concerned pixel).
  • the method of Figure 2 starts with a step S2 of determining a luminance value L(i,j) for each pixel (i,j) based on the pixel values, here R(i,j), G(i,j), B(i,j).
  • Step S2 may optionally include a step of linearizing the pixel values (e.g. by application of an inverse Opto-Electrical Transfer Function) prior to determining the luminance values.
  • a step of linearizing the pixel values e.g. by application of an inverse Opto-Electrical Transfer Function
  • the method of Figure 2 then includes a step S4 of computing image gradient magnitudes D(i,j) based on the determined luminance values L(i,j).
  • the method of Figure 2 then includes a step S6 of determining a local average luminance E(i,j) based on the luminance values L(i,j), e.g. by applying a convolution filter to the luminance values L(i,j).
  • the method of Figure 2 then includes a step S8 of determining the minimum visible contrast V(i,j) at each image location (i.e. at each pixel) (i,j) based on the local average luminance E(i,j).
  • step S8 use can be made for instance of the Contrast Sensitivity Function (CSF) model proposed in the article just mentioned:
  • the visible contrast V(i,j) can be determined as follows:
  • V(i,j) fcsF(u,E(i,j))
  • fcsF the function defined by the CSF model
  • u the frequency corresponding to a step edge in a given set of viewing conditions, measured in cycles per degree of visual angle.
  • the method of Figure 2 ends with a step S10 of producing the sought map C (corresponding either to CSDR or to CHDR in the description above) identifying areas of visible contrast, here by identifying pixels (i,j) for which the gradient magnitude
  • the method of Figure 2 may be performed repeatedly for different scales of the input image: in this case, the method of Figure 2 is not only applied to the input image I (thus producing a map Co), but also to at least one down-sampled version h of input image I (possibly to several down-sampled versions) of input image I (thus producing at least another map Ci) , thus evaluating the visibility of banding for different spatial frequencies.
  • Figure 3 illustrates the use of a module 12 having the architecture of Figure 1 to train an artificial neural network 14.
  • the module 12 constructed in accordance with Figure 1 is thus configured to produce a banding map BHDR based on first pixel values representing an image ISDR.
  • the artificial neural network 14 is adapted to receive the first pixel values as an input. For instance, the first pixel values are applied to an input layer of the artificial neural network 14. In this connection, each first pixel value may be applied to a corresponding node of the input layer of the artificial neural network 14.
  • the artificial neural network 14 produces as an output (i.e. on an output layer of the neural network 14) a predicted banding map B’.
  • the artificial neural network 14 may have for instance the structure explained below (i.e. the structure of a U-Net network) in connection with the later use of this artificial neural network 14 (after training).
  • a training module 16 is configured to train the artificial neural network 14 such as to reduce a distance between the banding map BHDR produced by the module 12 constructed in accordance with Figure 1 , and the predicted banding map B’ produced by the artificial neural network 14.
  • the training module 16 is for instance configured to adjust the weights of the nodes of the artificial neural network 14 so as to reduce the distance between the banding map BHDR and the predicted banding map B’ (for instance using a gradient descent algorithm). Training the artificial neural network 14 involves repeating this weight adjustment many times, while also successively applying a plurality of distinct images ISDR.
  • Figure 4 shows a second possible architecture for producing a banding map B’HDR from an image ISD represented by first pixel values.
  • This second possible architecture is an electronic device implementing the artificial neural network 14 trained as just mentioned in connection with Figure 3.
  • the (trained) artificial neural network 14 receives the first pixel values as inputs (for instance on an input layer 22 of the artificial neural network 14) and produces the banding map B’HDR as an output (for instance from an output layer 28 of the artificial neural network 14).
  • the electronic device is thus an image processing device.
  • the artificial neural network 14 is a U-Net network.
  • Such an artificial neural network 14 includes a plurality of convolution layers distributed in various levels as now explained.
  • the image I SD has a resolution m x m.
  • the input image ISD i.e. the first pixel values
  • a plurality of convolution layers here 2 convolution layers 22, 24
  • a feature map MH the first convolution layer 22 among these convolution layers being the input layer of the artificial neural network 14.
  • the dimensions of the feature map MH is here n x m x m, with n an integer (related to the convolution layers 22, 24 used).
  • a downscaling unit 32 receives the feature map MH, Mi produced in the level just above, scales this feature map down by 2 (i.e. divides by 2 the dimensions of the feature map in both the horizontal and vertical directions, e.g. by taking the maximum pixel value among each window of size 2 pixels x 2 pixels) and applies the down-scaled feature map to a series of convolution layers (here 2 convolution layers 34, 36) such that the down-scaled feature map is successively processed by these convolution layers 34, 36 to produce a feature map Mi.
  • 2 convolution layers 34, 36 a series of convolution layers
  • a concatenation unit 37 concatenates the feature map Mi produced by the convolution layers 34, 36 and an upscaled feature map produced by the level just below (as described below).
  • Each intermediate level 30 also includes another plurality of convolution layers (here 2 other convolution layers 38, 40) configured to process the concatenated feature map (output from the concatenation unit 37 of the concerned layer), so as to produce an output feature map (for the concerned intermediate level 30), that is scaled up by 2 (i.e. its dimensions are multiplied by 2 in both the horizontal and vertical directions) by an upscaling unit 42 (of the concerned intermediate level 30).
  • 2 other convolution layers 38, 40 configured to process the concatenated feature map (output from the concatenation unit 37 of the concerned layer), so as to produce an output feature map (for the concerned intermediate level 30), that is scaled up by 2 (i.e. its dimensions are multiplied by 2 in both the horizontal and vertical directions) by an upscaling unit 42 (of the concerned intermediate level 30).
  • the number of elements of the concatenated feature map is twice the number of elements of the feature Mi produced at the output of convolution layer 36, but is divided by two in the convolution layer 38 such that the number of elements of the feature map produced by convolution layer 38 is equal to the number of elements of the feature map Mi.
  • a downscaling unit 52 receives the feature map Mi produced in the (intermediate) level just above, scales this feature map down by 2 (i.e. divides by 2 the dimensions of the feature map in both the horizontal and vertical directions, e. g. by by taking the maximum pixel value among each window of size 2 pixels x 2 pixels) and applies the down-scaled feature map to a series of convolution layers (here 2 convolution layers 54, 56) such that the down-scaled feature map is successively processed by these convolution layers 34, 36 to produce an output feature map for the lowest level.
  • 2 convolution layers 54, 56 a series of convolution layers
  • This output feature map (for the lowest level) is scaled up by 2 (i.e. its dimensions are multiplied by 2 in both the horizontal and vertical directions) by an upscaling unit 58 (of the lowest level 50) so as to be used in the (intermediate) level just above as already mentioned.
  • the highest level 20 includes a concatenation unit 25 for concatenating the feature map MH and the upscaled feature map coming from the intermediate level 30 just below, and a series of convolution layers (here 2 convolution layers 26, 28) configured for processing the concatenated feature map, so as to produce the banding map B’HDR at the output of the last convolution layer 28 among the convolution layers 26, 28 (this last convolution layer 28 forming here the output layer of the artificial neural network 14).
  • a concatenation unit 25 for concatenating the feature map MH and the upscaled feature map coming from the intermediate level 30 just below
  • a series of convolution layers here 2 convolution layers 26, 28 configured for processing the concatenated feature map, so as to produce the banding map B’HDR at the output of the last convolution layer 28 among the convolution layers 26, 28 (this last convolution layer 28 forming here the output layer of the artificial neural network 14).
  • the number of elements of the concatenated feature map (here 2n x m x m) is twice the number of elements of the feature map MH produced at the output of convolution layer 24 (i.e. here n x m x m elements), but is divided by two in the convolution layer 26 such that the number of elements of the feature map produced by the convolution layer 26 is equal to the number of elements of the feature map MH.
  • concatenations units 25, 37 The interest of concatenations provided by concatenations units 25, 37 is that it allows for high frequency information from the encoding part of the network to be propagated to the decoding part. (This information would otherwise possibly be lost following the sequence of multiple downscaling and upscaling steps.)
  • Convolution layer 28 thus receives a feature map having dimensions n x m x m and produces at its output the banding map B’HDR with a resolution (here m x m) identical to the resolution of the image ISDR.
  • Each convolution layer of the artificial neural network 14 applies for instance 3x3 convolutions to the input values (first pixel values for the input layer 22 or values in the feature map for other layers) and a rectified linear unit to the result of convolutions.
  • the respective weights of the nodes of the convolution layers are adjusted during the training of the artificial neural network 14 as explained above referring to Figure 3.
  • Figure 5 is a diagram showing steps of a possible method for processing an image ISDR represented by first pixel values having the first dynamic range.
  • This method starts with a step S20 of producing, based on the first pixel values representing the image ISDR, a banding map BHDR, B’HDR identifying areas where at least a false contour appears when the image is displayed based on second pixel values respectively corresponding to the first pixel values and having the second dynamic range, but does not appear when the image is displayed based on the first pixel values.
  • the production of the banding map BHDR, B’HDR based on the first pixel values may use either the first architecture described above (referring to Figure 1 ), or the second architecture described above (referring to Figure 4), for instance.
  • this processing is not limited to pixels lying in areas identified in the banding map BHDR, B’HDR, but may also include pixels located near these areas (for instance at a distance lower than a given threshold from an area identified in the banding map BHDR, B’HDR).
  • This processing may for instance take the form of a combination of an increase in bit-depth followed by an averaging filter performed in the zones identified in the banding map, so as to break the appearance of the false contours.
  • Step S22 therefore produces processed pixel values (for pixels identified in the banding map BHDR, B’HDR), which, together with first pixel values which are not processed (for pixels not identified in the banding map BHDR, B’HDR), define a processed image I’SDR.
  • the method of Figure 5 then includes a step S24 of converting the pixel values of the processed image I’SDR (including the processed pixel values), which have the first dynamic range, here a Standard Dynamic Range, into third pixel values having the second dynamic range (here a High Dynamic Range).
  • the third value for a given pixel is obtained by applying the above-mentioned non-linear, continuous and increasing function to the corresponding pixel value (here a processed value or a first pixel value) for this given pixel in the processed image I’SDR. (This function is respectively applied to the value relating to each pixel of the processed image I’SDR to obtain a corresponding third value for the concerned pixel.)
  • step S24 is thus a dynamic range expansion, which may for instance be performed in accordance with the solutions provided in document EP 3 839 876.
  • the third values define an image I’HDR adapted for display on a HDR display apparatus.
  • the method of Figure 5 may also include a step S26 of displaying the image I’HDR using such a HDR display apparatus and based on the third values obtained in step S24.
  • the displayed image does not contain the banding artifacts that were previously identified in the banding map BHDR, B’HDR.
  • the method of Figure 5 may be implemented in an electronic device, which is thus an image processing device. All the steps S2-S10 of Figure 5 may then be implemented by this electronic device.
  • Figure 6 relates to a situation where a server S and a client device C (possibly equipped, or at least connected, with a HDR display apparatus) can exchange data via a communication network.
  • the server comprises for instance a processor 82 and a communication circuit 84 and/or the client device comprises a processor 86 and a communication circuit 88.
  • the communication circuit 84 and the communication circuit 88 makes it possible to exchange data between the server S (in particular the processor 82) and the client device C (in particular the processor 86) over the communication network.
  • step S40 the server S (here the processor 82) produces, based on first pixel values representing an image ISDR, a banding map BHDR, B’HDR identifying areas where at least a false contour appears when the image is displayed based on second pixel values respectively corresponding to the first pixel values and having the second dynamic range, but does not appear when the image is displayed based on the first pixel values.
  • step S42 the server S (here the processor 82) processes the first pixel values which are respectively associated with pixels lying in areas identified in the banding map BHD , B’HD .
  • a step S44 the server S sends (here using the communication circuit 84) the processed pixel values (as well as first pixel values for pixels that were not processed at step S42, i.e. for pixels not identified in the banding map BHD , B’HD ) to the client device C.
  • this step may involve broadcasting the processed pixel values (and first pixel values) to a plurality of client devices.
  • the client device C receives (here using the communication circuit 88) the processed pixel values (for pixels identified in the banding map BHD , B’HD ) and first pixel values (for pixels not identified in the banding map BHD , B’HD ), i.e. pixel values defining an image I’SD processed at the server by step S42.
  • steps S44 and S46 may be implemented by usual means for sending and receiving an SDR image.
  • the SDR image is not defined (at least partly) by processed pixel values as defined above.
  • step S48 the client device C (here the processor 86) converts the pixel values of the processed image I’SD (including the processed pixel values), which have the first dynamic range, here a Standard Dynamic Range, into third pixel values having the second dynamic range (here a High Dynamic Range), which is larger than the first dynamic range.
  • the client device C here the processor 86
  • step S50 (corresponding to step S26 in Figure 5), the image I’HDR defined by the third pixel values is displayed on the HDR display apparatus included in the client device or connected thereto.
  • Figure 7 also relates to a situation where a server S and a client device C (possibly equipped, or at least connected, with a HDR display apparatus) can exchange data via a communication network.
  • the server comprises for instance a processor 82 and a communication circuit 84 and/or the client device comprises a processor 86 and a communication circuit 88.
  • the communication circuit 84 and the communication circuit 88 makes it possible to exchange data between the server S (in particular the processor 82) and the client device C (in particular the processor 86) over the communication network.
  • step S60 the server S (here the processor 82) produces, based on first pixel values representing an image ISDR, a banding map BHDR, B’HDR identifying areas where at least a false contour appears when the image is displayed based on second pixel values respectively corresponding to the first pixel values and having the second dynamic range, but does not appear when the image is displayed based on the first pixel values.
  • a step S62 the server S sends (here using the communication circuit 84) the first pixel values representing the image ISD and data representing the banding map BHD , B’HD to the client device C.
  • this step may involve broadcasting the first pixel values and data representing the banding map BHD , B’HD to a plurality of client devices.
  • the client device C receives (here using the communication circuit 88) the first pixel values and data representing the banding map BHD , B’HD .
  • a step S66 (corresponding to step S22 in Figure 5), the client device C (here the processor 86) processes the first pixel values which are respectively associated with pixels lying in areas identified in the banding map BHD , B’HD .
  • step S68 the client device C (here the processor 86) converts the pixel values of the processed image I’SDR (including the processed pixel values), which have the first dynamic range, here a Standard Dynamic Range, into third pixel values having the second dynamic range (here a High Dynamic Range), which is larger than the first dynamic range.
  • the client device C here the processor 86
  • step S70 (corresponding to step S26 in Figure 5), the image I’HDR defined by the third pixel values is displayed on the HDR display apparatus included in the client device C or connected thereto.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

A method is described for processing an image (ISDR) represented by first pixel values having a first dynamic range and displayable based on second pixel values respectively corresponding to the first pixel values and having a second dynamic range larger than the first dynamic range. This method includes producing, based at least on the first pixel values, a banding map (BHDR) identifying areas where at least a false contour appears when the image is displayed based on the second pixel values but does not appear when the image is displayed based on the first pixel values. A corresponding processing device and corresponding systems are also described.

Description

Method for processing an image and corresponding image processing device and systems
Technical field of the invention
The invention relates to the field of image processing.
More particularly, the invention relates to a method for processing an image, to a corresponding image processing device and to corresponding systems.
Background information
In order to more fully benefit from displays offering extended brightness possibilities, it has been proposed to convert pixel values of images available in a conventional format, having a first dynamic range (for instance a “Standard Dynamic Range” or SDR), into pixel values having a second dynamic range (for instance a “High Dynamic Range” or HDR) that is higher than the first dynamic range. Such a conversion is generally called “tone expansion”.
This conversion may be performed for each pixel of the image by transforming the luminance value for this pixel using a function that maps the first dynamic range into the second dynamic range. This function is generally continuous and increasing in order not to alter the appearance of the image; the function may however be nonlinear to better take advantage of the spread of the second dynamic range.
Summary of the invention
In this context, the invention provides a method for processing an image represented by first pixel values having a first dynamic range and displayable based on second pixel values respectively corresponding to the first pixel values and having a second dynamic range larger than the first dynamic range, said method including producing, based at least on the first pixel values, a banding map identifying areas where at least a false contour appears when the image is displayed based on the second pixel values but does not appear when the image is displayed based on the first pixel values.
Such banding map interestingly identifies regions where a false contour or banding is likely occur, owing to the dynamic range expansion, when displaying the image using the second pixel values. This banding map may be used for instance to pre-process the image to avoid these banding artifacts upon display, as further explained below. According to a first possibility, the production of said banding map comprises the following steps:
- producing a first map identifying areas of visible contrast in the image based on the first pixel values;
- computing the second pixel values based on the first pixel values;
- producing a second map identifying areas of visible contrast in the image based on the second pixel values;
- based on the first map and the second map, producing said banding map by indicating therein areas of visible contrast identified by the second map and not identified by the first map.
The step of producing a first map, or respectively a second map, may for instance comprise the following sub-steps:
- determining, for each pixel of the image, a minimum visible contrast based on a local average luminance at the concerned pixel;
- identify a pixel as an area of visible contrast if a luminance gradient magnitude at the concerned pixel exceeds the minimum visible contrast determined for the concerned pixel.
Each second pixel value may for instance be computed by applying a non-linear, continuous and increasing function to a corresponding first pixel value, thereby obtaining the above-mentioned dynamic range expansion.
In a particular embodiment, the method may include a step of training an artificial neural network such as to reduce a distance between said banding map and a third map produced at an output of the artificial neural network when the first pixel values are applied at an input of the artificial neural network. In this situation, the produced banding map aims at training an artificial neural network.
According to a second possibility, said banding map is produced at an output of an artificial neural network receiving said first pixel values at an input of the artificial neural network. This artificial neural network may for instance be the artificial neural network trained as mentioned above.
In possible embodiments, the method may comprise a step of processing first pixel values respectively associated with pixels lying in areas identified in said banding map. The banding map may thus be used in this case to select pixels which should be (pre-)processed to avoid the production of false contours or banding artifacts when the image is displayed using the second pixel values, i.e. with an expanded dynamic range.
According to a possible embodiment, the method may comprise a step of sending the processed first pixel values to a client device.
According to another possible embodiment, the method may comprise a step of sending the first pixel values and data representative of said banding map to a client device. In this situation, the step of processing said first pixel values respectively associated with pixels lying in areas identified in said banding map may for instance be implemented by the client device.
In both situations, the client device may then convert the processed first pixel values into third pixel values having the second dynamic range.
The method may also comprise a step of displaying an image using the third pixel values, e.g. on a high dynamic range display apparatus..
The invention also provides an image processing device configured or programmed to implement a method as presented above.
In addition, the invention provides a system comprising:
- a server configured to implement a method as presented above and including a step of processing first pixel values respectively associated with pixels lying in areas identified in said banding map, the server including a communication circuit configured to send the processed first pixel values to a client device;
- the client device including a communication circuit to receive the processed first pixel values and a processor configured to convert the processed first pixel values into third pixel values having the second dynamic range.
Furthermore, the invention provides a system comprising:
- a server configured to implement a method as presented above, the server including a communication circuit configured to send the first pixel values and data representative of the banding map to a client device;
- the client device including a communication circuit to receive the first pixel values and said data representative of the banding map, the client device being configured to process first pixel values respectively associated with pixels lying in areas identified in the banding map and to convert the processed first pixel values into third pixel values having the second dynamic range. Both of these systems may further include a display apparatus for displaying an image based on the third pixel values.
The invention also provides a computer program comprising instructions executable by a processor and designed such that the processor carries out a method as presented above when these instructions are executed by the processor.
Lastly, the invention provides a (non-transitory) computer readable medium storing such a computer program.
Detailed description of example(s)
The following description with reference to the accompanying drawings will make it clear what the invention consists of and how it can be achieved. The invention is not limited to the embodiments illustrated in the drawings. Accordingly, it should be understood that where features mentioned in the claims are followed by reference signs, such signs are included solely for the purpose of enhancing the intelligibility of the claims and are in no way limiting on the scope of the claims.
In the accompanying drawings:
- Figure 1 shows elements of a first possible architecture for producing a banding map;
- Figure 2 is a diagram showing steps of a possible method for identifying areas of visible contrast in an image;
- Figure 3 illustrates the use of the architecture of Figure 1 to train an artificial neural network;
- Figure 4 shows a second possible architecture for producing a banding map;
- Figure 5 is a diagram showing steps of a possible method for processing an image;
- Figure 6 shows a first context in which the steps of Figure 4 may be implemented;
- Figure 7 shows a second context in which the steps of Figure 4 may be implemented; and
- Figure 8 show elements of a system where the method of Figure 6 or Figure 7 may be implemented.
A first possible architecture of an electronic device for producing a banding map BHDR from an image ISDR is shown in Figure 1 . Such an electronic device includes a first contrast visibility analysis module 2, a range expansion module 4, a second contrast visibility analysis module 6, a comparison module 8 and, possibly, a continuity analysis module 10. This electronic device is thus an image processing device.
Each of these modules is for instance implemented using a processor executing software instructions corresponding to the module concerned and stored in a memory associated with the processor. Several modules may in this context be implemented by a same processor, which may execute different software portions respectively corresponding to the various modules. According to a possible variation, some of the modules may each be implemented by an Application Specific Integrated Circuit.
The image ISDR is represented by first pixel values respectively associated with a set of pixels (generally a matrix of pixels) of the image ISDR.
In the present example, the first pixel values include three colour components RP, GP, BP for each pixel p (namely a red component RP, a green component GP and a blue component BP). Another representation may however be used for the image ISDR, such as for instance using a luminance component YP and two chrominance components Up, VP for each pixel p.
The first pixel values have a first dynamic range, here a Standard Dynamic Range. Such a Standard Dynamic Range extends between 0 and 100 nits (i.e. 100 cd/m2), for instance. In some embodiments, the first dynamic range may extend between 0 and a first maximum having a value between 50 nits and 200 nits.
The first contrast visibility analysis module 2 receives the first pixel values representing the image ISDR as an input, and is configured to produce a first map CSDR identifying areas of visible contrast in the image ISDR based on these first pixel values.
An exemplary method (possibly implemented by the first contrast visibility analysis module 2) for producing the first map CSDR based on the first pixel values (i.e. for identifying areas of visible contrast in an image) is given below referring to Figure 2.
In the present example, the first map CSDR has the same resolution (i.e. the same number of pixels) as the image ISDR, and/or is defined by binary values (i.e. the first map CSDR is defined by a binary value for each pixel of the image ISDR).
The range expansion module 4 also receives the first pixel values representing the image ISDR as an input, and is configured to produce second pixel values respectively corresponding to the first pixel values and having a second dynamic range larger than the first dynamic range. The second dynamic range is for instance a High Dynamic Range. Such a High Dynamic Range extends between 0 and 1000 nits (i.e. 1000 cd/m2), for instance. In some embodiments, the second dynamic range may extend between 0 and a second maximum having a value between 500 nits and 2000 nits.
The range expansion module 4 may use for instance a non-linear, continuous and increasing function which maps the first dynamic range into the second dynamic range (which is larger as the first dynamic range, as already indicated).
The range expansion module 4 is then for instance adapted to compute each second pixel value (relating to a given pixel) by applying the non-linear, continuous and increasing function to the corresponding first pixel value (relating to the same pixel).
The range expansion module 4 is for instance designed in accordance with the solutions provided in document EP 3 839 876.
The second pixel values thus represent an image I HDR having the same content (from a viewer’s perspective) as the image ISDR represented by the first pixel values, but with a larger dynamic range. As will be explained below, the image I HDR is intended for display on a dedicated HDR display apparatus.
In the present example, the image I HDR has the same resolution (i.e. the same number of pixels) as the image ISDR.
The second contrast visibility analysis module 6 receives the second pixel values (representing the image I HDR) as an input, and is configured to produce a second map CHDR identifying areas of visible contrast in the image IHDR based on these second pixel values.
An exemplary method (possibly implemented by the second contrast visibility analysis module 6) for producing the second map CHDR based on the second pixel values (i.e. for identifying areas of visible contrast in an image) is given below referring to Figure 2.
In the present example, the second map CHDR has the same resolution (i.e. the same number of pixels) as the image I HDR and hence as the image ISDR. The second map CHDR is here defined by binary values (i.e. the second map CHDR is defined by a binary value for each pixel of the image ISDR or IHDR).
The comparison module 8 receives the first map CSDR and the second map CHDR as inputs, and is configured to produce a banding map BHDR by detecting areas of visible contrast identified by the second map CHDR and not identified by the first map CSDR (based on the comparison of the first map CSDR with the second map CHDR), and by indicating these detected areas of visible contrast in the banding map BHDR.
The banding map BHDR thus identifies areas where at least a false contour is likely to appear when the image is displayed based on the second pixel values but is unlikely to appear when the image is displayed based on the first pixel values.
In the present example, the banding map BHDR thus has the same resolution (i.e. the same number of pixels) as the image ISDR, and/or is defined by binary values (i.e. the banding map BHDR is defined by a binary value for each pixel).
According to a possible embodiment, the banding map BHDR may be further processed by the continuity analysis module 10. The continuity analysis module 10 is configured to analyse the banding map BHDR in order to remove isolated pixels and pixels with texture in their neighbourhood. Reference can be made in this respect to the article “Bband Index: A No-Reference Banding Artifact Predictor”, by Z. Tu, J. Lin, Y. Wang, B. Adsumili and A.C. Bovik in ICASSP, IEEE Int. Cong. Acoust. Speech Sign. Proc., vol. 2020 May, pp. 2712-2716, 2020.
As noted above, a possible method for identifying areas of visible contrast in an image I is now described referring to Figure 2.
The image I is represented by pixel values (corresponding to the first pixel values for the first contrast visibility analysis module 2 and to the second pixel values for the second contrast visibility analyse module 6). In the present example, these pixel values include three component values R(i,j), G(i,j), B(i,j) for each pixel (i,j) of the image I (i denoting the line of the concerned pixel, j denoting the column of the concerned pixel).
The method of Figure 2 starts with a step S2 of determining a luminance value L(i,j) for each pixel (i,j) based on the pixel values, here R(i,j), G(i,j), B(i,j).
Step S2 may optionally include a step of linearizing the pixel values (e.g. by application of an inverse Opto-Electrical Transfer Function) prior to determining the luminance values.
The method of Figure 2 then includes a step S4 of computing image gradient magnitudes D(i,j) based on the determined luminance values L(i,j). Image gradient magnitudes D(i,j) may be computed for instance using the Prewitt Operator, whereby horizontal gradients Dx(i,j) are computed by convolving the image with a 3x3 kernel and vertical gradients Dy(i,j) are computed by convolving the image
Figure imgf000009_0001
with a 3x3 kernel Gy = The image gradient magnitudes D(i,j) are
Figure imgf000009_0002
computed as:
Figure imgf000009_0003
The method of Figure 2 then includes a step S6 of determining a local average luminance E(i,j) based on the luminance values L(i,j), e.g. by applying a convolution filter to the luminance values L(i,j).
The method of Figure 2 then includes a step S8 of determining the minimum visible contrast V(i,j) at each image location (i.e. at each pixel) (i,j) based on the local average luminance E(i,j).
This is because, as explained for instance in “Contrast Sensitivity of the Human Eye and Its Effects on Image Quality”, by P. G. Barten, in Contrast Sensit. Hum. Eye Its Eff. Image Qual., No. 1999, 2009, the contrast sensitivity is dependent upon the luminance.
In step S8, use can be made for instance of the Contrast Sensitivity Function (CSF) model proposed in the article just mentioned: the visible contrast V(i,j) can be determined as follows:
V(i,j) = fcsF(u,E(i,j)) where fcsF is the function defined by the CSF model and u is the frequency corresponding to a step edge in a given set of viewing conditions, measured in cycles per degree of visual angle. For an image of HD (1920 x 1080 pixels) resolution, viewed under typical conditions for instance, we might consider that a block of 32x32 pixels subtends 10 of visual angle, and thus u = 1/32.
The method of Figure 2 ends with a step S10 of producing the sought map C (corresponding either to CSDR or to CHDR in the description above) identifying areas of visible contrast, here by identifying pixels (i,j) for which the gradient magnitude |D(i,j)| (i.e. the modulus of the gradient magnitude D(i,j) determined in step S4) exceeds the minimum visible contrast V(i,j). Precisely, for each pixel (i,j):
C(i,j) = 0 if |D(i,j)| < V(i,j);
C(i,j) = 1 if |D(i,j)| > V(i,j).
According to a possible embodiment, to improve accuracy the method of Figure 2 may be performed repeatedly for different scales of the input image: in this case, the method of Figure 2 is not only applied to the input image I (thus producing a map Co), but also to at least one down-sampled version h of input image I (possibly to several down-sampled versions) of input image I (thus producing at least another map Ci) , thus evaluating the visibility of banding for different spatial frequencies.
In this case, the resulting map C (to be used as map CSDR or map CHDR in the description above) identifies pixels which are identified in at least one of the obtained maps Co, Ci : for each pixel (i,j), C(i,j) = Co(i,j) OR Ci(i,j).
Figure 3 illustrates the use of a module 12 having the architecture of Figure 1 to train an artificial neural network 14.
The module 12 constructed in accordance with Figure 1 is thus configured to produce a banding map BHDR based on first pixel values representing an image ISDR.
The artificial neural network 14 is adapted to receive the first pixel values as an input. For instance, the first pixel values are applied to an input layer of the artificial neural network 14. In this connection, each first pixel value may be applied to a corresponding node of the input layer of the artificial neural network 14.
The artificial neural network 14 produces as an output (i.e. on an output layer of the neural network 14) a predicted banding map B’.
The artificial neural network 14 may have for instance the structure explained below (i.e. the structure of a U-Net network) in connection with the later use of this artificial neural network 14 (after training).
A training module 16 is configured to train the artificial neural network 14 such as to reduce a distance between the banding map BHDR produced by the module 12 constructed in accordance with Figure 1 , and the predicted banding map B’ produced by the artificial neural network 14.
In practice, the training module 16 is for instance configured to adjust the weights of the nodes of the artificial neural network 14 so as to reduce the distance between the banding map BHDR and the predicted banding map B’ (for instance using a gradient descent algorithm). Training the artificial neural network 14 involves repeating this weight adjustment many times, while also successively applying a plurality of distinct images ISDR.
Figure 4 shows a second possible architecture for producing a banding map B’HDR from an image ISD represented by first pixel values.
This second possible architecture is an electronic device implementing the artificial neural network 14 trained as just mentioned in connection with Figure 3.
According to this second possible architecture, the (trained) artificial neural network 14 receives the first pixel values as inputs (for instance on an input layer 22 of the artificial neural network 14) and produces the banding map B’HDR as an output (for instance from an output layer 28 of the artificial neural network 14). The electronic device is thus an image processing device.
In the example shown in Figure 4, the artificial neural network 14 is a U-Net network.
Such an artificial neural network 14 includes a plurality of convolution layers distributed in various levels as now explained.
For the following explanations, it is considered that the image I SD has a resolution m x m.
At the highest level 20, the input image ISD (i.e. the first pixel values) is successively processed by a plurality of convolution layers (here 2 convolution layers 22, 24) producing a feature map MH (the first convolution layer 22 among these convolution layers being the input layer of the artificial neural network 14). The dimensions of the feature map MH is here n x m x m, with n an integer (related to the convolution layers 22, 24 used).
At each intermediate level 30, a downscaling unit 32 receives the feature map MH, Mi produced in the level just above, scales this feature map down by 2 (i.e. divides by 2 the dimensions of the feature map in both the horizontal and vertical directions, e.g. by taking the maximum pixel value among each window of size 2 pixels x 2 pixels) and applies the down-scaled feature map to a series of convolution layers (here 2 convolution layers 34, 36) such that the down-scaled feature map is successively processed by these convolution layers 34, 36 to produce a feature map Mi.
At each intermediate level, a concatenation unit 37 concatenates the feature map Mi produced by the convolution layers 34, 36 and an upscaled feature map produced by the level just below (as described below).
Each intermediate level 30 also includes another plurality of convolution layers (here 2 other convolution layers 38, 40) configured to process the concatenated feature map (output from the concatenation unit 37 of the concerned layer), so as to produce an output feature map (for the concerned intermediate level 30), that is scaled up by 2 (i.e. its dimensions are multiplied by 2 in both the horizontal and vertical directions) by an upscaling unit 42 (of the concerned intermediate level 30).
It may be noted that, in each intermediate level 30, the number of elements of the concatenated feature map is twice the number of elements of the feature Mi produced at the output of convolution layer 36, but is divided by two in the convolution layer 38 such that the number of elements of the feature map produced by convolution layer 38 is equal to the number of elements of the feature map Mi.
At the lowest level 50, a downscaling unit 52 receives the feature map Mi produced in the (intermediate) level just above, scales this feature map down by 2 (i.e. divides by 2 the dimensions of the feature map in both the horizontal and vertical directions, e. g. by by taking the maximum pixel value among each window of size 2 pixels x 2 pixels) and applies the down-scaled feature map to a series of convolution layers (here 2 convolution layers 54, 56) such that the down-scaled feature map is successively processed by these convolution layers 34, 36 to produce an output feature map for the lowest level.
This output feature map (for the lowest level) is scaled up by 2 (i.e. its dimensions are multiplied by 2 in both the horizontal and vertical directions) by an upscaling unit 58 (of the lowest level 50) so as to be used in the (intermediate) level just above as already mentioned.
The highest level 20 includes a concatenation unit 25 for concatenating the feature map MH and the upscaled feature map coming from the intermediate level 30 just below, and a series of convolution layers (here 2 convolution layers 26, 28) configured for processing the concatenated feature map, so as to produce the banding map B’HDR at the output of the last convolution layer 28 among the convolution layers 26, 28 (this last convolution layer 28 forming here the output layer of the artificial neural network 14).
As for convolution layers 38 in intermediate layers 30, the number of elements of the concatenated feature map (here 2n x m x m) is twice the number of elements of the feature map MH produced at the output of convolution layer 24 (i.e. here n x m x m elements), but is divided by two in the convolution layer 26 such that the number of elements of the feature map produced by the convolution layer 26 is equal to the number of elements of the feature map MH.
The interest of concatenations provided by concatenations units 25, 37 is that it allows for high frequency information from the encoding part of the network to be propagated to the decoding part. (This information would otherwise possibly be lost following the sequence of multiple downscaling and upscaling steps.)
Convolution layer 28 thus receives a feature map having dimensions n x m x m and produces at its output the banding map B’HDR with a resolution (here m x m) identical to the resolution of the image ISDR.
Each convolution layer of the artificial neural network 14 applies for instance 3x3 convolutions to the input values (first pixel values for the input layer 22 or values in the feature map for other layers) and a rectified linear unit to the result of convolutions. The respective weights of the nodes of the convolution layers are adjusted during the training of the artificial neural network 14 as explained above referring to Figure 3.
Figure 5 is a diagram showing steps of a possible method for processing an image ISDR represented by first pixel values having the first dynamic range.
This method starts with a step S20 of producing, based on the first pixel values representing the image ISDR, a banding map BHDR, B’HDR identifying areas where at least a false contour appears when the image is displayed based on second pixel values respectively corresponding to the first pixel values and having the second dynamic range, but does not appear when the image is displayed based on the first pixel values.
The production of the banding map BHDR, B’HDR based on the first pixel values may use either the first architecture described above (referring to Figure 1 ), or the second architecture described above (referring to Figure 4), for instance.
The method of Figure 5 continues with a step S22 of processing first pixel values respectively associated with pixels lying in areas identified in the banding map BHDR, B’HDR obtained in step S20. In some embodiments, this processing is not limited to pixels lying in areas identified in the banding map BHDR, B’HDR, but may also include pixels located near these areas (for instance at a distance lower than a given threshold from an area identified in the banding map BHDR, B’HDR).
This processing may for instance take the form of a combination of an increase in bit-depth followed by an averaging filter performed in the zones identified in the banding map, so as to break the appearance of the false contours.
Step S22 therefore produces processed pixel values (for pixels identified in the banding map BHDR, B’HDR), which, together with first pixel values which are not processed (for pixels not identified in the banding map BHDR, B’HDR), define a processed image I’SDR.
The method of Figure 5 then includes a step S24 of converting the pixel values of the processed image I’SDR (including the processed pixel values), which have the first dynamic range, here a Standard Dynamic Range, into third pixel values having the second dynamic range (here a High Dynamic Range).
In a possible embodiment, the third value for a given pixel is obtained by applying the above-mentioned non-linear, continuous and increasing function to the corresponding pixel value (here a processed value or a first pixel value) for this given pixel in the processed image I’SDR. (This function is respectively applied to the value relating to each pixel of the processed image I’SDR to obtain a corresponding third value for the concerned pixel.)
The conversion of step S24 is thus a dynamic range expansion, which may for instance be performed in accordance with the solutions provided in document EP 3 839 876. The third values define an image I’HDR adapted for display on a HDR display apparatus.
The method of Figure 5 may also include a step S26 of displaying the image I’HDR using such a HDR display apparatus and based on the third values obtained in step S24.
Thanks to the (pre)processing of step S22, the displayed image does not contain the banding artifacts that were previously identified in the banding map BHDR, B’HDR.
According to a possible embodiment, the method of Figure 5 may be implemented in an electronic device, which is thus an image processing device. All the steps S2-S10 of Figure 5 may then be implemented by this electronic device.
According to a first possible variation, the steps of Figure 5 may be implemented as now described referring to Figure 6.
Figure 6 relates to a situation where a server S and a client device C (possibly equipped, or at least connected, with a HDR display apparatus) can exchange data via a communication network.
As represented in Figure 8, the server comprises for instance a processor 82 and a communication circuit 84 and/or the client device comprises a processor 86 and a communication circuit 88. The communication circuit 84 and the communication circuit 88 makes it possible to exchange data between the server S (in particular the processor 82) and the client device C (in particular the processor 86) over the communication network.
In a step S40 (corresponding to step S20 in Figure 5), the server S (here the processor 82) produces, based on first pixel values representing an image ISDR, a banding map BHDR, B’HDR identifying areas where at least a false contour appears when the image is displayed based on second pixel values respectively corresponding to the first pixel values and having the second dynamic range, but does not appear when the image is displayed based on the first pixel values.
In a step S42 (corresponding to step S22 in Figure 5), the server S (here the processor 82) processes the first pixel values which are respectively associated with pixels lying in areas identified in the banding map BHD , B’HD .
In a step S44, the server S sends (here using the communication circuit 84) the processed pixel values (as well as first pixel values for pixels that were not processed at step S42, i.e. for pixels not identified in the banding map BHD , B’HD ) to the client device C. In some embodiments, this step may involve broadcasting the processed pixel values (and first pixel values) to a plurality of client devices.
In a step S46, the client device C receives (here using the communication circuit 88) the processed pixel values (for pixels identified in the banding map BHD , B’HD ) and first pixel values (for pixels not identified in the banding map BHD , B’HD ), i.e. pixel values defining an image I’SD processed at the server by step S42.
Interestingly, steps S44 and S46 may be implemented by usual means for sending and receiving an SDR image. Conventionally however, the SDR image is not defined (at least partly) by processed pixel values as defined above.
In step S48 (corresponding to step S24 in Figure 5), the client device C (here the processor 86) converts the pixel values of the processed image I’SD (including the processed pixel values), which have the first dynamic range, here a Standard Dynamic Range, into third pixel values having the second dynamic range (here a High Dynamic Range), which is larger than the first dynamic range.
In step S50 (corresponding to step S26 in Figure 5), the image I’HDR defined by the third pixel values is displayed on the HDR display apparatus included in the client device or connected thereto.
According to a second possible variation, the steps of Figure 5 may be implemented as now described referring to Figure 7.
Figure 7 also relates to a situation where a server S and a client device C (possibly equipped, or at least connected, with a HDR display apparatus) can exchange data via a communication network.
As represented in Figure 8, the server comprises for instance a processor 82 and a communication circuit 84 and/or the client device comprises a processor 86 and a communication circuit 88. The communication circuit 84 and the communication circuit 88 makes it possible to exchange data between the server S (in particular the processor 82) and the client device C (in particular the processor 86) over the communication network.
In a step S60 (corresponding to step S20 in Figure 5), the server S (here the processor 82) produces, based on first pixel values representing an image ISDR, a banding map BHDR, B’HDR identifying areas where at least a false contour appears when the image is displayed based on second pixel values respectively corresponding to the first pixel values and having the second dynamic range, but does not appear when the image is displayed based on the first pixel values.
In a step S62, the server S sends (here using the communication circuit 84) the first pixel values representing the image ISD and data representing the banding map BHD , B’HD to the client device C. In some embodiments, this step may involve broadcasting the first pixel values and data representing the banding map BHD , B’HD to a plurality of client devices.
In a step S64, the client device C receives (here using the communication circuit 88) the first pixel values and data representing the banding map BHD , B’HD .
In a step S66 (corresponding to step S22 in Figure 5), the client device C (here the processor 86) processes the first pixel values which are respectively associated with pixels lying in areas identified in the banding map BHD , B’HD . This makes it possible to define a processed image I’SDR including processed pixel values (for pixels identified in the banding map BHDR, B’HDR) and first pixel values which are not processed (for pixels not identified in the banding map BHD , B’HDR).
In step S68 (corresponding to step S24 in Figure 5), the client device C (here the processor 86) converts the pixel values of the processed image I’SDR (including the processed pixel values), which have the first dynamic range, here a Standard Dynamic Range, into third pixel values having the second dynamic range (here a High Dynamic Range), which is larger than the first dynamic range.
In step S70 (corresponding to step S26 in Figure 5), the image I’HDR defined by the third pixel values is displayed on the HDR display apparatus included in the client device C or connected thereto.

Claims

Claims
1. A method for processing an image represented by first pixel values having a first dynamic range and displayable based on second pixel values respectively corresponding to the first pixel values and having a second dynamic range larger than the first dynamic range, said method including producing, based at least on the first pixel values, a banding map identifying areas where at least a false contour appears when the image is displayed based on the second pixel values but does not appear when the image is displayed based on the first pixel values.
2. The method according to claim 1 , wherein the production of said banding map comprises the following steps:
- producing a first map identifying areas of visible contrast in the image based on the first pixel values;
- computing the second pixel values based on the first pixel values;
- producing a second map identifying areas of visible contrast in the image based on the second pixel values;
- based on the first map and the second map, producing said banding map by indicating therein areas of visible contrast identified by the second map and not identified by the first map.
3. A method according to claim 2, wherein the step of producing a first map, or respectively a second map, comprises the following sub-steps:
- determining, for each pixel of the image, a minimum visible contrast based on a local average luminance at the concerned pixel;
- identify a pixel as an area of visible contrast if a luminance gradient magnitude at the concerned pixel exceeds the minimum visible contrast determined for the concerned pixel.
4. A method according to claim 2 or 3, wherein each second pixel value is computed by applying a non-linear, continuous and increasing function to a corresponding first pixel value.
5. A method according to any of claims 1 to 4, comprising a step of training an artificial neural network such as to reduce a distance between said banding map and a third map produced at an output of the artificial neural network when the first pixel values are applied at an input of the artificial neural network.
6. A method according to claim 1 , wherein said banding map is produced at an output of an artificial neural network receiving said first pixel values at an input of the artificial neural network.
7. A method according to any of claims 1 , 2, 3, 4 and 6, comprising a step of processing first pixel values respectively associated with pixels lying in areas identified in said banding map.
8. A method according to claim 7, comprising a step of sending the processed first pixel values to a client device.
9. A method according to claim 7, comprising a step of sending the first pixel values and data representative of said banding map to a client device, wherein the step of processing said first pixel values respectively associated with pixels lying in areas identified in said banding map is implemented by the client device.
10. A method according to any of claims 8-9, wherein the client device converts the processed first pixel values into third pixel values having the second dynamic range.
1 1. An image processing device configured or programmed to implement a method according to any of claims 1 -8.
12. A system comprising:
- a server configured to implement a method according to claim 7 and including a communication circuit configured to send the processed first pixel values to a client device;
- the client device including a communication circuit to receive the processed first pixel values and a processor configured to convert the processed first pixel values into third pixel values having the second dynamic range.
13. A system comprising:
- a server configured to implement a method according to any of claims 1 , 2, 3, 4 and 6, the server including a communication circuit configured to send the first pixel values and data representative of the banding map to a client device; - the client device including a communication circuit to receive the first pixel values and said data representative of the banding map, the client device being configured to process first pixel values respectively associated with pixels lying in areas identified in the banding map and to convert the processed first pixel values into third pixel values having the second dynamic range.
14. A computer program comprising instructions executable by a processor and designed such that the processor carries out a method according to any one of claims 1 -10 when these instructions are executed by the processor.
15. A computer readable medium storing a computer program according to claim 14.
PCT/IB2022/000382 2022-06-28 2022-06-28 Method for processing an image and corresponding image processing device and systems WO2024003589A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/IB2022/000382 WO2024003589A1 (en) 2022-06-28 2022-06-28 Method for processing an image and corresponding image processing device and systems

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/IB2022/000382 WO2024003589A1 (en) 2022-06-28 2022-06-28 Method for processing an image and corresponding image processing device and systems

Publications (1)

Publication Number Publication Date
WO2024003589A1 true WO2024003589A1 (en) 2024-01-04

Family

ID=83355231

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2022/000382 WO2024003589A1 (en) 2022-06-28 2022-06-28 Method for processing an image and corresponding image processing device and systems

Country Status (1)

Country Link
WO (1) WO2024003589A1 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3839876A1 (en) 2019-12-20 2021-06-23 Fondation B-COM Method for converting an image and corresponding device

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3839876A1 (en) 2019-12-20 2021-06-23 Fondation B-COM Method for converting an image and corresponding device

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
"Wiley Encyclopedia of Electrical and Electronics Engineering", 18 April 2016, JOHN WILEY & SONS, INC., Hoboken, NJ, USA, ISBN: 978-0-471-34608-1, article JOHN G. WEBSTER ET AL: "High Dynamic Range Imaging", pages: 1 - 42, XP055412750, DOI: 10.1002/047134608X.W8265 *
KAPOOR AKSHAY ET AL: "Capturing Banding in Images: Database Construction and Objective Assessment", ICASSP 2021 - 2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), IEEE, 6 June 2021 (2021-06-06), pages 2425 - 2429, XP033955594, DOI: 10.1109/ICASSP39728.2021.9413884 *
MANTIUK RAFAL K ET AL: "PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR", 2021 PICTURE CODING SYMPOSIUM (PCS), IEEE, 29 June 2021 (2021-06-29), pages 1 - 5, XP033945039, DOI: 10.1109/PCS50896.2021.9477471 *
P. G. BARTEN: "Contrast Sensitivity of the Human Eye and Its Effects on Image Quality", CONTRAST SENSIT. HUM. EYE ITS EFF. IMAGE QUAL, no. 1999, 2009
SHUHONG YUAN ET AL: "Multiple clues inspired banding detector in high dynamic range videos", ELECTRONICS LETTERS, THE INSTITUTION OF ENGINEERING AND TECHNOLOGY, GB, vol. 58, no. 7, 18 February 2022 (2022-02-18), pages 280 - 282, XP006114607, ISSN: 0013-5194, DOI: 10.1049/ELL2.12443 *
Z. TUJ. LINY. WANGB. ADSUMILIA.C. BOVIK: "ICASSP, IEEE Int. Cong. Acoust. Speech Sign. Proc", vol. 2020, May 2020, article "Bband Index: A No-Reference Banding Artifact Predictor", pages: 2712 - 2716

Similar Documents

Publication Publication Date Title
US5109282A (en) Halftone imaging method and apparatus utilizing pyramidol error convergence
JP3328934B2 (en) Method and apparatus for fusing images
US6850642B1 (en) Dynamic histogram equalization for high dynamic range images
JP2006058879A (en) Bit-depth extension system and method of digital display via use of model of impulse response of visual system
CN113596573B (en) Image processing method, image processing device, electronic equipment and computer readable storage medium
Tourancheau et al. Impact of subjective dataset on the performance of image quality metrics
CN111784570A (en) Video image super-resolution reconstruction method and device
US8269898B2 (en) Image processing apparatus
JP2004166007A (en) Device, method and program for image processing, and storage medium
CN116188929A (en) Small target detection method and small target detection system
JPH011074A (en) Image processing device
CN107220934A (en) Image rebuilding method and device
CN117934408B (en) Light field image full-reference quality evaluation method based on lens diagram
US10984506B2 (en) Image anti-aliasing method and image anti-aliasing device
CN112200719B (en) Image processing method, electronic device, and readable storage medium
WO2024003589A1 (en) Method for processing an image and corresponding image processing device and systems
KR101152455B1 (en) Apparatus and method for sub-pixel rendering of image display device
US20030164896A1 (en) Information signal processing device, information signal processing method, image signal processing device, image signal processing method and image displaying method using it, coeficient kind data generating device and mehtod used therein, and computer readable meedium and program
US5481376A (en) Method and apparatus for improving image quality
CN112399069A (en) Image encoding method and apparatus, storage medium, and electronic device
CN107292829B (en) Image processing method and device
CN113689333A (en) Image enhancement method and device
CN113538265A (en) Image denoising method and device, computer readable medium and electronic device
CN114845093B (en) Image display method, image display device, and computer-readable storage medium
JP7262032B2 (en) Image processing device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22772567

Country of ref document: EP

Kind code of ref document: A1