CN117474816B - High dynamic range image tone mapping method, system and readable storage medium - Google Patents

High dynamic range image tone mapping method, system and readable storage medium Download PDF

Info

Publication number
CN117474816B
CN117474816B CN202311799099.1A CN202311799099A CN117474816B CN 117474816 B CN117474816 B CN 117474816B CN 202311799099 A CN202311799099 A CN 202311799099A CN 117474816 B CN117474816 B CN 117474816B
Authority
CN
China
Prior art keywords
dynamic range
image
range image
high dynamic
loss function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311799099.1A
Other languages
Chinese (zh)
Other versions
CN117474816A (en
Inventor
庄佳衍
易耕华
陈希玉
叶思超
肖江剑
宋康康
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ningbo Institute of Material Technology and Engineering of CAS
Original Assignee
Ningbo Institute of Material Technology and Engineering of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ningbo Institute of Material Technology and Engineering of CAS filed Critical Ningbo Institute of Material Technology and Engineering of CAS
Priority to CN202311799099.1A priority Critical patent/CN117474816B/en
Publication of CN117474816A publication Critical patent/CN117474816A/en
Application granted granted Critical
Publication of CN117474816B publication Critical patent/CN117474816B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20208—High dynamic range [HDR] image processing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a high dynamic range image tone mapping method, a system and a readable storage medium. The high dynamic range image tone mapping method includes: compressing the high dynamic range image into a low dynamic range image through global mapping; acquiring an enhanced neural network; performing on-line training on the low dynamic range image, and extracting enhancement factors; image enhancement is carried out through enhancement factors, and a mapping image is obtained; the loss function is a nonlinear loss function, and comprises a first type of loss function showing the difference between the image parameters of the iterative image and the statistical ideal value and a second type of loss function showing the difference between the iterative image and the high dynamic range image. The invention achieves high-quality mapping from the high dynamic range image to the low dynamic range display image, and solves the technical problem of limited high dynamic display; by combining traditional global mapping with learning-based image enhancement, learning difficulty and calculation cost are reduced.

Description

High dynamic range image tone mapping method, system and readable storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a high dynamic range image tone mapping method, system, and readable storage medium.
Background
Real scenes tend to have a wide dynamic range and conventional Low Dynamic (LDR) images cannot reproduce the appearance of the real world. In order to meet the requirement of high-quality images, multiple exposure fusion technology is adopted to fuse image information with different exposure levels into a single High Dynamic Range (HDR) image, and the HDR image with a higher and wider dynamic range contains richer scene information, however, due to high cost, the dynamic range supported by many display devices is limited at present, all details of the HDR image are difficult to present, and serious distortion phenomenon occurs in local areas.
In order to solve the problem of the visualization of HDR images, different tone mapping methods have been proposed in the past. Early conventional tone mapping methods had problems with local contrast loss or artifacts. With the rapid development of deep learning, a great breakthrough is made by applying the technical means of deep learning to the field of image processing.
Unlike other image processing problems, however, tone mapping problems do not have ground truth values, and it is difficult to form large amounts of tagged data, which is often required by most deep learning algorithms to learn the mapping relationship of input and output images. Some studies use the best results of multiple conventional tone mapping methods as labels, but the performance of the network is limited by the quality of labels generated by conventional methods.
Therefore, how to avoid the limitation of labels to implement tone mapping of high dynamic range images by using deep learning method is a technical problem to be solved in the art.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a high dynamic range image tone mapping method, a system and a readable storage medium, which solve the technical problems that tone mapping in the field depends on labeled data, a large amount of labeled data is difficult to obtain and the quality of labels is difficult to control.
In order to achieve the purpose of the invention, the technical scheme adopted by the invention comprises the following steps:
in a first aspect, the present invention provides a high dynamic range image tone mapping method based on deep learning, comprising:
providing a high dynamic range image, and compressing the high dynamic range image into a low dynamic range image through global mapping;
acquiring an enhanced neural network under the encoder-decoder structure, wherein the enhanced neural network takes jump connection as a connection mode;
performing on-line training on the low dynamic range image through the enhanced neural network, and extracting an enhancement factor;
image enhancement is carried out on the low dynamic range image through enhancement factors, and a mapping image is obtained;
the nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
Further, the first class of loss functions includes an intensity loss function and/or a contrast loss function.
Further, the second class of loss functions includes any one or a combination of two or more of hue loss functions, saturation loss functions and structure loss functions.
In a second aspect, the present invention also provides a high dynamic range image tone mapping system based on deep learning, comprising:
the global mapping module is used for compressing the high dynamic range image into a low dynamic range image through global mapping;
the network construction module is used for constructing an enhanced neural network under the encoder-decoder structure, and the enhanced neural network takes jump connection as a connection mode;
the on-line training module is used for on-line training the low dynamic range image through the enhanced neural network and extracting an enhanced factor;
the image enhancement module is used for enhancing the image of the low dynamic range image through the enhancement factors to obtain a mapping image;
the nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
In a third aspect, the present invention also provides a readable storage medium having stored therein a computer program which when executed performs the steps of the high dynamic range image tone mapping method described above.
Based on the technical scheme, compared with the prior art, the invention has the beneficial effects that:
aiming at the problem that tone mapping does not have ground truth value and deep learning needs a large number of real labels, the tone mapping method and system provided by the invention designs a group of non-reference loss functions which can be used for tone mapping tasks, effectively guides training of an enhanced neural network under the condition that no matched labels exist, realizes supervision of tone mapping directly through high dynamic range images, finally achieves high-quality mapping from high dynamic range images to low dynamic range display images, and effectively solves the technical problem that high dynamic range images are limited in display; on the other hand, tone mapping is converted into global mapping and image enhancement, and the learning difficulty and the calculation cost are reduced by combining the traditional global mapping and learning-based image enhancement.
The above description is only an overview of the technical solutions of the present invention, and in order to enable those skilled in the art to more clearly understand the technical means of the present application, the present invention may be implemented according to the content of the specification, the following description is given of the preferred embodiments of the present invention with reference to the accompanying drawings.
Drawings
FIG. 1 is a flow chart of a tone mapping method for high dynamic range images according to an exemplary embodiment of the present invention;
FIG. 2 is a diagram showing an exemplary implementation of a tone mapping method for high dynamic range images according to an exemplary embodiment of the present invention;
FIG. 3 is a schematic diagram of an enhanced neural network according to an exemplary embodiment of the present invention;
FIG. 4 is a schematic diagram showing the composition and structure of a tone mapping system for high dynamic range images according to an exemplary embodiment of the present invention
FIG. 5 is a graph showing an example of the effect of different loss functions on image quality in a high dynamic range image tone mapping method according to an exemplary embodiment of the present invention.
Detailed Description
In view of the shortcomings in the prior art, the inventor of the present invention has long studied and practiced in a large number of ways to propose the technical scheme of the present invention. The technical scheme, the implementation process, the principle and the like are further explained as follows.
Specifically, HDR (High-Dynamic Range) is simply a processing technology for improving brightness and contrast of images, and can lighten details of each dark part, darken the dark part, enrich more detail colors, and enable films and pictures to show excellent effects. Let the user more closely look at the visual perception in real environment when watching, this is what HDR exists. The traditional SDR (standard contrast) has the highest brightness of only 100nit, the part higher than 100nit in the picture is distorted (lost), the lowest debugging is 0.1nit, and the part lower than 0.1nit in the picture is lost. The HDR technology is developed, the highest brightness reaches thousands nit, the lowest brightness reaches 0.0005nit, the details of the parts with the brightness higher than 100nit and lower than 0.1nit in the picture are greatly expanded, and meanwhile, the whole picture is more transparent and clear and has rich details. More abundant scene information results in higher dynamic range for HDR, but some real-world devices do not support such high dynamic range displays, require tone mapping to adjust their dynamic range, and maintain as rich a level of detail as possible. In order to enable HDR to be applied in most current dynamic range limited devices, the present invention proposes a high dynamic range image tone mapping method based on deep learning.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced otherwise than as described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
Moreover, relational terms such as "first" and "second", and the like, may be used solely to distinguish one from another component or method step having the same name, without necessarily requiring or implying any actual such relationship or order between such components or method steps.
Referring to fig. 1, an embodiment of the present invention provides a high dynamic range image tone mapping method based on deep learning, which includes the steps of:
s1, providing a high dynamic range image, and compressing the high dynamic range image into a low dynamic range image through global mapping.
S2, acquiring an enhanced neural network under the encoder-decoder structure, wherein the enhanced neural network takes jump connection as a connection mode.
S3, performing online training on the low dynamic range image through the enhanced neural network, and extracting an enhanced factor.
S4, performing image enhancement on the low dynamic range image through the enhancement factors to obtain a mapping image.
In the step S3, the loss function of the online training is a nonlinear loss function, and the nonlinear loss function includes a first type of loss function that reflects a difference between an image parameter and a statistical ideal value of an iterative image formed in the online training process, and a second type of loss function that reflects a difference between the iterative image and the high dynamic range image.
The prior art is mainly based on supervised learning methods for tone mapping, and therefore often requires a large number of labeled datasets for training. The technical scheme provided by the invention designs a series of non-reference loss functions by taking the contrast, intensity, saturation, hue and brightness structure of the LDR image into consideration and utilizing the statistical information of the LDR image dataset and a given HDR image based on unsupervised consideration; based on the loss function, the mapping method provided by the invention realizes tone mapping of the HDR image in an on-line training mode, no tag data is needed, various problems caused by the tag data are avoided, and unsupervised tone mapping is realized.
And in particular in some embodiments the statistical ideal may comprise, for example, average parameter information obtained from existing image dataset statistics.
As an example, some of the statistical ideal values and intensities in the first class of loss functions are based on the statistical information of a public dataset Imagenet dataset, i.e. the mean, variance, etc. of the dataset image. Of course, even if statistics are made from other common data sets, it is possible to count the image average parameters of the higher quality image data collected by itself.
In some embodiments, the first class of loss functions includes an intensity loss function and/or a contrast loss function.
In some embodiments, the second class of loss functions includes any one or a combination of two or more of hue loss functions, saturation loss functions, structure loss functions.
In some embodiments, the intensity loss function may be expressed, for example, as:
；
wherein,as strength lossFailure of value, ->For the average intensity of the color channels of the iterative image, < >>The average ideal intensity of the color channels is obtained for statistics.
In some embodiments, the contrast loss function may be expressed, for example, as:
；
wherein,for contrast loss value, +.>For the standard deviation of the color channels of the iterative image,the mean ideal standard deviation of the color channels obtained for statistics.
In some embodiments, the hue loss function may be expressed, for example, as:
；
wherein,is hue loss value, N is total pixel point, i is constant with value of 1-N, < ->For the red-green component of the high dynamic range image in IPT space, +.>For yellow Lan Fenliang of the high dynamic range image in IPT space,for the red-green component of the iterative image in IPT space, +.>For the yellow-blue component of the iterative image in IPT space.
In some embodiments, the saturation loss function may be expressed, for example, as:
；
wherein,for saturation loss value, +_>For the saturation of the ith pixel point in the high dynamic range image +.>And the saturation of the ith pixel point in the iterative image.
In some embodiments, the structural loss function may be expressed, for example, as:
；
；
；
wherein,for the structural loss value, +.>Is a multi-scale structural similarity index>For the low dynamic range image, +.>For the iterative image; />Is the weight of the nth scale, sigma x 、σ y Local standard deviation, sigma, between corresponding blocks in the high dynamic range image and the iterative image, respectively xy Cross-correlation coefficients for the high dynamic range image and the iterative image>Is->Global mapped version, C 1 And C 2 Are all stable constants.
With respect to the utilization of a plurality of loss functions, in some embodiments, a plurality of the first type of loss functions and the second type of loss functions are added according to respective weights as the nonlinear loss functions. A specific embodiment may be, for example, a weighted average method, for example, sequentially giving different weights V1-V5 to the intensity loss function, the contrast loss function, the hue loss function, the saturation loss function, and the structure loss function, and then performing an addition operation to obtain a final total nonlinear loss function, so as to guide the online training.
Based on the practical experience of the inventor, in order to achieve better mapping quality and efficiency, the weight values of different loss functions are different, specifically, the weight range of the intensity loss function is generally set to 40-60, the weight range of the contrast loss function is generally set to 1-3, the weight range of the hue loss function is generally set to 80-120, the weight range of the saturation loss function is generally set to 1-3, and the weight range of the structure loss function is generally set to 1-3. Of course, scaling up and down the weight values in equal proportion is still equivalent to the range of values using the weight values.
Of course, the method is not limited to the above-mentioned range, but generally follows a similar rule of magnitude, as shown in fig. 5, the intensity loss and the contrast loss affect the brightness and the contrast, the saturation loss and the hue loss affect the true color of the image, and the structural loss results in loss of details, so based on experience, the invention summarizes and derives the weight range of each loss function applicable to the technical scheme provided by the embodiment of the invention.
While details regarding the remaining steps of the solution provided by the present invention, in some embodiments, the global mapping includes sequentially performing luminance compression and color restoration on the high dynamic range image.
The luminance compression is expressed as:
；
；
wherein,is an asymmetric parameter>For the brightness of the low dynamic range image, < >>For the brightness of the high dynamic range image, < >>For the average luminance of the high dynamic range image, +.>For the maximum brightness of the high dynamic range image,to be the instituteMinimum brightness of the high dynamic range image.
The color recovery is expressed as:
；
wherein,color channel intensity for color restored low dynamic range image, +.>Color channel intensity for high dynamic range images.
In some embodiments, the convolutional layer in the enhanced neural network adopts mixed hole convolution, the output layer in the active layer adopts sigmod as an activation function, and the rest layers adopt LeakyReLU as an activation function.
In some embodiments, the image enhancement employs pixel-level recursive enhancement expressed as:
；
wherein,for the enhancement factor, < >>For the output of the n-th step of recursion enhancement, n is the number of recursions, and when the number of recursions is 1,/is>Is the low dynamic range image input.
Of course, the manner of enhancement should be consistent with the final image enhancement when training is performed.
Corresponding to the above-mentioned high dynamic range image tone mapping method, the embodiment of the invention also provides a high dynamic range image tone mapping system based on deep learning, which comprises:
and the global mapping module is used for compressing the high dynamic range image into a low dynamic range image through global mapping.
And the network construction module is used for constructing an enhanced neural network under the encoder-decoder structure, and the enhanced neural network takes jump connection as a connection mode.
And the online training module is used for carrying out online training on the low dynamic range image through the enhanced neural network and extracting an enhanced factor.
And the image enhancement module is used for carrying out image enhancement on the low dynamic range image through the enhancement factors to obtain a mapping image.
The nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
Correspondingly, the embodiment of the invention also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed to execute the steps of the high dynamic range image tone mapping method.
The technical scheme of the invention is further described in detail below through a plurality of embodiments and with reference to the accompanying drawings. However, the examples are chosen to illustrate the invention only and are not intended to limit the scope of the invention.
Example 1
The present embodiment illustrates a process of a tone mapping method for a high dynamic range image, specifically as follows:
s1: the image is compressed from a high dynamic range image to a low dynamic range image by global mapping.
S2: an enhanced neural network under the encoder-decoder structure is constructed and takes a jump connection as a connection mode.
S3: the low dynamic range image is trained online through the enhanced neural network to extract the enhancement factors.
S4: the low dynamic range image is pixel-level enhanced by an enhancement factor.
Considering that the brightness at the same pixel point between the low dynamic range image and the high dynamic range image always has a linear relationship, in this embodiment, a compression object for global mapping is selected by using a brightness channel, and the compression is implemented through an asymmetric omentum response model, and the specific expression is as follows:
；
；
in the method, in the process of the invention,is an asymmetric parameter>For the brightness of low dynamic range images, +.>For the brightness of high dynamic range images, +.>For the average luminance of the high dynamic range image, +.>Maximum brightness for high dynamic range image, +.>Is the minimum brightness of the high dynamic range image.
However, in view of the calculation of luminance channels (taking a high dynamic range image as an example,) Is calculated according to the channel intensity of each color channel, thus, the brightness is switched onAfter the channels are compressed, the channels of each color are required to be restored, and the restoration formula is as follows:
；
in the method, in the process of the invention,color channel intensity for low dynamic range image after color recovery,/->Color channel intensity for high dynamic range images.
Further, as shown in fig. 3, in order to improve the receptive field of the enhancement neural network, in this embodiment, it is proposed that the encoder-decoder enhancement neural network constructed uses mixed-hole convolution instead of the common convolution layer, and the convolution kernels are all 3×3 in size and 1 in step length. Besides, the active layer uses the sigmod as an active function except the output layer, and the other layers use the LeakyReLU as an active function. Meanwhile, in order to avoid the problem of gradient disappearance, a jump connection mode is adopted among layers.
To solve the problem of tag loss, the online training in this embodiment is performed under the guidance of a series of nonlinear loss functions, which considers the constraint on training by each loss function, and ensures the reliability and quality of training even without tags. Specific loss functions include an intensity loss function, a contrast loss function, a hue loss function, a saturation loss function, and a structure loss function.
Wherein the intensity loss function is expressed as the following formula:
；
in the method, in the process of the invention,for strength loss->For the average intensity of the color channels of the iterative image formed in the training,is the ideal intensity of the color channel.
The contrast loss function is expressed as the following formula:
；
in the method, in the process of the invention,for contrast loss +.>To the standard deviation of the color channels of the iterative images formed in the training,is an ideal value for the color channel.
The hue loss function is expressed as the following formula:
；
in the method, in the process of the invention,for hue loss, N is the total number of pixels, i is a constant with a value of 1, N, P is the red-green component in IPT space, and T is yellow Lan Fenliang in IPT space (where subscript ldr represents the iterative image formed in training, rather than the low dynamic range image obtained via global mapping).
The saturation loss function is expressed as the following formula:
；
in the method, in the process of the invention,for saturation loss, +_>Saturation for the i-th pixel in the high dynamic range image, < >>Saturation for the i-th pixel in the low dynamic range image (similarly, the subscript ldr represents the iterative image formed in the training, rather than the low dynamic range image obtained via global mapping).
The structural loss function is expressed as the following formula:
；
in the method, in the process of the invention,for structural loss->Is a multi-scale structural similarity index.
；
Is the weight of the nth scale; (the value of n can be set to 5 in general, but can be adjusted up and down
；
Wherein sigma x 、σ y Sum sigma xy The local standard deviation and cross-correlation between corresponding blocks in the HDR and LDR images respectively,is->Reflecting the importance of signal strength, C 1 、C 2 Is a stable constant.
The multiple loss functions are added and integrated according to a certain weight, and the weighted average obtains a total nonlinear loss function to guide the training, specifically, the weight of the intensity loss function 50, the weight of the contrast loss function 1, the weight of the hue loss function 100, the weight of the saturation loss function 1 and the weight of the structure loss function 1 are given in the embodiment.
Finally, the enhancement factors obtained through training carry out pixel-level enhancement under the recursion enhancement on the low dynamic range image, so that the high dynamic range image is finally converted into a high-quality low dynamic range image, and the high dynamic range image is suitable for HDR image presentation of dynamic range limited equipment. The specific formula expression for pixel enhancement is as follows:
；
in the method, in the process of the invention,to enhance the factor->For the output of the n-th recursion enhancement, n is the number of recursions, when the number of recursions is 1 (i.e. the first round of recursion),>is an input low dynamic range image.
In the specific implementation process, referring to fig. 2, taking a high dynamic range image as an example, the display effect is as shown in the first image in fig. 2, and is limited by the dynamic range of the display device, so that details cannot be clearly presented; the second image is obtained through global mapping, so that the dynamic range is reduced, the whole is clearer, and a large number of defects still exist in local details; and the enhancement factors extracted through training are shown as a third image, the second image is enhanced in pixel level by utilizing the enhancement factors, and a final mapping image shown as a fourth chapter image is obtained, so that the local details are full and clear, and the display effect is better.
Example 2
For better understanding of the technical content of the present invention, the present embodiment illustrates the present invention by way of a system structure, as shown in fig. 4, a high dynamic range image tone mapping system based on deep learning, comprising:
and the global mapping module is used for mapping the image from the high dynamic range image to the low dynamic range image.
And the network construction module is used for constructing an enhancement network under the encoder-decoder structure and takes jump connection as a connection mode.
And the online training module is used for carrying out online training on the low dynamic range image through the enhancement network so as to extract the enhancement factors.
And the pixel enhancement module is used for carrying out pixel-level enhancement on the low dynamic range image through the enhancement factors.
Further, in the global mapping module, the global mapping is specifically to compress the brightness channel and then perform color recovery.
Further, in the pixel enhancement module, pixel-level enhancement is achieved by recursive enhancement.
Based on the above embodiments, it can be clear that, according to the high dynamic range image tone mapping method and system provided by the embodiments of the present invention, a set of non-reference loss functions that can be used for tone mapping tasks are designed, training of an enhanced neural network is effectively guided without pairing tags, supervision of tone mapping directly through high dynamic range images is achieved, and finally high quality mapping from high dynamic range images to low dynamic range display images is achieved, and the technical problem that high dynamic range images are limited in display is effectively solved; on the other hand, tone mapping is converted into global mapping and image enhancement, and the learning difficulty and the calculation cost are reduced by combining the traditional global mapping and learning-based image enhancement.
It should be understood that the above embodiments are merely for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the present invention and implement the same according to the present invention without limiting the scope of the present invention. All equivalent changes or modifications made in accordance with the spirit of the present invention should be construed to be included in the scope of the present invention.
It should be noted that all directional indicators (such as up, down, left, right, front, and rear … …) in the embodiments of the present invention are merely used to explain the relative positional relationship, movement, etc. between the components in a particular posture (as shown in the drawings), and if the particular posture is changed, the directional indicator is changed accordingly.
Furthermore, descriptions such as those referred to herein as "first," "second," "a," and the like are provided for descriptive purposes only and are not to be construed as indicating or implying a relative importance or an implicit indication of the number of features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
In the present invention, unless specifically stated and limited otherwise, the terms "connected," "affixed," and the like are to be construed broadly, and for example, "affixed" may be a fixed connection, a removable connection, or an integral body; can be mechanically or electrically connected; either directly or indirectly, through intermediaries, or both, may be in communication with each other or in interaction with each other, unless expressly defined otherwise. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In addition, the technical solutions of the embodiments of the present invention may be combined with each other, but it is necessary to be based on the fact that those skilled in the art can implement the technical solutions, and when the technical solutions are contradictory or cannot be implemented, the combination of the technical solutions should be considered as not existing, and not falling within the scope of protection claimed by the present invention.

Claims (6)

1. A high dynamic range image tone mapping method based on deep learning, comprising:
providing a high dynamic range image, and compressing the high dynamic range image into a low dynamic range image through global mapping;
acquiring an enhanced neural network under the encoder-decoder structure, wherein the enhanced neural network takes jump connection as a connection mode;
performing on-line training on the low dynamic range image through the enhanced neural network, and extracting an enhancement factor;
image enhancement is carried out on the low dynamic range image through enhancement factors, and a mapping image is obtained;
the method comprises the steps that a nonlinear loss function is selected as a loss function of online training, and the nonlinear loss function comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of iterative images formed in the online training process and a second type of loss function reflecting the difference between the iterative images and the high dynamic range images;
the first class of loss functions includes an intensity loss function and a contrast loss function; the second class of loss functions comprises hue loss functions, saturation loss functions and structure loss functions;
the intensity loss function is expressed as:
；
wherein,for the intensity loss value, +.>Is saidAverage intensity of color channels of iterative image, +.>The average ideal intensity of the color channels obtained for statistics;
the contrast loss function is expressed as:
；
wherein,for contrast loss value, +.>Standard deviation for color channel of the iterative image, +.>The average ideal standard deviation of the color channels obtained for statistics;
the hue loss function is expressed as:
；
wherein,is hue loss value, N is total pixel point, i is constant with value of 1-N, < ->For the red-green component of the high dynamic range image in IPT space, +.>Yellow Lan Fenliang in IPT space for the high dynamic range image, +.>For the red-green component of the iterative image in IPT space, +.>Yellow Lan Fenliang in IPT space for the iterative image;
the saturation loss function is expressed as:
；
wherein,for saturation loss value, +_>For the saturation of the ith pixel point in the high dynamic range image +.>The saturation of the ith pixel point in the iterative image is obtained;
the structural loss function is expressed as:
；
；
；
wherein,for the structural loss value, +.>Is a multi-scale structural similarity index>For the low dynamic range image, +.>For the iterative image; />Is the weight of the q-th scale, sigma x 、σ y Local standard deviation, sigma, between corresponding blocks in the high dynamic range image and the iterative image, respectively xy For the cross-correlation coefficients of the high dynamic range image and the iterative image, < >>Is->Global mapped version, C 1 And C 2 Are all stable constants;
adding the first class of loss functions and the second class of loss functions according to corresponding weights to serve as the nonlinear loss functions;
the image enhancement employs pixel-level recursive enhancement expressed as:
；
wherein,for the enhancement factor, < >>For the output of the mth step of recursion enhancement, m is the number of recursions, and when the number of recursions is 1, +.>Is the low dynamic range image input.
2. The high dynamic range image tone mapping method of claim 1, wherein said statistical idealities comprise average parameter information obtained from existing image dataset statistics.
3. The high dynamic range image tone mapping method according to claim 1, wherein the global mapping comprises sequentially performing luminance compression and color recovery on the high dynamic range image;
the luminance compression is expressed as:
；
；
wherein,is an asymmetric parameter>For the brightness of the low dynamic range image, < >>For the brightness of the high dynamic range image, < >>For the average luminance of the high dynamic range image, +.>For the maximum brightness of the high dynamic range image, < >>Minimum brightness for the high dynamic range image;
the color recovery is expressed as:
；
wherein,color channel intensity for color restored low dynamic range image, +.>Color channel intensity for high dynamic range images.
4. The high dynamic range image tone mapping method of claim 1, wherein the convolutional layer in the enhanced neural network employs mixed-hole convolutional, the output layer in the active layer employs sigmod as the active function, and the remaining layers employ LeakyReLU as the active function.
5. A deep learning-based high dynamic range image tone mapping system for implementing the high dynamic range image tone mapping method of any of claims 1-4, comprising:
the global mapping module is used for compressing the high dynamic range image into a low dynamic range image through global mapping;
the network construction module is used for constructing an enhanced neural network under the encoder-decoder structure, and the enhanced neural network takes jump connection as a connection mode;
the on-line training module is used for on-line training the low dynamic range image through the enhanced neural network and extracting an enhanced factor;
the image enhancement module is used for enhancing the image of the low dynamic range image through the enhancement factors to obtain a mapping image;
the nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
6. A readable storage medium, characterized in that the readable storage medium has stored therein a computer program which, when executed, performs the steps of the high dynamic range image tone mapping method of any of claims 1-4.
CN202311799099.1A 2023-12-26 2023-12-26 High dynamic range image tone mapping method, system and readable storage medium Active CN117474816B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311799099.1A CN117474816B (en) 2023-12-26 2023-12-26 High dynamic range image tone mapping method, system and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311799099.1A CN117474816B (en) 2023-12-26 2023-12-26 High dynamic range image tone mapping method, system and readable storage medium

Publications (2)

Publication Number Publication Date
CN117474816A CN117474816A (en) 2024-01-30
CN117474816B true CN117474816B (en) 2024-03-12

Family

ID=89625930

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311799099.1A Active CN117474816B (en) 2023-12-26 2023-12-26 High dynamic range image tone mapping method, system and readable storage medium

Country Status (1)

Country Link
CN (1) CN117474816B (en)

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015180854A1 (en) * 2014-05-28 2015-12-03 Koninklijke Philips N.V. Methods and apparatuses for encoding an hdr images, and methods and apparatuses for use of such encoded images
CN108198152A (en) * 2018-02-07 2018-06-22 广东欧珀移动通信有限公司 Image processing method and device, electronic equipment, computer readable storage medium
CN108681991A (en) * 2018-04-04 2018-10-19 上海交通大学 Based on the high dynamic range negative tone mapping method and system for generating confrontation network
CN109410126A (en) * 2017-08-30 2019-03-01 中山大学 A kind of tone mapping method of details enhancing and the adaptive high dynamic range images of brightness
CN110163808A (en) * 2019-03-28 2019-08-23 西安电子科技大学 A kind of single frames high dynamic imaging method based on convolutional neural networks
CN110197463A (en) * 2019-04-25 2019-09-03 深圳大学 High dynamic range image tone mapping method and its system based on deep learning
CN110796595A (en) * 2019-10-31 2020-02-14 北京大学深圳研究生院 Tone mapping method and device and electronic equipment
CN111709900A (en) * 2019-10-21 2020-09-25 上海大学 High dynamic range image reconstruction method based on global feature guidance
CN112837229A (en) * 2019-11-25 2021-05-25 英特尔公司 Unsupervised training of neural networks for high dynamic range image compression
CN113095991A (en) * 2021-04-15 2021-07-09 电子科技大学 High dynamic range image steganography and recovery method based on deep learning
CN113822830A (en) * 2021-08-30 2021-12-21 天津大学 Multi-exposure image fusion method based on depth perception enhancement
CN115526803A (en) * 2022-10-14 2022-12-27 中国石油大学(华东) Non-uniform illumination image enhancement method, system, storage medium and device
CN115619682A (en) * 2022-11-04 2023-01-17 南京大学 Deep learning-based denoising tone mapping method and device
CN116563183A (en) * 2023-04-06 2023-08-08 北京理工大学 High dynamic range image reconstruction method and system based on single RAW image

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015180854A1 (en) * 2014-05-28 2015-12-03 Koninklijke Philips N.V. Methods and apparatuses for encoding an hdr images, and methods and apparatuses for use of such encoded images
CN109410126A (en) * 2017-08-30 2019-03-01 中山大学 A kind of tone mapping method of details enhancing and the adaptive high dynamic range images of brightness
CN108198152A (en) * 2018-02-07 2018-06-22 广东欧珀移动通信有限公司 Image processing method and device, electronic equipment, computer readable storage medium
CN108681991A (en) * 2018-04-04 2018-10-19 上海交通大学 Based on the high dynamic range negative tone mapping method and system for generating confrontation network
CN110163808A (en) * 2019-03-28 2019-08-23 西安电子科技大学 A kind of single frames high dynamic imaging method based on convolutional neural networks
CN110197463A (en) * 2019-04-25 2019-09-03 深圳大学 High dynamic range image tone mapping method and its system based on deep learning
CN111709900A (en) * 2019-10-21 2020-09-25 上海大学 High dynamic range image reconstruction method based on global feature guidance
CN110796595A (en) * 2019-10-31 2020-02-14 北京大学深圳研究生院 Tone mapping method and device and electronic equipment
CN112837229A (en) * 2019-11-25 2021-05-25 英特尔公司 Unsupervised training of neural networks for high dynamic range image compression
CN113095991A (en) * 2021-04-15 2021-07-09 电子科技大学 High dynamic range image steganography and recovery method based on deep learning
CN113822830A (en) * 2021-08-30 2021-12-21 天津大学 Multi-exposure image fusion method based on depth perception enhancement
CN115526803A (en) * 2022-10-14 2022-12-27 中国石油大学(华东) Non-uniform illumination image enhancement method, system, storage medium and device
CN115619682A (en) * 2022-11-04 2023-01-17 南京大学 Deep learning-based denoising tone mapping method and device
CN116563183A (en) * 2023-04-06 2023-08-08 北京理工大学 High dynamic range image reconstruction method and system based on single RAW image

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
High Dynamic Range Image Tone Mapping: Literature review and performance benchmark;Xueyu Han;Digital Signal Processing;20230529;1-20 *
Real-time self-supervised tone curve estimation for HDR image;Xiyu Chen 等;Computers & Graphics;20230722;461-471 *
基于亮度自适应分段的高动态图像色调映射算法;刘颖;王倩;刘卫华;;电视技术;20180105(01);24-30 *
高动态范围图像色调映射算法研究;贾爱秀;中国优秀硕士学位论文全文数据库 信息科技辑;20200715(第07期);I138-612 *

Also Published As

Publication number Publication date
CN117474816A (en) 2024-01-30

Similar Documents

Publication Publication Date Title
US10970830B2 (en) Image style conversion method, apparatus and device
CN108830813A (en) A kind of image super-resolution Enhancement Method of knowledge based distillation
CN109559276B (en) Image super-resolution reconstruction method based on quality evaluation and feature statistics
CN107016415B (en) A kind of color image Color Semantic classification method based on full convolutional network
CN109671023A (en) A kind of secondary method for reconstructing of face image super-resolution
CN111681177B (en) Video processing method and device, computer readable storage medium and electronic equipment
CN111951164B (en) Image super-resolution reconstruction network structure and image reconstruction effect analysis method
CN113628152B (en) Dim light image enhancement method based on multi-scale feature selective fusion
CN109257168A (en) A kind of generation method of picture validation code, device, equipment and medium
CN104408697B (en) Image Super-resolution Reconstruction method based on genetic algorithm and canonical prior model
CN112950640A (en) Video portrait segmentation method and device, electronic equipment and storage medium
CN116958534A (en) Image processing method, training method of image processing model and related device
CN114140786A (en) Scene text recognition method based on HRNet coding and double-branch decoding
CN115393227A (en) Self-adaptive enhancing method and system for low-light-level full-color video image based on deep learning
US20240205376A1 (en) Image processing method and apparatus, computer device, and storage medium
CN117474816B (en) High dynamic range image tone mapping method, system and readable storage medium
CN111753714A (en) Multidirectional natural scene text detection method based on character segmentation
CN110489584B (en) Image classification method and system based on dense connection MobileNet model
CN111754412A (en) Method and device for constructing data pairs and terminal equipment
CN113627342B (en) Method, system, equipment and storage medium for video depth feature extraction optimization
CN114782249A (en) Super-resolution reconstruction method, device and equipment for image and storage medium
CN114299105A (en) Image processing method, image processing device, computer equipment and storage medium
CN113592765A (en) Image processing method, device, equipment and storage medium
CN111062862A (en) Color-based data enhancement method and system, computer device and storage medium
Liang et al. Grayscale Image Colorization with GAN and CycleGAN in Different Image Domain

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant