CN112508815A - Model training method and device, electronic equipment and machine-readable storage medium - Google Patents
Model training method and device, electronic equipment and machine-readable storage medium Download PDFInfo
- Publication number
- CN112508815A CN112508815A CN202011449091.9A CN202011449091A CN112508815A CN 112508815 A CN112508815 A CN 112508815A CN 202011449091 A CN202011449091 A CN 202011449091A CN 112508815 A CN112508815 A CN 112508815A
- Authority
- CN
- China
- Prior art keywords
- image
- brightening
- hsv
- channel
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000012549 training Methods 0.000 title claims abstract description 61
- 238000005282 brightening Methods 0.000 claims abstract description 157
- 238000005286 illumination Methods 0.000 claims abstract description 57
- 238000012545 processing Methods 0.000 claims abstract description 21
- 230000006870 function Effects 0.000 claims description 84
- 230000002776 aggregation Effects 0.000 claims description 15
- 238000004220 aggregation Methods 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 30
- 238000010586 diagram Methods 0.000 description 15
- 238000013528 artificial neural network Methods 0.000 description 5
- 239000011159 matrix material Substances 0.000 description 5
- 238000007796 conventional method Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000001965 increasing effect Effects 0.000 description 3
- 230000002708 enhancing effect Effects 0.000 description 2
- 230000005284 excitation Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000003042 antagnostic effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000265 homogenisation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 230000002207 retinal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
A training method of a low-illumination image enhancement model is provided, and comprises the following steps: receiving a low-illumination RGB image, and converting the low-illumination RGB image into an HSV image; carrying out brightening treatment on the HSV image by using a depth brightening network to obtain an HSV brightening image, and converting the HSV brightening image into an RGB brightening image; and calculating to obtain a loss value according to the RGB brightening image, the normal illumination image and the loss function, and updating the network parameters of the depth brightening network according to the calculated loss value. A training apparatus, an electronic device, and a machine-readable storage medium for a low-illumination image enhancement model are also provided. The method can ensure that the model obtained by training can not generate the phenomena of color cast and overexposure on the predicted image obtained by processing the low-illumination image, can better predict the details of the image and the whole image more accurately, and can obtain better visual intuition.
Description
Technical Field
The invention belongs to the technical field of image processing and machine learning, and particularly relates to a training method and a training device for a low-illumination image enhancement model, electronic equipment and a machine-readable storage medium.
Background
The photographing is one of the most convenient ways to record various memorable moments in our lives. It is conventionally unavoidable to take pictures in low light, but pictures taken in low light conditions are often very dark, which makes it difficult to discern a scene or object. In the case where it is daily desired to obtain a high-visibility image and in some public places, flash light cannot be used, only sensitivity and exposure can be increased, but increasing sensitivity causes much noise and increasing exposure blurs the image. A number of conventional methods have been proposed to mitigate the degradation caused by weak light, such as HE (histogram equalization), but neural networks have better characterization than conventional methods.
The traditional method for processing the low-illumination image by using the neural network mainly comprises the following steps: based on retinal theory (Retinex theory) and on Generative Antagonistic Networks (GANs). However, both conventional methods do not accurately restore the edges and partial details of low-light images well.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides a training method and a training device for a low-illumination image enhancement model, an electronic device and a machine-readable storage medium, wherein the training method and the training device can accurately restore the edges and the details of a low-illumination image.
According to an aspect of an embodiment of the present invention, there is provided a training method for a low-light image enhancement model, the training method including: receiving a low-illumination RGB image, and converting the low-illumination RGB image into an HSV image; carrying out brightening treatment on the HSV image by using a depth brightening network to obtain an HSV brightening image, and converting the HSV brightening image into an RGB brightening image; and calculating to obtain a loss value according to the RGB brightening image, the normal illumination image and the loss function, and updating the network parameters of the depth brightening network according to the calculated loss value.
In an example of the training method for the low-illumination image enhancement model provided in the above aspect, the loss function includes a structural similarity function, a perceptual loss function, and a total variation function.
In one example of the training method of the low-illumination image enhancement model provided in the above-described aspect, the loss function is expressed as the following equation 1,
[ equation 1] L ═ L _ ssim + L _ per +0.001 × L _ tv
Wherein L represents the loss function, L _ ssim represents the structural similarity function, L _ per represents the perceptual loss function, and L _ tv represents the total variation function.
In an example of the training method for the low-illumination image enhancement model provided in the above aspect, the perceptual loss function includes an image loss function, a feature reconstruction loss function, and a style reconstruction loss function, and the image loss function represents a mean square error between the RGB enhanced image and the normal-illumination image.
In an example of the training method for the low-illumination image enhancement model provided in the above aspect, the perceptual loss function is equal to a sum of the image loss function, the feature reconstruction loss function, and the style reconstruction loss function.
In an example of the training method for the low-illumination image enhancement model provided in the foregoing aspect, the brightening the HSV image by using the depth brightening network to obtain an HSV brightening image includes: carrying out brightening treatment on the S channel image and the V channel image of the HSV image by using the depth brightening network to respectively obtain an S channel brightening image and a V channel brightening image; and acquiring an HSV (hue, saturation, value) brightening image according to the H channel image, the S channel brightening image and the V channel brightening image of the HSV image.
In an example of the training method for the low-illumination image enhancement model provided in the foregoing aspect, the performing, by using the depth brightening network, brightening the S-channel image and the V-channel image of the HSV image respectively includes: performing shallow feature extraction on the input channel image by using a shallow feature extraction layer of the depth brightening network to obtain a corresponding feature map; cumulatively brightening the feature map by utilizing a plurality of brightening back projection layers with feature aggregation of the depth brightening network to obtain a brightening feature map; processing the brightening characteristic graph by using a brightening layer of the depth brightening network to obtain a channel brightening image; wherein the input channel image is the S-channel image, and the channel-enhancing image is the S-channel enhancing image, or the input channel image is the V-channel image, and the channel-enhancing image is the V-channel enhancing image.
According to another aspect of the embodiments of the present invention, there is provided a training apparatus for a low-light image enhancement model, the training apparatus including: the receiving module is used for receiving the low-illumination RGB image; a first conversion module for converting the low-illumination RGB image into an HSV image; the depth brightening network module is used for brightening the HSV image by using a depth brightening network to obtain an HSV brightening image; a second conversion module for converting the HSV-brightened image into an RGB-brightened image; the loss value calculation module is used for calculating a loss value according to the RGB brightening image, the normal illumination image and the loss function; and the updating module is used for updating the network parameters of the deep brightening network according to the loss values obtained by calculation.
According to still another aspect of embodiments of the present invention, there is provided an electronic apparatus, including: at least one processor, and a memory coupled with the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method of training a low-light image enhancement model as described above.
According to a further aspect of embodiments of the present invention there is provided a machine readable storage medium having stored thereon executable instructions that, when executed, cause the machine to perform the training method of the low-light image enhancement model as described above.
Has the advantages that: the training method and the training device for the low-illumination image enhancement model can ensure that the model obtained through training can not generate color cast and overexposure phenomena on a predicted image (namely an output image) obtained by processing the low-illumination image, can better predict the details of the image and the whole image more accurately, and can obtain better visual intuition.
Drawings
The above and other aspects, features and advantages of embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow diagram of a method of training a low-light image enhancement model according to an embodiment of the invention;
FIG. 2 is a flowchart of a method for performing a brightening process on an S-channel image and a V-channel image of the HSV image respectively by using a deep brightening network in a training method according to an embodiment of the present invention;
FIG. 3 is a block diagram of a training apparatus for a low-light image enhancement model according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating an electronic device implementing a method of training a model in accordance with an embodiment of the present invention.
Detailed Description
Hereinafter, specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the specific embodiments set forth herein. Rather, these embodiments are provided to explain the principles of the invention and its practical application to thereby enable others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.
As used herein, the term "include" and its variants mean open-ended terms in the sense of "including, but not limited to. The terms "based on," based on, "and the like mean" based at least in part on, "" based at least in part on. The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first," "second," and the like may refer to different or the same object. Other definitions, whether explicit or implicit, may be included below. The definition of a term is consistent throughout the specification unless the context clearly dictates otherwise.
As described above, the conventional method for processing the low-light image by using the neural network cannot accurately restore the edge and the partial details of the low-light image. In order to solve such a technical problem, embodiments according to the present invention provide a training method and a training apparatus for a low-light image enhancement model, which can accurately restore edges and details of a low-light image. The training method can comprise the following steps: receiving a low-illumination RGB image, and converting the low-illumination RGB image into an HSV image; carrying out brightening treatment on the HSV image by using a depth brightening network to obtain an HSV brightening image, and converting the HSV brightening image into an RGB brightening image; and calculating to obtain a loss value according to the RGB brightening image, the normal illumination image and the loss function, and updating the network parameters of the depth brightening network according to the calculated loss value.
Therefore, the model trained according to the training method converts the low-illumination RGB image into the HSV image, and then carries out brightening treatment on the HSV image, so that the phenomena of color cast and over exposure of the RGB brightening image (namely the RGB brightening image converted from the brightened HSV image, namely the predicted normal-illumination image or the predicted output image) can be avoided, the image details and the image whole can be more accurately predicted, and better visual intuition can be obtained.
The following describes in detail a training method and a training apparatus of a low-light image enhancement model capable of accurately restoring edges and details of a low-light image according to an embodiment of the present invention with reference to the drawings. .
The training method of the low-illumination image enhancement model according to the embodiment of the invention can be executed by an electronic device. The electronic device may include a smartphone, a tablet, a personal computer, a cloud server, a server, and the like.
FIG. 1 is a flow diagram of a method of training a low-light image enhancement model according to an embodiment of the invention.
Referring to fig. 1, at block 101, a low-light RGB image is received and converted into an HSV image.
In one example, since the model training is performed in a loop before the loop end condition is not satisfied, the low-light RGB image received here is obtained by performing a darkening process on the RGB brightened image (which may also be referred to as a predicted normal-light image or a predicted output image) obtained in the previous loop process.
In this case, the RGB brightened image obtained in the previous cycle is subjected to the darkening processing to obtain the low-light RGB image in the present cycle. Further, the low-illumination RGB image in the circulation process is converted into the HSV image in the circulation process.
At block 103, the HSV image is brightened by a depth brightening network to obtain an HSV brightened image, and the HSV brightened image is converted into an RGB brightened image.
In one example, the HSV brightening image in the current cycle process is subjected to the brightening process by using the depth brightening network to obtain the HSV brightening image in the current cycle process, and the HSV brightening image in the current cycle process is converted into the RGB brightening image in the current cycle process.
Specifically, in one example, a depth brightening network is used to respectively brighten an S-channel image and a V-channel image of the HSV image, so as to respectively obtain an S-channel brightening image and a V-channel brightening image. The following describes in detail a specific process of performing a brightening process on the S-channel image and the V-channel image of the HSV image respectively by using a depth brightening network.
Fig. 2 is a flowchart of a method for performing a brightening process on an S-channel image and a V-channel image of the HSV image respectively by using a deep brightening network in a training method according to an embodiment of the present invention.
Referring to fig. 2, in step S202, shallow feature extraction is performed on the input channel image by using a shallow feature extraction layer of the depth brightening network to obtain a corresponding feature map.
In one example, the shallow feature extraction layer includes two convolutional layers, each having 64 filters of 3 x 3 with a step size of 1 and a fill of 1. In this case, the input channel image is convolved with the two convolution layers to obtain the corresponding feature map.
In step S204, the feature map is cumulatively brightened by using a plurality of brightening back projection layers with feature aggregation of the depth brightening network to obtain a brightened feature map.
In one example, the process of brightened backprojection of an image by a brightened backprojection layer comprises: firstly, performing first brightening processing on an input image (such as the characteristic diagram) to obtain a first brightening image; then, performing darkening treatment on the first brightening image to obtain a first darkening image; then, obtaining a residual error between the first darkening graph and the input image to obtain a residual error graph; then, carrying out secondary brightening treatment on the residual error image to obtain a brightening residual error image; and finally, obtaining a brightening output image by utilizing the brightening residual image and the first brightening image.
Therefore, brightening the rear projection layer to process the image to obtain a brightened output image can be represented by equation 1 below.
[ formula 1]Y＝λ2L1(X)+L2(D(L1(X))-λ1X)
Where X represents the input image (low-light image), Y represents the brightened output image, and L1() Indicating a first brightening treatment, L2() Represents the second brightening process, D () represents the darkening process, lambda1And λ2Are two weight parameters that balance the residual update.
In one example, the feature aggregation is to aggregate the feature map processed by the unlighted backprojection layer and the feature map processed by the brighted backprojection layer.
In one example, three brightening backprojection layers and two feature aggregation layers of a depth brightening network. Wherein the second brightening rear projection layer is preceded by a first characteristic aggregation layer and the third brightening rear projection layer is preceded by a second characteristic aggregation layer. The following describes a process of processing an input feature map by using three brightening back-projection layers and two feature aggregation layers, specifically: firstly, a first brightening back-projection layer carries out first brightening back-projection processing on an input feature map to obtain a first map; then, the first feature aggregation layer carries out first feature aggregation processing on the input feature graph and the first graph to obtain a graph obtained after the first feature aggregation processing; then, the second brightening back projection layer carries out second brightening back projection processing on the image obtained after the first characteristic polymerization processing to obtain a second image; then, the second characteristic aggregation layer carries out second characteristic aggregation processing on the input characteristic diagram, the first diagram and the second diagram to obtain a diagram obtained after the second characteristic aggregation processing; then, the third brightening back projection layer carries out third brightening back projection processing on the image obtained after the first feature aggregation processing to obtain a third image; and finally, accumulating the input feature map, the first map, the second map and the third map together to form a brightening feature map.
In step S206, the brightening feature map is processed by the brightening layer of the depth brightening network to obtain a channel brightening image.
In one example, the brightening layer includes two convolutional layers, each having 64 filters of 3 x 3 with a step size of 1 and a fill of 1. In this case, the highlight feature map is convolved with the two convolution layers to obtain a residual between the highlight feature map and a normal-luminance image (target image).
And then, overlapping the product of the residual error and the interaction factor and the input feature map to obtain a channel brightening image.
In the above steps S202 to S206, the input channel image is the S-channel image, and the channel-up image is the S-channel-up image, or the input channel image is the V-channel image, and the channel-up image is the V-channel-up image.
Continuing to refer to fig. 1, at block 105, a loss value is calculated according to the RGB highlight image, the normal illumination image, and the loss function, and a network parameter of the depth highlight network is updated according to the calculated loss value.
In one example, a loss value in the current cycle process is calculated according to the RGB brightening image, the normal illumination image and the loss function in the current cycle process, and the network parameter of the depth brightening network is updated according to the loss value in the current cycle process, so as to serve as the depth brightening network in the next cycle process when the cycle condition is not satisfied.
In one example, the loss functions include a structural similarity function, a perceptual loss function, and a full variational function. In this case, the loss function is expressed as the following equation 2.
[ equation 2] L ═ L _ ssim + L _ per +0.001 × L _ tv
Wherein L represents the loss function, L _ ssim represents the structural similarity function, L _ per represents the perceptual loss function, and L _ tv represents the total variation function.
In one example, images taken under low light conditions often have significant structural distortion problems, and in order to qualitatively and quantitatively improve the quality of the estimation, a Structural Similarity (SSIM) function, which is expressed as equation 3 below, is used in the loss function according to an embodiment of the present invention.
[ formula 3 ]]
Where x and y are the two images to be compared, i.e. the RGB brightened image and the normal illumination image (i.e. the target image), μxAnd muyIs the average of two images, σxAnd σyIs the variance of the two images. c. C1And c2Are two constants to prevent the denominator from being zero.
In one example, a perceptual loss function is also used in the loss function according to embodiments of the present invention. Here, the perceptual loss function includes an image loss function, a feature reconstruction loss function, and a style reconstruction loss function, wherein the image loss function represents a mean square error between the RGB-enhanced image and a normal-illumination image. The feature reconstruction loss function and the style reconstruction loss function will be described below.
Feature reconstruction loss function: rather than having the pixels of the output image (i.e., the RGB-brightened image) exactly match the pixels of the target image (i.e., the normal-illumination image), they have similar feature representations. The characteristic reconstruction loss function can be expressed as the following equation 4.
[ formula 4]
Wherein phi isj(x) Is the excitation of the j-th layer of the neural network phi (e.g., VGG network) when processing low-light images. Where j is a convolution layer, phij(x) Representative shape is Cj×Hj×WjThe characteristic diagram of (1).
Is the RGB-brightened image and y is the normal-light image (i.e., the target image). The feature reconstruction loss function is the squared normalized euclidean distance of the two feature representations.
Style reconstruction loss function: when the output image (i.e., the RGB-brightened image) deviates in content from the target image (i.e., the normal-illumination image), the feature reconstruction loss function trains the output image well. But to get better color, texture, etc., a style reconstruction penalty function is used. As described above, will phij(x) As x excitation of the input at the j-th layer on the neural network phi, the shape of the feature diagram is Cj×Hj×Wj. Defining a Gram (Gram) matrix
Is Cj×CjThe matrix, its elements are represented by equation 5 below.
[ formula 5]
If understand phij(x) Is given as CjDimensional features, one H for each input pointj×WjOn a grid, then
Is to Cj-homogenization of the non-central covariance of the dimensional features, each grid position being treated as an independent sample. Thus, it obtains feature information that each feature tends to deactivate together. This gram matrix can be computed very efficiently by fitting phij(x) Is converted into a shape of Cj×HjWjMatrix phi of, then
Then, this stylistic reconstruction loss function is the squared Flobenius norm between the output image of the gram matrix and the target image, and is represented by equation 6 below.
[6]
Equation 6 shows that generating an image that minimizes the loss of stylistic reconstruction preserves the stylistic characteristics of the target image, but does not preserve its spatial structure.
In summary, the perceptual loss function may be expressed as the following equation 7.
[7]L_per＝image_loss+L_feat+L_style
Wherein L _ per represents the perceptual loss function, image _ loss represents the image loss function, L _ feat represents the characteristic reconstruction loss function, and L _ style represents the stylistic reconstruction loss function.
In one example, a full variogram is also used in the loss function according to embodiments of the present invention. A normally illuminated image that is restored by a low-illuminated image may have unstable illumination and noise, which may degrade visual quality. A Total Variation (TV) function is used as a smoothness prior to minimize the gradient of the entire image. The full variational function L _ tv is represented as the following equation 8.
[ formula 8]
Where p denotes a pixel value, i and j denote pixel reference numerals, and W and H denote feature size.
Continuing with FIG. 1, in block 107, a determination is made as to whether a loop over condition is satisfied. If yes, ending the training; if not, block 101 is entered.
Here, the loop end condition may be specified. In one example, the loop-over condition may include reaching a predetermined number of loops.
In another example, the loop-over condition may include: and updating the depth brightening network in the previous cycle to obtain the low-illumination image enhancement model in the current cycle, and judging that the low-illumination image enhancement model meets the requirement without continuing training.
Fig. 3 is a block diagram of a training apparatus for a low-light image enhancement model according to an embodiment of the present invention.
Referring to fig. 3, a training apparatus 300 of a low-light image enhancement model according to an embodiment of the present invention includes: a receiving module 310, a first converting module 320, a deep brightening network module 330, a second converting module 340, a loss value calculating module 350, and an updating module 360. The receiving module 310, the first converting module 320, the depth brightening network module 330, the second converting module 340, the loss value calculating module 350, and the updating module 360 operate in a loop until a loop end condition is satisfied.
The receiving module 310 is used for receiving a low-illumination RGB image. The first conversion module 320 is used to convert the low-light RGB image into an HSV image. The depth brightening network module 330 is configured to brighten the HSV image by using a depth brightening network to obtain an HSV brightened image. The second conversion module 340 is configured to convert the HSV highlight image into an RGB highlight image. The loss value calculating module 350 is configured to calculate a loss value according to the RGB enhanced image, the normal illumination image, and the loss function. The updating module 360 is configured to update the network parameters of the deep lightening network according to the calculated loss value.
Wherein the cycle end condition includes: reaching a predetermined cycle number; or the low-illumination image enhancement model in the cycle process obtained by updating the depth brightening network in the previous cycle process is judged to meet the requirement without continuing training. .
The training method and the training apparatus for the low-light image enhancement model according to the embodiment of the present invention are described above with reference to fig. 1 to 3.
The training device for the low-illumination image enhancement model according to the embodiment of the invention can be realized by hardware, software or a combination of hardware and software. The software implementation is taken as an example, and is formed by reading corresponding computer program instructions in the storage into the memory for operation through the processor of the device where the software implementation is located as a logical device. In an embodiment of the invention, the use of the apparatus for model training may be implemented, for example, with an electronic device.
FIG. 4 is a block diagram illustrating an electronic device implementing a method of training a model in accordance with an embodiment of the present invention.
Referring to fig. 4, the electronic device 400 may include at least one processor 410, a memory (e.g., a non-volatile memory) 420, a memory 430, and a communication interface 440, and the at least one processor 410, the memory 420, the memory 430, and the communication interface 440 are connected together via a bus 450. The at least one processor 410 executes at least one computer-readable instruction (i.e., an element described above as being implemented in software) stored or encoded in memory.
In one example, computer-executable instructions are stored in the memory that, when executed, cause the at least one processor 410 to perform the following loop process until a loop-ending condition is satisfied: receiving a low-illumination RGB image, and converting the low-illumination RGB image into an HSV image; carrying out brightening treatment on the HSV image by using a depth brightening network to obtain an HSV brightening image, and converting the HSV brightening image into an RGB brightening image; and calculating a loss value according to the RGB brightening image, the normal illumination image and the loss function, and updating the network parameters of the depth brightening network according to the calculated loss value, wherein when the circulation condition is not met, the RGB brightening image obtained in the previous circulation process is subjected to darkening treatment to obtain a low-illumination RGB image in the circulation process, and the depth brightening network in the circulation process is the depth brightening network after the network parameters in the previous circulation process are updated.
It should be understood that the computer-executable instructions stored in the memory, when executed, cause the at least one processor 410 to perform various operations and functions described in conjunction with fig. 1-3 above in various embodiments in accordance with the present invention.
According to one embodiment, a program product, such as a machine-readable medium, is provided. A machine-readable medium may have instructions (i.e., elements described above as being implemented in software) that, when executed by a machine, cause the machine to perform various operations and functions described in connection with fig. 1-3 above in various embodiments of the invention.
Specifically, a system or an apparatus may be provided which is provided with a readable storage medium on which software program code implementing the functions of any of the above embodiments is stored, and causes a computer or a processor of the system or the apparatus to read out and execute instructions stored in the readable storage medium.
In this case, the program code itself read from the readable medium can realize the functions of any of the above-described embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the embodiments of the present invention.
Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-R, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or the cloud over a communications network.
The foregoing description has described certain embodiments of this invention. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Not all steps and elements in the above flows and system structure diagrams are necessary, and some steps or elements may be omitted according to actual needs. The execution order of the steps is not fixed, and can be determined as required. The device structures described in the above embodiments may be physical structures or logical structures, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some units may be implemented by some components in multiple independent devices.
The terms "exemplary," "example," and the like, as used throughout this specification, mean "serving as an example, instance, or illustration," and do not mean "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described technology. However, the techniques may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.
Alternative embodiments of the present invention are described in detail with reference to the drawings, however, the embodiments of the present invention are not limited to the specific details in the above embodiments, and within the technical idea of the embodiments of the present invention, many simple modifications may be made to the technical solution of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the description is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A training method of a low-illumination image enhancement model is characterized by comprising the following steps:
receiving a low-illumination RGB image, and converting the low-illumination RGB image into an HSV image;
carrying out brightening treatment on the HSV image by using a depth brightening network to obtain an HSV brightening image, and converting the HSV brightening image into an RGB brightening image;
and calculating to obtain a loss value according to the RGB brightening image, the normal illumination image and the loss function, and updating the network parameters of the depth brightening network according to the calculated loss value.
2. The training method of claim 1, wherein the loss function comprises a structural similarity function, a perceptual loss function, and a fully-variant function.
3. The training method of claim 1, wherein the loss function is represented by the following equation 1,
[ equation 1] L ═ L _ ssim + L _ per +0.001 × L _ tv
Wherein L represents the loss function, L _ ssim represents the structural similarity function, L _ per represents the perceptual loss function, and L _ tv represents the total variation function.
4. Training method according to claim 2 or 3, wherein the perceptual loss function comprises an image loss function, a feature reconstruction loss function and a style reconstruction loss function, wherein the image loss function represents a mean square error between the RGB brightened image and a normal-illumination image.
5. The training method of claim 4, wherein the perceptual loss function is equal to a sum of the image loss function, the feature reconstruction loss function, and the style reconstruction loss function.
6. The training method of claim 1, wherein the brightening the HSV image with a deep brightening network to obtain an HSV brightened image comprises:
carrying out brightening treatment on the S channel image and the V channel image of the HSV image by using the depth brightening network to respectively obtain an S channel brightening image and a V channel brightening image;
and acquiring the HSV brightening image according to the H channel image, the S channel brightening image and the V channel brightening image of the HSV image.
7. The training method according to claim 6, wherein the brightening the S-channel image and the V-channel image of the HSV image by using the depth brightening network comprises:
performing shallow feature extraction on the input channel image by using a shallow feature extraction layer of the depth brightening network to obtain a corresponding feature map;
cumulatively brightening the feature map by using a plurality of brightening back projection layers with feature aggregation of the depth brightening network to obtain a brightening feature map;
processing the brightening characteristic graph by using a brightening layer of the depth brightening network to obtain a channel brightening image;
the input channel image is the S-channel image, the channel brightening image is the S-channel brightening image, or the input channel image is the V-channel image, and the channel brightening image is the V-channel brightening image.
8. A training device for a low-light image enhancement model, the training device comprising:
the receiving module is used for receiving the low-illumination RGB image;
a first conversion module for converting the low-illumination RGB image into an HSV image;
the depth brightening network module is used for brightening the HSV image by using a depth brightening network to obtain an HSV brightening image;
the second conversion module is used for converting the HSV brightening image into an RGB brightening image;
the loss value calculation module is used for calculating a loss value according to the RGB brightening image, the normal illumination image and the loss function;
and the updating module is used for updating the network parameters of the deep brightening network according to the loss values obtained by calculation.
9. An electronic device, comprising:
at least one processor, and
a memory coupled with the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method of training a low-light image enhancement model of any of claims 1 to 7.
10. A machine readable storage medium storing executable instructions that when executed cause the machine to perform a method of training a low-light image enhancement model according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011449091.9A CN112508815A (en) | 2020-12-09 | 2020-12-09 | Model training method and device, electronic equipment and machine-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011449091.9A CN112508815A (en) | 2020-12-09 | 2020-12-09 | Model training method and device, electronic equipment and machine-readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112508815A true CN112508815A (en) | 2021-03-16 |
Family
ID=74971439
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011449091.9A Pending CN112508815A (en) | 2020-12-09 | 2020-12-09 | Model training method and device, electronic equipment and machine-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112508815A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114119422A (en) * | 2021-12-03 | 2022-03-01 | 深圳大学 | Method, system and related components for enhancing image quality of no-reference low-illumination endoscope |
| WO2023092386A1 (en) * | 2021-11-25 | 2023-06-01 | 中国科学院深圳先进技术研究院 | Image processing method, terminal device, and computer readable storage medium |
| WO2024055398A1 (en) * | 2022-09-16 | 2024-03-21 | 北京大学 | Unsupervised low-illumination-domain adaptive training method and detection method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105096273A (en) * | 2015-08-20 | 2015-11-25 | 北京邮电大学 | Automatic adjustment method of color image brightness |
| CN107103590A (en) * | 2017-03-22 | 2017-08-29 | 华南理工大学 | A kind of image for resisting generation network based on depth convolution reflects minimizing technology |
| CN107705242A (en) * | 2017-07-20 | 2018-02-16 | 广东工业大学 | A kind of image stylization moving method of combination deep learning and depth perception |
| CN109816608A (en) * | 2019-01-22 | 2019-05-28 | 北京理工大学 | A kind of low-light (level) image adaptive brightness enhancement based on noise suppressed |
-
2020
- 2020-12-09 CN CN202011449091.9A patent/CN112508815A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105096273A (en) * | 2015-08-20 | 2015-11-25 | 北京邮电大学 | Automatic adjustment method of color image brightness |
| CN107103590A (en) * | 2017-03-22 | 2017-08-29 | 华南理工大学 | A kind of image for resisting generation network based on depth convolution reflects minimizing technology |
| CN107705242A (en) * | 2017-07-20 | 2018-02-16 | 广东工业大学 | A kind of image stylization moving method of combination deep learning and depth perception |
| CN109816608A (en) * | 2019-01-22 | 2019-05-28 | 北京理工大学 | A kind of low-light (level) image adaptive brightness enhancement based on noise suppressed |
Non-Patent Citations (5)
| Title |
|---|
| JUSTIN JOHNSON ET AL: "Perceptual Losses for Real-Time Style Transfer and Super-Resolution", pages 694 - 711 * |
| LI-WEN WANG ET AL: "Lightening Network for Low-Light Image Enhancement", IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 29, pages 7984 - 7996, XP011800962, DOI: 10.1109/TIP.2020.3008396 * |
| MINGSONG CHEN ET AL: "BEN: Brightness Enhancement Network for Low-Light Image Enhancement in Complex Environment", 2021 INTERNATIONAL CONFERENCE ON SENSING, MEASUREMENT & DATA ANALYTICS IN THE ERA OF ARTIFICIAL INTELLIGENCE (ICSMD), pages 10 * |
| 吴海冰 等: "基于低照度三基色图像去噪及融合彩色图像增强方法研究", 应用光学, vol. 39, no. 1, pages 57 - 63 * |
| 米恒 等: "基于改进生成式对抗网络的图像超分辨率重建", 计算机应用与软件, no. 09, pages 139 - 145 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023092386A1 (en) * | 2021-11-25 | 2023-06-01 | 中国科学院深圳先进技术研究院 | Image processing method, terminal device, and computer readable storage medium |
| CN114119422A (en) * | 2021-12-03 | 2022-03-01 | 深圳大学 | Method, system and related components for enhancing image quality of no-reference low-illumination endoscope |
| CN114119422B (en) * | 2021-12-03 | 2022-07-15 | 深圳大学 | Method, system and related components for enhancing image quality of no-reference low-illumination endoscope |
| WO2024055398A1 (en) * | 2022-09-16 | 2024-03-21 | 北京大学 | Unsupervised low-illumination-domain adaptive training method and detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10339643B2 (en) | Algorithm and device for image processing | |
| Zhang et al. | Zero-shot restoration of back-lit images using deep internal learning | |
| US9275445B2 (en) | High dynamic range and tone mapping imaging techniques | |
| Ying et al. | A bio-inspired multi-exposure fusion framework for low-light image enhancement | |
| CN108694705B (en) | Multi-frame image registration and fusion denoising method | |
| Kim et al. | Optimized contrast enhancement for real-time image and video dehazing | |
| US7933454B2 (en) | Class-based image enhancement system | |
| CN112508815A (en) | Model training method and device, electronic equipment and machine-readable storage medium | |
| US9053558B2 (en) | Method and system for fusing multiple images | |
| Li et al. | Detail-preserving multi-exposure fusion with edge-preserving structural patch decomposition | |
| KR101361644B1 (en) | Image processing device and image defogging method | |
| CN107172354B (en) | Video processing method and device, electronic equipment and storage medium | |
| CN113129236B (en) | Single low-light image enhancement method and system based on Retinex and convolutional neural network | |
| Kim | Low-light image enhancement: A comparative review and prospects | |
| Celebi et al. | Fuzzy fusion based high dynamic range imaging using adaptive histogram separation | |
| Chen et al. | End-to-end single image enhancement based on a dual network cascade model | |
| Zhao et al. | Multi-scene image enhancement based on multi-channel illumination estimation | |
| CN117152182A (en) | Ultralow-illumination network camera image processing method and device and electronic equipment | |
| CN111105369A (en) | Image processing method, image processing apparatus, electronic device, and readable storage medium | |
| CN116228553A (en) | Image enhancement method capable of simultaneously enhancing definition of high-illumination and low-illumination areas | |
| Gasparyan et al. | Iterative retinex-based decomposition framework for low light visibility restoration | |
| Zini et al. | Back to the future: a night photography rendering ISP without deep learning | |
| Ma et al. | Image Dehazing Based on Improved Color Channel Transfer and Multiexposure Fusion | |
| Lee et al. | Ghost and noise removal in exposure fusion for high dynamic range imaging | |
| CN112365553A (en) | Human body image generation model training, human body image generation method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |