CN115115527A - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN115115527A CN115115527A CN202110302555.1A CN202110302555A CN115115527A CN 115115527 A CN115115527 A CN 115115527A CN 202110302555 A CN202110302555 A CN 202110302555A CN 115115527 A CN115115527 A CN 115115527A
- Authority
- CN
- China
- Prior art keywords
- image
- neural network
- convolutional neural
- illumination
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 141
- 238000005286 illumination Methods 0.000 claims abstract description 116
- 238000013528 artificial neural network Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000005070 sampling Methods 0.000 claims abstract description 10
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 9
- 239000013598 vector Substances 0.000 claims description 11
- 230000003321 amplification Effects 0.000 claims description 6
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an image processing method and a device, the method is used for an image processing model, firstly, a first convolution neural network in the image processing model is used for extracting a reflection component and an illumination component from initial image information, and then a second convolution neural network in the image processing model is used for synthesizing the reflection component and the illumination component into intermediate image information, so that illumination enhancement processing of an initial image is completed; and extracting intermediate image features from the intermediate image information by using a third convolutional neural network in the image processing model, amplifying the intermediate image features by using an up-sampling layer in the image processing model, and finally performing resolution enhancement processing on the amplified intermediate image features by using a fourth convolutional neural network in the image processing model to obtain an illumination and resolution enhanced image. The image processing method provided by the embodiment of the application can improve the image resolution and avoid image color distortion and information loss while improving the image illuminance.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing method and apparatus.
Background
Due to the limitation of weather, illumination conditions and the performance of image acquisition equipment when the image is acquired, the information retention condition of the image is difficult to meet the subsequent processing requirement. To solve this problem, image enhancement methods are usually used to enhance the image to highlight important information in the image while attenuating unimportant information.
In the field of image processing, for low-illumination images, an illumination compensation method is generally adopted to perform enhancement processing on the low-illumination images, so as to enhance the image contrast and dark details, thereby solving the problem of low illumination of the images. However, most illumination compensation methods can improve the problem of too low illumination of the image, but they are prone to cause color distortion of the image and lose some important information in the image.
Disclosure of Invention
The application provides an image processing method and device, which are used for solving the problems that the existing illumination compensation method is easy to cause image color distortion and can lose part of important information in an image.
In a first aspect, the present application provides an image processing method for an image processing model, the method comprising:
extracting a reflection component and an illumination component from initial image information by using a first convolution neural network in an image processing model;
synthesizing the reflection component and the illumination component by using a second convolutional neural network in the image processing model, and outputting intermediate image information;
extracting intermediate image features from the intermediate image information by using a third convolutional neural network in an image processing model, and amplifying the intermediate image features by using an up-sampling layer in the image processing model;
and performing resolution enhancement processing by using the intermediate image features amplified by the fourth convolutional neural network in the image processing model to obtain an illumination and resolution enhancement image.
In a second aspect, the present application also provides an image processing apparatus, comprising:
the low illumination enhancement module is used for extracting a reflection component and an illumination component from the initial image information; synthesizing the reflection component and the illumination component, and outputting intermediate image information;
a resolution enhancement module for extracting intermediate image features from the intermediate image information; amplifying the intermediate image characteristics; and performing resolution enhancement processing on the intermediate image features after the amplification processing to obtain an illumination and resolution enhanced image.
According to the technical scheme, the embodiment of the application provides an image processing method and device, the method comprises the steps of firstly extracting a reflection component and an illumination component from initial image information by using a first convolutional neural network, and then synthesizing the reflection component and the illumination component into intermediate image information by using a second convolutional neural network, so that illumination enhancement processing of the initial image is completed; and extracting intermediate image features from the intermediate image information by using a third convolutional neural network, amplifying the intermediate image features by using an up-sampling layer, and finally performing resolution enhancement processing on the amplified intermediate image features by using a fourth convolutional neural network to obtain an illumination and resolution enhanced image. The image processing method provided by the embodiment of the application can improve the image resolution and avoid image color distortion and information loss while improving the image illuminance.
Drawings
In order to more clearly explain the technical solution of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without any creative effort.
Fig. 1 is a schematic structural diagram of a low illumination enhancement module and a resolution enhancement module according to an exemplary embodiment of the present application;
FIG. 2 is a flow chart of an image processing method according to an exemplary illustration of the present application;
FIG. 3 is a schematic diagram of a second convolutional neural network, as exemplary shown in the present application;
FIG. 4 is a schematic illustration of a low illumination initial image and an intermediate image according to an exemplary illustration of the present application;
FIG. 5 is a schematic illustration of an intermediate image and an illumination and resolution enhancement image according to an exemplary illustration of the present application;
FIG. 6 is a flow chart of an image processing method illustrated in accordance with an exemplary embodiment of the present application;
fig. 7 is a schematic diagram of an image processing apparatus according to an exemplary embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application provides an image processing method, which comprises the steps of firstly constructing an image processing model based on a neural network for a low-illumination image, then using the trained image processing model to process the low-illumination image, aiming at improving the illumination of the low-illumination image, improving the resolution of the image, avoiding color distortion and information loss, and outputting an illumination and resolution enhanced image.
Fig. 1 is a schematic structural diagram of an image processing model according to an exemplary embodiment of the present application. As shown in fig. 1, the image processing model includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network, at least one upsampling layer, and a fourth convolutional neural network, which are connected in sequence. The first convolutional neural network is used for extracting a reflection component and an illumination component from input initial image (low-illumination image) information, and the second convolutional neural network is used for synthesizing the reflection component and the illumination component output by the first convolutional neural network into intermediate image information. Notably, the illuminance of the intermediate image is enhanced compared to the initial image information. The third convolutional neural network is used for extracting intermediate image features from intermediate image information output by the second convolutional neural network, at least one up-sampling layer is used for amplifying the intermediate image features output by the third convolutional neural network at least once, and the fourth convolutional neural network is used for enhancing the resolution of the amplified intermediate image features and finally outputting an illumination and resolution enhanced image. It is noted that the final output illumination and resolution enhancement image is enlarged in size and resolution enhanced compared to the intermediate image information.
Fig. 2 is a flowchart of an image processing method according to an exemplary embodiment of the present application, and as shown in fig. 2, the method may include:
s210, a first convolution neural network in the image processing model is utilized to extract a reflection component and an illumination component from initial image information.
In the embodiment of the present application, the initial image information is a vectorized representation of an initial image, wherein the initial image is a processing object of the image processing method of the present application and should be a low-illumination image with a brightness parameter meeting a preset condition. For example, for an initial image with a size of a × b and a number of channels of c, the corresponding initial image information can be represented as a vector a × b × c. For another example, for an RGB picture with a size of 64 × 64, the corresponding initial image information may be represented as a 64 × 64 × 3 vector, where the number of channels of the RGB picture is 3.
In the present application, a first convolutional neural network is used to segment the initial image information into a reflectance component and an illumination component. The first convolutional neural network may include n sequentially connected convolutional layers, and the first convolutional neural network is used to extract a reflection component and an illumination component from the initial image information, that is, n times of convolution processing is performed on the initial image information by using the n convolutional layers, where an input of a 1 st convolutional layer is the initial image information, an output result of an i th convolutional layer is an input of an i +1 th convolutional layer, an output result of the n th convolutional layer is 4 feature maps, i is 1, 2, … …, and n-1, 4 is the number of channels of the n th convolutional layer. As a possible implementation manner, the 1 st to 3 rd feature maps may be used as the reflection component extracted from the initial image information, and the 4 th feature map may be used as the illumination component extracted from the initial image information. In addition, to ensure that the input image and the output image are of the same size, the fill pixel size of each convolution layer is set to half (convolution kernel size-1).
In one example, the initial image information is a 64 × 64 × 3 vector, and the first convolutional neural network includes 3 sequentially connected convolutional layers (i.e., n — 3). The number of channels of the 1 st convolutional layer and the 2 nd convolutional layer is 64, and the number of channels of the 3 rd convolutional layer is 4. In this example, the output result of the first convolutional neural network is 4 feature maps, the first 3 feature maps are reflection components of the original image information, which may be specifically represented as vectors 64 × 64 × 3, and the 4 th feature map is illumination components of the original image information, which is specifically vectors 64 × 64 × 1.
And S220, synthesizing the reflection component and the illumination component by using a second convolution neural network in the image processing model, and outputting intermediate image information.
The intermediate image information is the vectorized representation of the intermediate image. In this application, the second convolutional neural network is used to synthesize the reflection component and the illumination component output by the first convolutional neural network into intermediate image information. Specifically, the second convolutional neural network comprises a plurality of convolutional layers based on a ResNet structure, when the second convolutional neural network is used for processing the reflection component and the illumination component, the reflection component and the illumination component are used as the input of the 1 st convolutional layer, the input of each of the other layers is the sum of the input and the output of the previous layer, the result of each 3 convolutional layers is stacked to be used as the input of the next convolutional layer, and finally 3 continuous convolutional layers are connected. Among the 3 last connected convolutional layers, the 1 st convolutional layer reduces the number of channels by one third of the stacking input, and the other 2 convolutional layers and the 1 st convolutional layer form a small bottle neck result, and the number of channels is 1 and 3 respectively. The output result of the last convolution layer is the intermediate image information.
Fig. 3 is a schematic diagram of a second convolutional neural network exemplarily illustrated in the present application, and as shown in fig. 3, the second convolutional neural network includes 6 convolutional layers based on a ResNet structure, which are convolutional layers 1 to 6, respectively. When the reflection component and the illumination component are processed by the second convolutional neural network, the reflection component and the illumination component are used as the input of the convolutional layer 1; adding the input and output of convolutional layer 1 as the input of convolutional layer 2; adding the input and output of convolutional layer 2 as the input of convolutional layer 3; stacking the output of the convolutional layer 1, the output of the convolutional layer 2 and the output of the convolutional layer 3 as the input of the convolutional layer 4, wherein the number of channels of the convolutional layer 4 is one third of the number of channels of the stacking result; the output of the convolutional layer 4 is taken as the input of the convolutional layer 5, the output of the convolutional layer 5 is taken as the input of the convolutional layer 6, and the convolutional layer 6 outputs the intermediate image information, wherein the number of channels of the convolutional layer 5 is 1, and the number of channels of the convolutional layer 6 is 3.
It should be understood that the present application is not limited to the number of convolutional layers included in the second convolutional neural network. For example, the second convolutional neural network may include 6 convolutional layers shown in fig. 3, or may include more convolutional layers, such as 9 convolutional layers, which is not described herein.
Fig. 4 is a schematic diagram of a low-illuminance initial image and an intermediate image according to an exemplary embodiment of the present application, and it can be seen from fig. 4 that after the low-illuminance initial image a is processed by the first convolutional neural network and the second convolutional neural network, an intermediate image B is obtained, and the illuminance of the image B is significantly enhanced relative to the image a.
And S230, extracting intermediate image features from the intermediate image information by using a third convolutional neural network in the image processing model, and amplifying the intermediate image features by using an up-sampling layer.
And S230, performing resolution enhancement processing by using the intermediate image features amplified by the fourth convolutional neural network in the image processing model to obtain an illumination and resolution enhanced image.
In an embodiment of the present application, the third convolutional neural network includes several convolutional layers, a last convolutional layer of which is connected to one or more upsampling layers, the number of channels of the last convolutional layer is 32, and the number of channels of the remaining convolutional layers is 64. The up-sampling layers are used for amplifying the characteristics of the intermediate image, and the number of the up-sampling layers determines the amplification factor of the initial image. For example, when the last convolution layer connects one upsampled layer, the processed image is enlarged by two times with respect to the original image, and when the last convolution layer connects two consecutive upsampled layers, the processed image is enlarged by four times with respect to the original image. It should be understood that a person skilled in the art may determine the number of upsampling layers according to the image processing target, and details are not described herein.
And the fourth convolutional neural network is used for carrying out resolution enhancement processing on the amplified intermediate image.
Fig. 5 is a schematic diagram of an intermediate image and an illuminance and resolution enhanced image according to an exemplary illustration of the present application, and it can be seen from fig. 5 that after the intermediate image B is processed by the third convolutional neural network, the upsampling layer and the fourth convolutional neural network, an illuminance and resolution enhanced image C is obtained, and with respect to the image B, the resolution of the image C is enhanced, the size is enlarged, and the illuminance is further optimized.
It can be known from the above embodiments that the present application provides an image processing method, in which a first convolutional neural network in an image processing model is first utilized to extract a reflection component and an illumination component from initial image information, and then a second convolutional neural network in the image processing model is utilized to synthesize the reflection component and the illumination component into intermediate image information, thereby completing illumination enhancement processing on an initial image; and extracting intermediate image features from the intermediate image information by using a third convolutional neural network in the image processing model, amplifying the intermediate image features by using an up-sampling layer in the image processing model, and finally performing resolution enhancement processing on the amplified intermediate image features by using a fourth convolutional neural network in the image processing model, thereby completing the amplification and resolution enhancement processing of the intermediate image. The image processing method provided by the embodiment of the application can improve the image resolution and avoid image color distortion and information loss while improving the image illuminance.
Fig. 6 is a flowchart of an image processing method according to an exemplary embodiment of the present application, which specifically illustrates a training process of the image processing model shown in fig. 1, that is, a training process of a first convolutional neural network, a second convolutional neural network, a third convolutional neural network, an upsampling layer, and a fourth convolutional neural network. As shown in fig. 6, the method may include:
s610, a training data set is obtained, and the training data set comprises a plurality of groups of corresponding input images, intermediate target images and target images.
The size of the intermediate target image is the same as that of the input image, and the illuminance of the intermediate target image is higher than that of the input image; the illumination of the target image is the same as that of the intermediate target image, and the size of the target image is larger than that of the intermediate target image.
In one possible implementation of S610, several sets of original images are first obtained, where each set of original images includes a corresponding low-illumination image and a normal-illumination image. The corresponding low-illumination image and the normal closed-lighting image refer to images respectively acquired in the same shooting scene and different lighting environments, and the two images have the same size. For example, the position of the image capturing device and various parameters are kept unchanged, a low-illumination image is captured under a low-illumination condition, and a normal-illumination image is captured under a normal-illumination condition. And then generating training data respectively corresponding to each group of original images according to each group of original images, wherein the input image in each group of training data is an image corresponding to the low-illumination image reduced by k times, the intermediate target image is an image corresponding to the normal-illumination image reduced by k times, the target image is a corresponding normal-illumination image, and k is a positive number. It should be understood that the reduction in size by a factor of k as referred to herein refers to the simultaneous reduction in width and height of the image by a factor of k.
In one example, a set of original images includes a first low-light image and a first normal-light image, and training data corresponding to the set of original images includes a first input image, a first intermediate target image, and a first target image. As shown in S110, the first input image is a reduced k-times image of the first low-illumination image, the first intermediate target image is a reduced k-times image of the first normal-illumination image, and the first target image is a first normal-illumination image.
It should be noted that the specific value of k can be designed by those skilled in the art according to the requirement. For example, when the initial image to be processed needs to be enlarged by 4 times by using the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network, the input image in each set of training data is an image reduced by 4 times corresponding to the low-illumination image, and the intermediate target image is an image reduced by 4 times corresponding to the normal-illumination image.
And S620, performing joint training on the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network by using the training data set.
During training, input images of each group of training data are input into the first convolutional neural network, and the output of the previous neural network in the image processing model is input into the next adjacent neural network. Specifically, the input image in each set of training data is used as the input of a first convolutional neural network, the input reflection component and the input illumination component output by the first convolutional neural network are used as the input of a second convolutional neural network, the intermediate training image output by the second convolutional neural network is used as the input of a third convolutional neural network, the output of the third convolutional neural network is used as the input of an upsampling layer, and the output of the upsampling layer is used as the input of a fourth convolutional neural network.
The method comprises the steps of inputting an input image into a first convolution neural network, inputting a corresponding intermediate target image into an auxiliary convolution neural network at the same time, so as to extract a target reflection component and a target illumination component from the intermediate target image by using the auxiliary convolution neural network, wherein the auxiliary convolution neural network has the same structure as the first convolution neural network, and the parameters of the auxiliary convolution neural network are optimized synchronously with the parameters of the first convolution neural network, so that the parameters of the auxiliary convolution neural network and the parameters of the first convolution neural network are always kept the same.
And after each round of training is finished, calculating training loss according to a preset loss function, wherein the training loss comprises a first local loss generated at the first convolutional neural network, a second local loss generated at the second convolutional neural network, and a third local loss generated at the third convolutional neural network, the upper sampling layer and the fourth convolutional neural network. And optimizing parameters of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network according to the training loss until a preset verification condition is met.
In one possible implementation, the preset loss function is an MSE function, and in this implementation, the training loss may be calculated according to the following formula:
wherein N is the group number of training data in the training round;
ω 1 、ω 2 、ω 3 the weights are respectively corresponding to the local losses.
In this application, the output of the first convolutional neural network is the input reflection component and the input illumination component extracted from the input image. An image restored according to the input reflection component and the input illumination component is called a first restored image, an image restored according to the target reflection component and the target illumination component is called a second restored image, and the method comprises the following steps:
a first restored image is input reflection component × input illumination component;
the second restored image is the target reflection component × the target illumination component.
In this application, the first local loss
The method can comprise the following steps: a loss of the first restored image relative to the corresponding input image; a loss of the second restored image relative to the corresponding intermediate target image; and, a loss of the input reflection vector relative to the corresponding target reflection vector. Therefore, through continuous training, the input reflection component output by the first convolution neural network is continuously close to the corresponding target reflection component, meanwhile, the input reflection component and the input illumination component output by the first convolution neural network can restore the corresponding input image as much as possible, and the target reflection component and the target illumination component can restore the corresponding intermediate target image as much as possible.
In addition, a second local loss is calculated from an output of the second convolutional neural network and the corresponding intermediate target image, and a third local loss is calculated from an output of the fourth convolutional neural network and the corresponding target image.
In the embodiment of the present application, the weight ω corresponding to each local loss may be preset according to the gradients of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, and the fourth convolutional neural network 1 、ω 2 、ω 3 Generally, the smaller the gradient of the neural network, the smaller the corresponding weight.
In one implementation, ω 1 Less than omega 2 And ω 3 。
In the training process, when the iteration result of a certain time is superior to the iteration result of the previous time, the parameters used by the iteration are stored as the current optimal parameters. And synchronously outputting the loss corresponding to the training set and the loss corresponding to the verification set in the training process, and stopping training when the loss on the verification set meets the preset verification condition.
It can be known from the foregoing S610-S620 that, in the image processing method provided in the present application, an image processing model based on a neural network is constructed for a low-illuminance image, where the image processing model includes a first convolution neural network, a second convolution neural network, a third convolution neural network, at least one upsampling layer, and a fourth convolution neural network, which are connected in sequence, and by performing joint training on the first convolution neural network, the second convolution neural network, the third convolution neural network, the upsampling layer, and the fourth convolution neural network, the image processing model can have a capability of improving illuminance of the low-illuminance image and a capability of improving resolution of an intermediate image, and can avoid color distortion and information loss.
According to the image processing method provided by the above embodiment, an embodiment of the present application further provides an image processing apparatus, as shown in fig. 7, the apparatus may include:
a low illumination enhancement module 710 for extracting a reflection component and an illumination component from the initial image information; and synthesizing the reflection component and the illumination component, and outputting intermediate image information. A resolution enhancement module 720 for extracting intermediate image features from the intermediate image information; and carrying out amplification processing and resolution enhancement processing on the intermediate image characteristics to obtain an illumination and resolution enhancement image.
In some embodiments, the low illumination enhancement module 710 includes a first enhancement unit for extracting a reflection component and an illumination component from the initial image information; and the second enhancement unit is used for carrying out synthesis processing on the reflection component and the illumination component and outputting intermediate image information. The resolution enhancement module 720 comprises a third enhancement unit and a fourth enhancement unit, wherein the third enhancement unit is used for extracting intermediate image features from the intermediate image information and amplifying the intermediate image features; and the fourth enhancement unit is used for carrying out resolution enhancement processing on the intermediate image features after the amplification processing to obtain an illumination and resolution enhanced image.
In some embodiments, the first enhancement unit is embodied as a convolutional neural network comprising a plurality of convolutional layers, the second enhancement unit is embodied as a convolutional neural network comprising a plurality of convolutional layers, the third enhancement unit is embodied as a convolutional neural network comprising a plurality of convolutional layers and at least one upsampling layer, and the fourth enhancement unit is embodied as a convolutional neural network comprising a plurality of convolutional layers.
In some embodiments, the first enhancement unit comprises n convolutional layers, the first enhancement unit being specifically configured to: carrying out convolution processing on the initial image information for n times by using the n convolution layers, wherein the output result of the ith convolution layer is the input of the (i + 1) th convolution layer, the output result of the nth convolution layer is 4 feature maps, and i is 1, 2, … … and n-1; the 1 st to 3 rd feature maps are used as the reflection component, and the 4 th feature map is used as the illumination component.
In some embodiments, the apparatus further comprises a training module comprising a data preparation unit for obtaining a training data set comprising sets of corresponding input images, intermediate target images and target images, wherein the intermediate target images are the same size as the input images and the intermediate target images are brighter than the input images, and the target images are the same luminance as the intermediate target images and the target images are larger in size than the intermediate target images; a training unit configured to perform joint training on the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit using the training data set.
In some embodiments, the data preparation unit is specifically configured to: acquiring a plurality of groups of original images, wherein each group of original images comprises a corresponding low-illumination image and a corresponding normal-illumination image; generating training data respectively corresponding to each group of original images, wherein the input image in each group of training data is an image which is reduced by k times corresponding to the low-illumination image, the intermediate target image is an image which is reduced by k times corresponding to the normal-illumination image, the target image is the corresponding normal-illumination image, and k is a positive number.
In some embodiments, the training unit is specifically configured to: inputting the input image of each group of training data into the first convolution neural network, and inputting the output of the previous neural network in the image processing model into the next adjacent neural network; calculating a training loss according to a preset loss function, wherein the training loss comprises a first local loss calculated according to the output of the first convolutional neural network and a target reflection component and a target illumination component extracted from a corresponding intermediate target image, a second local loss calculated according to the output of the second convolutional neural network and a corresponding intermediate target image, and a third local loss calculated according to the output of the fourth convolutional neural network and a corresponding target image; and optimizing parameters of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network according to the training loss until a preset verification condition is met. In some embodiments, the training unit is further to: inputting the corresponding intermediate target image into an auxiliary convolutional neural network to extract a target reflection component and a target illumination component from the first intermediate target image by using the auxiliary convolutional neural network, wherein the auxiliary convolutional neural network has the same structure as the first enhancement unit; and synchronously optimizing the parameters of the auxiliary convolutional neural network when the parameters of the first enhancement unit are optimized according to the training loss so as to enable the parameters of the auxiliary convolutional neural network to be the same as the parameters of the first enhancement unit.
In some embodiments, the output of the first convolutional neural network is an input reflection component and an input illumination component extracted from the input image; the local loss corresponding to the first enhancement unit comprises: a loss of a first restored image relative to a corresponding input image, the first restored image being restored from the input reflection component and the input illumination component; the loss of a second restored image relative to a corresponding intermediate target image is obtained by restoring the second restored image according to the target reflection component and the target illumination component; and, a loss of the input reflection vector relative to the target reflection vector.
In some embodiments, the training loss is calculated as follows:
wherein N is the number of groups of training data;
ω 1 、ω 2 、ω 3 the weights are respectively corresponding to the local losses.
In some embodiments, ω is 1 Less than said ω 2 And ω 3 。
In a specific implementation, the present invention further provides a computer storage medium, where the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments of the image processing method provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The same and similar parts in the various embodiments in this specification may be referred to each other. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the description in the method embodiment.
The above-described embodiments of the present invention should not be construed as limiting the scope of the present invention.
Claims (10)
1. An image processing method for an image processing model, the method comprising:
extracting a reflection component and an illumination component from initial image information by using a first convolution neural network in an image processing model;
synthesizing the reflection component and the illumination component by using a second convolutional neural network in the image processing model, and outputting intermediate image information;
extracting intermediate image features from the intermediate image information by using a third convolutional neural network in an image processing model, and amplifying the intermediate image features by using an up-sampling layer in the image processing model;
and performing resolution enhancement processing by using the intermediate image features amplified by the fourth convolutional neural network in the image processing model to obtain an illumination and resolution enhancement image.
2. The method of claim 1, wherein the first convolutional neural network comprises n convolutional layers, and wherein extracting the reflection component and the illumination component from the initial image information using the first convolutional neural network comprises:
performing convolution processing on the initial image information for n times by using the n convolution layers, wherein the output result of the ith convolution layer is the input of the (i + 1) th convolution layer, the output result of the nth convolution layer is 4 characteristic graphs, and i is 1, 2, … … and n-1;
the 1 st to 3 rd feature maps are used as the reflection component, and the 4 th feature map is used as the illumination component.
3. The method of claim 1, wherein the image processing model is trained by:
acquiring a training data set, wherein the training data set comprises a plurality of groups of corresponding input images, intermediate target images and target images, the intermediate target images and the input images are the same in size, the illuminance of the intermediate target images is higher than that of the input images, the target images and the intermediate target images are the same in illuminance, and the sizes of the target images are larger than that of the intermediate target images;
jointly training the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network in the image processing model using the training data set.
4. The method of claim 3, wherein the obtaining a training data set comprises:
acquiring a plurality of groups of original images, wherein each group of original images comprises a corresponding low-illumination image and a corresponding normal-illumination image;
generating training data respectively corresponding to each group of original images, wherein an input image in each group of training data is an image corresponding to a low-illumination image reduced by k times, a middle target image in each group of training data is an image corresponding to a normal-illumination image reduced by k times, a target image in each group of training data is the corresponding normal-illumination image, and k is a positive number.
5. The method of claim 3, wherein the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network are connected in sequence, and wherein jointly training the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network in the image processing model using the training data set comprises:
inputting the input image of each group of training data into the first convolution neural network, and inputting the output of the previous neural network in the image processing model into the next adjacent neural network;
calculating training losses according to a preset loss function, wherein the training losses include a first local loss calculated according to the output of the first convolutional neural network and a target reflection component and a target illumination component extracted from a corresponding intermediate target image, a second local loss calculated according to the output of the second convolutional neural network and a corresponding intermediate target image, and a third local loss calculated according to the output of the fourth convolutional neural network and a corresponding target image;
and optimizing parameters of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network according to the training loss until a preset verification condition is met.
6. The method of claim 5, wherein while inputting a set of input images of training data to the first convolutional neural network, the method further comprises:
inputting an intermediate target image in the set of training data into an auxiliary convolutional neural network so as to extract a target reflection component and a target illumination component from the intermediate target image by using the auxiliary convolutional neural network, wherein the auxiliary convolutional neural network has the same structure as the first convolutional neural network;
and synchronously optimizing the parameters of the auxiliary convolutional neural network when the parameters of the first convolutional neural network are optimized according to the training loss so as to enable the parameters of the auxiliary convolutional neural network to be the same as the parameters of the first convolutional neural network.
7. The method of claim 5, wherein the output of the first convolutional neural network is an input reflection component and an input illumination component extracted from the input image; calculating a first local loss according to the output of the first convolution neural network and a target reflection component and a target illumination component extracted from a corresponding intermediate target image, including:
restoring according to the input reflection component and the input illumination component to obtain a first restored image, and calculating the loss of the first restored image relative to the corresponding input image;
restoring according to the target reflection component and the target illumination component to obtain a second restored image, and calculating the loss of the second restored image relative to the corresponding intermediate target image;
and calculating a loss of the input reflection vector relative to the target reflection vector.
8. The method of claim 5, wherein the training loss is calculated according to the following equation:
wherein N is the number of groups of training data;
ω 1 、ω 2 、ω 3 the weights are respectively corresponding to the local losses.
9. The method of claim 8, wherein ω is said ω is determined by a weight of said block 1 Less than said ω 2 And ω 3 。
10. An image processing apparatus, characterized in that the apparatus comprises:
the low illumination enhancement module is used for extracting a reflection component and an illumination component from the initial image information; synthesizing the reflection component and the illumination component, and outputting intermediate image information;
a resolution enhancement module for extracting intermediate image features from the intermediate image information; amplifying the intermediate image characteristics; and performing resolution enhancement processing on the intermediate image features after the amplification processing to obtain an illumination and resolution enhanced image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110302555.1A CN115115527A (en) | 2021-03-22 | 2021-03-22 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110302555.1A CN115115527A (en) | 2021-03-22 | 2021-03-22 | Image processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115115527A true CN115115527A (en) | 2022-09-27 |
Family
ID=83323239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110302555.1A Pending CN115115527A (en) | 2021-03-22 | 2021-03-22 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115115527A (en) |
-
2021
- 2021-03-22 CN CN202110302555.1A patent/CN115115527A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102281017B1 (en) | Neural network model training method, apparatus and storage medium for image processing | |
| CN107123089B (en) | Remote sensing image super-resolution reconstruction method and system based on depth convolution network | |
| US20190294931A1 (en) | Systems and Methods for Generative Ensemble Networks | |
| CN113454680A (en) | Image processor | |
| CN111835983B (en) | Multi-exposure-image high-dynamic-range imaging method and system based on generation countermeasure network | |
| CN111784570A (en) | Video image super-resolution reconstruction method and device | |
| WO2023010754A1 (en) | Image processing method and apparatus, terminal device, and storage medium | |
| US11887218B2 (en) | Image optimization method, apparatus, device and storage medium | |
| CN111951164B (en) | Image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| US11972543B2 (en) | Method and terminal for improving color quality of images | |
| CN111260580A (en) | Image denoising method based on image pyramid, computer device and computer readable storage medium | |
| CN110148088A (en) | Image processing method, image rain removing method, device, terminal and medium | |
| CN112508812A (en) | Image color cast correction method, model training method, device and equipment | |
| CN111612722A (en) | Low-illumination image processing method based on simplified Unet full-convolution neural network | |
| CN111145102A (en) | Synthetic aperture radar image denoising method based on convolutional neural network | |
| CN111724312A (en) | Method and terminal for processing image | |
| CN114240767A (en) | Image wide dynamic range processing method and device based on exposure fusion | |
| CN107220934A (en) | Image rebuilding method and device | |
| CN115311149A (en) | Image denoising method, model, computer-readable storage medium and terminal device | |
| CN106296614B (en) | Image processing apparatus and image processing method | |
| CN110717913B (en) | Image segmentation method and device | |
| CN110942425A (en) | Reconstruction method and reconstruction system of super-resolution image and electronic equipment | |
| CN115115527A (en) | Image processing method and device | |
| CN114219725A (en) | Image processing method, terminal equipment and computer readable storage medium | |
| Korkmaz et al. | Two-stage domain adapted training for better generalization in real-world image restoration and super-resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |