CN115115527A - Image processing method and device - Google Patents

Abstract

The application discloses an image processing method and a device, the method is used for an image processing model, firstly, a first convolution neural network in the image processing model is used for extracting a reflection component and an illumination component from initial image information, and then a second convolution neural network in the image processing model is used for synthesizing the reflection component and the illumination component into intermediate image information, so that illumination enhancement processing of an initial image is completed; and extracting intermediate image features from the intermediate image information by using a third convolutional neural network in the image processing model, amplifying the intermediate image features by using an up-sampling layer in the image processing model, and finally performing resolution enhancement processing on the amplified intermediate image features by using a fourth convolutional neural network in the image processing model to obtain an illumination and resolution enhanced image. The image processing method provided by the embodiment of the application can improve the image resolution and avoid image color distortion and information loss while improving the image illuminance.

Description

Image processing method and device

technical field

The present application relates to the technical field of image processing, and in particular, to an image processing method and apparatus.

Background technique

Limited by the weather, lighting conditions and the performance of the image acquisition device when the image is collected, the information retention of the image is often difficult to meet the subsequent processing requirements. To solve this problem, the image enhancement method is usually used to enhance the image to highlight the important information in the image, while weakening the unimportant information.

In the field of image processing, for low-illumination images, illumination compensation methods are usually used to enhance them, in order to enhance the image contrast and dark details, so as to solve the problem of too low image illumination. However, although most illumination compensation methods can improve the problem of too low image illumination, it is easy to cause image color distortion and lose some important information in the image.

SUMMARY OF THE INVENTION

The present application provides an image processing method and device to solve the problem that the existing illumination compensation method easily causes image color distortion and loses some important information in the image.

In a first aspect, the present application provides an image processing method for an image processing model, the method comprising:

Using the first convolutional neural network in the image processing model to extract the reflection component and the illumination component from the initial image information;

Use the second convolutional neural network in the image processing model to synthesize the reflection component and the illumination component, and output intermediate image information;

Use the third convolutional neural network in the image processing model to extract intermediate image features from the intermediate image information, and use the upsampling layer in the image processing model to amplify the intermediate image features;

The image processing model uses the enlarged intermediate image features of the fourth convolutional neural network to perform resolution enhancement processing to obtain an image with enhanced illumination and resolution.

In a second aspect, the present application further provides an image processing device, the device comprising:

A low-illuminance enhancement module, used for extracting a reflection component and an illumination component from the initial image information; synthesizing the reflection component and the illumination component, and outputting intermediate image information;

A resolution enhancement module is used for extracting intermediate image features from the intermediate image information; performing amplification processing on the intermediate image features; performing resolution enhancement processing on the enlarged intermediate image features to obtain illumination and resolution enhanced images .

It can be seen from the above technical solutions that the embodiments of the present application provide an image processing method and device. The method first uses a first convolutional neural network to extract reflection components and illumination components from initial image information, and then uses a second convolutional neural network. The reflection component and the illumination component are synthesized into the intermediate image information to complete the illumination enhancement processing of the initial image; the third convolutional neural network is used to extract the intermediate image features from the intermediate image information, and then the intermediate image features are processed by the upsampling layer. Enlargement processing, and finally, the fourth convolutional neural network is used to perform resolution enhancement processing on the enlarged intermediate image features to obtain illumination and resolution enhanced images. The image processing method provided by the embodiments of the present application can improve the image illuminance while improving the image resolution, avoiding image color distortion and information loss.

Description of drawings

In order to illustrate the technical solutions of the present application more clearly, the accompanying drawings that need to be used in the embodiments will be briefly introduced below. Other drawings can also be obtained from these drawings.

1 is a schematic structural diagram of a low-illuminance enhancement module and a resolution enhancement module according to an exemplary embodiment of the present application;

FIG. 2 is a flowchart of an image processing method according to an exemplary illustration of the present application;

3 is a schematic diagram of a second convolutional neural network exemplarily shown in the application;

FIG. 4 is a schematic diagram of a low-illumination initial image and an intermediate image according to an exemplary illustration of the present application;

FIG. 5 is a schematic diagram of an intermediate image and an illuminance and resolution enhanced image according to an exemplary illustration of the present application;

6 is a flowchart of an image processing method according to an exemplary embodiment of the present application;

FIG. 7 is a schematic diagram of an image processing apparatus according to an exemplary embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described The embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

The embodiment of the present application provides an image processing method. The method first constructs an image processing model based on a neural network for low-light images, and then uses the trained image processing model to process the low-light images, so as to improve the quality of the low-light images. Illumination, improve image resolution, avoid color distortion and information loss, and output illumination and resolution enhancement images.

FIG. 1 is a schematic structural diagram of an image processing model according to an exemplary embodiment of the present application. As shown in FIG. 1 , the image processing model includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network, at least one upsampling layer, and a fourth convolutional neural network connected in sequence. Among them, the first convolutional neural network is used to extract the reflection component and the illumination component from the input initial image (low-light image) information, and the second convolutional neural network is used to extract the reflection component and the illumination component output by the first convolutional neural network. The illumination components are synthesized into intermediate image information. It is worth noting that the illuminance of the intermediate image is enhanced compared to the initial image information. The third convolutional neural network is used for extracting intermediate image features from the intermediate image information output by the second convolutional neural network, and at least one upsampling layer is used for at least one amplification of the intermediate image features output by the third convolutional neural network The fourth convolutional neural network is used to enhance the resolution of the enlarged intermediate image features, and finally output the luminance and resolution enhanced images. It is worth noting that, compared with the intermediate image information, the size of the final output illumination and resolution-enhanced image is enlarged and the resolution is enhanced.

FIG. 2 is a flowchart of an image processing method according to an exemplary illustration of the present application. As shown in FIG. 2 , the method may include:

S210, using the first convolutional neural network in the image processing model to extract the reflection component and the illumination component from the initial image information.

In the embodiment of the present application, the initial image information is a vectorized representation of the initial image, wherein the initial image, as the processing object of the image processing method of the present application, should be a low-illuminance image whose brightness parameter meets preset conditions. For example, for an initial image whose size is a×b and the number of channels is c, the corresponding initial image information can be represented as a vector a×b×c. For another example, for an RGB picture with a size of 64×64, the corresponding initial image information can be represented as a 64×64×3 vector, wherein the number of channels of the RGB picture is 3.

In this application, the first convolutional neural network is used to segment the initial image information into reflection components and illumination components. The first convolutional neural network may include n consecutively connected convolutional layers, and the first convolutional neural network is used to extract the reflection component and the illumination component from the initial image information, that is, using the n convolutional layers to perform n times on the initial image information. Convolution processing, where the input of the first convolutional layer is the initial image information, the output of the i-th convolutional layer is the input of the i+1-th convolutional layer, and the output of the n-th convolutional layer is 4 feature maps, i is 1, 2, ..., n-1, 4 is the number of channels of the nth convolutional layer. As a possible implementation, the first to third feature maps may be used as the reflection component extracted from the initial image information, and the fourth feature map may be used as the illumination component extracted from the initial image information. In addition, to ensure that the input image is the same size as the output image, the padding pixel size of each convolutional layer is set to half of (convolution kernel size - 1).

In one example, the initial image information is a 64×64×3 vector, and the first convolutional neural network includes 3 sequentially connected convolutional layers (ie, n=3). Among them, the number of channels of the first convolutional layer and the second convolutional layer are both 64, and the number of channels of the third convolutional layer is 4. In this example, the output result of the first convolutional neural network is 4 feature maps. The first 3 feature maps are the reflection components of the initial image information. The feature map is the illumination component of the initial image information, and the illumination component is specifically a vector of 64×64×1.

S220. Use the second convolutional neural network in the image processing model to perform synthesis processing on the reflection component and the illumination component, and output intermediate image information.

The intermediate image information is the vectorized representation of the intermediate image. In this application, the second convolutional neural network is used to synthesize the reflection component and the illumination component output by the first convolutional neural network into intermediate image information. Specifically, the second convolutional neural network includes a plurality of convolutional layers based on the ResNet structure, and when using the second convolutional neural network to process the reflection component and the illumination component, the reflection component and the illumination component are used as the first The input of the convolutional layer, the input of each other layer is the sum of the input and output of the previous layer, the results of each 3 convolutional layers are stacked as the input of the next convolutional layer, and finally connected to 3 consecutive volumes Laminate. Among them, among the last 3 convolutional layers connected, the first convolutional layer reduces the number of channels to one-third of the stacked input, and the other two convolutional layers and the first convolutional layer form a small bottleneck result, and its channel The numbers are 1 and 3, respectively. The output of the last convolutional layer is the intermediate image information.

FIG. 3 is a schematic diagram of a second convolutional neural network exemplarily shown in the application. As shown in FIG. 3 , the second convolutional neural network includes 6 convolutional layers based on the ResNet structure, which are convolutional layers 1- 6. When using the second convolutional neural network to process the reflection component and the illumination component, the reflection component and the illumination component are used as the input of the convolutional layer 1; the input and output of the convolutional layer 1 are added, as the convolutional layer 2 The input of convolution layer 2 and the output are added as the input of convolution layer 3; the output of convolution layer 1, the output of convolution layer 2 and the output of convolution layer 3 are stacked as the convolution layer 4, and the number of channels of convolutional layer 4 is one third of the number of channels of the previous stacking result; the output of convolutional layer 4 is used as the input of convolutional layer 5, and the output of convolutional layer 5 is used as the convolutional layer. The input of 6, the convolutional layer 6 outputs the intermediate image information, wherein the number of channels of the convolutional layer 5 is 1, and the number of channels of the convolutional layer 6 is 3.

It should be understood that the present application does not limit the number of convolutional layers included in the second convolutional neural network. For example, the second convolutional neural network may include the 6 convolutional layers shown in FIG. 3 , or may include more convolutional layers, for example, may include 9 convolutional layers, which will not be repeated here.

FIG. 4 is a schematic diagram of a low-illumination initial image and an intermediate image according to an exemplary illustration of the present application. It can be seen from FIG. 4 that after the low-illumination initial image A is processed by the first convolutional neural network and the second convolutional neural network, An intermediate image B is obtained. Compared with the image A, the illumination of the image B is significantly enhanced.

S230 , using the third convolutional neural network in the image processing model to extract intermediate image features from the intermediate image information, and using an upsampling layer to perform amplification processing on the intermediate image features.

S230 , performing resolution enhancement processing on the intermediate image features amplified by the fourth convolutional neural network in the image processing model to obtain an image with enhanced illumination and resolution.

In the embodiment of the present application, the third convolutional neural network includes several convolutional layers, the last convolutional layer is connected to one or more upsampling layers, the number of channels of the last convolutional layer is 32, and the remaining convolutional layers are The number of channels for the layer is 64. The upsampling layer is used to enlarge the intermediate image features, and the number of the upsampling layers determines the magnification of the initial image. For example, when the last convolutional layer connects an upsampling layer, the processed image is enlarged twice relative to the original image, and when the last convolutional layer connects two consecutive upsampling layers, the processed image is enlarged relative to the original image The image is magnified four times. It should be understood that those skilled in the art can determine the number of up-sampling layers according to the image processing target, and details are not described herein.

The fourth convolutional neural network is used to perform resolution enhancement processing on the enlarged intermediate image.

FIG. 5 is a schematic diagram of an intermediate image and an illuminance and resolution-enhanced image according to an exemplary illustration of the present application. It can be seen from FIG. 5 that the intermediate image B is processed by the third convolutional neural network, the upsampling layer and the fourth convolutional neural network. After processing, an image C with enhanced illumination and resolution is obtained. Compared with the image B, the resolution of the image C is enhanced, the size is enlarged, and the illumination is further optimized.

It can be seen from the above embodiments that the embodiments of the present application provide an image processing method. The method first uses the first convolutional neural network in the image processing model to extract the reflection component and the illumination component from the initial image information, and then uses the image processing model. The second convolutional neural network in the image processing model combines the reflection component and the illumination component into the intermediate image information, so as to complete the illumination enhancement processing of the initial image; the third convolutional neural network in the image processing model is used to extract the intermediate image information from the intermediate image information. image features, and then use the upsampling layer in the image processing model to amplify the intermediate image features, and finally use the fourth convolutional neural network in the image processing model to perform resolution enhancement processing on the enlarged intermediate image features to complete the image processing. Enlargement and resolution enhancement of intermediate images. The image processing method provided by the embodiments of the present application can improve the image illuminance while improving the image resolution, avoiding image color distortion and information loss.

FIG. 6 is a flowchart of an image processing method according to an exemplary embodiment of the present application, which specifically shows the training process of the image processing model shown in FIG. 1 , that is, the first convolutional neural network, the second convolutional The training process of the neural network, the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network. As shown in Figure 6, the method may include:

S610: Acquire a training data set, where the training data set includes several sets of corresponding input images, intermediate target images, and target images.

Wherein, the size of the intermediate target image is the same as that of the input image, and the luminance of the intermediate target image is higher than that of the input image; the luminance of the target image is the same as that of the intermediate target image, and the size of the target image is larger than that of the intermediate target image.

In a possible implementation manner of S610, several groups of original images are obtained first, and each group of original images includes corresponding low-illumination images and normal-illumination images. The corresponding low-illumination image and normal care image refer to images respectively collected in the same shooting scene and under different lighting environments, and the sizes of the two images are the same. For example, keeping the position and various parameters of the image acquisition device unchanged, a low-illumination image is acquired under low-light conditions, and a normal-illumination image is acquired under normal illumination conditions. Then, according to each group of original images, training data corresponding to each group of original images is generated, wherein the input image in each group of training data is an image reduced by k times corresponding to the low-light image, and the intermediate target image is the corresponding normal light image reduced by k times After the image, the target image is the corresponding normal illumination image, and k is a positive number. It should be understood that the size reduction referred to here by k times means that the width and height of the image are simultaneously reduced by k times.

In one example, a certain group of original images includes a first low-illuminance image and a first normal illumination image, and the training data corresponding to the group of original images includes a first input image, a first intermediate target image, and a first target image. According to S110, the first input image is an image reduced by k times of the first low illumination image, the first intermediate target image is an image reduced by k times of the first normal illumination image, and the first target image is the first normal illumination image.

It should be noted that those skilled in the art can design a specific value of k according to requirements. For example, when the initial image to be processed needs to be enlarged by 4 times using the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network, then each group The input image in the training data is an image reduced by 4 times corresponding to the low illumination image, and the intermediate target image is an image corresponding to the normal illumination image reduced by 4 times.

S620. Perform joint training on the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network by using the training data set.

During training, the input images of each set of training data are input into the first convolutional neural network, and the output of the previous neural network in the image processing model is input into the adjacent next neural network. Specifically, the input image in each set of training data is used as the input of the first convolutional neural network, the input reflection component and the input illumination component output by the first convolutional neural network are used as the input of the second convolutional neural network, and the The intermediate training image output by the second convolutional neural network is used as the input of the third convolutional neural network, the output of the third convolutional neural network is used as the input of the up-sampling layer, and the output of the above-sampling layer is used as the output of the fourth convolutional neural network. enter.

Wherein, while inputting the input image into the first convolutional neural network, the corresponding intermediate target image is input into the auxiliary convolutional neural network, so as to use the auxiliary convolutional neural network to extract the target reflection component from the intermediate target image and the target illumination component, the auxiliary convolutional neural network has the same structure as the first convolutional neural network, and its parameters are optimized synchronously with the parameters of the first convolutional neural network, so that the parameters of the auxiliary convolutional neural network are the same as those of the first convolutional neural network. The parameters of the product neural network always remain the same.

After each round of training, the training loss is calculated according to the preset loss function, and the training loss includes the first local loss generated at the first convolutional neural network, the second local loss generated at the second convolutional neural network, and, The third local loss produced at the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network. Thus, the parameters of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network are optimized according to the training loss until the preset verification conditions are met.

In a possible implementation, the preset loss function is the MSE function. In this implementation, the training loss can be calculated according to the following formula:

Among them, N is the number of groups of training data in this round of training;

表示第i组训练数据在第一卷积神经网络中产生的局部损失，即第一局部损失；

Represents the local loss generated by the i-th group of training data in the first convolutional neural network, that is, the first local loss;

表示第i组训练数据在第二卷积神经网络中产生的局部损失，即第二局部损失；

Represents the local loss generated by the i-th group of training data in the second convolutional neural network, that is, the second local loss;

表示第i组训练数据在第三卷积神经网络、上采样层和第四卷积神经网络中产生的局部损失，即第三局部损失；

represents the local loss generated by the i-th group of training data in the third convolutional neural network, the upsampling layer and the fourth convolutional neural network, that is, the third local loss;

ω ₁ , ω ₂ , and ω ₃ are the weights corresponding to each local loss, respectively.

In this application, the output of the first convolutional neural network is the input reflection component and the input illumination component extracted from the input image. The image restored according to the input reflection component and the input illumination component is called the first restored image, and the image restored according to the target reflection component and the target illumination component is called the second restored image, wherein:

The first restored image=input reflection component×input illumination component;

Second restored image=target reflection component×target illumination component.

可以包括：第一还原图像相对于相应的输入图像的损失；第二还原图像相对于相应的中间目标图像的损失；以及，输入反射向量相对于对应目标反射向量的损失。这样，可以通过不断训练，使得第一卷积神经网络输出的输入反射分量不断接近对应目标反射分量，同时，使得第一卷积神经网络输出的输入反射分量和输入光照分量可以尽量还原出对应的输入图像，以及目标反射分量与目标光照分量可以尽量还原出对应的中间目标图像。In this application, the first partial loss

It may include: a loss of the first restored image relative to the corresponding input image; a loss of the second restored image relative to the corresponding intermediate target image; and a loss of the input reflection vector relative to the corresponding target reflection vector. In this way, through continuous training, the input reflection component output by the first convolutional neural network can be continuously approached to the corresponding target reflection component, and at the same time, the input reflection component and the input illumination component output by the first convolutional neural network can be restored to the corresponding The input image, as well as the target reflection component and the target illumination component can restore the corresponding intermediate target image as much as possible.

In addition, the second local loss is calculated from the output of the second convolutional neural network and the corresponding intermediate target image, and the third local loss is calculated from the output of the fourth convolutional neural network and the corresponding target image.

In this embodiment of the present application, the weights ω ₁ and ω ₂ corresponding to the local losses may be preset according to the gradients of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, and the fourth convolutional neural network. , ω ₃ , generally, the smaller the gradient of the neural network, the smaller the corresponding weight.

In one implementation, ω ₁ is less than ω ₂ and ω ₃ .

During the training process, when the result of an iteration is better than the result of the previous iteration, the parameters used in this iteration are saved as the current optimal parameters. The training process synchronously outputs the loss corresponding to the training set and the loss corresponding to the validation set, and stops training when the loss on the validation set meets the preset validation conditions.

It can be seen from the above S610-S620 that the image processing method provided by the present application constructs an image processing model based on a neural network for low-illumination images, and the image processing model includes a first convolutional neural network and a second convolutional neural network connected in sequence. , a third convolutional neural network, at least one upsampling layer and a fourth The joint training of convolutional neural networks can make the image processing model have the ability to improve the illumination of low-light images and the ability to improve the resolution of intermediate images, and can avoid color distortion and information loss.

According to the image processing method provided by the above embodiment, the embodiment of the present application further provides an image processing apparatus. As shown in FIG. 7 , the apparatus may include:

The low-illuminance enhancement module 710 is configured to extract the reflection component and the illumination component from the initial image information; perform synthesis processing on the reflection component and the illumination component, and output intermediate image information. The resolution enhancement module 720 is used for extracting intermediate image features from the intermediate image information; performing amplification processing and resolution enhancement processing on the intermediate image features to obtain an image with enhanced illumination and resolution.

In some embodiments, the low-illuminance enhancement module 710 includes a first enhancement unit and a second enhancement unit, the first enhancement unit is used for extracting the reflection component and the illumination component from the initial image information; the second enhancement unit is used for the reflection component The component and the illumination component are synthesized, and the intermediate image information is output. The resolution enhancement module 720 includes a third enhancement unit and a fourth enhancement unit, and the third enhancement unit is used for extracting intermediate image features from the intermediate image information, and performing enlarging processing on the intermediate image features; the fourth enhancement unit uses Performing resolution enhancement processing on the enlarged intermediate image features to obtain illumination and resolution enhancement images.

In some embodiments, the first enhancement unit is specifically a convolutional neural network including a plurality of convolutional layers, the second enhancement unit is specifically a convolutional neural network including a plurality of convolutional layers, and the third enhancement unit is specifically a convolutional neural network including a plurality of convolutional layers. A convolutional neural network of two convolutional layers and at least one upsampling layer, and the fourth enhancement unit is specifically a convolutional neural network including a plurality of convolutional layers.

In some embodiments, the first enhancement unit includes n convolution layers, and the first enhancement unit is specifically configured to: perform n convolution processing on the initial image information by using the n convolution layers, wherein the ith The output result of each convolutional layer is the input of the i+1th convolutional layer, and the output result of the nth convolutional layer is 4 feature maps, i is 1, 2, ..., n-1; The third feature map is used as the reflection component, and the fourth feature map is used as the illumination component.

In some embodiments, the apparatus further includes a training module, the training module includes a data preparation unit for acquiring a training data set, the training data set including several sets of corresponding input images, intermediate target images and target images , wherein the intermediate target image has the same size as the input image, and the intermediate target image has a higher illuminance than the input image, and the target image and the intermediate target image have the same illuminance, and the target image is larger in size than the intermediate target an image; a training unit for jointly training the first enhancement unit, the second enhancement unit, the third enhancement unit and the fourth enhancement unit by using the training data set.

In some embodiments, the data preparation unit is specifically configured to: acquire several groups of original images, each group of which includes corresponding low-light images and normal-light images; generate training data corresponding to each group of original images, each The input image in the set of training data is an image corresponding to a low-illumination image reduced by k times, the intermediate target image is an image corresponding to a normal illumination image reduced by k times, and the target image is the corresponding normal illumination image, k is a positive number.

In some embodiments, the training unit is specifically configured to: input the input image of each set of training data into the first convolutional neural network, and input the output of the previous neural network in the image processing model to the adjacent in the next neural network of the The first local loss of the local loss; optimizing the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer, and the fourth convolutional neural network according to the training loss parameters until the preset verification conditions are met. In some embodiments, the training unit is further configured to: input the corresponding intermediate target image into an auxiliary convolutional neural network, so as to extract a target reflection component from the first intermediate target image by using the auxiliary convolutional neural network and the target illumination component, wherein the auxiliary convolutional neural network has the same structure as the first enhancement unit; and, when optimizing the parameters of the first enhancement unit according to the training loss, the auxiliary volume is optimized synchronously parameters of the convolutional neural network so that the auxiliary convolutional neural network has the same parameters as the first enhancement unit.

In some embodiments, the output of the first convolutional neural network is the input reflection component and the input illumination component extracted from the input image; the local loss corresponding to the first enhancement unit includes: the first restored image is relatively Corresponding to the loss of the input image, the first restored image is restored according to the input reflection component and the input illumination component; the second restored image is relative to the loss of the corresponding intermediate target image, and the second restored image is obtained according to the target. The reflection component and the target illumination component are restored; and, the loss of the input reflection vector relative to the target reflection vector.

In some embodiments, the training loss is calculated as:

Among them, N is the number of groups of training data;

表示第i组训练数据在第一增强单元中产生的局部损失；

represents the local loss generated by the i-th group of training data in the first enhancement unit;

表示第i组训练数据在第二增强单元中产生的局部损失；

represents the local loss generated by the i-th group of training data in the second augmentation unit;

表示第i组训练数据在第三增强单元和第四增强单元中产生的局部损失；

Represents the local loss generated by the i-th group of training data in the third enhancement unit and the fourth enhancement unit;

ω ₁ , ω ₂ , and ω ₃ are the weights corresponding to each local loss, respectively.

In some embodiments, the ω ₁ is smaller than the ω ₂ and ω ₃ .

In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, it can include some or all of the steps in each embodiment of the image processing method provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (English: read-only memory, ROM for short) or a random access memory (English: random access memory, RAM for short).

Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products may be stored in a storage medium, such as ROM/RAM , magnetic disk, optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

It is sufficient to refer to each other for the same and similar parts among the various embodiments in this specification. In particular, as for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for related parts, please refer to the descriptions in the method embodiments.

The embodiments of the present invention described above do not limit the protection scope of the present invention.

Claims (10)

1. an image processing method, is characterized in that, for image processing model, described method comprises: Using the first convolutional neural network in the image processing model to extract the reflection component and the illumination component from the initial image information; Use the second convolutional neural network in the image processing model to synthesize the reflection component and the illumination component, and output intermediate image information; Use the third convolutional neural network in the image processing model to extract intermediate image features from the intermediate image information, and use the upsampling layer in the image processing model to amplify the intermediate image features; The image processing model uses the enlarged intermediate image features of the fourth convolutional neural network to perform resolution enhancement processing to obtain an image with enhanced illumination and resolution. 2. The method according to claim 1, wherein the first convolutional neural network comprises n convolutional layers, and the first convolutional neural network is used to extract the reflection component and the illumination component from the initial image information, including : Use the n convolutional layers to perform n convolution processing on the initial image information, wherein the output of the i-th convolutional layer is the input of the i+1-th convolutional layer, and the n-th convolutional layer The output result is 4 feature maps, i is 1, 2, ..., n-1; The first to third feature maps are used as the reflection component, and the fourth feature map is used as the illumination component. 3. The method according to claim 1, wherein the image processing model is obtained by training according to the following steps: Acquire a training data set, the training data set includes several sets of corresponding input images, intermediate target images and target images, wherein the intermediate target image is the same size as the input image, and the intermediate target image has a higher luminance than the input image image, and the target image and the intermediate target image have the same illuminance, and the size of the target image is larger than the intermediate target image; The first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the third convolutional neural network in the image processing model are analyzed using the training data set. Four convolutional neural networks are jointly trained. 4. The method according to claim 3, wherein the acquiring a training data set comprises: Acquiring several groups of original images, each group of the original images includes corresponding low-light images and normal-light images; Generate training data corresponding to each group of original images. The input image in each group of training data is an image reduced by k times for the corresponding low-light image, and the intermediate target image in each group of training data is the corresponding normal light image. Reduced by k times image, the target image in each set of training data is the corresponding normal illumination image, and k is a positive number. 5. The method of claim 3, wherein the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the The fourth convolutional neural network is connected in sequence, and the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network in the image processing model are analyzed by using the training data set. , the upsampling layer and the fourth convolutional neural network are jointly trained, including: The input image of each group of training data is input into the first convolutional neural network, and the output of the previous neural network in the image processing model is input into the adjacent next neural network; The training loss is calculated according to the preset loss function, and the training loss includes: the first local loss calculated according to the output of the first convolutional neural network and the target reflection component and the target illumination component extracted from the corresponding intermediate target image; the output of the second convolutional neural network and the second local loss calculated corresponding to the intermediate target image, and the third local loss calculated according to the output of the fourth convolutional neural network and the corresponding target image; Parameters of the first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the upsampling layer and the fourth convolutional neural network are optimized according to the training loss, until the preset verification conditions are met. 6. The method according to claim 5, characterized in that, while inputting a set of input images of training data into the first convolutional neural network, the method further comprises: The intermediate target image in the set of training data is input to the auxiliary convolutional neural network, so as to extract the target reflection component and the target illumination component from the intermediate target image using the auxiliary convolutional neural network, wherein the auxiliary volume The structure of the convolutional neural network is the same as that of the first convolutional neural network; and, when optimizing the parameters of the first convolutional neural network according to the training loss, simultaneously optimizing the parameters of the auxiliary convolutional neural network, so that the auxiliary convolutional neural network and the first convolutional neural network The parameters of the network are the same. 7. The method according to claim 5, wherein the output of the first convolutional neural network is the input reflection component and the input illumination component extracted from the input image; The output of the neural network and the target reflection component and target illumination component extracted from the corresponding intermediate target image calculate the first local loss, including: Restoring the first restored image according to the input reflection component and the input illumination component, and calculating the loss of the first restored image relative to the corresponding input image; Obtain a second restored image according to the target reflection component and the target illumination component, and calculate the loss of the second restored image relative to the corresponding intermediate target image; and, calculating the loss of the input reflection vector relative to the target reflection vector. 8. The method according to claim 5, wherein the training loss is calculated according to the following formula:

Among them, N is the number of groups of training data;

represents the first local loss generated in the first convolutional neural network by the i-th set of training data;

represents the second local loss generated by the i-th group of training data in the second convolutional neural network;

represents the third local loss generated in the third convolutional neural network, the upsampling layer and the fourth convolutional neural network for the i-th group of training data; ω ₁ , ω ₂ , and ω ₃ are the weights corresponding to each local loss, respectively. 9. The method of claim 8, wherein the ω ₁ is smaller than the ω ₂ and ω ₃ . 10. An image processing device, wherein the device comprises: A low-illuminance enhancement module, used for extracting a reflection component and an illumination component from the initial image information; synthesizing the reflection component and the illumination component, and outputting intermediate image information; A resolution enhancement module is used for extracting intermediate image features from the intermediate image information; performing amplification processing on the intermediate image features; performing resolution enhancement processing on the enlarged intermediate image features to obtain illumination and resolution enhanced images .

Images