CN113674169A

CN113674169A - Image processing method, image processing device, electronic equipment and computer readable storage medium

Info

Publication number: CN113674169A
Application number: CN202110924588.XA
Authority: CN
Inventors: 何慕威
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-08-12
Filing date: 2021-08-12
Publication date: 2021-11-19

Abstract

The application relates to an image processing method, an image processing device, an electronic device and a computer readable storage medium, comprising: performing convolution processing on an image to be processed to respectively obtain a tone template image and a noise template image; carrying out fusion processing on the tone template image and the image to be processed to obtain a tone mapping image; and carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a first target image. By adopting the method, the definition of the image can be improved and the calculation amount can be reduced.

Description

Image processing method, image processing device, electronic equipment and computer readable storage medium

Technical Field

The present application relates to the field of image technologies, and in particular, to an image processing method and apparatus, an electronic device, and a computer-readable storage medium.

Background

With the development of imaging technology, tone mapping algorithms have emerged. The tone mapping algorithm can adjust the gray scale of the image by mapping and transforming the colors of the image, so that the processed image looks more comfortable. However, conventional tone mapping tends to result in unclear images.

Disclosure of Invention

The embodiment of the application provides an image processing method and device, electronic equipment and a computer readable storage medium, which can improve the definition of an image and save the computing resources of image processing.

An image processing method comprising:

performing convolution processing on an image to be processed to respectively obtain a tone template image and a noise template image;

carrying out fusion processing on the tone template image and the image to be processed to obtain a tone mapping image;

and carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a first target image.

An image processing apparatus comprising:

the convolution module is used for performing convolution processing on the image to be processed to respectively obtain a tone template image and a noise template image;

the fusion module is used for fusing the tone template image and the image to be processed to obtain a tone mapping image;

and the noise reduction module is used for carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a first target image.

A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

A computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions are executed by a processor for the following steps:

By performing convolution processing on the image to be processed, the tone template image containing tone information and the noise template image containing noise information can be separated from the convolution processing of the same image, so that the calculation amount can be reduced, and the calculation resources can be saved. By fusing the tone template image and the image to be processed, the tone mapping can be performed on the image to be processed, and meanwhile, the detail information and the features in the original image can be better expressed. And carrying out noise reduction treatment on the tone mapping image based on the noise template image, so that the definition of the image can be improved, and the first target image which not only retains the detail characteristics of the image to be processed but also ensures the definition is obtained.

A method of training an image processing model, comprising:

acquiring a training sample image, and a tone label image and a noise label image which are respectively corresponding to the training sample image;

carrying out convolution processing on the training sample image through an image processing model to respectively obtain a tone sample image and a noise sample image;

carrying out fusion processing on the tone sample image and the training sample image to obtain a tone mapping predicted image;

carrying out noise reduction processing on the tone mapping predicted image according to the noise sample image to obtain a noise predicted image;

and adjusting parameters of the image processing model and continuing training based on the difference between the tone mapping predicted image and the corresponding tone label image and the difference between the noise predicted image and the corresponding noise label image until a training stopping condition is met, so as to obtain the trained image processing model.

An apparatus for training an image processing model, comprising:

the acquisition module is used for acquiring a training sample image, and a tone label image and a noise label image which respectively correspond to the training sample image;

the sample convolution module is used for carrying out convolution processing on the training sample image through an image processing model to respectively obtain a tone sample image and a noise sample image;

the sample fusion module is used for carrying out fusion processing on the tone sample image and the training sample image to obtain a tone mapping predicted image;

the sample denoising module is used for denoising the tone mapping predicted image according to the noise sample image to obtain a noise predicted image;

and the adjusting module is used for adjusting the parameters of the image processing model and continuing training based on the difference between the tone mapping predicted image and the corresponding tone label image and the difference between the noise predicted image and the corresponding noise label image until the training stopping condition is met, so that the trained image processing model is obtained.

By acquiring the training sample image and the tone label image and the noise label image respectively corresponding to the training sample image, and performing convolution processing on the training sample image through the image processing model, the tone sample image containing tone information and the noise sample image containing noise information are respectively obtained, and the calculation amount can be reduced. The method comprises the steps of carrying out fusion processing on a tone sample image and a training sample image to obtain a tone mapping predicted image, carrying out noise reduction processing on the tone mapping predicted image according to the noise sample image to obtain a noise predicted image, adjusting parameters of an image processing model and continuing training on the basis of the difference between the tone mapping predicted image and a corresponding tone label image and the difference between the noise predicted image and a corresponding noise label image until a training stopping condition is met, and obtaining a trained image processing model.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a diagram of an exemplary embodiment of an image processing method;

FIG. 2 is a flow diagram of a method of image processing in one embodiment;

FIG. 3 is a schematic diagram of a convolutional neural network in one embodiment;

FIG. 4 is a flow diagram of obtaining a tonal template image and a noisy template image in one embodiment;

FIG. 5 is a flow diagram for obtaining a first target image after obtaining a tonal template image and a noisy template image in one embodiment;

FIG. 6 is a flowchart of an image processing method in another embodiment;

FIG. 7 is a flow diagram that illustrates a method for training an image processing model, according to one embodiment;

FIG. 8 is a flowchart illustrating the acquisition of a tonal label image and a noise label image corresponding to a training sample image, respectively, in one embodiment;

FIG. 9 is a block diagram showing the configuration of an image processing apparatus according to an embodiment;

FIG. 10 is a block diagram showing the construction of an image processing model training apparatus according to an embodiment;

FIG. 11 is a diagram illustrating the internal architecture of an electronic device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first target image may be referred to as a second target image, and similarly, a second target image may be referred to as a first target image, without departing from the scope of the present application. Both the first target image and the second target image are target images, but they are not the same target image.

Fig. 1 is a schematic diagram of an application environment of an image processing method in an embodiment. As shown in fig. 1, the application environment includes an electronic device 110 and a server 120. In one embodiment, the electronic device 110 and the server 120 may each separately execute the image processing method, and the electronic device 110 and the server 120 may also cooperatively execute the image processing method. When the electronic device 110 and the server 120 cooperatively execute the image processing method, the electronic device 110 acquires the image to be processed and transmits the image to be processed to the server 120. The server 120 performs convolution processing on the image to be processed to obtain a tone template image and a noise template image, respectively. And the server performs fusion processing on the tone template image and the image to be processed to obtain a tone mapping image. The server performs noise reduction processing on the tone-mapped image based on the noise template image to obtain a first target image, and returns the first target image to the electronic device 110.

Wherein the electronic device 110 communicates with the server 120 over a network. The electronic device 110 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 120 may be implemented by a stand-alone server or a server cluster composed of a plurality of servers.

FIG. 2 is a flow diagram of a method of image processing in one embodiment. The image processing method in this embodiment is described by taking the electronic device in fig. 1 as an example. As shown in fig. 2, the image processing method includes steps 202 to 206.

Step 202, performing convolution processing on the image to be processed to respectively obtain a tone template image and a noise template image.

The image to be processed may be a High-Dynamic Range (HDR) image, and specifically may be any one of an RGB (Red, Green, Blue) image, a RAW image, a grayscale image, a depth image, an image corresponding to a Y component in a YUV image, and the like. The RAW image is RAW data obtained by converting a captured light source signal into a digital signal by an image sensor. "Y" in YUV images represents brightness (Luma) and gray scale value, and "U" and "V" represent Chrominance (Chroma) and saturation, which are used to describe the color and saturation of the image and to specify the color of the pixel.

The tone template image is an image separated from the image to be processed and related to tones, and the noise template image is a noise image separated from the image to be processed.

Specifically, the electronic device may obtain the image to be processed from a local device or other devices or a network, or the electronic device may obtain the image to be processed by shooting a scene with a camera. The image to be processed may also be a video frame in a video acquired from a local or other device or network, or a video frame in a video shot by a camera.

The electronic equipment performs convolution processing on the image to be processed to respectively obtain a tone template image and a noise template image. Further, the electronic device performs different convolution processes on the image to be processed respectively to obtain a tone template image and a noise template image respectively.

In one embodiment, the image to be processed may be convolved, pooled, deconvolved, etc. by a convolutional neural network as shown in fig. 3, resulting in a tone template image and a noise template image, respectively.

In one implementation, the electronic device inputs an image to be processed into a tone mapping network, the tone mapping network is composed of at least two convolutional layers, and the image to be processed is subjected to convolution processing through the tone mapping network to obtain a tone template image. The electronic equipment inputs the image to be processed into a noise network, the noise network consists of at least two layers of convolution layers, and the image to be processed is subjected to convolution processing through the noise network to obtain a noise template image.

And 204, fusing the tone template image and the image to be processed to obtain a tone mapping image.

Specifically, the electronic device may perform fusion processing on the tone template image and the image to be processed, and obtain a tone mapping image after the fusion processing.

In one embodiment, the electronic device may perform weighted averaging on the tone template image and the image to be processed to obtain a tone mapped image.

In one embodiment, the fusion processing of the tone template image and the image to be processed to obtain a tone mapping image includes: and carrying out pixel dot multiplication on the tone template image and the image to be processed to obtain a tone mapping image.

And step 206, carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a first target image.

Specifically, the electronic device performs noise reduction processing on the tone mapping image based on the noise template image to obtain a noise-reduced first target image. The first target image may be a Low-Dynamic Range (LDR) image.

In one embodiment, the electronic device may perform weighted averaging on the noise template image and the tone-mapped image to obtain the first target image after noise reduction.

In one embodiment, denoising the tone-mapped image based on the noise template image to obtain a first target image, comprises: and carrying out pixel addition processing on the noise template image and the tone mapping image to obtain a first target image.

In the image processing method in this embodiment, by performing convolution processing on an image to be processed, a tone template image including tone information and a noise template image including noise information can be separated from the convolution processing of the same image, so that the amount of calculation can be reduced, and calculation resources can be saved. By fusing the tone template image and the image to be processed, the tone mapping can be performed on the image to be processed, and meanwhile, the detail information and the features in the original image can be better expressed. The tone mapping image is subjected to noise reduction processing based on the noise template image, and the definition of the image can be improved, so that the first target image which not only retains the detail characteristics of the image to be processed but also ensures the definition is obtained.

In one embodiment, performing convolution processing on an image to be processed to obtain a tone template image and a noise template image respectively includes: normalizing the image to be processed, and performing convolution processing on the image after the normalization processing to respectively obtain a tone template image and a noise template image;

carrying out fusion processing on the tone template image and the image to be processed to obtain a tone mapping image, wherein the fusion processing comprises the following steps: carrying out fusion processing on the tone template image and the normalized image to obtain a tone mapping image;

based on the noise template image, carrying out noise reduction processing on the tone mapping image to obtain a first target image, wherein the noise reduction processing comprises the following steps: carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a noise reduction image; and carrying out image restoration processing on the noise-reduced image to obtain a first target image.

Specifically, the electronic device may perform normalization processing on the image to be processed to obtain an image after the normalization processing.

In one embodiment, the electronic device obtains a maximum value max and a minimum value min of pixels in an image to be processed, and then subtracts the minimum value min from each pixel value pixel in the image to be processed, and divides the difference between the maximum value max and the minimum value min to obtain a normalized pixel value pixel _ out, so as to obtain the image after normalization processing, wherein a specific calculation formula of the normalized pixel value pixel _ out is as follows:

then, the electronic device can perform different convolution processes on the normalized image to obtain a tone template image and a noise template image respectively. And the electronic equipment performs fusion processing on the tone template image and the normalized image to obtain a tone mapping image.

And carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a noise reduction image, and then carrying out image restoration processing on the noise reduction image to obtain a first target image. Further, after obtaining the noise-reduced image, performing clipping processing on the noise-reduced image, that is, setting a value greater than 1 as 1 and a value less than 0 as 0, and then multiplying the image by 255 pixel by pixel to obtain a first target image, wherein a reduction formula is as follows:

pixel_out＝max(0,min(1.0，pixel))×255.0

where pixel is an input pixel and pixel _ out is an output pixel.

In this embodiment, the image to be processed is normalized, so that the calculation amount of subsequent processing is reduced, and the calculation efficiency is improved. The normalized image is subjected to convolution processing, and a tone template image including tone information and a noise template image including noise information can be separated. The tone mapping image is obtained by fusing the tone template image containing tone information and the normalized image, so that the tone of the image to be processed can be effectively compressed from a high dynamic range to a low dynamic range, and the detailed characteristics of the image can be reserved. And carrying out noise reduction processing and image restoration processing on the tone mapping image based on the noise template image, so that the noise of the image can be reduced, and a clearer first target image can be obtained.

In one embodiment, the convolution processing is performed on the normalized image to obtain a tone template image and a noise template image, respectively, and the method includes:

performing convolution processing on the image after the normalization processing based on the tone mapping parameters to obtain a corresponding tone template image; and performing convolution processing on the normalized image based on the noise extraction parameters to obtain a corresponding noise template image.

Wherein the tone mapping parameters and the noise extraction parameters are parameters obtained from a trained image processing model.

Specifically, the electronic device may obtain the tone mapping parameter, and perform convolution processing on the normalized image based on the tone mapping parameter to obtain the tone mapping template. The electronic device can obtain the tone mapping parameters, and perform convolution processing on the normalized image based on the noise extraction parameters to obtain a noise template image.

In one embodiment, as shown in fig. 4, the image to be processed is a high dynamic range image, the trained image processing model includes a tone mapping network and a noise network, the tone mapping network includes tone mapping parameters, and the noise network includes noise extraction parameters.

Step 402, the electronic device obtains a high dynamic range image and inputs the high dynamic range image into a trained image processing model.

And 404, performing normalization processing on the high dynamic range image through the image processing model to obtain an image after the normalization processing.

And step 406, sending the normalized image to an encoder network of the image processing model, wherein the encoder network consists of a plurality of convolution layers and pooling layers.

Step 408, the image features obtained through the encoder network are sent to a decoder network, wherein the decoder network consists of a plurality of convolution layers and a plurality of deconvolution layers. The image characteristics after passing through the decoder network are respectively sent to the tone mapping network and the noise network.

Step 410, convolving the decoded image features with the tone mapping parameters of the tone mapping network to obtain a tone template image 414.

Step 412, performing convolution processing on the decoded image features through a noise network to obtain a noise template image 416.

In this embodiment, the normalized image is convolved based on the tone mapping parameter, so that the amount of calculation can be reduced, and the tone template image containing tone information can be quickly separated from the image to be processed. The image after the normalization processing is subjected to convolution processing based on the noise extraction parameters, the calculation amount can be reduced, and therefore the noise template image containing the image noise can be separated from the image to be processed quickly.

In one embodiment, the fusion processing of the tone template image and the image to be processed to obtain a tone-mapped image includes: and carrying out pixel dot multiplication on the tone template image and the image to be processed to obtain a tone mapping image.

Specifically, the electronic device may perform pixel matching on the noise template image and the tone mapping image, and determine pixel points matched between the noise template image and the tone mapping image. And the electronic equipment performs point multiplication on the pixel values of the matched pixel points in the noise template image and the pixel values in the tone mapping image to obtain the pixel values of the corresponding pixel points in the tone mapping image. And performing dot multiplication on the pixel values of the matched pixel points according to the same processing mode to obtain the pixel values corresponding to the pixel points in the tone mapping image, thereby obtaining the tone mapping image.

In the embodiment, the pixel dot multiplication processing is performed on the tone template image and the image to be processed, the brightness range of the image to be processed can be adjusted, and the detail information and the color information in the image to be processed can be effectively reserved in the obtained tone mapping image.

Specifically, the electronic device may perform pixel matching on the noise template image and the tone mapping image, and determine pixel points matched between the noise template image and the tone mapping image. The electronic equipment adds the pixel values of the matched pixel points in the noise template image and the pixel values of the matched pixel points in the tone mapping image to fuse the matched pixel points to obtain the pixel values of the corresponding pixel points in the first target image. According to the same processing mode, each pixel point in the first target image and the pixel value corresponding to each pixel point can be obtained, and therefore the first target image is obtained.

In the embodiment, the noise template image and the tone mapping image are subjected to pixel addition processing to obtain the first target image, so that the noise of the image can be simply and effectively reduced, and the definition of the image is improved.

FIG. 5 is a flow diagram illustrating obtaining a first target image after obtaining a tonal template image and a noisy template image, according to one embodiment. The normalized image 502 and the tone template image 504 are subjected to the pixel dot multiplication process of step 506 to obtain a tone-mapped image 508. The tone-mapped image 508 and the noise template image 510 are subjected to the pixel-addition denoising process of step 512 to obtain a denoised image 514. The noise-reduced image 514 is subjected to image restoration processing in step 516, and a first target image 518 is obtained.

Fig. 6 is a flowchart of an image processing method according to an embodiment. And acquiring an image to be processed 602, and performing normalization processing in the step 604 on the image to be processed 602 to obtain an image after normalization processing. The normalized image is input to a convolutional neural network 606. The convolutional neural network 606 includes an encoder network, a decoder network, a tone mapping network, and a noise network, and the convolutional neural network 606 performs convolutional processing on the normalized image to obtain a tone template image 608 and a noise template image 610, respectively.

The tone mapping process of step 612, i.e., the pixel dot multiplication process, is performed on the normalized image and the tone template image 608 to obtain a tone mapped image. And performing noise reduction processing on the tone mapping image and the noise template image 610, namely performing pixel addition denoising processing to obtain a noise reduction image. The noise-reduced image is subjected to image restoration processing in step 616 to obtain a first target image 618.

In an embodiment, an image processing method is provided, where the image processing method in this embodiment is described by taking the electronic device in fig. 1 as an example, and includes:

performing convolution processing on an image to be processed to respectively obtain a tone template image and a noise template image; and performing noise reduction processing on the image to be processed based on the noise template image, and fusing the tone template image and the image subjected to the noise reduction processing to obtain a second target image.

Specifically, the electronic device performs different convolution processes on the image to be processed respectively to obtain a tone template image and a noise template image respectively.

In one embodiment, performing convolution processing on an image to be processed to obtain a tone template image and a noise template image respectively includes: and carrying out normalization processing on the image to be processed, and carrying out convolution processing on the image after the normalization processing to respectively obtain a tone template image and a noise template image.

In one embodiment, the convolution processing is performed on the normalized image to obtain a tone template image and a noise template image, respectively, and the method includes: performing convolution processing on the image after the normalization processing based on the tone mapping parameters to obtain a corresponding tone template image; and performing convolution processing on the normalized image based on the noise extraction parameters to obtain a corresponding noise template image.

After obtaining the tone template image and the noise template image, the electronic device may perform noise reduction processing on the image to be processed based on the noise template image. Further, the noise template image and the image to be processed may be subjected to pixel addition processing to reduce noise of the image to be processed. And fusing the tone template image and the image subjected to noise reduction processing to obtain a second target image. Further, the color tone template image and the image subjected to the noise reduction processing are subjected to pixel dot multiplication processing to obtain a second target image.

In one embodiment, denoising an image to be processed based on a noise template image, and fusing the tone template image and the denoised image to obtain a second target image, includes:

and performing noise reduction processing on the normalized image based on the noise template image, and performing fusion processing on the tone template image and the noise-reduced image to obtain a second target image.

In one embodiment, denoising the normalized image based on the noise template image comprises: carrying out pixel addition processing on the noise template image and the normalized image to obtain an image subjected to noise reduction processing;

and carrying out fusion processing on the tone template image and the image subjected to noise reduction processing to obtain a second target image, wherein the fusion processing comprises the following steps: and carrying out pixel dot multiplication on the tone template image and the image subjected to noise reduction processing to obtain a second target image.

In this embodiment, by performing convolution processing on an image to be processed, a tone template image including tone information and a noise template image including noise information can be separated from convolution processing of the same image, so that the amount of calculation can be reduced, and calculation resources can be saved. And the noise reduction processing is carried out on the image to be processed based on the noise template image, so that the noise of the image can be reduced, and the definition of the image is improved. The tone template image and the image subjected to noise reduction processing are fused, so that the detail information and the features in the original image can be better expressed.

It can be understood that the specific processing procedures of normalization processing, fusion, denoising processing, pixel addition processing, pixel dot multiplication processing, and the like in this embodiment may refer to the corresponding processing in each of the above embodiments, and are not described herein again.

FIG. 7 is a flow diagram that illustrates a methodology for training an image processing model, in one embodiment. The training method of the image processing model in this embodiment is described by taking the electronic device in fig. 1 as an example. As shown in fig. 7, the training method of the image processing model includes:

step 702, obtaining a training sample image, and a tone label image and a noise label image respectively corresponding to the training sample image.

The training sample image may be a high dynamic range image, and specifically may be any one of an RGB image, a RAW image, a grayscale image, a depth image, an image corresponding to a Y component in a YUV image, and the like. The tone label image is a real image for characterizing tones in the training sample image, and the noise label image is a real image for characterizing noise in the training sample image.

Specifically, the electronic device may obtain each training sample image from a local device or other devices or networks, or the electronic device may obtain each training sample image by shooting a scene through a camera. The training sample image may also be a video frame in a video taken from a local or other device or network, or a video frame in a video taken by a camera.

Under the condition that a plurality of training sample images exist, the electronic equipment respectively acquires a tone label image and a noise label image which respectively correspond to each training sample image.

Step 704, performing convolution processing on the training sample image through the image processing model to obtain a tone sample image and a noise sample image respectively.

Specifically, the electronic device inputs a training sample image into an image processing model, and performs convolution processing on the image to be processed through the image processing model to obtain a tone sample image and a noise sample image respectively. Further, different convolution processes are carried out on the training sample image through the image processing model so as to respectively obtain the tone sample image and the noise sample image.

In one embodiment, the image processing model includes an encoder network, a decoder network, a tone mapping network, and a noise network. The image to be processed can be normalized through the image processing model, and the normalized image is obtained. And sending the normalized image into an encoder network of an image processing model for encoding, and sending the image characteristics obtained by encoding into a decoder network for decoding. And the image characteristics after passing through the decoder network are respectively sent into a tone mapping network and a noise network, and the decoded image characteristics are subjected to convolution processing through the tone mapping network to obtain a tone sample image. And carrying out convolution processing on the decoded image characteristics through a noise network to obtain a noise sample image.

And step 706, performing fusion processing on the tone sample image and the training sample image to obtain a tone mapping predicted image.

Specifically, the electronic device may perform fusion processing on the tone sample image and the training sample image, and obtain the tone mapping predicted image after the fusion processing.

In one embodiment, the fusing the tone sample image and the training sample image to obtain a tone-mapped predicted image includes: and carrying out pixel dot multiplication on the tone sample image and the training sample image to obtain a tone mapping predicted image.

And step 708, performing noise reduction processing on the tone mapping predicted image according to the noise sample image to obtain a noise predicted image.

Specifically, the electronic device performs noise reduction processing on the tone mapping predicted image based on the noise sample image to obtain a noise predicted image after noise reduction. The noise-predicted image may be a low dynamic range image.

In one embodiment, denoising a tone-mapped predicted image based on a noise sample image to obtain a noise predicted image comprises: and carrying out pixel addition processing on the noise sample image and the tone mapping predicted image to obtain a noise predicted image.

And step 710, adjusting parameters of the image processing model and continuing training based on the difference between the tone mapping predicted image and the corresponding tone label image and the difference between the noise predicted image and the corresponding noise label image until the training stopping condition is met, so as to obtain the trained image processing model.

Specifically, the electronic device calculates the difference between the tone mapping predicted image and the corresponding tone label image, calculates the difference between the noise predicted image and the corresponding noise label image, adjusts the parameters of the image processing model according to the two differences, and continues training the image processing model after the parameters are adjusted until the training stopping condition is met, so as to obtain the trained image processing model.

In one embodiment, the training stop condition may be that the loss error is smaller than an error threshold, or that the number of training iterations reaches a preset number of iterations, or the like.

For example, the electronic device calculates a first loss error between the tone-mapped predicted image and the corresponding tone-labeled image and calculates a second loss error between the noise predicted image and the corresponding noise-labeled image, and adjusts parameters of the image processing model based on the first loss error and the second loss error. And further, when at least one of the first loss error is larger than a first error threshold value and the second loss error is larger than a second error threshold value exists, adjusting parameters of the image processing model, and continuing training the image processing model after the parameters are adjusted until the first loss error is not larger than the first error threshold value and the second loss error is not larger than the second error threshold value, and stopping training to obtain the trained image processing model.

In this embodiment, a training sample image, and a tone label image and a noise label image corresponding to the training sample image are obtained, and the training sample image is convolved by an image processing model to obtain a tone sample image containing tone information and a noise sample image containing noise information, respectively, which can reduce the amount of calculation. The method comprises the steps of carrying out fusion processing on a tone sample image and a training sample image to obtain a tone mapping predicted image, carrying out noise reduction processing on the tone mapping predicted image according to the noise sample image to obtain a noise predicted image, adjusting parameters of an image processing model and continuing training on the basis of the difference between the tone mapping predicted image and a corresponding tone label image and the difference between the noise predicted image and a corresponding noise label image until a training stopping condition is met, and obtaining a trained image processing model.

In one embodiment, performing convolution processing on a training sample image through an image processing model to obtain a tone sample image and a noise sample image respectively includes:

performing convolution processing on the training sample image based on the initial tone mapping parameter of the image processing model to obtain a corresponding tone sample image; performing convolution processing on the training sample image based on the initial noise extraction parameters of the image processing model to obtain a corresponding noise sample image;

adjusting parameters of the image processing model and continuing training based on differences between the tone-mapped predicted image and the corresponding tone-labeled image and differences between the noise predicted image and the corresponding noise-labeled image, comprising: adjusting initial tone mapping parameters of the image processing model based on differences between the tone-mapped predicted image and the corresponding tone-labeled image, and adjusting initial noise extraction parameters of the image processing model based on differences between the noise predicted image and the corresponding noise-labeled image and continuing training.

Specifically, the untrained image processing model includes an initial tone mapping parameter and an initial noise extraction parameter, where the initial tone mapping parameter may refer to an original tone mapping parameter included in the untrained image processing model or may be an adjusted tone mapping parameter. The initial noise extraction parameter may refer to an original noise extraction parameter included in the untrained image processing model, or may be an adjusted noise extraction parameter.

And carrying out convolution processing on the training sample image through the initial tone mapping parameters of the tone mapping network in the image processing model to obtain a corresponding tone sample image. And performing convolution processing on the training sample image through the initial noise extraction parameters of the noise network in the image processing model to obtain a corresponding noise sample image.

In one embodiment, the normalized image is convolved with the initial tone mapping parameters of the tone mapping network to obtain a corresponding tone sample image. And performing convolution processing on the normalized image through the initial noise extraction parameters of the noise network to obtain a corresponding noise sample image.

Initial tone mapping parameters of the tone mapping network are adjusted based on a difference between the tone mapped predicted image and the corresponding tone tagged image. Adjusting an initial noise extraction parameter of the noise network based on a difference between the noise prediction image and the corresponding noise label image. And continuing training the image processing model after the initial tone mapping parameters and the initial noise extraction parameters are adjusted until the training stopping conditions are met, and obtaining the trained image processing model. The trained image processing model comprises tone mapping parameters and noise extraction parameters.

In this embodiment, the training sample image is convolved based on the initial tone mapping parameter, so that the amount of calculation can be reduced, and the tone sample image including tone information can be quickly separated from the original image. The convolution processing is carried out on the training sample image based on the noise extraction parameters, the calculation amount can be reduced, and therefore the noise sample image containing the image noise can be separated from the original image quickly. The initial tone mapping parameters of the image processing model are adjusted based on the difference between the tone mapping predicted image and the corresponding tone label image, and the initial noise extraction parameters of the image processing model are adjusted based on the difference between the noise predicted image and the corresponding noise label image, so that the tone mapping parameters and the noise extraction parameters of the model can be accurately adjusted, and the processing precision of the model can be effectively improved.

In one embodiment, acquiring a training sample image, and a tone label image and a noise label image respectively corresponding to the training sample image, includes:

acquiring each frame of training sample image, and performing registration processing on each frame of training sample image to obtain each registration image; carrying out tone mapping processing on each registration image, and selecting a tone label image from the image subjected to the tone mapping processing; and carrying out multi-frame noise reduction processing on the image subjected to tone mapping processing to obtain a noise label image.

Specifically, the electronic device may acquire a plurality of frames of training sample images, and select a reference image from the plurality of frames of training sample images. Further, the first frame training sample image may be selected as the reference image.

And carrying out registration processing on each frame of training sample image through the reference image to obtain each frame of registration image. The electronic device can perform tone mapping processing on each frame of registration image to obtain corresponding images subjected to tone mapping processing.

The electronic equipment can select the tone label image from the images subjected to the tone mapping processing, and perform multi-frame noise reduction processing on the images subjected to the tone mapping processing to obtain the noise label image. Further, the electronic device may average matched pixel values in each tone mapped processed image to generate a new image, which may be used as a noise label image.

In one embodiment, the reference image may be tone mapped to obtain a tone-tagged image.

In one embodiment, the registration image and the reference image of each frame may be tone mapped by a local tone mapping (local tonemapping) algorithm or a global tone mapping algorithm.

In this embodiment, each frame of training sample image is obtained, each frame of training sample image is subjected to registration processing to obtain each registration image, each registration image is subjected to tone mapping processing, a tone label image is selected from the images subjected to tone mapping processing, and the images subjected to tone mapping processing are subjected to multi-frame noise reduction processing to obtain a noise label image, so that the reliability of the tone label image and the noise label image serving as the comparison labels can be ensured.

Fig. 8 is a flowchart illustrating obtaining a tone label image and a noise label image corresponding to a training sample image, respectively, in an embodiment.

Step 802, 10 training sample images of the same scene are obtained, and the first training sample image is taken as a reference image. 10 still images may be acquired for each scene, or more than 10, with the acquisition being continuous on a foot rest. The continuous images on the foot rest have small displacement, and are continuously collected on the foot rest, so that the subsequent registration and multi-frame processing are facilitated. Multiple sets of training sample images, e.g., 50 sets of 10, may also be acquired for different scenarios.

And step 804, respectively carrying out image registration on the rest 9 training sample images through the reference image to obtain 9 registration images.

Step 806, performing tone mapping processing on the reference image and the 9 registered images respectively to obtain 10 images subjected to tone mapping processing, and using the images subjected to tone mapping processing of the reference image as tone label images 808.

Step 810, performing pixel addition and averaging on 10 images subjected to tone mapping processing to obtain a noise label image 812.

In one embodiment, there is provided an image processing method, performed by an electronic device, including:

and acquiring each frame of training sample image, and performing registration processing on each frame of training sample image to obtain each registration image.

Then, carrying out tone mapping processing on each registration image, and selecting a tone label image from the image subjected to the tone mapping processing; and carrying out multi-frame noise reduction processing on the image subjected to tone mapping processing to obtain a noise label image.

Further, convolution processing is carried out on the training sample image based on the initial tone mapping parameters of the tone mapping network in the image processing model to obtain a corresponding tone sample image, and convolution processing is carried out on the training sample image based on the initial noise extraction parameters of the noise network in the image processing model to obtain a corresponding noise sample image.

And then, carrying out fusion processing on the tone sample image and the training sample image to obtain a tone mapping predicted image.

Further, the tone mapping predicted image is subjected to noise reduction processing according to the noise sample image to obtain a noise predicted image.

Further, based on the difference between the tone mapping predicted image and the corresponding tone label image, adjusting the initial tone mapping parameters of the image processing model, and based on the difference between the noise predicted image and the corresponding noise label image, adjusting the initial noise extraction parameters of the image processing model and continuing training until the training stopping condition is met, so as to obtain the trained image processing model.

And acquiring an image to be processed, and inputting the image to be processed into the trained image processing model. The image processing model includes a tone mapping network and a noise network.

And carrying out normalization processing on the image to be processed through an image processing model, and carrying out convolution processing on the image after the normalization processing based on tone mapping parameters of a tone mapping network to obtain a corresponding tone template image.

And then, performing convolution processing on the normalized image based on the noise extraction parameters of the noise network to obtain a corresponding noise template image.

And then, carrying out pixel dot multiplication on the tone template image and the normalized image to obtain a tone mapping image.

Further, the tone mapping image is subjected to pixel addition processing based on the noise template image to obtain a noise reduction image.

Further, the noise-reduced image is subjected to image restoration processing to obtain a first target image.

In this embodiment, each frame of training sample image is subjected to registration processing, each registration image is subjected to tone mapping processing, a tone label image is selected from the images subjected to tone mapping processing, and the images subjected to tone mapping processing are subjected to multi-frame noise reduction processing to obtain a noise label image, so that the reliability of the tone label image and the noise label image serving as comparison labels can be ensured.

The training sample image is convolved by the image processing model to obtain a tone sample image containing tone information and a noise sample image containing noise information, respectively, so that the amount of calculation can be reduced. The method comprises the steps of carrying out fusion processing on a tone sample image and a training sample image to obtain a tone mapping predicted image, carrying out noise reduction processing on the tone mapping predicted image according to a noise sample image to obtain a noise predicted image, adjusting parameters of an image processing model and continuing training based on the difference between the tone mapping predicted image and a corresponding tone label image and the difference between the noise predicted image and a corresponding noise label image until a training stopping condition is met, and obtaining a trained image processing model.

By performing convolution processing on the image to be processed, the tone template image containing tone information and the noise template image containing noise information can be separated from the convolution processing of the same image, so that the calculation amount can be reduced, and the calculation resources can be saved. By fusing the tone template image and the image to be processed, the tone mapping can be performed on the image to be processed, and meanwhile, the detail information and the features in the original image can be better expressed. The tone mapping image is subjected to noise reduction processing based on the noise template image, and the definition of the image can be improved, so that the first target image which not only retains the detail characteristics of the image to be processed but also ensures the definition is obtained.

It should be understood that although the various steps in the flowcharts of fig. 2-8 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-8 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.

Fig. 9 is a block diagram showing the configuration of an image processing apparatus according to an embodiment. As shown in fig. 9, the image processing apparatus includes:

and a convolution module 902, configured to perform convolution processing on the image to be processed to obtain a tone template image and a noise template image, respectively.

And a fusion module 904, configured to perform fusion processing on the tone template image and the image to be processed to obtain a tone mapping image.

And a noise reduction module 906, configured to perform noise reduction processing on the tone-mapped image based on the noise template image to obtain a first target image.

In this embodiment, by performing convolution processing on an image to be processed, a tone template image including tone information and a noise template image including noise information can be separated from convolution processing of the same image, so that the amount of calculation can be reduced, and calculation resources can be saved. By fusing the tone template image and the image to be processed, the tone mapping can be performed on the image to be processed, and meanwhile, the detail information and the features in the original image can be better expressed. The tone mapping image is subjected to noise reduction processing based on the noise template image, and the definition of the image can be improved, so that the first target image which not only retains the detail characteristics of the image to be processed but also ensures the definition is obtained.

In an embodiment, the convolution module 902 is further configured to perform normalization on the image to be processed, and perform convolution on the image after the normalization to obtain a tone template image and a noise template image, respectively;

the fusion module 904 is further configured to perform fusion processing on the tone template image and the normalized image to obtain a tone mapping image;

the denoising module 906 is further configured to perform denoising processing on the tone mapping image based on the noise template image to obtain a denoised image; and carrying out image restoration processing on the noise-reduced image to obtain a first target image.

In an embodiment, the convolution module 902 is further configured to perform convolution processing on the normalized image based on the tone mapping parameter to obtain a corresponding tone template image; and performing convolution processing on the normalized image based on the noise extraction parameters to obtain a corresponding noise template image.

In an embodiment, the fusion module 904 is further configured to perform pixel dot multiplication on the tone template image and the image to be processed to obtain a tone mapping image;

In one embodiment, the denoising module 906 is further configured to perform pixel addition processing on the noise template image and the tone-mapped image to obtain the first target image.

In one embodiment, an image processing apparatus is provided that includes a convolution module 902 and a processing module, wherein,

And the processing module is used for carrying out noise reduction processing on the image to be processed based on the noise template image and fusing the tone template image and the image subjected to the noise reduction processing to obtain a second target image.

The division of the modules in the image processing apparatus is merely for illustration, and in other embodiments, the image processing apparatus may be divided into different modules as needed to complete all or part of the functions of the image processing apparatus.

Fig. 10 is a block diagram showing a configuration of an image processing model training apparatus according to an embodiment. As shown in fig. 10, the apparatus includes:

an obtaining module 1002, configured to obtain a training sample image, and a tone label image and a noise label image corresponding to the training sample image respectively;

a sample convolution module 1004, configured to perform convolution processing on the training sample image through the image processing model to obtain a hue sample image and a noise sample image, respectively;

a sample fusion module 1006, configured to perform fusion processing on the tone sample image and the training sample image to obtain a tone mapping predicted image;

the sample denoising module 1008 is configured to perform denoising processing on the tone mapping predicted image according to the noise sample image to obtain a noise predicted image;

and the adjusting module 1010 is configured to adjust parameters of the image processing model and continue training based on a difference between the tone mapping prediction image and the corresponding tone label image and a difference between the noise prediction image and the corresponding noise label image, until a training stop condition is met, and obtain a trained image processing model.

In this embodiment, the training sample image, and the tone label image and the noise label image corresponding to the training sample image are obtained, and the training sample image is subjected to convolution processing by the image processing model, so that the tone sample image including tone information and the noise sample image including noise information are obtained, and the amount of calculation can be reduced. The method comprises the steps of carrying out fusion processing on a tone sample image and a training sample image to obtain a tone mapping predicted image, carrying out noise reduction processing on the tone mapping predicted image according to the noise sample image to obtain a noise predicted image, adjusting parameters of an image processing model and continuing training on the basis of the difference between the tone mapping predicted image and a corresponding tone label image and the difference between the noise predicted image and a corresponding noise label image until a training stopping condition is met, and obtaining a trained image processing model.

In one embodiment, the sample convolution module 1004 is further configured to perform convolution processing on the training sample image based on the initial tone mapping parameter of the image processing model to obtain a corresponding tone sample image; performing convolution processing on the training sample image based on the initial noise extraction parameters of the image processing model to obtain a corresponding noise sample image;

the adjusting module 1010 is further configured to adjust an initial tone mapping parameter of the image processing model based on a difference between the tone mapping prediction image and the corresponding tone label image, and adjust an initial noise extraction parameter of the image processing model based on a difference between the noise prediction image and the corresponding noise label image and continue training.

In an embodiment, the obtaining module 1002 is further configured to obtain each frame of training sample image, and perform registration processing on each frame of training sample image to obtain each registration image; carrying out tone mapping processing on each registration image, and selecting a tone label image from the image subjected to the tone mapping processing; and carrying out multi-frame noise reduction processing on the image subjected to tone mapping processing to obtain a noise label image.

The division of the modules in the training apparatus for the image processing model is only for illustration, and in other embodiments, the training apparatus for the image processing model may be divided into different modules as needed to complete all or part of the functions of the training apparatus for the image processing model.

For specific limitations of the image processing apparatus and the training apparatus of the image processing model, reference may be made to the above limitations of the image processing method and the training method of the image processing model, which are not described herein again. The modules in the image processing apparatus and the training apparatus for image processing models may be implemented in whole or in part by software, hardware, or a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

Fig. 11 is a schematic diagram of an internal structure of an electronic device in one embodiment. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, and a wearable device. The electronic device includes a processor and a memory connected by a system bus. The processor may include one or more processing units, among others. The processor may be a CPU (Central Processing Unit), a DSP (Digital Signal processor), or the like. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor for implementing an image processing method and an image processing model training method provided in the following embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium.

The image processing apparatus provided in the embodiments of the present application, and the implementation of each module in the training apparatus of the image processing model may be in the form of a computer program. The computer program may be run on a terminal or a server. Program modules constituted by such computer programs may be stored on the memory of the electronic device. Which when executed by a processor, performs the steps of the method described in the embodiments of the present application.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the image processing method, and the training method for the image processing model.

Embodiments of the present application also provide a computer program product containing instructions which, when run on a computer, cause the computer to perform a method of image processing, and a method of training an image processing model.

Any reference to memory, storage, database, or other medium used herein may include non-volatile and/or volatile memory. The nonvolatile Memory may include a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a flash Memory. Volatile Memory can include RAM (Random Access Memory), which acts as external cache Memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDR SDRAM (Double Data Rate Synchronous Random Access Memory), ESDRAM (Enhanced Synchronous Dynamic Random Access Memory), SLDRAM (Synchronous Link Dynamic Random Access Memory), RDRAM (Random Dynamic Random Access Memory), and DRmb DRAM (Dynamic Random Access Memory).

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. An image processing method, comprising:

2. The method according to claim 1, wherein the performing convolution processing on the image to be processed to obtain a tone template image and a noise template image respectively comprises:

normalizing the image to be processed, and performing convolution processing on the image after the normalization processing to respectively obtain a tone template image and a noise template image;

the fusing the tone template image and the image to be processed to obtain a tone mapping image, comprising:

carrying out fusion processing on the tone template image and the normalized image to obtain a tone mapping image;

the denoising processing is performed on the tone mapping image based on the noise template image to obtain a first target image, and the denoising processing comprises:

carrying out noise reduction processing on the tone mapping image based on the noise template image to obtain a noise reduction image;

and carrying out image restoration processing on the noise-reduced image to obtain a first target image.

3. The method according to claim 2, wherein the convolving the normalized image to obtain a tone template image and a noise template image respectively comprises:

performing convolution processing on the image after the normalization processing based on the tone mapping parameters to obtain a corresponding tone template image;

and performing convolution processing on the normalized image based on the noise extraction parameters to obtain a corresponding noise template image.

4. The method according to claim 1, wherein the fusing the tone template image and the image to be processed to obtain a tone-mapped image comprises:

carrying out pixel dot multiplication on the tone template image and the image to be processed to obtain a tone mapping image;

and carrying out pixel addition processing on the noise template image and the tone mapping image to obtain a first target image.

5. An image processing method, comprising:

and carrying out noise reduction treatment on the image to be processed based on the noise template image, and fusing the tone template image and the image subjected to the noise reduction treatment to obtain a second target image.

6. A method for training an image processing model, comprising:

7. The method of claim 6, wherein the convolving the training sample image with the image processing model to obtain a hue sample image and a noise sample image respectively comprises:

performing convolution processing on the training sample image based on the initial tone mapping parameter of the image processing model to obtain a corresponding tone sample image;

performing convolution processing on the training sample image based on the initial noise extraction parameters of the image processing model to obtain a corresponding noise sample image;

adjusting parameters of the image processing model and continuing training based on a difference between the tone-mapped predicted image and a corresponding tone-labeled image and a difference between the noise predicted image and a corresponding noise-labeled image, comprising:

adjusting initial tone mapping parameters of the image processing model based on a difference between the tone-mapped predicted image and a corresponding tone-labeled image, and adjusting initial noise extraction parameters of the image processing model based on a difference between the noise predicted image and a corresponding noise-labeled image and continuing training.

8. The method according to claim 6 or 7, wherein the obtaining of the training sample image and the tone label image and the noise label image respectively corresponding to the training sample image comprises:

acquiring training sample images of each frame, and performing registration processing on the training sample images of each frame to obtain registration images;

performing tone mapping processing on each of the registered images, and selecting a tone label image from the images subjected to the tone mapping processing;

and carrying out multi-frame noise reduction processing on the image subjected to tone mapping processing to obtain a noise label image.

9. An image processing apparatus characterized by comprising:

10. An image processing apparatus characterized by comprising:

11. An apparatus for training an image processing model, comprising:

12. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the computer program, when executed by the processor, causes the processor to perform the steps of the method according to any of claims 1 to 8.

13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.

14. A computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the steps of the method according to any of claims 1 to 8.