WO2021109867A1 - Image processing method and apparatus, computer readable storage medium and electronic device - Google Patents

Abstract

Provided are an image processing method, an image processing apparatus, a computer readable storage medium, and an electronic device, relating to the technical field of image processing. The image processing method includes: obtaining an image to be processed, and using the image to be processed to perform an iterative process, until the similarity between a first intermediate image and a second intermediate image is greater than the similarity threshold (S42), both the first intermediate image and the second intermediate image are images generated during the denoising process of the image to be processed; after finishing the iterative process, outputting the first intermediate image or the second intermediate image as the processed image corresponding to the image to be processed (S44); wherein, the iterative process includes: determining the second intermediate image using the image to be processed and the first intermediate image based on the objective function; using the noise estimation model and the second intermediate image to determine a third intermediate image; using the third intermediate image as the first intermediate image. The noise in the image can be reduced.

Description

Image processing method and device, computer readable medium and electronic equipment

Cross-references to related applications

This application claims the priority of a Chinese patent application filed on December 4, 2019, with the application number 201911228475.5 and titled "Image processing method and device, computer readable medium and electronic equipment", and the entire content of the Chinese patent application is approved All references are incorporated into this article.

Technical field

The present disclosure relates to the field of image processing technology, and in particular, to an image processing method, an image processing device, a computer-readable medium, and an electronic device.

Background technique

With the development of mobile terminals, more and more attention has been paid to image functions. As a result, the optical sensor, lens and overall structural design of the camera module have all been rapidly developed. From CCD (Charge Coupled Device) to CMOS (Complementary Metal Oxide Semiconductor), from tens of thousands of pixels to hundreds of millions of pixels, from plastic lenses to sapphire lenses, from ordinary lenses to periscope models Group, all reflect the pursuit of image functions from manufacturers to users.

Integrating high-pixel sensors on mobile terminals has become a trend in the development of mobile terminals. In order to obtain better image resolution, in the process of continuous iteration of mobile terminals, the total number of pixels of the sensor has doubled. However, the increase in the actual photosensitive size of the sensor is limited. This has caused the problem of increasing pixel density and weakening of the signal received by each pixel and the more serious electronic crosstalk. As a result, the output image has more noise and low signal-to-noise ratio, which severely limits high-pixel sensors. Application scenarios.

Summary of the invention

According to a first aspect of the present disclosure, there is provided an image processing method, including: acquiring an image to be processed, and performing an iterative process using the image to be processed until the similarity between the first intermediate image and the second intermediate image is greater than the similarity Up to the threshold, the first intermediate image and the second intermediate image are both images generated in the denoising process of the image to be processed; after the iterative process is ended, the first intermediate image or the second intermediate image is output as the image corresponding to the image to be processed The processed image; wherein, the iterative process includes: based on the objective function, the second intermediate image is determined using the image to be processed and the first intermediate image; the third intermediate image is determined using the noise estimation model and the second intermediate image; the third intermediate image is determined by the noise estimation model and the second intermediate image. The image serves as the first intermediate image.

According to a second aspect of the present disclosure, there is provided an image processing device, including: an image denoising module for acquiring an image to be processed, and using the image to be processed to perform an iterative process until the difference between the first intermediate image and the second intermediate image Until the similarity between the two is greater than the similarity threshold, the first intermediate image and the second intermediate image are both images generated in the denoising process of the image to be processed; the image output module is used to output the first intermediate image after the iterative process is completed Or the second intermediate image, as the processed image corresponding to the image to be processed; wherein, the iterative process includes: based on the objective function, the second intermediate image is determined by using the image to be processed and the first intermediate image; and the noise estimation model and the first intermediate image are used to determine the second intermediate image. The second intermediate image determines the third intermediate image; the third intermediate image is used as the first intermediate image.

According to a third aspect of the present disclosure, there is provided a computer-readable medium on which a computer program is stored, and the computer program is executed by a processor to implement the above-mentioned image processing method.

According to a fourth aspect of the present disclosure, there is provided an electronic device, including: one or more processors; a storage device, for storing one or more programs, when one or more programs are executed by one or more processors , Enabling one or more processors to implement the above-mentioned image processing method.

Description of the drawings

FIG. 1 shows a schematic diagram of an exemplary system architecture of an image processing method or image processing apparatus to which an embodiment of the present disclosure can be applied;

FIG. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of a process of determining an optimal solution after introducing auxiliary variables according to an exemplary embodiment of the present disclosure;

Fig. 4 schematically shows a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;

FIG. 5 schematically shows a flowchart of an iterative process according to an exemplary embodiment of the present disclosure;

Fig. 6 shows a schematic structural diagram of a noise estimation model according to an exemplary embodiment of the present disclosure;

FIG. 7 shows a schematic diagram of visualized iterative processing according to an exemplary embodiment of the present disclosure;

FIG. 8 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure;

FIG. 9 schematically shows a block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure;

FIG. 10 schematically shows a block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure;

FIG. 11 schematically shows a block diagram of an image processing apparatus according to still another exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms, and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed To those skilled in the art. The described features, structures or characteristics can be combined in one or more embodiments in any suitable way. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, the well-known technical solutions are not shown or described in detail in order to avoid overwhelming the crowd and obscure all aspects of the present disclosure.

In addition, the drawings are only schematic illustrations of the present disclosure, and are not necessarily drawn to scale. The same reference numerals in the figures denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

The flowchart shown in the drawings is only an exemplary description, and does not necessarily include all the steps. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may be changed according to actual conditions. In addition, all the terms "first", "second", "third", etc. below are only for the purpose of distinction and should not be taken as a limitation of the present disclosure.

FIG. 1 shows a schematic diagram of an exemplary system architecture of an image processing method or image processing apparatus to which an embodiment of the present disclosure can be applied.

As shown in FIG. 1, the system architecture 1000 may include one or more of terminal devices 1001, 1002, 1003, a network 1004 and a server 1005. The network 1004 is used to provide a medium for communication links between the terminal devices 1001, 1002, 1003 and the server 1005. The network 1004 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.

It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks, and servers according to implementation needs. For example, the server 1005 may be a server cluster composed of multiple servers.

The user can use the terminal devices 1001, 1002, 1003 to interact with the server 1005 through the network 1004 to receive or send messages and so on. The terminal devices 1001, 1002, 1003 may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and so on.

For example, the terminal device 1001, 1002, 1003 may obtain the image to be processed. Specifically, the image captured by the terminal device 1001, 1002, 1003 through its camera module may be used as the image to be processed. Next, the terminal device 1001, 1002, 1003 may perform the following iterative process until the similarity between the first intermediate image and the second intermediate image associated with the image to be processed is less than the similarity threshold, and the iterative process ends After that, the first intermediate image or the second intermediate image is used as the processed image.

The iterative process may include: the first step is to substitute the image to be processed and the first intermediate image into a pre-configured objective function to determine the second intermediate image; the second step is to use the noise estimation model and the second intermediate image The third intermediate image is determined, and the third intermediate image is used as the first intermediate image to update the first intermediate image. Therefore, the first and second steps above are repeated continuously to realize the iterative process.

For the noise estimation model, a machine learning model such as a convolutional neural network can be used. The training process of the noise estimation model can be performed by the server 1005. The server 1005 transmits the trained model parameters to the terminal devices 1001, 1002, 1003 through the network 1004. , Thus better solve the problem of insufficient processing capacity of the terminal equipment 1001, 1002, 1003.

However, it should be understood that the main steps of the image processing method involved in the present disclosure may also be executed by the server 1005. Specifically, the terminal devices 1001, 1002, and 1003 send the image taken by the camera module to the server 1005 via the network 1004, and the image is the image to be processed. The server 1005 uses the image to be processed to perform the above iterative process until the first intermediate image The similarity with the second intermediate image is greater than the similarity threshold. After the iterative process is over, the first intermediate image or the second intermediate image is used as the processed image, and the determined processed image is sent to the terminal devices 1001, 1002, 1003 through the network 1004, so that the user can view the denoised Image.

It should be noted that the image processing method of the exemplary embodiment of the present disclosure is generally executed by the terminal device 1001, 1002, 1003, and specifically, is usually executed by a mobile terminal such as a mobile phone. Correspondingly, the image processing apparatus described below is generally configured in the terminal equipment 1001, 1002, 1003.

Fig. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an exemplary embodiment of the present disclosure. This electronic device corresponds to a terminal device that executes the image processing method of the exemplary embodiment of the present disclosure.

It should be noted that the computer system 200 of the electronic device shown in FIG. 2 is only an example, and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 2, the computer system 200 includes a central processing unit (CPU) 201, which can be based on a program stored in a read-only memory (ROM) 202 or a program loaded from a storage portion 208 into a random access memory (RAM) 203 And perform various appropriate actions and processing. In RAM 203, various programs and data required for system operation are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input/output (I/O) interface 205 is also connected to the bus 204.

The following components are connected to the I/O interface 205: the input part 206 including keyboard, mouse, touch screen, etc.; including the output part 207 such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers; including hard disk, etc. The storage section 208; and the communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the Internet. The drive 210 is also connected to the I/O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that the computer program read from it is installed into the storage section 208 as needed.

In the case of implementing the solution of the present disclosure with a terminal device such as a mobile phone, the system structure may also include a camera module. Specifically, it may include dual-camera, triple-camera, quad-camera, etc., to enrich shooting modes to meet the needs of different shooting scenes.

In particular, according to an embodiment of the present disclosure, the process described below with reference to a flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication section 209, and/or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, various functions defined in the system of the present application are executed.

It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable removable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more for realizing the specified logic function. Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown one after another can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by It is realized by a combination of dedicated hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented in software or hardware, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

As another aspect, this application also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist alone without being assembled into the electronic device. in. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device realizes the method described in the following embodiments.

In some technologies, statistical models based on preset prior constraints are used to suppress image noise. This method can flexibly solve various noise-related inverse problems, but its solution requires a large number of iterative processes, which takes a long time, and the denoising effect is heavily dependent on the preset initial value, and it is very easy to fall into a local optimum or affect the algorithm convergence.

In other technologies, machine learning models are used to estimate noise. Although this method can be used to fit a more complex noise model to achieve a better processing effect, and the processing time is short. However, the processing effect of this method is heavily dependent on the sample size and conditions in the model training process.

In the exemplary embodiment of the present disclosure, the image denoising problem can be considered as the main branch of the image restoration field. Based on the concept of the image restoration field, the image denoising can be represented by a degradation model, which can be represented by Formula 1. :

y=Hx+n (Formula 1)

Among them, y represents the image before denoising, x represents the image after denoising, H represents the identity matrix, and n represents additive white Gaussian noise with a standard deviation of σ.

Solving x when only y is known is an ill-conditioned problem. The solution to this ill-conditioned problem can be transformed from the perspective of the Bayesian school of statistics into a method for obtaining the maximum posterior probability (MAP, Maximum A Posterior) Process, expressed as formula 2:

among them,

Represents the estimate of x, log p(y|x) represents the logarithm of the likelihood function, and log p(x) represents the logarithm of the prior probability. Further, the above problem can be transformed into formula 3:

Among them, λ represents the regularization parameter, which is used to measure the importance of the former constraint and the latter constraint. If λ is larger, it means that the latter item in the entire constraint is more important. If λ is smaller (for example, much less than two One part), it means that the previous constraint is more important. Φ(x) is a general representation of the prior distribution of the signal. It represents a pre-judgmental constraint on the signal distribution. For example, it can be a constraint on the gradient, a constraint on the space, or Constraints in the frequency domain are not limited in this disclosure.

In the present disclosure, the function corresponding to Formula 3 may be referred to as an intermediate function. In addition,

It is also called the fidelity term, and Φ(x) can be called the regularization term.

For formula 3, some time-consuming iterative optimization algorithms can be used to approximate the optimal solution. For the model learning method, the solution process is to obtain a set of prior parameters Θ, this set of prior parameters Θ are related parameters of the loss function to be optimized, by using a large-capacity training set with a one-to-one correspondence between a noise image and a noiseless image To determine the best parameters that meet the corresponding relationship between the two

Use this loss function to estimate the noise-free image corresponding to the noise image. Therefore, for the model learning method, formula 3 can be rewritten as formula 4:

Among them, l represents the loss function. Therefore, the above-mentioned MAP problem is transformed into solving a nonlinear equation.

Starting from the solution of the statistical model, the HQS (Half Quadratic Splitting) method can be used to solve this problem. Specifically, it can be seen from the above parameters that it is difficult to find the global optimal solution directly through the direction of x, and the amount of calculation is large. Therefore, the present disclosure introduces an auxiliary variable (that is, a direction different from the direction of x), and approaches the optimal solution from two directions by continuously iterating the auxiliary variable and x. It should be understood that these two directions are similar to each other.

In this case, introduce the auxiliary variable z, assume that z is an alternative solution to x, and add constraints to make the two as close as possible. Therefore, the above formula 3 can be transformed into formula 5:

The restrictions and conditions here can be appropriately relaxed and transformed into a regularization constraint of 2 norm, as shown in formula 6:

Among them, u represents a regularization parameter, which is used to represent the importance of the constraint item, and is a constraint to ensure that x and z are similar.

Based on Equation 6, the estimates of x can be solved separately

And the estimate of z

Gradually approach the optimal solution x, which can be expressed as Formula 7:

This processing strategy can be understood as a process of exploring "downhill". As shown in Figure 3, although it is not known from which direction from the initial point to find the optimal solution, it is known that there are two directions from which to approach the optimal solution (minimum objective function).

For (i) of formula 7, it can be solved by the way of finding the extreme value of the quadratic term. As for (ii) of formula 7, it returns to the solution of a standard statistical model, and the solution of this equation depends on the prior situation. The previous method of solving this problem believes that a certain transform domain dimension of z (frequency domain, difference domain, etc.) has certain sparse characteristics. However, the noise is not sparse, so (ii) in Equation 7 can be transformed into a formula 8:

Among them, R represents some possible transform operators (Fourier transform operators, difference operators, etc.), and p represents a form of norm constraint, for example, p=1 or 0.

However, this calculation method depends heavily on the choice of transform domain and initial position and the tightness of regularization constraints. Too tight constraints will make the solution more time-consuming and difficult, and a poor initial position can easily lead to a local optimum, and the algorithm stability is poor.

In view of this, in terms of image denoising, exemplary embodiments of the present disclosure provide a new image processing method.

FIG. 4 schematically shows a flowchart of an image processing method of an exemplary embodiment of the present disclosure. Referring to FIG. 4, the image processing method may include the following steps:

S42. Obtain the image to be processed, and use the image to be processed to perform an iterative process until the similarity between the first intermediate image and the second intermediate image is greater than the similarity threshold, and the first intermediate image and the second intermediate image are both in the to-be-processed image. Process the image generated in the denoising process of the image.

In the exemplary embodiment of the present disclosure, the image to be processed may be an image taken by a camera module of a terminal device, or may be an image obtained from another terminal device or the network. In addition, the image to be processed may also be any image to be denoised in the video. The present disclosure does not limit the source, size, shooting scene, etc. of the image to be processed.

After acquiring the image to be processed, the terminal device can use the image to be processed to perform an iterative process. The iterative process involved in the present disclosure will be described below with reference to steps S52 to S56 in FIG. 5.

In step S52, based on the objective function, the second intermediate image is determined using the image to be processed and the first intermediate image.

It should be noted that steps S52 to S56 only describe one iteration process. In the process of the first iteration, the process of initializing the first intermediate image is included. Specifically, the image to be processed may be filtered to obtain the initialized first intermediate image, which is used as the first intermediate image to perform the iterative process for the first time. For example, one of a high-pass filter, a low-pass filter, or a combination thereof may be used. To achieve the above filtering process.

After the first intermediate image is initialized, the second intermediate image can be determined using the image to be processed and the first intermediate image. In the exemplary embodiment of the present disclosure, the second intermediate image may be determined based on an objective function.

The objective function for the exemplary embodiment of the present disclosure corresponds to (i) in Equation 7 above. That is to say, according to the exemplary embodiment of the present disclosure, firstly, an intermediate function (see formula 3) can be constructed based on the degradation model of image restoration (see formula 1); next, the fidelity of the intermediate function can be determined by the auxiliary variable z. The term is decoupled from the regularization term to determine (i) in Equation 7.

It should be understood that in the objective function, the auxiliary variable corresponds to the first intermediate image, that is, the auxiliary variable z can reflect all the information of the first intermediate image. In addition, as an estimate after denoising of the image to be processed,

It can be used as the second intermediate image in the exemplary embodiment of the present disclosure.

Regarding the process of solving (i) in Equation 7, two quadratic terms can be used to solve the extreme value for processing. Through the method of derivation, we can get:

x _k+1 = (H ^T H+uI) ^-1 (H ^T y+uz _k ) (Equation 9)

Among them, I is the identity matrix.

In view of the denoising situation, H is the identity matrix, therefore, formula 9 can also be rewritten as:

x _k+1 = (H+uI) ^-1 (y+uz _k ) (Equation 10)

Therefore, when y represents the image to be processed, H and I are unit matrices, and u represents the regularization parameter, the second intermediate image x _{k+1 can be determined if the first intermediate image z k is} known.

In step S54, a third intermediate image is determined using the noise estimation model and the second intermediate image.

In an exemplary embodiment of the present disclosure, the noise estimation model may be a model based on a convolutional neural network. Figure 6 schematically shows the network structure of the model. The model can be a 7-layer convolutional neural network, including a first layer 61, a second layer 62, a third layer 63, a fourth layer 64, and a fifth layer. 65. The sixth layer 66 and the seventh layer 67.

The network structure can be constructed based on dilated convolution, for example, the first layer 61 is composed of dilated convolution units and modified linear units (ReLU), the second layer 62, the third layer 63, the fourth layer 64, and the fifth layer 65. The sixth layer 66 is composed of an expanded convolution unit, a batch normalization unit (BN), and a modified linear unit (ReLU), and the seventh layer 67 is composed of an expanded convolution unit. The size of the sensor of the expanded convolution unit in the first layer 61 is 3×3, that is, the size of the convolution kernel is 3×3. For the entire network, when the expansion coefficient is 1, 2, 3, 4, 3, 2, 1, the corresponding sensor size of each layer is (2s+1)*(2s+1), where s It is the coefficient of expansion. From this, the size of the susceptor of each layer can be determined to be 3×3, 5×5, 7×7, 9×9, 7×7, 5×5, 3×3, respectively. In addition, the dimension of each layer can be set to 64, that is, the number of feature maps (feature maps) of each layer is set to 64.

Using a convolutional neural network based on dilated convolution as the noise estimation model in the present disclosure can obtain semantic information more effectively, thereby ensuring the accuracy of the denoising result.

However, it should be understood that, in addition to the noise estimation model exemplarily shown above, other convolutional neural networks or other network configurations may also be used to implement the noise estimation model. This disclosure does not limit this.

After the network structure of the noise estimation model is determined, the model training process can be performed on the server in advance.

First, the server can obtain the training set. The training set may include multiple noise images and denoising images corresponding to each noise image, and the difference in noise intensity between each noise image is within a difference threshold, where the difference threshold can be set by the developer according to pre-conducted experiments. Certainly, this disclosure does not limit its specific value.

By controlling the difference in noise intensity within a certain range, the noise level of each noise image in the training set is consistent, which is convenient to improve the training effect.

Next, the images in the training set can be used to train the noise estimation model to obtain the trained model.

Specifically, for each noise image and the corresponding denoised image in the training set, the noise image is input into the convolutional neural network. In this case, the output of the convolutional neural network is the image corresponding to the noise image. Output images for training. Next, when the loss function of the convolutional neural network is determined, the training output image corresponding to the noise image and the corresponding denoising image can be used to calculate the loss function. The above process is performed by continuously inputting samples to minimize the loss function to complete the training process of the convolutional neural network.

After the server trains the noise estimation model, the server can send the parameter information of the model to the terminal device so that the terminal device can use the noise estimation model to perform an iterative process.

The use of server for model training solves the problem of insufficient processing capacity of terminal equipment.

However, it should be noted that if the processing capacity of the terminal device is sufficient for model training, the training process of the model can also be performed in the above-mentioned terminal device, which is not limited in the present disclosure.

After determining the trained noise estimation model, the terminal device may input the second intermediate image determined in step S52 into the trained noise estimation model to determine the noise estimation value corresponding to the second intermediate image. Next, the third intermediate image can be determined based on the second intermediate image and its noise estimate.

Specifically, formula 11 may be used to determine the third intermediate image:

z _k+1 = x _k+1- f(x _k+1 ; Θ) (Equation 11)

Among them, f(x _k+1 ; Θ) represents the noise estimation value for the second intermediate image, and Θ here represents the model parameter.

In step S56, the third intermediate image is used as the first intermediate image to realize the update of the first intermediate image.

Therefore, steps S52 to S56 are repeatedly executed in this way, and during the execution process, the similarity between the first intermediate image and the second intermediate image is continuously determined until the similarity between the first intermediate image and the second intermediate image is determined The iterative process ends until the degree is greater than the similarity threshold. Wherein, the similarity threshold can be set by the developer according to the result of the experiment, which is not limited in the present disclosure. In the case where the similarity between the first intermediate image and the second intermediate image is greater than the similarity threshold, it can be considered that an optimal solution has been found, and the optimal solution is the denoised image.

It should be noted that, during the iterative process from step S52 to step S56 performed by the terminal device, each time the iterative process is executed, the model parameters are updated, and the updated parameters are used to execute the next iterative process. That is to say, during the iterative process, the parameters of the noise estimation model will change to ensure that the iterative process of formula 7 (1) and formula 11 is used to continuously approach the optimal solution.

In addition, the foregoing determines whether the iterative process is over by the similarity between the first intermediate image and the second intermediate image. It is easy to understand that when the difference between the first intermediate image and the second intermediate image is small, the iterative process ends. . In this case, the index of image difference can also be used to determine whether the iterative process is over. For example, when the image difference between the first intermediate image and the second intermediate image is less than a preset threshold, the iteration can be determined The process is over.

S44. After finishing the iterative process, output the first intermediate image or the second intermediate image as the processed image corresponding to the image to be processed.

After the iterative process involved in step S42 ends, according to some embodiments of the present disclosure, since the difference between the first intermediate image and the second intermediate image is small, the terminal device may output the first intermediate image or the second intermediate image As the processed image corresponding to the image to be processed.

According to other embodiments of the present disclosure, after the first intermediate image or the second intermediate image is updated, a similarity determination process is performed. For example, after the first intermediate image is updated, if the similarity between the first intermediate image and the second intermediate image is less than the similarity threshold, the first intermediate image is output as the processed image. For another example, after the second intermediate image is updated, if the similarity between the first intermediate image and the second intermediate image is less than the similarity threshold, the second intermediate image is output as the processed image.

The processed image output can be directly saved to the terminal, and can also be displayed for the user to view.

With reference to Fig. 7, the above process of implementing image denoising can be understood as: “going down” from the starting point, walking on one foot (solving the noise-free image directly) is difficult and the local optimum is prone to occur. In this case, another foot is introduced (the auxiliary variable z, which is the first intermediate image above), and the whole process becomes a two-step solution. However, for the problem of how to determine the auxiliary variable, the exemplary embodiment of the present disclosure may be solved by using a convolutional neural network. It should also be noted that the whole process is _{constrained by ||xz k} || ² , which guarantees that x and z maintain a large degree of similarity, that is, in the above example, both feet are guaranteed to move forward.

In Figure 7, the angle α between x and z, these two variables is actually a very small angle to ensure a high degree of similarity between the two. Therefore, the entire two-dimensional information expansion is a long and narrow curved surface, which makes it more advantageous to solve the global optimization.

Based on the image processing method of the exemplary embodiment of the present disclosure, on the one hand, the present disclosure combines a noise estimation model to complete the iterative process. Compared with some technologies that only use regularization constraints to continuously optimize the iterative process, the complexity is greatly reduced. While better denoising effects can be obtained, the time-consuming is short; on the other hand, the solution of the present disclosure can effectively remove image noise, so that the high-pixel camera module can be used in low-light environments, greatly expanding the high-pixel camera model The application scenario of the group; on the other hand, the disclosed solution does not require auxiliary tools or hardware changes, and is easy to implement.

It should be noted that although the various steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that these steps must be performed in the specific order, or that all the steps shown must be performed to achieve the desired the result of. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution, etc.

Further, an image processing device is also provided in this exemplary embodiment.

FIG. 8 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. Referring to FIG. 8, the image processing device 8 according to an exemplary embodiment of the present disclosure may include an image denoising module 81 and an image output module 83.

Specifically, the image denoising module 81 may be used to obtain the image to be processed, and use the image to be processed to perform an iterative process until the similarity between the first intermediate image and the second intermediate image is greater than the similarity threshold, the first intermediate image Both the second intermediate image and the second intermediate image are images generated during the denoising process of the image to be processed; the iterative process includes: based on the objective function, the second intermediate image is determined by using the image to be processed and the first intermediate image; and the noise estimation model is used And the second intermediate image determine the third intermediate image; the third intermediate image is used as the first intermediate image.

The image output module 83 may be used to output the first intermediate image or the second intermediate image as the processed image corresponding to the image to be processed after the iterative process is ended.

Using the image processing device of the exemplary embodiment of the present disclosure, on the one hand, the present disclosure combines a noise estimation model to complete the iterative processing process. Compared with the process of continuous optimization and iteration that only uses regularization constraints in some technologies, the complexity is greatly reduced. While it is possible to obtain a better denoising effect, it takes a short time; on the other hand, the solution of the present disclosure can effectively remove image noise, so that the high-pixel camera module can be used in a low-light environment, greatly expanding the high-pixel camera model The application scenario of the group; on the other hand, the disclosed solution does not require auxiliary tools or hardware changes, and is easy to implement.

According to an exemplary embodiment of the present disclosure, the process of determining the third intermediate image by the image denoising module 81 using the noise estimation model and the second intermediate image may be configured to execute: input the second intermediate image into the noise estimation model, and determine the difference between the second intermediate image and the second intermediate image. The noise estimation value corresponding to the intermediate image; the third intermediate image is determined according to the second intermediate image and the noise estimation value.

According to an exemplary embodiment of the present disclosure, referring to FIG. 9, compared to the image processing device 8, the image processing device 9 may further include a model training module 91.

Specifically, the model training module 91 may be configured to execute: obtain a training set; wherein the training set includes multiple noise images and denoised images corresponding to each noise image, and the noise intensity difference between each noise image is a difference Within the threshold; input the noise image in the training set into a convolutional neural network, and the convolutional neural network outputs the training output image corresponding to the noise image; use the training output image and denoising image corresponding to the noise image to calculate the loss of the convolutional neural network Function to train the convolutional neural network; determine the trained convolutional neural network as the noise estimation model.

According to an exemplary embodiment of the present disclosure, the image denoising module 81 may also be configured to execute: each time the iterative process is executed, the parameters of the convolutional neural network are updated, and the next iterative process is executed using the updated parameters.

According to an exemplary embodiment of the present disclosure, a convolutional neural network includes a cascaded plurality of convolutional layers, and each convolutional layer includes an expanded convolution unit.

According to an exemplary embodiment of the present disclosure, referring to FIG. 10, compared to the image processing device 8, the image processing device 10 may further include an initialization module 101.

Specifically, the initialization module 101 may be configured to perform: filter processing on the image to be processed to obtain the initialized first intermediate image, which is used as the first intermediate image for the first execution of the iterative process.

According to an exemplary embodiment of the present disclosure, referring to FIG. 11, compared to the image processing device 8, the image processing device 11 may further include an objective function determining module 111.

Specifically, the objective function determination module 111 may be configured to execute: construct an intermediate function based on the degradation model of image restoration, the intermediate function includes a fidelity term and a regularization term; use an auxiliary variable to combine the fidelity term and regularization of the intermediate function The terms are decoupled, and the objective function is determined according to the decoupling result; among them, the auxiliary variable corresponds to the first intermediate image.

Since each functional module of the image processing device in the embodiment of the present disclosure is the same as in the above method embodiment, it will not be repeated here.

Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described here can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

In addition, the above-mentioned drawings are merely schematic illustrations of the processing included in the method according to the exemplary embodiments of the present disclosure, and are not intended for limitation. It is easy to understand that the processing shown in the above drawings does not indicate or limit the time sequence of these processings. In addition, it is easy to understand that these processes can be executed synchronously or asynchronously in multiple modules, for example.

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Those skilled in the art will easily think of other embodiments of the present disclosure after considering the description and practicing the content disclosed herein. This application is intended to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field that are not disclosed in the present disclosure. . The description and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

It should be understood that the present disclosure is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims (20)

1. An image processing method, including:

   Obtain the image to be processed, and use the image to be processed to perform an iterative process until the similarity between the first intermediate image and the second intermediate image is greater than the similarity threshold, the first intermediate image and the second intermediate image Are all images generated during the denoising process of the image to be processed;

   After finishing the iterative process, output the first intermediate image or the second intermediate image as a processed image corresponding to the image to be processed;

   Wherein, the iterative process includes:

   Determine the second intermediate image by using the image to be processed and the first intermediate image based on the objective function;

   Determining a third intermediate image by using the noise estimation model and the second intermediate image;

   Use the third intermediate image as the first intermediate image.
2. The image processing method according to claim 1, wherein determining the third intermediate image using the noise estimation model and the second intermediate image comprises:

   Inputting the second intermediate image into the noise estimation model, and determining a noise estimation value corresponding to the second intermediate image;

   A third intermediate image is determined according to the second intermediate image and the noise estimation value.
3. The image processing method according to claim 1 or 2, wherein the image processing method further comprises:

   Obtain a training set; wherein the training set includes a plurality of noise images and denoising images corresponding to each of the noise images, and the difference in noise intensity between each of the noise images is within a difference threshold;

   Input the noise image in the training set to a convolutional neural network, and the convolutional neural network outputs a training output image corresponding to the noise image;

   Using the training output image and the denoising image corresponding to the noise image to calculate the loss function of the convolutional neural network to train the convolutional neural network;

   The trained convolutional neural network is determined as the noise estimation model.
4. The image processing method according to claim 3, wherein the image processing method further comprises:

   Each time the iterative process is executed, the parameters of the convolutional neural network are updated, and the updated parameters are used to execute the next iterative process.
5. 3. The image processing method according to claim 3, wherein the convolutional neural network includes a plurality of convolutional layers cascaded, and each of the convolutional layers includes an expanded convolution unit.
6. The image processing method according to claim 1, wherein the image processing method further comprises:

   Filtering the image to be processed is performed to obtain the initialized first intermediate image, which is used as the first intermediate image for performing the iterative process for the first time.
7. The image processing method according to claim 1, wherein the image processing method further comprises:

   An intermediate function is constructed based on the degradation model of image restoration, and the intermediate function includes a fidelity term and a regularization term;

   Use an auxiliary variable to decouple the fidelity term and regularization term of the intermediate function, and determine the objective function according to the decoupling result;

   Wherein, the auxiliary variable corresponds to the first intermediate image.
8. The image processing method according to claim 1, wherein the image to be processed is an image taken by a camera module of a terminal device.
9. The image processing method according to claim 1, wherein the image processing method further comprises:

   The processed image is saved to the terminal device and displayed.
10. An image processing device, including:

    The image denoising module is configured to obtain the image to be processed, and use the image to be processed to perform an iterative process until the similarity between the first intermediate image and the second intermediate image is greater than the similarity threshold, the first intermediate image Both the image and the second intermediate image are images generated during the denoising process of the image to be processed;

    An image output module configured to output the first intermediate image or the second intermediate image as a processed image corresponding to the image to be processed after the iterative process is ended;

    Wherein, the iterative process includes: determining the second intermediate image using the image to be processed and the first intermediate image based on an objective function; determining a third intermediate image using a noise estimation model and the second intermediate image ; Use the third intermediate image as the first intermediate image.
11. The image processing device according to claim 10, wherein the process of determining the third intermediate image by the image denoising module using the noise estimation model and the second intermediate image is configured to perform: inputting the second intermediate image The noise estimation model determines a noise estimation value corresponding to the second intermediate image; and determines a third intermediate image according to the second intermediate image and the noise estimation value.
12. The image processing device according to claim 10 or 11, wherein the image processing device further comprises:

    The model training module is configured to obtain a training set; wherein the training set includes a plurality of noise images and a denoising image corresponding to each of the noise images, and the noise intensity difference between each of the noise images is a difference Within the threshold; input the noise image in the training set into a convolutional neural network, and the convolutional neural network outputs the training output image corresponding to the noise image; use the training output image and the denoising image corresponding to the noise image Calculate the loss function of the convolutional neural network to train the convolutional neural network; determine the trained convolutional neural network as the noise estimation model.
13. The image processing device according to claim 12, wherein the image denoising module is further configured to execute: each time the iterative process is executed, the parameters of the convolutional neural network are updated, and the updated parameters are used to execute the next Describe the iterative process.
14. The image processing device according to claim 12, wherein the convolutional neural network includes a plurality of convolutional layers cascaded, and each of the convolutional layers includes an expanded convolution unit.
15. The image processing device according to claim 10, wherein the image processing device further comprises:

    The initialization module is configured to perform filtering processing on the image to be processed to obtain an initialized first intermediate image as the first intermediate image for performing the iterative process for the first time.
16. The image processing device according to claim 10, wherein the image processing device further comprises:

    The objective function determination module is configured to construct an intermediate function based on the degradation model of image restoration, the intermediate function including a fidelity term and a regularization term; and an auxiliary variable is used to resolve the fidelity and regularization terms of the intermediate function And determine the objective function according to the decoupling result; wherein, the auxiliary variable corresponds to the first intermediate image.
17. 10. The image processing device according to claim 10, wherein the image to be processed is an image taken by a camera module of a terminal device.
18. 11. The image processing device according to claim 10, wherein the processed image is included and displayed by a terminal device.
19. A computer readable medium having a computer program stored thereon, and when the program is executed by a processor, the image processing method according to any one of claims 1 to 9 is realized.
20. An electronic device including:

    One or more processors;

    The storage device is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the implementation as in claims 1 to 9 Any one of the image processing methods.

Images