CN117788946A

CN117788946A - Image processing method, device, electronic equipment and storage medium

Info

Publication number: CN117788946A
Application number: CN202410014654.3A
Authority: CN
Inventors: 邱镇; 周逸平; 陶俊; 刘璟; 郭庆; 黄晓光; 聂文萍; 崔迎宝; 王晓东; 张琳瑜; 陈振宇; 刘识; 李博; 李明; 董小菱
Original assignee: Big Data Center Of State Grid Corp Of China; State Grid Information and Telecommunication Co Ltd; Information and Telecommunication Branch of State Grid Anhui Electric Power Co Ltd
Current assignee: Big Data Center Of State Grid Corp Of China; State Grid Information and Telecommunication Co Ltd; Information and Telecommunication Branch of State Grid Anhui Electric Power Co Ltd
Priority date: 2024-01-03
Filing date: 2024-01-03
Publication date: 2024-03-29

Abstract

The application provides an image processing method, an image processing device, electronic equipment and a storage medium, which comprise the following steps: acquiring an image to be processed, and determining the effect requirement of image processing; inputting the image to be processed into a training model which is adjusted by utilizing a convolutional neural network to perform illumination enhancement processing on the image to be processed according to the model, and generating an enhanced image corresponding to the effect requirement; and outputting the enhanced image.

Description

Image processing method, device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing device, an electronic device, and a storage medium.

Background

The operation, maintenance and overhaul of the power industry are very important to ensure the normal operation of the power system. The image recognition method based on deep learning has been widely applied to test points in the field of power operation and maintenance, and a certain application effect is obtained. However, in a low-illumination environment, the acquired power operation and detection images are limited by the environment, acquisition equipment and other factors, the quality of the acquired power operation and detection images is often poor, the problems of detail degradation, color distortion, serious noise and the like exist, the subsequent image recognition and target detection processes are seriously influenced, the power failure cannot be timely and accurately recognized, and the safe and efficient operation of a power system is influenced.

Disclosure of Invention

In view of the foregoing, the present application proposes an image processing method, an image processing apparatus, an electronic device, and a storage medium, so as to solve or partially solve the above-mentioned problems.

Based on the above object, the present application provides an image processing method, including:

acquiring an image to be processed, and determining the effect requirement of image processing;

inputting the image to be processed into a training-completed model for adjusting a Monte Carlo tree search algorithm by using a convolutional neural network, so as to carry out illumination enhancement processing on the image to be processed according to the model, and generating an enhancement image corresponding to the effect requirement;

and outputting the enhanced image.

In some exemplary embodiments, the training process of the model for adjusting the Monte Carlo tree search algorithm by using the convolutional neural network comprises the following steps:

acquiring a training set composed of image pairs composed of at least one group of first images with the illumination intensity higher than a first threshold value and second images with the illumination intensity lower than a second threshold value, which correspond to the same images;

selecting any image pair in the training set, generating a decision tree for the selected second image by utilizing a Monte Carlo tree search algorithm based on a preset adjustment action, and obtaining a triplet for training a deep convolutional neural network through repeated iterative simulation; wherein the triplet includes a second image, a corresponding selection policy, and a benefit value;

Inputting the triples into a deep convolutional neural network, outputting intermediate parameters for generating the decision tree by using the Monte Carlo tree search algorithm, and repeating the generation of the decision tree through the intermediate parameters, so as to circulate until a preset stopping condition is met, thereby completing the training of the model; wherein the intermediate parameters include the adjusted selection policy and the benefit value.

In some exemplary embodiments, the generating a decision tree based on the preset adjustment action using a monte carlo tree search algorithm includes:

constructing an initial decision tree according to the Monte Carlo tree search algorithm based on a preset adjustment action;

traversing the initial decision tree by using an upper confidence limit algorithm selection rule, and determining any leaf node which is not searched;

determining all child nodes of the leaf node, and adding the all child nodes to the leaf node;

determining an optimal adjustment action according to the probability distribution of each preset adjustment action, simulating any node in all the child nodes by using a convolutional neural network based on the optimal adjustment action, and circulating until a preset stop condition is reached, and calculating a current node profit value;

And updating the current node benefit value to the root node by calculating the average benefit of any node.

In some exemplary embodiments, the upper confidence limit algorithm selects rules, specifically:

wherein UCB (x) is an upper confidence-bound algorithm selection rule for an image x as a child node, x represents the image of the child node, x _p An image of a parent node representing a child node,for the average benefit in historical iterations that has propagated through x, pi (x _p A) is the father node x _p Generating probability distribution estimation of a regulation action of a child node x, wherein a epsilon A is a preset regulation action set, N () is the access times of the corresponding node in the history selection step, and c is a preset control exploration degree parameter;

the current node profit value is specifically:

r(x)＝exp(-α‖x-y‖ ² )

wherein r (x) is the benefit value of the image of the current child node when the preset stop condition is reached, y is the first image corresponding to the selected second image, and alpha is a preset weight parameter;

the statistical probability of the probability distribution is specifically:

wherein ρ (x, a) is the statistical probability of the child node image x performing the a adjustment, and a (x) is the parameter corresponding to the child node image x performing the a adjustment.

In some exemplary embodiments, before the inputting the triplets into the deep convolutional neural network, the method further comprises:

Replacing the last layer of the complete connection classification layer of the deep convolutional neural network with two parallel feedforward head layers; wherein, a feedforward head layer comprises a layer of completely connected linear layer and normalized exponential function; the other feed-forward layer comprises a fully connected linear layer and an S-shaped growth curve function.

In some exemplary embodiments, the deep convolutional neural network is at least one of a ResNet-18 network, a ResNet34 network, a ResNet50 network, a ResNet101 network, or a DenseNet121 network.

In some exemplary embodiments, the preset adjustment actions include at least: brightness adjustment, contrast adjustment, gamma correction, saturation adjustment, hue rotation, gray adjustment, maximum three primary color adjustment, median filter adjustment, gaussian blur filter adjustment, sharpening filter adjustment, detail filter adjustment, edge enhancement filter adjustment, and/or smoothing filter adjustment.

In some exemplary embodiments, the preset stop condition includes: the loop reaches a preset number of times and/or satisfies a preset loss function.

In some exemplary embodiments, the effect requirements include targeting effects;

the generating the enhanced image corresponding to the effect requirement comprises the following steps:

And responding to the effect requirement, taking the effect as a guide, establishing a decision tree of the Monte Carlo tree search algorithm once in the model according to the image to be processed, and outputting the enhanced image according to the adjusted algorithm.

In some exemplary embodiments, the effect requirements include efficiency oriented;

and responding to the effect requirement, wherein the effect requirement is guided by efficiency, and sequentially adjusting the image to be processed according to the adjustment action with the highest profit value of each layer of the decision tree of the Monte Carlo tree search algorithm of the trained model, so as to generate the enhanced image.

Based on the same conception, the present application also provides an image processing apparatus including:

the acquisition module is used for acquiring the image to be processed and determining the effect requirement of image processing;

the processing module is used for inputting the image to be processed into a training-completed model for adjusting a Monte Carlo tree search algorithm by using a convolutional neural network so as to carry out illumination enhancement processing on the image to be processed according to the model and generate an enhancement image corresponding to the effect requirement;

And the output module is used for outputting the enhanced image.

In some exemplary embodiments, the training process of the model for adjusting the monte carlo tree searching algorithm by using the convolutional neural network in the processing module includes:

wherein UCB (x) is an upper confidence-bound algorithm selection rule for an image x as a child node, x represents the image of the child node, x _p An image of a parent node representing a child node, For the average benefit in historical iterations that has propagated through x, pi (x _p A) is the father node x _p Generating probability distribution estimation of a regulation action of a child node x, wherein a epsilon A is a preset regulation action set, N () is the access times of the corresponding node in the history selection step, and c is a preset control exploration degree parameter;

the current node profit value is specifically:

r(x)＝exp(-α‖x-y‖ ² )

the statistical probability of the probability distribution is specifically:

the processing module is further configured to:

The processing module is further configured to:

Based on the same conception, the application also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method according to any one of the above when executing the program.

Based on the same conception, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to implement the method as described in any one of the above.

As can be seen from the foregoing, the image processing method, apparatus, electronic device and storage medium provided in the present application include: acquiring an image to be processed, and determining the effect requirement of image processing; inputting the image to be processed into a training model which is adjusted by utilizing a convolutional neural network to perform illumination enhancement processing on the image to be processed according to the model, and generating an enhanced image corresponding to the effect requirement; and outputting the enhanced image. The method constructs a low-illumination electric image enhancement method based on an improved MCTS algorithm by utilizing a convolutional neural network to search a Monte Carlo tree, selects an enhancement processing step with highest enhancement processing income each time by utilizing a model algorithm to process an image, and determines user requirements by acquiring effect requirements in consideration of the condition that the quality requirements and the efficiency requirements cannot be compatible at present, so that an enhancement image with higher processing quality or higher processing speed is correspondingly provided, the current brightness enhancement processing effect of the low-brightness image is integrally improved, and the accuracy and the processing efficiency are improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for those skilled in the art.

FIG. 1 is a flow diagram of an exemplary method provided by embodiments of the present application;

fig. 2 is a schematic structural diagram of an adjusted deep convolutional neural network according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an exemplary device provided in an embodiment of the present application;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the present specification will be further described in detail below with reference to the accompanying drawings.

It should be noted that unless otherwise defined, technical or scientific terms used in the embodiments of the present application should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present application belongs. The terms "first," "second," and the like, as used in embodiments of the present application, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements, articles, or method steps preceding the word are included in the listed elements, articles, or method steps following the word, and equivalents thereof, without precluding other elements, articles, or method steps. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.

As described in the background section, the low-light image enhancement aims to recover images under normal light from images collected in dark light, night and other environments, so as to meet task requirements of downstream image recognition, target detection, semantic segmentation and the like. With the rapid development of artificial intelligence techniques represented by deep learning, the field of low-light image enhancement has made great progress. At present, a common low-illumination image enhancement method mainly comprises the following steps: histogram equalization-based methods, retinex theory-based methods, generation of an countermeasure network-based methods, diffusion model-based methods, and so forth.

In the related art, the histogram equalization-based method specifically includes: and performing low-illumination image enhancement by using algorithms such as contrast limited self-adaptive histogram equalization, self-adaptive histogram correction and the like. The basic principle is that the contrast and definition of the image are enhanced by calculating the image histogram and carrying out equalization processing on the histogram to change the gray distribution of the image. The method based on Retinex theory specifically comprises the following steps: and performing low-illumination image enhancement by using algorithms such as a multi-scale Retinex model, a weighted variation Retinex model and the like. The basic principle is to decompose the image into an illumination component and a reflection component, and to generate an enhanced low-light image by enhancing the illumination component and fusing with the reflection component. The method based on generating the countermeasure network comprises the following steps: and (5) performing low-illumination image enhancement by using a Cycle-GAN, enlighten-GAN algorithm and the like. The basic principle is that the generator learns how to generate an image very similar to the training image, and the discriminator learns how to distinguish between the real training image and the generated image, and through the countermeasure learning between the two, the discriminator is optimal when the discriminator cannot distinguish between the image in training and the image generated by the generator. The diffusion model-based method specifically comprises the following steps: and carrying out low-illumination image enhancement by using algorithms such as a conditional diffusion model, a denoising diffusion probability model and the like. The basic principle is to add gaussian noise in the data distribution through a forward process so that the original data gradually loses its characteristics, and then restore the original data through a reverse process, thereby optimizing the generator and the discriminator to generate a target image.

However, when using the foregoing low-light image enhancement method, the applicant found that the related art mainly has the following drawbacks: (1) The histogram equalization-based method may cause loss of context details, poor color recovery and noise interference, and is difficult to solve the problem of color shift; (2) The method based on Retinex theory may cause problems such as color distortion, halation flickering, noise amplification, etc.; (3) The image enhancement result is unstable based on the method for generating the countermeasure network, the noise artifact problem exists, and the method has weak interpretability; (4) Diffusion model based methods require significant computational resources.

In combination with the above practical situation, the embodiment of the application provides an image processing scheme. The method constructs a low-illumination electric image enhancement method based on an improved MCTS algorithm by utilizing a convolutional neural network to search a Monte Carlo tree, selects an enhancement processing step with highest enhancement processing income each time by utilizing a model algorithm to process an image, and determines user requirements by acquiring effect requirements in consideration of the condition that the quality requirements and the efficiency requirements cannot be compatible at present, so that an enhancement image with higher processing quality or higher processing speed is correspondingly provided, the current brightness enhancement processing effect of the low-brightness image is integrally improved, and the accuracy and the processing efficiency are improved.

Referring to fig. 1, a flowchart of an image processing method according to an exemplary embodiment of the present application is shown.

The image processing method specifically comprises the following steps:

step 102, obtaining an image to be processed, and determining the effect requirement of image processing.

In this step, the image to be processed is a low-brightness image that needs brightness enhancement, such as an image obtained from a darker scene. The effect requirement can be actively set by the user according to the requirement of the user, wherein the user can set the image processing with higher quality according to the generated quality requirement and the calculation resource condition of the current system, namely, the effect is used as a guide; there is still a need for faster image processing, i.e. efficiency oriented. Of course, according to specific application scenarios, different effect requirements can also be set to meet different scenario requirements.

And 104, inputting the image to be processed into a training model which is adjusted by a convolutional neural network to perform a Monte Carlo tree search algorithm, so as to perform illumination enhancement processing on the image to be processed according to the model and generate an enhancement image corresponding to the effect requirement.

In the step, the image to be processed can be transmitted to a model which is trained and used for adjusting a Monte Carlo tree search algorithm by using a convolutional neural network after the image to be processed is acquired, and illumination enhancement processing can be carried out on the image to be processed through the model. The model establishes a decision tree for carrying out one-step enhancement adjustment on the image through a Monte Carlo tree search algorithm, calculates the overall benefit of the whole selected adjustment action sequence while adjusting the benefit value of each enhancement action in the training process, and finally selects a group of adjustment action sequences with highest benefit through continuous iteration. The processing gain for the image to be processed by means of this sequence of adjustment actions is generally high.

In some embodiments, the specific training process for the model that uses convolutional neural networks to adjust the Monte Carlo tree search algorithm may be: acquiring a training set composed of image pairs composed of at least one group of first images with the illumination intensity higher than a first threshold value and second images with the illumination intensity lower than a second threshold value, which correspond to the same images; selecting any image pair in the training set, generating a decision tree for the selected second image by utilizing a Monte Carlo tree search algorithm based on a preset adjustment action, and obtaining a triplet for training a deep convolutional neural network through repeated iterative simulation; wherein the triplet includes a second image, a corresponding selection policy, and a benefit value; inputting the triples into a deep convolutional neural network, outputting intermediate parameters for generating the decision tree by using the Monte Carlo tree search algorithm, and repeating the generation of the decision tree through the intermediate parameters, so as to circulate until a preset stopping condition is met, thereby completing the training of the model; wherein the intermediate parameters include the adjusted selection policy and the benefit value.

The training set formed by the image pairs formed by the first image with the illumination intensity higher than the first threshold value and the second image with the illumination intensity lower than the second threshold value corresponding to the same image can be simply understood as the image pair formed by the first image with the illumination intensity lower than the second image when the illumination of the same image is normal, and the plurality of groups of image pairs form a training set. The first threshold and the second threshold can be specifically set according to specific scenes. The preset adjustment action is a specific action for adjusting the image, and may be any one or more of brightness adjustment, contrast adjustment, gamma correction, saturation adjustment, tone rotation, gray adjustment, maximum three primary color adjustment, median filter adjustment, gaussian blur filter adjustment, sharpening filter adjustment, detailed filter adjustment, edge enhancement filter adjustment and/or smoothing filter adjustment. The Monte Carlo tree search algorithm (Monte Carlo Tree Search, MCTS) is then a heuristic search algorithm based on a tree data structure that can be large in search space yet relatively efficient.

And then, in order to perform optimization iteration on the Monte Carlo tree search algorithm, the deep convolutional neural network can be utilized to perform convolution iteration on the triplets to obtain adjusted intermediate parameters corresponding to the adjustment action, the intermediate parameters can be utilized to adjust and update the corresponding parameters in the Monte Carlo tree search algorithm, and the iteration is circulated until a preset termination condition is reached, so that training of the model can be completed. Among these, deep convolutional neural networks (Deep Convolution Neural Network, DCNN) are mathematical or computational models that mimic the structure and function of biological neural networks (the central nervous system of animals, particularly the brain). DCNN plays an important role in the field of visual target detection with its excellent learning ability and feature extraction ability. In some embodiments, the deep convolutional neural network is at least one of a ResNet-18 network, a ResNet34 network, a ResNet50 network, a ResNet101 network or a DenseNet121 network, although some other classical convolutional neural network architecture is also possible, such as VGG, inception, etc. Then, in order to enable the deep convolutional neural network to be applied to the current model, the last layer of the deep convolutional neural network can be modified to adapt to a specific image enhancement scene. As shown in fig. 2, the specific way of improvement can be as follows: replacing the last layer of the complete connection classification layer (FC) of the deep convolutional neural network with two parallel feedforward head layers; wherein a feed-forward header layer comprises a fully connected linear layer (FC), and a normalized exponential function (softmax function); the other feed-forward layer comprises a fully connected linear layer (FC), and an S-shaped growth curve function (sigmoid function). That is, in some embodiments, before the inputting the triplets into the deep convolutional neural network, the method further comprises: replacing the last layer of the complete connection classification layer of the deep convolutional neural network with two parallel feedforward head layers; wherein, a feedforward head layer comprises a layer of completely connected linear layer and normalized exponential function; the other feed-forward layer comprises a fully connected linear layer and an S-shaped growth curve function. L is the loss function of the adjusted deep convolutional neural network.

Then, for the stopping condition of training, in different application scenarios, different settings may be performed, for example, a condition that the specific number of loops is completed, or the second image is enhanced to reach a similarity with the first image to meet a certain threshold, or a preset loss function is met may be set. That is, in some embodiments, the preset stop condition includes: the loop reaches a preset number of times and/or satisfies a preset loss function. In some specific scenarios, the stop condition of the decision tree generation stage may be that a defined number of iterations (number of simulations) is reached, such as 10000 times; or the convolutional neural network training stage is to reach the limit cycle times (such as 160 times); or the loss function L reaches a certain threshold (e.g., 5 consecutive times are not falling).

In some specific embodiments, the specific process of generating the decision tree by utilizing the monte carlo tree searching algorithm based on the preset adjustment action can be that the initial decision tree is built according to the monte carlo tree searching algorithm based on the preset adjustment action; traversing the initial decision tree by using an upper confidence limit algorithm selection rule, and determining any leaf node which is not searched; determining all child nodes of the leaf node, and adding the all child nodes to the leaf node; determining an optimal adjustment action according to the probability distribution of each preset adjustment action, simulating any node in all the child nodes by using a convolutional neural network based on the optimal adjustment action, and circulating until a preset stop condition is reached, and calculating a current node profit value; and updating the current node benefit value to the root node by calculating the average benefit of any node. The upper confidence boundary algorithm selection rule is UCB (Upper Confidence Bound) selection rule. It may be specifically Wherein UCB (x) is an upper confidence-bound algorithm selection rule for an image x as a child node, x represents the image of the child node, x _p An image of a parent node representing a child node, +.>For the average benefit in historical iterations that has propagated through x, pi (x _p A) is the father node x _p Generating probability distribution estimation of a regulation action of the child node x, wherein a epsilon A, A is a preset regulation action set, N () is the access times of the corresponding node in the history selection step, and c is a preset control exploration degree parameter. The current node benefit value may be r (x) =exp (- α|x-y||) ² ). Wherein r (x) is the benefit value of the image of the current child node when the preset stop condition is reached, y is the first image corresponding to the selected second image, and alpha is the preset weight parameter. The statistical probability of the probability distribution may be +.>Wherein ρ (x, a) is the statistical probability of the child node image x performing the a adjustment, and a (x) is the parameter corresponding to the child node image x performing the a adjustment.

In a specific application scenario, the training process of the model for adjusting the monte carlo tree searching algorithm by using the convolutional neural network may specifically include: (1) A low-light power operation inspection image construction dataset is obtained, which contains a low-light image (i.e., a second image) and a corresponding normal-light image (i.e., a first image), with the dataset as a training set. (2) Decision trees are generated using a modified MCTS algorithm (monte carlo tree search algorithm) based on the randomly selected training set image pairs. (3) Convolutional neural networks for MCTS (monte carlo tree search algorithm) node selection are optimized based on the images obtained in the generation stage. (4) And (3) alternately and circularly carrying out the step (2) and the step (3) based on the training set data to obtain the trained convolutional neural network.

Wherein for step (2), specifically for each random image pair, the modified MCTS algorithm first builds an initial decision tree. The decision tree is composed of nodes and actions, wherein the nodes represent images of each layer, the actions represent preset adjustment actions (in this embodiment, 5 kinds of adjustment are taken as examples, namely, brightness adjustment, contrast adjustment, gamma correction, saturation adjustment and tone rotation, and specifically, the adjustment method of each adjustment action is shown in table 1, where i and j are the abscissa of coordinates, c is any color channel of a pixel), and each node corresponds to a benefit value. The goal of decision tree construction is to find the action sequence corresponding to the highest benefit value.

TABLE 1.5 specific adjustment methods for adjustment actions

Thereafter, for a specific construction process of the decision tree, the following steps may be included: 1) Selecting: the UCB selection rules are used as tree policies to choose a leaf node (i.e., child node) throughout the decision tree. 2) Expansion: a new child node is found for the selected leaf node. 3) Simulation: and starting from the selected node or a newly added child node, performing action track simulation, and returning the benefits of the node. 4) Backtracking: the benefit value is passed back from the selected node to the root of the tree and the action value is updated or initialized.

Wherein the leaf node (i.e., child node) selection process is performed by a convolutional neural network. This is also an improvement of the MCTS algorithm in the foregoing improvement of the MCTS algorithm, that is, adding a convolutional neural network to the MCTS algorithm selects child nodes. As shown in fig. 2, the convolutional neural network takes as input a node image x and as output a policy pi (x,) and a value v (x). Where the policy pi (x,) is an estimate of the probability distribution of all possible actions and the value v (x) is an estimate of the expected benefit r (·) of the offspring of x.

The UCB selection rules used in this step focus on both exploring nodes with fewer visits and those that take high revenue on average, as follows:

then, when the maximum depth is reached, the search fails and the profit is zero. When a stop operation is selected (i.e., a preset stop condition is reached), the benefit depends on the distance between the resulting image x and the target image y (i.e., the first image):

r(x)＝exp(-α‖x-y‖ ² )

after a series of MCTS iterations, the adjustment action probability ρ of the root node may be obtained:

the probability is a statistical probability representation of the child node access probability distribution of the root node. A random sample is extracted from this probability distribution and the corresponding action is applied to the root image. Starting from the new image (becoming the new root node of the decision tree), the above operations are re-performed and run repeatedly until the end node is reached.

Of course, in some embodiments, in order to enable a certain benefit value when the maximum depth is reached, so that the data at the maximum depth can be used, a small amount of Dirichlet noise (Dirichlet noise) may be added to the UCB selection rules. When an end node is found (maximum depth is reached or a special stop operation is selected), the noise is used to replace the convolutional neural network for estimation.

And (3) for the step (3), forming a training set of the convolutional neural network by the root node accessed in the step (2). Each training sample contains an image x, a benefit r (x) and a random strategy probability ρ (x,) for a triplet. Taking a deep convolutional neural network of ResNet-18 network architecture as an example, a modified version of the ResNet-18 network architecture is shown in FIG. 3. Except for the last layer, a normal structure layer, which may specifically include a fully connected linear layer (FC), a linear rectification function ReLU function (an activation function), a random deactivation function (dropout). And the last layer is replaced by two feedforward head layers, both of which contain two fully connected linear layers (FC), the first feedback head containing the softmax function generating value v (x). The second feedback header contains the sigmoid function generation policy pi (x,). The value v (x) and the policy pi (x,) correspond to intermediate parameters, and the parameters of the Monte Carlo tree search algorithm can be updated by using the two intermediate parameters. Then, the loss function L used by the deep convolutional neural network comprises square estimation errors of gain values, cross entropy between a random strategy rho and a network strategy pi, and the method is concretely as follows:

Where λ is an adjustable parameter that can be used to balance these two losses.

After model training is completed according to the above embodiments, the trained model may be applied to the image processing process. When image processing is performed, the effect requirements corresponding to different scenes are different, so that a certain difference may exist in the effect of the generated enhanced image.

In application, after the image to be processed is acquired, the image to be processed is further adjusted according to different effect requirements.

If the effect is the effect requirement with the effect as the guide, the obtained parameters can be returned to the Monte Carlo tree search algorithm after one-time calculation is carried out according to the trained model, so that the Monte Carlo tree search algorithm is utilized to select the targeted adjustment action of each layer according to the image to be processed again, namely, decision tree establishment is carried out again, the establishment process is similar to the training process, an iteration loop is not needed, and the image output after completion can be used as an enhanced image for output. That is, in some embodiments, the effect requirements include targeting effects; the generating the enhanced image corresponding to the effect requirement comprises the following steps: and responding to the effect requirement, taking the effect as a guide, establishing a decision tree of the Monte Carlo tree search algorithm once in the model according to the image to be processed, and outputting the enhanced image according to the adjusted algorithm. This approach provides better image enhancement but requires more computing resources and computing time. In an embodiment, the value v (x) may be used as a return value in the previous embodiment, similar to the process of step (2), but without access to the first image when the final node is reached. After the decision tree is reconstructed, traversing the decision tree downwards by maximizing the strategy probability rho (x,) at each node to obtain an action sequence, and directly carrying out image enhancement on the obtained action sequence.

If the efficiency is used as a guide, the image to be processed can be adjusted according to the adjustment action with the highest profit value of each layer of the decision tree of the Monte Carlo tree search algorithm of the trained model, and an enhanced image is generated. For the trained model, the adjustment actions determined by each layer of the decision tree are generally the adjustment actions with the highest profit value, so that the images to be processed can be processed in sequence directly according to the determined adjustment action sequences in implementation, and the enhanced images can be obtained. That is, in some embodiments, the effect requirements include efficiency oriented; the generating the enhanced image corresponding to the effect requirement comprises the following steps: and responding to the effect requirement, wherein the effect requirement is guided by efficiency, and sequentially adjusting the image to be processed according to the adjustment action with the highest profit value of each layer of the decision tree of the Monte Carlo tree search algorithm of the trained model, so as to generate the enhanced image. This approach has relatively poor image enhancement, but saves computational resources and computation time.

It can be seen that, in the embodiment of the application, the problems of color shift, noise artifact, halation flicker, loss of detail information, unstable enhancement result and the like faced in the low-illumination image enhancement process are considered, and the low-illumination electric image enhancement method based on the improved MCTS algorithm is constructed by means of the MCTS algorithm and the enhanced deep learning algorithm; considering that the existing method cannot simultaneously give consideration to the generation quality requirement and the calculation resource condition, two inference strategies with effect as a guide and efficiency as a guide are provided, and selection can be performed according to actual conditions; considering the problem of poor interpretability of the existing method, the image editing operation sequence is used for low-light image enhancement.

And step 106, outputting the enhanced image.

In this step, the enhanced image is generated, and the enhanced image is output and displayed, so that the operator can see the enhanced image. In other embodiments, the output mode of the enhanced image is not limited to output display, and the enhanced image can be stored, displayed, used or reprocessed. The specific output mode of the enhanced image can be flexibly selected according to different application scenes and implementation requirements.

For example, for an application scenario in which the method of the present embodiment is executed on a single device, the enhanced image may be directly output in a display manner on a display section (display, projector, etc.) of the current device, so that an operator of the current device can directly see the content of the enhanced image from the display section.

For another example, for an application scenario of the method of the embodiment executed on a system formed by a plurality of devices, the enhanced image may be sent to other preset devices as a receiving party in the system, that is, the synchronization terminal, through any data communication manner (such as wired connection, NFC, bluetooth, wifi, cellular mobile network, etc.), so that the synchronization terminal may perform subsequent processing on the enhanced image. Optionally, the synchronization terminal may be a preset server, where the server is generally disposed in the cloud, and is used as a data processing and storage center, and capable of storing and distributing the enhanced image; the receiving party of the distribution is terminal equipment, and the holders or operators of the terminal equipment can be superordinate supervisory personnel, auditors and the like of the image acquisition terminal.

For another example, for an application scenario executed by the method of the present embodiment on a system formed by a plurality of devices, the enhanced image may be directly sent to a preset terminal device through any data communication manner, where the terminal device may be one or more of the foregoing paragraphs.

As can be seen from the foregoing embodiments, the image processing method provided in the embodiments of the present application includes: acquiring an image to be processed, and determining the effect requirement of image processing; inputting the image to be processed into a training model which is adjusted by utilizing a convolutional neural network to perform illumination enhancement processing on the image to be processed according to the model, and generating an enhanced image corresponding to the effect requirement; and outputting the enhanced image. The method constructs a low-illumination electric image enhancement method based on an improved MCTS algorithm by utilizing a convolutional neural network to search a Monte Carlo tree, selects an enhancement processing step with highest enhancement processing income each time by utilizing a model algorithm to process an image, and determines user requirements by acquiring effect requirements in consideration of the condition that the quality requirements and the efficiency requirements cannot be compatible at present, so that an enhancement image with higher processing quality or higher processing speed is correspondingly provided, the current brightness enhancement processing effect of the low-brightness image is integrally improved, and the accuracy and the processing efficiency are improved.

It should be noted that, the method of the embodiments of the present application may be performed by a single device, for example, a computer or a server. The method of the embodiment of the application can also be applied to a distributed scene, and is completed by mutually matching a plurality of devices. In the case of such a distributed scenario, one of the devices may perform only one or more steps of the methods of embodiments of the present application, and the devices may interact with each other to complete the methods.

It should be noted that the foregoing describes specific embodiments of the present application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments described above and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Based on the same conception, the application also provides an image processing device corresponding to the method of any embodiment.

Referring to fig. 3, the image processing apparatus includes:

the acquiring module 210 is configured to acquire an image to be processed, and determine an effect requirement of image processing.

The processing module 220 is configured to input the image to be processed into a training-completed model that adjusts a monte carlo tree search algorithm by using a convolutional neural network, so as to perform illumination enhancement processing on the image to be processed according to the model, and generate an enhanced image corresponding to the effect requirement.

And an output module 230 for outputting the enhanced image.

the current node profit value is specifically:

r(x)＝exp(-α‖x-y‖ ² )

the statistical probability of the probability distribution is specifically:

the processing module 220 is further configured to:

For convenience of description, the above devices are described as being functionally divided into various modules, respectively. Of course, the functions of each module may be implemented in one or more pieces of software and/or hardware when implementing the embodiments of the present application.

The device of the foregoing embodiment is configured to implement the corresponding image processing method in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which is not described herein.

Based on the same conception, the application also provides electronic equipment corresponding to the method of any embodiment, comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the image processing method of any embodiment when executing the program.

Fig. 4 shows a more specific hardware architecture of an electronic device according to this embodiment, where the device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 implement communication connections therebetween within the device via a bus 1050.

The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit ), microprocessor, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc. for executing relevant programs to implement the technical solutions provided in the embodiments of the present disclosure.

The Memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory ), static storage device, dynamic storage device, or the like. Memory 1020 may store an operating system and other application programs, and when the embodiments of the present specification are implemented in software or firmware, the associated program code is stored in memory 1020 and executed by processor 1010.

The input/output interface 1030 is used to connect with an input/output module for inputting and outputting information. The input/output module may be configured as a component in a device (not shown in the figure) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.

Communication interface 1040 is used to connect communication modules (not shown) to enable communication interactions of the present device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).

Bus 1050 includes a path for transferring information between components of the device (e.g., processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).

It should be noted that although the above-described device only shows processor 1010, memory 1020, input/output interface 1030, communication interface 1040, and bus 1050, in an implementation, the device may include other components necessary to achieve proper operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.

The electronic device of the foregoing embodiment is configured to implement the corresponding image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which is not described herein.

Based on the same conception, corresponding to any of the above embodiments, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the image processing method according to any of the above embodiments.

The computer readable media of the present embodiments, including both permanent and non-permanent, removable and non-removable media, may be any method or technology for information storage. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.

The storage medium of the above embodiment stores computer instructions for causing the computer to perform the image processing method according to any one of the above embodiments, and has the advantages of the corresponding method embodiments, which are not described herein.

Based on the same conception, the application also provides a computer program product corresponding to the method of any embodiment, and the computer program product comprises the computer program instructions. In some embodiments, the computer program instructions may be executed by one or more processors of a computer to cause the computer and/or the processor to perform the image processing method. Corresponding to the execution subject corresponding to each step in each embodiment of the image processing method, the processor for executing the corresponding step may belong to the corresponding execution subject.

The computer program product of the above embodiment is configured to enable the computer and/or the processor to perform the image processing method according to any one of the above embodiments, and has the advantages of corresponding method embodiments, which are not described herein again.

Those of ordinary skill in the art will appreciate that: the discussion of any of the embodiments above is merely exemplary and is not intended to suggest that the scope of the application (including the claims) is limited to these examples; the technical features of the above embodiments or in the different embodiments may also be combined within the idea of the present application, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.

Additionally, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures, in order to simplify the illustration and discussion, and so as not to obscure the embodiments of the present application. Furthermore, the devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present application, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform on which the embodiments of the present application are to be implemented (i.e., such specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the application, it should be apparent to one skilled in the art that embodiments of the application can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative in nature and not as restrictive.

While the present application has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of those embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

The present embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Accordingly, any omissions, modifications, equivalents, improvements and/or the like which are within the spirit and principles of the embodiments are intended to be included within the scope of the present application.

Claims

1. An image processing method, comprising:

and outputting the enhanced image.

2. The method of claim 1, wherein the training process of the model for adjusting the monte carlo tree search algorithm using the convolutional neural network comprises:

3. The method of claim 2, wherein the generating a decision tree based on the preset adjustment action using a monte carlo tree search algorithm comprises:

4. A method according to claim 3, wherein the upper confidence limit algorithm selects rules, in particular:

wherein UCB (x) is an upper confidence-bound algorithm selection rule for an image x as a child node, x represents the image of the child node, x _p An image of a parent node representing a child node,for the average benefit in historical iterations that has propagated through x, pi (x _p A) is the father node x _p Generating a probability distribution estimate of an a-tuning action of child node xA epsilon A, A is a preset set of adjustment actions, N () is the access times of the corresponding node in the history selection step, and c is a preset control exploration degree parameter;

The current node profit value is specifically:

r(x)＝exp(-α‖x-y‖ ² )

the statistical probability of the probability distribution is specifically:

5. The method of claim 2, wherein prior to said inputting the triplets into a deep convolutional neural network, the method further comprises:

6. The method of claim 2, wherein the deep convolutional neural network is at least one of a res net-18 network, a res net34 network, a res net50 network, a res net101 network, or a densnet 121 network.

7. The method according to claim 2, wherein the preset adjustment actions comprise at least: brightness adjustment, contrast adjustment, gamma correction, saturation adjustment, hue rotation, gray adjustment, maximum three primary color adjustment, median filter adjustment, gaussian blur filter adjustment, sharpening filter adjustment, detail filter adjustment, edge enhancement filter adjustment, and/or smoothing filter adjustment.

8. The method according to claim 2, wherein the preset stop condition comprises: the loop reaches a preset number of times and/or satisfies a preset loss function.

9. The method of claim 1, wherein the effect requirements include effect oriented;

10. The method of claim 1, wherein the effect requirement comprises efficiency oriented;

11. An image processing apparatus, comprising:

and the output module is used for outputting the enhanced image.

12. The apparatus of claim 11, wherein the training process of the model in the processing module that uses a convolutional neural network to adjust the monte carlo tree search algorithm comprises:

13. The apparatus of claim 12, wherein the generating a decision tree based on the preset adjustment action using a monte carlo tree search algorithm comprises:

14. The apparatus of claim 13, wherein the upper confidence algorithm selection rule is specifically:

the current node profit value is specifically:

r(x)＝exp(-α||x-y|| ² )

the statistical probability of the probability distribution is specifically:

15. The apparatus of claim 12, wherein prior to inputting the triplets into a deep convolutional neural network, further comprising:

16. The apparatus of claim 12, wherein the deep convolutional neural network is at least one of a res net-18 network, a res net34 network, a res net50 network, a res net101 network, or a densnet 121 network.

17. The apparatus of claim 12, wherein the preset adjustment action comprises at least: brightness adjustment, contrast adjustment, gamma correction, saturation adjustment, hue rotation, gray adjustment, maximum three primary color adjustment, median filter adjustment, gaussian blur filter adjustment, sharpening filter adjustment, detail filter adjustment, edge enhancement filter adjustment, and/or smoothing filter adjustment.

18. The apparatus of claim 12, wherein the preset stop condition comprises: the loop reaches a preset number of times and/or satisfies a preset loss function.

19. The apparatus of claim 11, wherein the effect requirements include effect oriented;

the processing module is further configured to:

20. The apparatus of claim 11, wherein the effect demand includes efficiency oriented;

the processing module is further configured to:

21. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 10 when the program is executed.

22. A non-transitory computer readable storage medium storing computer instructions for causing a computer to implement the method of any one of claims 1 to 10.