CN108898557B

CN108898557B - Image restoration method and apparatus, electronic device, computer program, and storage medium

Info

Publication number: CN108898557B
Application number: CN201810541974.9A
Authority: CN
Inventors: 庞家昊; 曾进; 孙文秀; 肖瑞超
Original assignee: Sensetime Group Ltd
Current assignee: Sensetime Group Ltd
Priority date: 2018-05-30
Filing date: 2018-05-30
Publication date: 2021-08-06
Anticipated expiration: 2038-05-30
Also published as: CN108898557A

Abstract

The embodiment of the invention discloses an image recovery method and device, electronic equipment, a computer program and a storage medium, wherein the method comprises the following steps: constructing a prior model according to a feature map of a current image to be processed, wherein the prior model is used for learning information of the current image to be processed, and the current image to be processed comprises at least one image degradation phenomenon; and based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a target image. The embodiment of the invention can obtain a good image recovery result.

Description

Image restoration method and apparatus, electronic device, computer program, and storage medium

Technical Field

The present invention relates to computer vision technologies, and in particular, to an image restoration method and apparatus, an electronic device, a computer program, and a storage medium.

Background

Due to the imperfection of an imaging system, a recording device, a processing method and a transmission medium, the image can generate degradation phenomena such as noise, blurring and partial pixel missing during the forming, recording, processing and transmission processes, so that the image quality is reduced.

Image restoration, which is a type of problem that is widely concerned in computer vision, is proposed for an image degradation phenomenon, and aims to reconstruct or restore an image with reduced quality to obtain an original image without degradation.

Disclosure of Invention

The embodiment of the invention provides an image recovery technical scheme.

According to an aspect of an embodiment of the present invention, there is provided an image restoration method including:

constructing a prior model according to a feature map of a current image to be processed, wherein the prior model is used for learning information of the current image to be processed, and the current image to be processed comprises at least one image degradation phenomenon;

and based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image.

Optionally, in the above method embodiment of the present invention, the image degradation phenomenon includes at least one of: image noise, image blur and image pixel dropout;

the image restoration process includes at least one of: image de-noising processing, image de-blurring processing and image pixel completion processing.

Optionally, in any one of the above method embodiments of the present invention, before constructing the prior model according to the feature map of the current image to be processed, the method further includes:

extracting the features of the current image to be processed to obtain a group of feature maps with the same size as the current image to be processed;

the method for constructing the prior model according to the feature map of the current image to be processed comprises the following steps:

and constructing the prior model according to a group of characteristic graphs with the same size as the current image to be processed.

Optionally, in any one of the above method embodiments of the present invention, before constructing the prior model according to a set of feature maps having the same size as the current image to be processed, the method further includes:

performing image segmentation processing on each feature map in a group of feature maps with the same size as the current image to be processed to obtain a group of local feature maps, wherein each feature map corresponds to each preset area in the current image to be processed;

the constructing the prior model according to a set of feature maps with the same size as the current image to be processed comprises:

determining a group of prior models respectively corresponding to each preset region in the current image to be processed according to the local feature maps corresponding to the same preset region in the current image to be processed in each group of local feature maps;

the image restoration processing is performed on the image degradation phenomenon of the current image to be processed based on the prior model, and before obtaining a corresponding target image, the method further includes:

performing image segmentation processing on the current image to be processed to obtain a group of local images to be processed corresponding to each preset area in the current image to be processed;

the image restoration processing is performed on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, and the method comprises the following steps:

performing image restoration processing on each preset region in the current image to be processed respectively based on the corresponding prior model aiming at the image degradation phenomenon of the corresponding local image to be processed to obtain a group of local target images respectively corresponding to each preset region in the current image to be processed;

after the image restoration processing is performed on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, the method further includes:

and carrying out splicing processing on a group of local target images respectively corresponding to each preset area in the current image to be processed to obtain the target image.

Optionally, in any one of the method embodiments of the present invention, determining, according to the local feature maps corresponding to the same preset region in the current image to be processed in the sets of local feature maps, a set of prior models respectively corresponding to the preset regions in the current image to be processed includes:

respectively constructing a group of local undirected graphs corresponding to each preset region in the current image to be processed according to the local feature graphs corresponding to the same preset region in the current image to be processed in each group of local feature graphs;

and determining a group of prior models respectively corresponding to each preset region in the current image to be processed according to a group of local undirected graphs respectively corresponding to each preset region in the current image to be processed.

Optionally, in any one of the method embodiments of the present invention, the respectively constructing a set of local undirected graphs corresponding to the preset regions in the current image to be processed according to the local feature graphs corresponding to the same preset regions in the current image to be processed in each set of local feature graphs includes:

for each preset area in the current image to be processed, determining the weight between corresponding pixels in the local undirected graph according to the distance between the pixels in each local feature image corresponding to each preset area;

and constructing a local undirected graph corresponding to each preset region according to the weight between the pixels determined by the local feature graphs corresponding to each preset region.

Optionally, in any one of the method embodiments of the present invention, the determining, according to a distance between pixels in the local feature maps corresponding to each preset region, a weight between corresponding pixels in the local undirected graph includes:

and determining the weight between corresponding pixels in the local undirected graph according to the distance between pixels which are smaller than or equal to a preset distance threshold in each local feature graph corresponding to each preset region.

Optionally, in any one of the method embodiments of the present invention, before performing image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, the method further includes:

performing primary recovery processing on the image degradation phenomenon of the current image to be processed, wherein the image subjected to the primary recovery processing and the current image to be processed have the same size;

and based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the image subjected to the primary recovery processing to obtain the target image.

Optionally, in any of the above method embodiments of the present invention, the prior model includes: the graph laplacian matrix.

Optionally, in any one of the method embodiments of the present invention, the performing, based on the prior model, image restoration processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image includes:

and solving a quadratic programming problem of the current image to be processed by taking the graph Laplacian matrix as a regularization item to obtain the target image.

and determining the graph Laplacian matrix as a coefficient of a regularization item according to the current image to be processed.

Optionally, in any one of the method embodiments of the present invention, after performing image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, the method further includes:

and (3) performing iteration: taking the target image as a current image to be processed, and constructing a prior model according to a feature map of the current image to be processed;

Optionally, in any of the above method embodiments of the present invention, the method is applied to a mobile terminal and/or a driving assistance system.

According to another aspect of an embodiment of the present invention, there is provided an image restoration apparatus including:

the modeling unit is used for constructing a prior model according to a feature map of a current image to be processed, the prior model is used for learning the information of the current image to be processed, and the current image to be processed comprises at least one image degradation phenomenon;

and the processing unit is used for carrying out image recovery processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image.

Optionally, in the above apparatus embodiment of the present invention, the image degradation phenomenon includes at least one of: image noise, image blur and image pixel dropout;

Optionally, in any one of the apparatus embodiments of the present invention, the apparatus further includes:

the extraction unit is used for extracting the features of the current image to be processed to obtain a group of feature maps with the same size as the current image to be processed;

the modeling unit is used for constructing the prior model according to a group of characteristic graphs with the same size as the current image to be processed.

the segmentation unit is used for carrying out image segmentation processing on each feature map in a group of feature maps with the same size as the current image to be processed to obtain a group of local feature maps, wherein each feature map corresponds to each preset area in the current image to be processed; performing image segmentation processing on the current image to be processed to obtain a group of local images to be processed corresponding to each preset area in the current image to be processed;

the modeling unit is used for determining a group of prior models respectively corresponding to each preset area in the current image to be processed according to the local feature maps corresponding to the same preset area in the current image to be processed in each group of local feature maps;

the processing unit is configured to perform image restoration processing on each preset region in the current image to be processed, based on the corresponding prior model, for the image degradation phenomenon of the corresponding local image to be processed, so as to obtain a group of local target images corresponding to each preset region in the current image to be processed;

the device further comprises:

and the splicing unit is used for splicing a group of local target images respectively corresponding to each preset area in the current image to be processed to obtain the target image.

Optionally, in an embodiment of any one of the above apparatuses of the present invention, the modeling unit is configured to respectively construct a set of local undirected graphs corresponding to each preset region in the current image to be processed according to the local feature graphs corresponding to the same preset region in the current image to be processed in each set of local feature graphs; and determining a group of prior models respectively corresponding to each preset region in the current image to be processed according to a group of local undirected graphs respectively corresponding to each preset region in the current image to be processed.

Optionally, in any one of the apparatus embodiments of the present invention, the modeling unit is configured to determine, for each preset region in the current image to be processed, a weight between corresponding pixels in the local undirected graph according to a distance between pixels in each local feature map corresponding to each preset region; and constructing a local undirected graph corresponding to each preset region according to the weight between the pixels determined by the local feature graphs corresponding to each preset region.

Optionally, in an embodiment of the apparatus according to the present invention, the modeling unit is configured to determine a weight between corresponding pixels in the local undirected graph according to a distance between pixels, smaller than or equal to a preset distance threshold, in each of the local feature maps corresponding to each preset region.

the preprocessing unit is used for performing primary recovery processing on the image degradation phenomenon of the current image to be processed, and the image subjected to the primary recovery processing and the current image to be processed have the same size;

and the processing unit is used for carrying out image recovery processing on the image degradation phenomenon of the image subjected to the primary recovery processing based on the prior model to obtain the target image.

Optionally, in any of the above apparatus embodiments of the present invention, the prior model includes: the graph laplacian matrix.

Optionally, in any one of the apparatus embodiments of the present invention, the processing unit is configured to obtain the target image by solving a quadratic programming problem of the current image to be processed with the graph laplacian matrix as a regularization term.

and the determining unit is used for determining the graph Laplacian matrix as a coefficient of a regularization item according to the current image to be processed.

Optionally, in any of the above apparatus embodiments of the present invention, the apparatus further comprises: at least two cascaded recovery modules, the recovery modules comprising: the modeling unit and the processing unit;

the first recovery module is used for constructing a prior model according to the feature map of the current image to be processed; based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image;

the recovery modules except the first recovery module are used for taking the target image as a current image to be processed and constructing a prior model according to a feature map of the current image to be processed; and based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image.

Optionally, in any of the above device embodiments of the present invention, the device is applied to a mobile terminal and/or a driving assistance system.

According to another aspect of the embodiments of the present invention, there is provided an electronic device including the apparatus according to any of the above embodiments.

According to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus including:

a memory for storing executable instructions; and

a processor configured to execute the executable instructions to perform the method according to any of the above embodiments.

According to a further aspect of embodiments of the present invention, there is provided a computer program comprising computer readable code which, when run on a device, executes instructions for implementing the method of any one of the above embodiments.

According to yet another aspect of embodiments of the present invention, there is provided a computer storage medium for storing computer-readable instructions which, when executed, implement the method of any of the above embodiments.

Based on the image restoration method and apparatus, the electronic device, the computer program, and the storage medium provided in the above embodiments of the present invention, a prior model is constructed according to a feature map of a current image to be processed, and based on the prior model, image restoration processing is performed for an image degradation phenomenon of the current image to be processed, so as to obtain a corresponding target image, where the current image to be processed includes at least one image degradation phenomenon, the prior model is used to learn information of the current image to be processed, and the information of the current image to be processed is learned by using the feature map to construct the prior model, so that the prior model can more truly reflect an image restoration requirement of the current image to be processed, and thus, by combining machine learning with the prior model, a good image restoration effect for the current image to be processed can be obtained.

The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.

The invention will be more clearly understood from the following detailed description, taken with reference to the accompanying drawings, in which:

FIG. 1 is a flow chart of an image restoration method of some embodiments of the present invention;

FIG. 2 is a flow chart of an image restoration method according to further embodiments of the present invention;

FIG. 3 is a schematic diagram of a network architecture implementing image restoration methods according to further embodiments of the present invention;

FIG. 4 is a schematic diagram of a network architecture implementing image restoration methods of further embodiments of the present invention;

FIG. 5 is a schematic diagram of an image restoration apparatus according to some embodiments of the present invention;

FIG. 6 is a schematic diagram of an image restoration apparatus according to further embodiments of the present invention;

fig. 7 is a schematic structural diagram of an electronic device according to some embodiments of the present invention.

Detailed Description

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

Embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the computer system/server include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

The computer system/server may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

FIG. 1 is a flow chart of an image restoration method of some embodiments of the present invention. It should be understood that the example shown in fig. 1 is only for helping those skilled in the art to better understand the technical solution of the present invention, and should not be construed as limiting the present invention. Those skilled in the art can make various changes on the basis of fig. 1, and such changes should also be understood to form part of the present invention.

As shown in fig. 1, the method includes:

and 102, constructing a prior model according to the feature map of the current image to be processed.

The image to be processed in the embodiment of the present invention may be an original image acquired from an image acquisition device, or may be an image acquired after processing a certain image in the image processing process. In an embodiment of the present invention, the prior model is used to learn information of an image to be processed, and the image to be processed includes at least one image degradation phenomenon, for example, the image degradation phenomenon includes at least one of the following: the embodiment of the invention does not limit the types of image degradation phenomena in the image to be processed, such as image noise, image blur, image pixel missing and the like.

Optionally, before constructing the prior model according to the feature map of the current image to be processed, feature extraction may be performed on the current image to be processed to obtain a set of feature maps of the current image to be processed, where each feature map in the set of feature maps has the same size as the current image to be processed, and then the prior model is constructed according to the set of feature maps having the same size as the current image to be processed. Optionally, a neural network or other machine learning method may be adopted to perform feature extraction on the current image to be processed, so as to obtain a set of feature maps having the same size as the current image to be processed. In an alternative example, the neural network may employ a convolutional neural network. The embodiment of the present invention does not limit the manner of obtaining the feature map according to the current image to be processed.

The embodiment of the invention does not limit the number of characteristic graphs for constructing the prior model. For example, a current image to be processed is input into a convolutional neural network, a three-dimensional array having the same length and width as the image to be processed and having three channels is obtained, each channel in the three-dimensional array is used as a feature map, three feature maps can be output, and a prior model can be constructed based on the three output feature maps.

Optionally, the embodiment of the present invention does not limit the form of the prior model, for example: the prior model can adopt a full variation model, a sparse expression model and the like. In an alternative example, the prior model may use a Graph laplacian model, i.e., a Graph laplacian model (Graph Laplacians).

And 104, based on the prior model, performing image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image.

The method for carrying out the image recovery processing on the image degradation phenomenon of the image to be processed can be determined according to the type of the prior model, and the method for carrying out the image recovery processing on the image degradation phenomenon of the image to be processed based on the prior model is not limited in the embodiment of the invention. In an optional example, the prior model may adopt an image laplacian model, and the image degradation phenomenon of the current image to be processed may be subjected to image recovery processing by solving a quadratic programming problem of the current image to be processed with an image laplacian matrix as a regularization term, so as to obtain a target image.

Alternatively, the image restoration process may be an image process corresponding to an image degradation phenomenon contained in the current image to be processed, for example, when the image degradation phenomenon includes at least one of: when image noise, image blurring, image pixel missing and the like exist, the corresponding image restoration processing includes at least one of the following: image denoising processing, image deblurring processing, image pixel completion processing and the like.

Optionally, before performing image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain the corresponding target image, preliminary restoration processing may be performed on the image degradation phenomenon of the current image to be processed, and then image restoration processing is performed on the image degradation phenomenon of the image subjected to the preliminary restoration processing based on the prior model to obtain the target image, where the image subjected to the preliminary restoration processing and the current image to be processed have the same size. Optionally, a preprocessing network or other image restoration processing methods may be adopted to perform preliminary restoration processing on the image degradation phenomenon of the current image to be processed. In an alternative example, the preprocessing network may employ a convolutional neural network. The embodiment of the present invention does not limit the method of the preliminary recovery processing.

Based on the image restoration method provided by the above embodiment of the present invention, a prior model is constructed according to the feature map of the current image to be processed, based on the prior model, image restoration processing is performed on the image degradation phenomenon of the current image to be processed, so as to obtain a corresponding target image, wherein the current image to be processed includes at least one image degradation phenomenon, the prior model is used for learning the information of the current image to be processed, and the information of the current image to be processed is learned by using the feature map to construct the prior model, so that the prior model can more truly reflect the image restoration requirement of the current image to be processed, and particularly for the situation that image degradation is complex in an actual application scene, by combining machine learning with the prior model, a good image restoration effect on the current image to be processed can be obtained.

Compared with a method for recovering an image by a neural network, the method needs to completely depend on the quantity and quality of training data, when the quantity and quality of the training data of the neural network are not robust enough, overfitting is easy to occur and a good image recovery result cannot be obtained. Compared with a method for recovering an image by simply adopting a prior model, the method has the advantages that some assumptions need to be made on the properties of the image, the flexibility is poor, and the image recovery effect can be influenced when the assumed conditions cannot be met.

The embodiment of the invention can be applied to image recovery of mobile terminals such as mobile phones and the like, auxiliary driving systems and the like.

Fig. 2 is a flow chart of an image restoration method according to further embodiments of the present invention. It should be understood that the example shown in fig. 2 is only for helping those skilled in the art to better understand the technical solution of the present invention, and should not be construed as limiting the present invention. Those skilled in the art can make various changes on the basis of fig. 2, and such changes should also be understood to form part of the present invention.

As shown in fig. 2, the method includes:

202, extracting the features of the current image to be processed to obtain a group of feature maps with the same size as the current image to be processed.

Optionally, a neural network or other machine learning method may be adopted to perform feature extraction on the current image to be processed, so as to obtain a set of feature maps having the same size as the current image to be processed. In an alternative example, the neural network may employ a convolutional neural network. The embodiment of the present invention does not limit the manner of obtaining the feature map according to the current image to be processed.

And 204, performing image segmentation processing on each feature map in a group of feature maps with the same size as the current image to be processed to obtain a group of local feature maps, wherein each feature map corresponds to each preset area in the current image to be processed.

Optionally, image segmentation processing may be performed on each feature map according to a preset image segmentation method, so that a group of local feature maps obtained by image segmentation of each feature map respectively corresponds to each preset region in the current image to be processed, and the whole current image to be processed may be completely covered, where adjacent local feature maps may have the same region, that is, an overlapping region. In an alternative example, the image segmentation processing may be performed by using a neural network, or may be performed by using another method other than the neural network.

The embodiment of the invention does not limit the number and size of the local feature maps obtained by image segmentation processing, and the larger the size of the local feature maps obtained by image segmentation processing is, the better the recovery effect of the finally obtained image is, but the longer the processing time of each local feature map is, so that the effect and efficiency of image recovery can be comprehensively considered in practical application to determine the size and number of the local feature maps obtained by image segmentation processing. For example, each feature map may be partitioned into a set of local feature maps of 26x26 pixels.

And 206, determining a group of prior models respectively corresponding to the preset regions in the current image to be processed according to the local feature maps corresponding to the same preset regions in the current image to be processed in the local feature maps.

Optionally, a set of local undirected graphs corresponding to the preset regions in the current image to be processed may be respectively constructed according to the local feature graphs corresponding to the same preset regions in the current image to be processed in the sets of local feature graphs, and then a set of prior models corresponding to the preset regions in the current image to be processed may be determined according to the set of local undirected graphs corresponding to the preset regions in the current image to be processed.

Optionally, for each preset region in the current image to be processed, the weight between corresponding pixels in the local undirected graph may be determined according to the distance between pixels in each local feature map corresponding to each preset region, and then the local undirected graph corresponding to the preset region may be constructed according to the weight between the pixels determined by each local feature map corresponding to each preset region.

In an alternative example, the weights between corresponding pixels in the local undirected graph may be determined according to the distances between pixels in the local feature maps corresponding to each preset region, which are smaller than or equal to a preset distance threshold. For example, the local feature map is an eight-connectivity map, in each local feature map, each pixel is connected to eight pixels around it, and when the distance between each pixel and its four pixels above, below, left, and right is 1, and the distance between each pixel and its four pixels above, below, left, and below, i.e., the distance between the pixels on the diagonal is the square root of 2, then the preset distance threshold may be the square root of 2. Generally, the larger the preset threshold value is, the more connected the obtained undirected graph will have, the better the recovery effect of the finally obtained image will be, but the more complicated the processing will be.

In an alternative example, the prior model may use an image laplacian model, and a set of image laplacian models respectively corresponding to preset regions in the current image to be processed may be determined according to a set of local undirected graphs respectively corresponding to the preset regions in the current image to be processed. For example: an adjacency matrix corresponding to each local undirected graph can be constructed according to the weight among the pixels in each local undirected graph, a corresponding degree matrix is constructed according to each adjacency matrix, and then a group of graph Laplacian matrices is obtained according to the adjacency matrix and the degree matrix corresponding to each local undirected graph.

And 208, performing image segmentation processing on the current image to be processed to obtain a group of local images to be processed corresponding to each preset area in the current image to be processed.

Alternatively, operation 208 may employ a preset image segmentation method the same as that in operation 204, so that a set of local images to be processed obtained by image segmentation of the current image to be processed respectively corresponds to each preset region in the current image to be processed, and may completely cover the entire current image to be processed, where adjacent local images to be processed may also have the same region, that is, an overlapping region, and the set of local images to be processed obtained by image segmentation of the current image to be processed and the set of local feature maps obtained by image segmentation of each feature map have the same size and number. In an alternative example, the image segmentation processing may be performed by using a neural network, or may be performed by using another method other than the neural network.

And 210, performing image recovery processing on each preset region in the current image to be processed based on the corresponding prior model respectively according to the image degradation phenomenon of the corresponding local image to be processed, so as to obtain a group of local target images respectively corresponding to each preset region in the current image to be processed.

The method for performing the image recovery processing on the image degradation phenomenon of each local to-be-processed image in the embodiment of the present invention may be determined according to the type of the prior model, and the method for performing the image recovery processing on the image degradation phenomenon of the corresponding local to-be-processed image based on the corresponding prior model in the embodiment of the present invention is not limited. In an alternative example, the prior model may use a graph laplacian model, and a set of local target images may be obtained by respectively solving a set of quadratic programming problems corresponding to preset regions in the current image to be processed, where each quadratic programming problem respectively uses a graph laplacian matrix corresponding to the same preset region in the current image to be processed as a regularization term.

Optionally, before performing image restoration processing on the image degradation phenomenon of each local image to be processed, a graph laplacian matrix may be determined according to the current image to be processed as a coefficient of a regularization term, that is, a group of graph laplacian regularization term coefficients, where the graph laplacian regularization term is a group of graph laplacian matrices respectively corresponding to preset regions in the current image to be processed, which are determined according to local feature maps in the groups of local feature maps corresponding to the same preset region in the current image to be processed, as the regularization term.

Optionally, a neural network or other methods may be used to process the current image to be processed to obtain a group of data serving as the graph laplacian regularization term coefficient.

And 212, splicing a group of local target images respectively corresponding to each preset area in the current image to be processed to obtain a target image.

Optionally, the local target images may be stitched by a preset image stitching method to obtain a final target image, where the target image and the current image to be processed have the same size, and for the local target images having the same area, that is, an overlapping area, the image of the overlapping area may be determined by averaging the overlapping area. Alternatively, the preset image stitching method may be reciprocal to the preset image segmentation method. In an alternative example, the images may be stitched by using a neural network, or the images may be stitched by using a method other than the neural network. The embodiment of the invention does not limit the image splicing method.

Based on the image restoration method provided by the above embodiment of the present invention, the current image to be processed is divided into a plurality of local images to be processed, and a plurality of prior models are respectively constructed according to the local feature maps corresponding to the local images to be processed, so that the image restoration problem of the current image to be processed can be quantized into the image restoration problem of the local images to be processed, and when the image degradation phenomenon of the local images to be processed is restored, the prior models can be constructed by combining machine learning and the prior models and utilizing the information of the local images to be processed for learning the local images to be processed, so that the prior models can more truly reflect the image restoration requirements of the local images to be processed, and a better image restoration effect on the current image to be processed can be obtained.

Fig. 3 is a schematic diagram of a network architecture implementing the image restoration method of some embodiments of the present invention. It should be understood that the example shown in fig. 3 is only for helping those skilled in the art to better understand the technical solution of the present invention, and should not be construed as limiting the present invention. Those skilled in the art can make various changes on the basis of fig. 3, and such changes should also be understood to form part of the present invention.

As shown in fig. 3, the network structure includes: three convolutional neural networks CNN_F、CNN_YAnd CNN_μThe image segmentation module, the undirected graph construction module, the quadratic programming solving module and the image splicing module are arranged, wherein the undirected graph construction module and the quadratic programming solving module form a graph Laplace regularization layer. The input of the network structure is a current image Y to be processed, the current image to be processed can be an image with image degradation phenomena such as a noise image or a blurred image, and the output of the network structure is a corresponding target image X^*The target image may be an image from which the network degradation phenomenon is removed after the network degradation phenomenon is restored.

Firstly, inputting the current image Y to be processed into a first convolutional neural network CNN_FExtracting the features, and outputting N feature images F with the same size as the current image to be processed Y₁,F₂,…,F_N. At the same time, the current image Y to be processed may be input into a second convolutional neural network CNN_YAnd performing primary recovery processing on the current image Y to be processed, and outputting an image Y' with the same size as the current image Y to be processed.

Then, N feature maps F may be generated₁,F₂,…,F_NAnd the image Y' subjected to the primary recovery processing is input into an image segmentation module, and each feature map and the image subjected to the primary recovery processing are segmented into K square maps by the image segmentation module according to the same preset mode, wherein F is used for segmenting each feature map and each image subjected to the primary recovery processing into K square maps_n(N is not less than 1 and not more than N) and the kth square divided is denoted as f_n ^kThe kth positive divided by YThe square is denoted by y_k，f_n ^kAnd y_kVectorization may be used, i.e. the columns in the matrix corresponding to the image are connected first and last in sequence to form a vector, where the length of the vectors is set to m.

Then, each local image Y to be processed output by the image segmentation module for the image Y' after the preliminary recovery processing can be obtained_k(K is more than or equal to 1 and less than or equal to K) establishing an undirected graph and solving a corresponding graph Laplacian matrix L_kFor y to_k"graph laplace regularization" is performed. Wherein, N characteristic maps F₁,F₂,…,F_NEach local feature map output by the input image segmentation module is input into the undirected graph construction module, and the undirected graph construction module can utilize the local feature maps corresponding to the same local image y to be processed in each local feature map_kN local feature maps f₁ ^k,f₂ ^k,…,f_N ^kAn undirected graph with m nodes is established together, wherein each node in the undirected graph represents a pixel. The establishment method of the undirected graph comprises the following steps:

wherein, w_ijIs the weight between the ith pixel and the jth pixel in the local feature map, and w is the weight between the ith pixel and the jth pixel in the local feature map if the ith pixel and the jth pixel are located at a relatively far distance in the image, e.g., the distance exceeds a distance threshold d_ijThat is, there is no connection between the ith pixel and the jth pixel. After the undirected graph is obtained, the corresponding graph Laplace matrix L can be calculated_k。

Next, the current image Y to be processed may be input into a third convolutional neural network CNN_μProcessing and outputting K data mu₁,μ₂,…,μ_KWherein, mu_kCorrespond toIn the local image y to be processed_kAnd (K is more than or equal to 1 and less than or equal to K), the weight of the graph Laplace regularization term when the quadratic programming problem is solved subsequently, namely the coefficient.

Thereafter, each local image to be processed y may be processed_kSolving the following quadratic programming problem:

wherein, the local target image after the recovery processing is marked as X_k ^*The second term in the quadratic programming problem is the graph laplacian regularization term.

Finally, all the restored local target images can be recorded as X_k ^*And inputting an image splicing module, and splicing to obtain a final target image after image recovery.

In the above process, the current image Y to be processed passes through only the second convolutional neural network CNN_YAnd performing image recovery operation twice on the graph Laplace regularization layer to obtain a target image after image recovery.

The undirected graph construction module and the quadratic programming solving module form a graph Laplace regularization layer, and two modules in the graph Laplace regularization layer are both conductive, so that the network structure of the image restoration method can be trained end to end.

In practical application, the first convolutional neural network CNN_FCan adopt a multilayer structure, so that the current image to be processed can be more effectively learned to construct a suitable undirected graph, and a second convolutional neural network CNN_YThe number of layers (c) may be small, for example, a structure such as 4 to 6 layers is adopted, so that the possibility of overfitting brought by the network itself can be reduced, and the function of the subsequent graph laplacian regularization layer can be exerted as much as possible. First convolutional neural network CNN_μThe number of layers of (2) may also be smaller.

In the above embodiments of the present invention, an iterative processing method may also be adopted, that is, after a prior model is constructed according to the feature map of the current image to be processed, and based on the prior model, image recovery processing is performed on the image degradation phenomenon of the current image to be processed, so as to obtain a corresponding target image, the following operations may also be iteratively performed: and taking the target image as the current image to be processed, constructing a prior model according to the characteristic diagram of the current image to be processed, and performing image recovery processing on the current image to be processed based on the prior model to obtain a corresponding target image. By carrying out iterative recovery processing on the input image to be processed, the quality of the image after each iteration can be further improved, and a high-quality target image can be recovered after multiple iterations.

Fig. 4 is a schematic diagram of a network structure implementing an image restoration method according to another embodiment of the present invention. It should be understood that the example shown in fig. 4 is only for helping those skilled in the art to better understand the technical solution of the present invention, and should not be construed as limiting the present invention. Those skilled in the art can make various changes on the basis of fig. 4, and such changes should also be understood to form part of the present invention.

As shown in fig. 4, the network structure includes: a plurality of cascaded image recovery modules, the input of the network structure is a current image to be processed Y, the current image to be processed can be an image with image degradation phenomena such as a noise image or a blurred image, and the output of the network structure is a target image X_TBy cascading the T image recovery modules, the image recovery processing is performed on the input current image Y to be processed for multiple times, so that a better image recovery effect can be obtained. For example, each stage of image restoration module may adopt the structure in fig. 3, and the cascaded image restoration modules may share parameters of the neural network, that is, the corresponding neural networks in each stage of image restoration module adopt the same parameters, so as to reduce the number of network parameters and prevent the occurrence of over-fitting.

Fig. 5 is a schematic structural diagram of an image restoration apparatus according to some embodiments of the present invention. It should be understood that the example shown in fig. 5 is only for helping those skilled in the art to better understand the technical solution of the present invention, and should not be construed as limiting the present invention. Those skilled in the art can make various changes on the basis of fig. 5, and such changes should also be understood to form part of the present invention.

As shown in fig. 5, the apparatus includes: a modeling unit 510 and a processing unit 520. Wherein the content of the first and second substances,

and the modeling unit 510 is used for constructing a prior model according to the feature map of the current image to be processed.

Optionally, the apparatus may further include an extraction unit, where the extraction unit may perform feature extraction on the current image to be processed to obtain a set of feature maps of the current image to be processed, where each feature map in the set of feature maps has the same size as the current image to be processed, and the modeling unit 510 constructs the prior model according to the set of feature maps having the same size as the current image to be processed. Alternatively, the extraction unit may perform feature extraction on the current image to be processed by using a neural network or other machine learning methods, so as to obtain a set of feature maps having the same size as the current image to be processed. In an alternative example, the neural network may employ a convolutional neural network. The embodiment of the present invention does not limit the manner in which the extraction unit obtains the feature map according to the current image to be processed.

And the processing unit 520 is configured to perform image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model, so as to obtain a corresponding target image.

The method for performing the image recovery processing on the image degradation phenomenon of the image to be processed by the processing unit 520 according to the embodiment of the present invention may be determined according to the type of the prior model, and the method for performing the image recovery processing on the image degradation phenomenon of the image to be processed by the processing unit 520 based on the prior model is not limited in the embodiment of the present invention. In an optional example, the prior model may adopt an image laplacian model, and the processing unit 520 may perform image recovery processing on an image degradation phenomenon of the current image to be processed by solving a quadratic programming problem of the current image to be processed with the image laplacian matrix as a regularization term, so as to obtain the target image.

Alternatively, the image restoration processing of the processing unit 520 may be image processing corresponding to an image degradation phenomenon contained in the current image to be processed, for example, when the image degradation phenomenon includes at least one of: when image noise, image blurring, image pixel missing and the like exist, the corresponding image restoration processing includes at least one of the following: image denoising processing, image deblurring processing, image pixel completion processing and the like.

Optionally, the apparatus may further include a preprocessing unit, where the preprocessing unit may perform preliminary recovery processing on an image degradation phenomenon of the current image to be processed, and the processing unit 520 performs image recovery processing on the image degradation phenomenon of the image subjected to the preliminary recovery processing based on the prior model to obtain a target image, where the image subjected to the preliminary recovery processing and the current image to be processed have the same size. Alternatively, the preprocessing unit may perform a preliminary restoration process on the image degradation phenomenon of the current image to be processed by using a preprocessing network or other image restoration processing methods. In an alternative example, the preprocessing network may employ a convolutional neural network. The embodiment of the invention does not limit the method for the preliminary recovery processing of the preprocessing unit.

Based on the image restoration device provided by the above embodiment of the present invention, a prior model is constructed according to the feature map of the current image to be processed, based on the prior model, image restoration processing is performed on the image degradation phenomenon of the current image to be processed, so as to obtain a corresponding target image, wherein the current image to be processed includes at least one image degradation phenomenon, the prior model is used for learning the information of the current image to be processed, and the information of the current image to be processed is learned by using the feature map to construct the prior model, so that the prior model can more truly reflect the image restoration requirement of the current image to be processed, and particularly for the situation that image degradation is complex in an actual application scene, by combining machine learning with the prior model, a good image restoration effect on the current image to be processed can be obtained.

Compared with a device for recovering images only through a neural network, the device needs to completely depend on the quantity and quality of training data, when the quantity and quality of the training data of the neural network are not robust enough, overfitting is easy to happen and a good image recovery result cannot be obtained. Compared with a method for recovering an image by simply adopting a prior model, the method has the advantages that some assumptions need to be made on the properties of the image, the flexibility is poor, and the image recovery effect can be influenced when the assumed conditions cannot be met.

Fig. 6 is a schematic structural diagram of an image restoration apparatus according to another embodiment of the present invention. It should be understood that the example shown in fig. 6 is only for helping those skilled in the art to better understand the technical solution of the present invention, and should not be construed as limiting the present invention. Those skilled in the art can make various changes on the basis of fig. 6, and such changes should also be understood to form part of the present invention.

As shown in fig. 6, the apparatus includes: an extraction unit 610, a segmentation unit 620, a modeling unit 630, a processing unit 640, and a concatenation unit 650. Wherein the content of the first and second substances,

the extracting unit 610 is configured to perform feature extraction on the current image to be processed, so as to obtain a group of feature maps having the same size as the current image to be processed.

Alternatively, the extraction unit 610 may perform feature extraction on the current image to be processed by using a neural network or other machine learning methods, so as to obtain a set of feature maps having the same size as the current image to be processed. In an alternative example, the neural network may employ a convolutional neural network. The embodiment of the present invention does not limit the manner in which the extracting unit 610 obtains the feature map according to the current image to be processed.

A segmentation unit 620, configured to perform image segmentation on each feature map in a group of feature maps having the same size as the current image to be processed, so as to obtain a group of local feature maps, where each feature map corresponds to each preset region in the current image to be processed; and performing image segmentation processing on the current image to be processed to obtain a group of local images to be processed corresponding to each preset area in the current image to be processed.

Alternatively, the segmentation unit 620 may perform image segmentation processing on each feature map according to a preset image segmentation method, so that a set of local feature maps obtained by image segmentation of each feature map respectively corresponds to each preset region in the current image to be processed, and may completely cover the entire current image to be processed, where adjacent local feature maps may have the same region, that is, an overlapping region. In an alternative example, the segmentation unit 620 may perform the image segmentation processing by using a neural network, or may perform the image segmentation processing by using a method other than the neural network, and the method for performing the image segmentation processing by the segmentation unit 620 is not limited in the embodiment of the present invention.

The embodiment of the present invention does not limit the number and size of the local feature maps obtained by the image segmentation processing performed by the segmentation unit 620, and generally, the larger the size of the local feature map obtained by the image segmentation processing is, the better the recovery effect of the finally obtained image is, but the longer the processing time of each local feature map is, and in practical application, the effect and efficiency of image recovery can be comprehensively considered, so as to determine the size and number of the local feature maps obtained by the image segmentation processing. For example, each feature map may be partitioned into a set of local feature maps of 26x26 pixels.

Alternatively, the segmentation unit 620 may use the same preset image segmentation method, so that a set of local images to be processed obtained by image segmentation of the current image to be processed respectively corresponds to each preset region in the current image to be processed, and may completely cover the entire current image to be processed, where adjacent local images to be processed may also have the same region, that is, an overlapping region, and the set of local images to be processed obtained by image segmentation of the image to be processed and the set of local feature maps obtained by image segmentation of each feature map have the same size and number.

The modeling unit 630 is configured to determine a set of prior models respectively corresponding to the preset regions in the current image to be processed according to the local feature maps corresponding to the same preset regions in the current image to be processed in the sets of local feature maps.

Alternatively, the modeling unit 630 may respectively construct a set of local undirected graphs corresponding to the preset regions in the current image to be processed according to the local feature graphs corresponding to the same preset regions in the current image to be processed in the sets of local feature graphs, and then determine a set of prior models respectively corresponding to the preset regions in the current image to be processed according to the set of local undirected graphs corresponding to the preset regions in the current image to be processed.

Alternatively, for each preset region in the current image to be processed, the modeling unit 630 may determine the weight between corresponding pixels in the local undirected graph according to the distance between the pixels in the local feature maps corresponding to each preset region, and then construct the local undirected graph corresponding to each preset region according to the weight between the pixels determined by the local feature maps corresponding to each preset region.

In an alternative example, the modeling unit 630 may determine the weight between corresponding pixels in the local undirected graph according to the distance between pixels in the local feature maps corresponding to each preset region, which are smaller than or equal to a preset distance threshold. For example, the local feature map is an eight-connectivity map, in each local feature map, each pixel is connected to eight pixels around it, and when the distance between each pixel and its four pixels above, below, left, and right is 1, and the distance between each pixel and its four pixels above, below, left, and below, i.e., the distance between the pixels on the diagonal is the square root of 2, then the preset distance threshold may be the square root of 2. Generally, the larger the preset threshold value is, the more connected the obtained undirected graph will have, the better the recovery effect of the finally obtained image will be, but the more complicated the processing will be.

In an alternative example, the prior model may adopt a graph laplacian model, and the modeling unit 630 may determine a set of graph laplacian models respectively corresponding to preset regions in the current image to be processed according to a set of local undirected graphs respectively corresponding to the preset regions in the current image to be processed. For example: the modeling unit 630 may construct an adjacency matrix corresponding to each local undirected graph according to the weight between the pixels in each local undirected graph, construct a corresponding degree matrix according to each adjacency matrix, and obtain a set of graph laplacian matrices according to the adjacency matrix and the degree matrix corresponding to each local undirected graph.

The processing unit 640 is configured to perform image restoration processing on each preset region in the current image to be processed, based on the corresponding prior model, for the image degradation phenomenon of the corresponding local image to be processed, so as to obtain a set of local target images corresponding to each preset region in the current image to be processed.

The method for the processing unit 640 to perform the image recovery processing on the image degradation phenomenon of each local image to be processed according to the embodiment of the present invention may be determined according to the type of the prior model, and the method for the processing unit 640 to perform the image recovery processing on the image degradation phenomenon of the corresponding local image to be processed based on the corresponding prior model is not limited in the embodiment of the present invention. In an alternative example, the prior model may use a graph laplacian model, and the processing unit 640 may obtain a set of local target images by respectively solving a set of quadratic programming problems corresponding to preset regions in the current image to be processed, where each quadratic programming problem respectively uses a graph laplacian matrix corresponding to the same preset region in the current image to be processed as a regularization term.

Optionally, the apparatus may further include a determining unit, where the determining unit may determine, according to the current image to be processed, a graph laplacian matrix as a coefficient of the regularization term, that is, a set of graph laplacian regularization term coefficients, where the graph laplacian regularization term is a set of graph laplacian matrices respectively corresponding to preset regions in the current image to be processed, which are determined according to local feature maps in the sets of local feature maps that correspond to the same preset region in the current image to be processed, and which respectively correspond to the preset regions in the current image to be processed, as the regularization term.

Optionally, the determining unit may process the current image to be processed by using a neural network or other methods to obtain a group of data serving as the graph laplacian regularization term coefficient.

And the splicing unit 650 is configured to splice a group of local target images respectively corresponding to each preset region in the current image to be processed, so as to obtain a target image.

Optionally, the stitching unit 650 may perform stitching processing on each local target image by using a preset image stitching method to obtain a final target image, where the target image and the current image to be processed have the same size, and for the local target images having the same area, that is, an overlapping area, the stitching unit 650 may determine the image of the overlapping area by averaging the overlapping area. Alternatively, the preset image stitching method of the stitching unit 650 may be reciprocal to the preset image segmentation method of the segmentation unit 620. In an alternative example, the stitching unit 650 may perform the stitching processing on the images by using a neural network, or may perform the stitching processing on the images by using a method other than the neural network. The method for performing image stitching processing on the stitching unit 650 in the embodiment of the present invention is not limited.

Based on the image restoration device provided by the above embodiment of the present invention, the current image to be processed is divided into the plurality of local images to be processed, and the plurality of prior models are respectively constructed according to the local feature maps corresponding to the respective local images to be processed, so that the image restoration problem of the current image to be processed can be quantized into the image restoration problem of the plurality of local images to be processed, and when the image degradation phenomenon of the plurality of local images to be processed is restored, the prior models can be constructed by combining machine learning with the prior models and utilizing the information of the local images to be processed for learning the local images to be processed, so that the prior models can more truly reflect the image restoration requirements of the local images to be processed, and a better image restoration effect on the current image to be processed can be obtained.

In the above embodiments of the present invention, the image restoration apparatus may further include at least two cascaded restoration modules, where the restoration modules include units for image restoration, such as a modeling unit and a processing unit. The first recovery module is used for constructing a prior model according to a feature map of a current image to be processed; and based on the prior model, carrying out image recovery processing aiming at the image degradation phenomenon of the current image to be processed to obtain a corresponding target image. The recovery modules except the first recovery module are used for taking the target image as the current image to be processed and constructing a prior model according to the feature map of the current image to be processed; and based on the prior model, carrying out image restoration processing on the current image to be processed according to the image degradation phenomenon to obtain a corresponding target image. By adopting the mode of restoring module base connection to carry out iterative restoring processing on the input image to be processed, the quality of the image after each iteration can be further improved, and a high-quality target image can be restored after multiple iterations.

The embodiment of the invention also provides electronic equipment, which can be a mobile terminal, a Personal Computer (PC), a tablet computer, a server and the like. Referring now to fig. 7, shown is a schematic diagram of an electronic device 700 suitable for use in implementing a terminal device or server of an embodiment of the present application: as shown in fig. 7, the electronic device 700 includes one or more processors, communication sections, and the like, for example: one or more Central Processing Units (CPUs) 701, and/or one or more image processors (GPUs) 713, etc., which may perform various suitable actions and processes according to executable instructions stored in a Read Only Memory (ROM)702 or loaded from a storage section 708 into a Random Access Memory (RAM) 703. Communications portion 712 may include, but is not limited to, a network card, which may include, but is not limited to, an ib (infiniband) network card,

the processor may communicate with the read-only memory 702 and/or the random access memory 730 to execute executable instructions, connect with the communication part 712 through the bus 704, and communicate with other target devices through the communication part 712, so as to complete operations corresponding to any method provided by the embodiments of the present application, for example, constructing a prior model according to a feature map of a current image to be processed, where the prior model is used to learn information of the current image to be processed, and the current image to be processed includes at least one image degradation phenomenon; and based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image.

In addition, in the RAM703, various programs and data necessary for the operation of the device can also be stored. The CPU701, the ROM702, and the RAM703 are connected to each other via a bus 704. The ROM702 is an optional module in case of the RAM 703. The RAM703 stores or writes executable instructions into the ROM702 at runtime, and the executable instructions cause the central processing unit 701 to perform operations corresponding to the above-described communication methods. An input/output (I/O) interface 705 is also connected to bus 704. The communication unit 712 may be integrated, or may be provided with a plurality of sub-modules (e.g., a plurality of IB network cards) and connected to the bus link.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

It should be noted that the architecture shown in fig. 7 is only an optional implementation manner, and in a specific practical process, the number and types of the components in fig. 7 may be selected, deleted, added or replaced according to actual needs; in different functional component settings, separate settings or integrated settings may also be used, for example, GPU713 and CPU701 may be separately provided or GPU713 may be integrated on CPU701, the communication part may be separately provided or integrated on CPU701 or GPU713, and so on. These alternative embodiments are all within the scope of the present disclosure.

In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the method illustrated in the flowchart, the program code may include instructions corresponding to performing the method steps provided in the embodiments of the present application, for example, constructing an a priori model from a feature map of a current image to be processed, the a priori model being used for learning information of the current image to be processed, the current image to be processed including at least one image degradation phenomenon; and based on the prior model, carrying out image recovery processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program, when executed by a Central Processing Unit (CPU)701, performs the above-described functions defined in the method of the present application.

In one or more alternative embodiments, the embodiment of the present invention further provides a computer program product for storing computer readable instructions, which when executed, cause a computer to execute the image restoration method in any one of the possible implementations.

The computer program product may be embodied in hardware, software or a combination thereof. In one alternative, the computer program product is embodied in a computer storage medium, and in another alternative, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

In one or more optional implementation manners, embodiments of the present invention further provide an image restoration method, and a corresponding apparatus, system, and electronic device, computer storage medium, computer program, and computer program product, where the method includes: the first device sending an image restoration instruction to the second device, the instruction causing the second device to execute the image restoration method in any of the above possible embodiments; the first device receives the result of the image restoration sent by the second device.

In some embodiments, the image restoration instruction may be embodied as a call instruction, and the first device may instruct the second device to perform image restoration by calling, and accordingly, in response to receiving the call instruction, the second device may perform the steps and/or flows of any of the above-described image restoration methods.

It is to be understood that the terms "first", "second", and the like in the embodiments of the present invention are used for distinguishing and not to limit the embodiments of the present invention.

It is also understood that in the present invention, "a plurality" may mean two or more, and "at least one" may mean one, two or more.

It is also to be understood that any reference to any component, data, or structure in the present disclosure is generally intended to mean one or more, unless explicitly defined otherwise or indicated to the contrary hereinafter.

It should also be understood that the description of the embodiments of the present invention emphasizes the differences between the embodiments, and the same or similar parts may be referred to each other, so that the descriptions thereof are omitted for brevity.

The method and apparatus of the present invention may be implemented in a number of ways. For example, the methods and apparatus of the present invention may be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustrative purposes only, and the steps of the method of the present invention are not limited to the order specifically described above unless specifically indicated otherwise. Furthermore, in some embodiments, the present invention may also be embodied as a program recorded in a recording medium, the program including machine-readable instructions for implementing a method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. An image restoration method, comprising:

determining a group of prior models respectively corresponding to the preset regions in the current image to be processed according to the local feature maps corresponding to the same preset regions in the current image to be processed in the groups of local feature maps, wherein the prior models are used for learning the information of the current image to be processed, and the current image to be processed comprises at least one image degradation phenomenon;

based on the prior model, performing image restoration processing on the image degradation phenomenon of the current image to be processed to obtain a corresponding target image, including: obtaining the target image by solving a quadratic programming problem of the current image to be processed with an image Laplacian matrix as a regularization item;

the image restoration processing is performed on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, and the method includes:

2. The method of claim 1, wherein the image degradation phenomenon comprises at least one of: image noise, image blur and image pixel dropout;

3. The method according to claim 1, wherein the determining a set of prior models respectively corresponding to preset regions in the current image to be processed according to the local feature maps corresponding to the same preset regions in the current image to be processed in the sets of local feature maps comprises:

4. The method according to claim 3, wherein the constructing a set of local undirected graphs corresponding to the preset regions in the current image to be processed respectively according to the local feature maps corresponding to the same preset regions in the current image to be processed in the sets of local feature maps comprises:

5. The method of claim 4, wherein determining the weight between corresponding pixels in the local undirected graph according to the distance between the pixels in the local eigenmaps corresponding to each preset region comprises:

6. The method according to any one of claims 1 to 5, wherein before performing image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, the method further includes:

7. The method of any of claims 1 to 5, wherein the prior model comprises: the graph laplacian matrix.

8. The method according to claim 7, wherein before performing image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, the method further comprises:

9. The method according to any one of claims 1 to 5, wherein, after performing image restoration processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, the method further comprises:

and (3) performing iteration: determining a group of prior models respectively corresponding to each preset region in the current image to be processed according to the local feature maps corresponding to the same preset region in the current image to be processed in each group of local feature maps;

10. The method according to any one of claims 1 to 5, applied to a mobile terminal and/or a driving assistance system.

11. An image restoration apparatus, comprising:

the modeling unit is used for determining a group of prior models respectively corresponding to all preset regions in the current image to be processed according to the local feature maps corresponding to the same preset regions in the current image to be processed in all groups of local feature maps, wherein the prior models are used for learning the information of the current image to be processed, and the current image to be processed comprises at least one image degradation phenomenon;

the processing unit is used for carrying out image recovery processing on the image degradation phenomenon of the current image to be processed based on the prior model to obtain a corresponding target image, and is specifically used for obtaining the target image by solving a quadratic programming problem of the current image to be processed with an image Laplace matrix as a regularization term;

the processing unit is specifically configured to perform image restoration processing on each preset region in the current image to be processed, based on the corresponding prior model, for the image degradation phenomenon of the corresponding local image to be processed, so as to obtain a set of local target images corresponding to each preset region in the current image to be processed;

12. The apparatus of claim 11, wherein the image degradation phenomenon comprises at least one of: image noise, image blur and image pixel dropout;

13. The apparatus according to claim 11, wherein the modeling unit is configured to respectively construct a set of local undirected graphs corresponding to preset regions in the current image to be processed according to local feature graphs corresponding to the same preset regions in the current image to be processed in the sets of local feature graphs; and determining a group of prior models respectively corresponding to each preset region in the current image to be processed according to a group of local undirected graphs respectively corresponding to each preset region in the current image to be processed.

14. The apparatus according to claim 13, wherein the modeling unit is configured to determine, for each preset region in the current image to be processed, a weight between corresponding pixels in the local undirected graph according to a distance between pixels in each local feature map corresponding to each preset region; and constructing a local undirected graph corresponding to each preset region according to the weight between the pixels determined by the local feature graphs corresponding to each preset region.

15. The apparatus of claim 14, wherein the modeling unit is configured to determine the weight between corresponding pixels in the local undirected graph according to a distance between pixels in the local feature maps corresponding to each of the preset regions that are smaller than or equal to a preset distance threshold.

16. The apparatus of any one of claims 11 to 15, further comprising:

17. The apparatus of any of claims 11 to 15, wherein the prior model comprises: the graph laplacian matrix.

18. The apparatus of claim 17, further comprising:

19. The apparatus according to any one of claims 11 to 15, comprising: at least two cascaded recovery modules, the recovery modules comprising: the modeling unit and the processing unit;

20. The device according to any one of claims 11 to 15, applied to a mobile terminal and/or a driving assistance system.

21. An electronic device, characterized in that it comprises the apparatus of any of claims 11 to 20.

22. An electronic device, comprising:

a memory for storing executable instructions; and

a processor for executing the executable instructions to perform the method of any one of claims 1 to 10.

23. A computer storage medium storing computer readable instructions that, when executed, implement the method of any one of claims 1 to 10.