CN111784580A - Super-resolution method and device for image and server - Google Patents
Super-resolution method and device for image and server Download PDFInfo
- Publication number
- CN111784580A CN111784580A CN202010623605.1A CN202010623605A CN111784580A CN 111784580 A CN111784580 A CN 111784580A CN 202010623605 A CN202010623605 A CN 202010623605A CN 111784580 A CN111784580 A CN 111784580A
- Authority
- CN
- China
- Prior art keywords
- image
- distortion
- super
- processed
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000012545 processing Methods 0.000 claims abstract description 86
- 238000012549 training Methods 0.000 claims abstract description 27
- 238000004458 analytical method Methods 0.000 claims description 14
- 230000000694 effects Effects 0.000 description 6
- 238000013136 deep learning model Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention provides a super-resolution method, a device and a server of an image, firstly, distortion parameters of the image to be processed are determined; according to the distortion parameters, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed after restoration; and performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed, wherein the super-resolution network model is obtained by performing distortion repairing processing on the sample image based on the distortion parameter of the sample image and then training according to the repaired image of the sample image. In the method, only one super-resolution network model is required to be trained, and a plurality of network models do not need to be trained aiming at different distortion types, so that the model training cost is reduced, and the generalization of the model is also improved; meanwhile, the method does not need manual intervention, can automatically obtain the super-resolution image of the image to be processed, and saves the labor cost.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a super-resolution method, a super-resolution device and a server for an image.
Background
The natural scene image generally refers to an image shot by people in a natural environment through equipment such as a camera and a video camera, and the natural scene image has the characteristics of wide distribution and multiple types, so that the natural scene image has more distortion types.
In order to obtain an ultra-high-definition image of a natural scene image with distortion, in the related art, corresponding super-resolution networks are generally trained respectively according to different distortion types of the image, and when the ultra-high-definition image of the natural scene image is obtained, the corresponding super-resolution network can be manually selected for processing according to the different distortion types of the natural scene image to obtain the ultra-high-definition image; or the images of the natural scene can be respectively input into the super-resolution networks corresponding to different distortion types to obtain the output result of each super-resolution network, and the image with the best effect is selected from all the output results to serve as the final ultra-high definition image. The above method requires training a plurality of super-resolution networks, which results in high network training cost.
Disclosure of Invention
The invention aims to provide a super-resolution method, a super-resolution device and a super-resolution server of images, so as to reduce the cost of network training.
In a first aspect, an embodiment of the present invention provides a super-resolution method for an image, where the method includes: determining a distortion parameter of an image to be processed; according to the distortion parameter, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed; performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image.
In an alternative embodiment, the distortion parameter includes a distortion type; the step of determining the distortion parameter of the image to be processed includes: performing quality analysis on the image to be processed to obtain the distortion type of the image to be processed and the distortion intensity corresponding to the distortion type; if the distortion type of the image to be processed comprises a plurality of types, determining the final distortion type of the image to be processed according to the distortion intensity of each type of distortion.
In an optional embodiment, the step of determining a final distortion type of the image to be processed according to the distortion strength of each distortion type includes: determining the distortion type with the maximum distortion intensity as the final distortion type of the image to be processed; or determining the distortion type with the distortion intensity higher than a preset intensity threshold value as the final distortion type of the image to be processed.
In an alternative embodiment, the distortion parameter includes a distortion type; the above-mentioned step of performing distortion restoration processing on the image to be processed according to the distortion parameter to obtain a restored image after restoration of the image to be processed includes: and performing distortion restoration processing on the image to be processed by a distortion restoration mode corresponding to the distortion type of the image to be processed to obtain a restored image after restoration of the image to be processed.
In an optional embodiment, the step of performing super-resolution processing on the repaired image through the pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed includes: and inputting the repaired image into a super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
In an alternative embodiment, the super-resolution network model is obtained by training in the following manner: determining a sample image based on a preset sample set; determining a distortion parameter of the sample image; performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image; and inputting the repaired image of the sample image into a preset initial network model so as to train the initial network model and obtain a super-resolution network model.
In an optional embodiment, the step of performing super-resolution processing on the repaired image through the pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed includes: and inputting the repaired image and the image to be processed into a super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
In an alternative embodiment, the super-resolution network model is obtained by training in the following manner: determining a sample image based on a preset sample set; determining a distortion parameter of the sample image; performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image; and inputting the repaired image of the sample image and the sample image into a preset initial network model so as to train the initial network model and obtain a super-resolution network model.
In a second aspect, an embodiment of the present invention provides a super-resolution apparatus for an image, the apparatus including: the distortion parameter determining module is used for determining the distortion parameter of the image to be processed; the distortion restoration module is used for carrying out distortion restoration processing on the image to be processed according to the distortion parameters to obtain a restored image after the image to be processed is restored; the super-resolution processing module is used for carrying out super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image.
In a third aspect, embodiments of the present invention provide a server, which includes a processor and a memory, where the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the super resolution method for the image.
In a fourth aspect, embodiments of the invention provide a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement a super-resolution method for images as described above.
The embodiment of the invention has the following beneficial effects:
the invention provides a super-resolution method, a device and a server of an image, firstly, distortion parameters of the image to be processed are determined; then according to the distortion parameter, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed; then, performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image. The method comprises the steps of carrying out distortion restoration processing on an image to be processed according to distortion parameters of the image to be processed, and then carrying out super-resolution processing on the restored image of the image to be processed through a super-resolution network model to obtain a super-resolution image of the image to be processed; meanwhile, the method does not need manual intervention, can automatically obtain the super-resolution image of the image to be processed, and saves the labor cost.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention as set forth above.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a super-resolution method for an image according to an embodiment of the present invention;
FIG. 2 is a flow chart of another super-resolution method for images according to an embodiment of the present invention;
FIG. 3 is a flow chart of another super-resolution method for images according to an embodiment of the present invention;
FIG. 4 is a flow chart of another super-resolution method for images according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a super-resolution device for images according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
With the development of multimedia technology, users are pursuing high-definition images more and more. Due to the influence of shooting equipment and shooting environment, a large number of low-resolution and low-definition images exist in the existing media and networks. At present, deep learning is widely applied in many fields, and has a good effect in many application fields, however, deep learning is mostly based on sample driving or training, and has a good effect only in a specific environment or a specific scene, but has a poor effect when other scenes are changed. In the face of the characteristics of wide image distribution and multiple types of natural scenes, a deep learning network is often difficult to obtain a satisfactory result in all scenes, so how to solve the generalization problem of deep learning becomes a problem to be solved urgently at present.
In order to obtain an ultra-high-definition image of a natural scene image, in the related art, corresponding super-resolution networks are usually trained respectively according to different distortion types of the image, and when the ultra-high-definition image of the natural scene image is obtained, the corresponding super-resolution network can be manually selected for processing according to the different distortion types of the natural scene image, so that the ultra-high-definition image is obtained; or the images of the natural scene can be respectively input into the super-resolution networks corresponding to different distortion types to obtain the output result of each super-resolution network, and the image with the best effect is selected from all the output results to serve as the final ultra-high definition image. The mode of manually selecting the network for processing according to the distortion type is lack of automation, and the mode of selecting the best image from all over-grading results as the final result has the problems of poor real-time performance and difficult quality evaluation; meanwhile, because the above-mentioned methods all need to train a plurality of super-resolution networks, the cost of network training is high.
Based on the above, the embodiments of the present invention provide a super-resolution method, apparatus and server for images, which can be applied to super-resolution scenes for various images, especially super-resolution scenes for images of natural scenes. To facilitate understanding of the embodiment, a detailed description will be first given of a super-resolution method for images disclosed in the embodiment of the present invention, as shown in fig. 1, the method includes the following specific steps:
step S102, determining distortion parameters of the image to be processed.
The image to be processed may be an image in a natural scene, an image in a monitoring scene, or an image in other scenes. The image to be processed may be an image captured by a video camera or a still camera, or may be a video frame in a recorded video. The above distortion parameters are generally parameters causing distortion of the image to be processed, and the distortion parameters may include an image blur parameter, an image noise parameter, an image aliasing parameter, and the like.
During specific implementation, quality analysis may be performed on the image to be processed to obtain distortion parameters corresponding to the image to be processed, for example, the distortion parameters of the image to be processed may be obtained through subjective quality analysis, that is, through assessment by naked eyes, or the quality of the image to be processed may be calculated through objective quality analysis, that is, through a preset formula to obtain the distortion parameters; and distortion parameters corresponding to the image to be processed can be obtained through a deep learning model, a neural network model and the like.
And step S104, performing distortion restoration processing on the image to be processed according to the distortion parameters to obtain a restored image of the image to be processed after restoration.
During specific implementation, a distortion restoration algorithm matched with the distortion parameters can be adopted to perform distortion restoration processing on the image to be processed so as to eliminate the distortion phenomenon of the image to be processed and obtain a restored image of the image to be processed. For example, when the distortion parameter indicates that the image to be processed has image blur, the image to be processed may be preprocessed by using a deblurring algorithm to eliminate a blur phenomenon in the image to be processed, so as to obtain a restored image with no distortion or less distortion. The repair image is typically the same picture size as the image to be processed.
Step S106, performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image.
When the super-resolution network model is trained, a preset sample image needs to be acquired from a preset sample set, a distortion parameter of the sample image is determined, distortion restoration processing is performed on the sample image according to the distortion parameter to obtain a restored image of the sample image, then the super-resolution network model is trained based on the restored image of the sample image, and the step of acquiring the preset sample image from the preset sample set is continuously executed until the super-resolution network model is converged to obtain the trained super-resolution network model. Specifically, when the super-resolution network model is trained, the super-resolution network models with slightly different input layer structures can be obtained according to different network input data.
In specific implementation, the super-resolution network model is used for carrying out super-resolution processing on the repaired image of the image to be processed to obtain a super-resolution image (also called as a high-definition image or a high-resolution image) corresponding to the image to be processed.
The super-resolution method of the image provided by the embodiment of the invention comprises the steps of firstly determining a distortion parameter of the image to be processed; then according to the distortion parameter, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed; then, performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image. The method comprises the steps of carrying out distortion restoration processing on an image to be processed according to distortion parameters of the image to be processed, and then carrying out super-resolution processing on the restored image of the image to be processed through a super-resolution network model to obtain a super-resolution image of the image to be processed; meanwhile, the method does not need manual intervention, can automatically obtain the super-resolution image of the image to be processed, and saves the labor cost.
The embodiment of the invention also provides another super-resolution method of the image, which is realized on the basis of the method of the embodiment; the method mainly describes a specific process of determining a distortion parameter of an image to be processed (realized by the following steps S202-S204) when the distortion parameter includes a distortion type, and a specific process of performing distortion restoration processing on the image to be processed according to the distortion parameter to obtain a restored image after restoration of the image to be processed (realized by the following step S206), as shown in fig. 2, the method includes the following steps:
step S202, performing quality analysis on the image to be processed to obtain the distortion type of the image to be processed and the distortion intensity corresponding to the distortion type.
The distortion parameter in the present embodiment may represent a distortion type including, but not limited to, image blur, image aliasing, image gaussian noise, image compression noise, and the like. The quality analysis is mainly to perform characteristic analysis research on the image and then evaluate the image quality (which can also be called as image distortion degree); the quality analysis method usually calculates the quality of the image by a formula, that is, calculates the distortion type existing in the image and the distortion strength corresponding to the distortion type, for example, PSNR (Peak-Signal to Noise Ratio) method, MSE (Mean square error) method, or calculation method (Mean, standard deviation) based on the image statistical characteristics may be adopted. The above distortion strength generally refers to a rating of the distortion magnitude of the image to be processed under a specific distortion type, and the larger the rating is, the larger the distortion is.
In specific implementation, by performing quality analysis on the image to be processed, one or more distortion types existing in the image to be processed can be obtained, and the distortion intensity corresponding to each distortion type can be obtained.
In some embodiments, the distortion type of the image to be processed and the distortion strength corresponding to each distortion type can also be obtained through a deep learning model or a neural network model.
Step S204, if the distortion types of the image to be processed comprise a plurality of types, determining the final distortion type of the image to be processed according to the distortion intensity of each type of distortion.
When the distortion types of the image to be processed are various, the dominant distortion type needs to be determined as the final distortion type. The final distortion type may be one type or multiple types, for example, the final distortion type may be image blur, image blur and image compression noise, and the distortion intensity of other distortion types in the image to be processed, except the final distortion type, is usually small, that is, the distortion intensity has little or no influence on the image distortion.
In a specific implementation, when determining the final distortion type of the image to be processed according to the distortion strength of each distortion type, the method can be implemented in one or two of the following manners:
and determining the distortion type with the maximum distortion intensity as the final distortion type of the image to be processed. In a specific implementation, the distortion types may be sorted according to the distortion strength of each distortion type, and one of the distortion types with the largest distortion strength is selected as the final distortion type of the image to be processed.
And determining the distortion type with the distortion intensity higher than the preset intensity threshold value as the final distortion type of the image to be processed.
The preset intensity threshold value can be set according to the requirements of a user, when the preset intensity threshold value is set to be higher, the number of the final distortion types which are possibly determined is less, and when the preset intensity threshold value is set to be lower, the number of the final distortion types which are possibly determined is more. In a specific implementation, all distortion types with distortion intensity higher than the preset intensity threshold may be determined as the final distortion type of the image to be processed, and it may also be understood that the final distortion type of at least one image to be processed may be determined in the above manner two.
And step S206, performing distortion restoration processing on the image to be processed by the distortion restoration mode corresponding to the final distortion type of the image to be processed to obtain a restored image after restoration of the image to be processed.
In a specific implementation, the distortion repairing processing needs to be performed on the image to be processed based on the distortion repairing mode corresponding to the final distortion type of the image to be processed, that is, for different distortion types, different distortion repairing modes need to be adopted for performing the distortion repairing processing on the image to be processed. Usually, the distortion restoration method to be adopted for each distortion type is preset, and a traditional image distortion processing algorithm or a pre-trained deep learning model can be adopted. If the final distortion type is one, distortion repairing treatment can be carried out only according to the distortion repairing mode corresponding to the distortion type; if the final distortion types are multiple, distortion repairing processing needs to be performed on the image to be processed by sequentially calling distortion repairing modes corresponding to the multiple distortion types so as to eliminate the distortion of the image to be processed, for example, image blurring or noise is removed.
And S208, performing super-resolution processing on the repaired image through the pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed.
Firstly, performing quality analysis on an image to be processed to obtain a distortion type of the image to be processed and distortion intensity corresponding to the distortion type; if the distortion types of the image to be processed comprise a plurality of types, determining the final distortion type of the image to be processed according to the distortion intensity of each type of distortion; then, performing distortion restoration processing on the image to be processed through a distortion restoration mode corresponding to the final distortion type of the image to be processed to obtain a restored image after restoration of the image to be processed; and performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed. The super-resolution network model in the method can perform super-resolution processing on images of any distortion type, and different network models do not need to be trained respectively aiming at images of different distortion types in advance, so that the generalization of the model is improved, and meanwhile, the method is applicable to images of different distortion types in any scene.
The embodiment of the invention also provides another super-resolution method of the image, which is realized on the basis of the method of the embodiment; the method mainly describes a specific process (realized by the following step S306) of performing super-resolution processing on a repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to an image to be processed, as shown in fig. 3, the method comprises the following steps:
step S302, determining the distortion parameter of the image to be processed.
And step S304, performing distortion restoration processing on the image to be processed according to the distortion parameters to obtain a restored image of the image to be processed after restoration.
And S306, inputting the repaired image into a pre-trained super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
During specific implementation, the repaired image corresponding to the image to be processed can be input into the pre-trained super-resolution network model for super-resolution processing, and the super-resolution image corresponding to the image to be processed is output. The super-resolution network model can be obtained by training through the following steps 10-13:
and step 10, determining a sample image based on a preset sample set.
The preset sample set comprises a large number of sample images, and the sample images can be images in a natural scene, images in a monitoring scene or images in other scenes; the sample set also includes a super-resolution image corresponding to each sample image.
And step 11, determining the distortion parameters of the sample image.
The distortion parameters comprise distortion types, and when the method is specifically implemented, quality analysis can be performed on the sample image to obtain the distortion types of the sample image and distortion intensities corresponding to the distortion types; if the distortion type of the sample image includes a plurality of types, the final distortion type of the sample image is determined according to the distortion intensity of each type of distortion, which can be seen in the above steps S202-S204.
And step 12, performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image.
In specific implementation, the sample image may be subjected to distortion restoration processing in a distortion restoration manner corresponding to the distortion type of the sample image, so as to eliminate the distortion of the sample image itself, and obtain a restored image of the sample image, where the restored image of the sample image is usually an image with little distortion or no distortion.
And step 13, inputting the repaired image of the sample image into a preset initial network model so as to train the initial network model to obtain a super-resolution network model.
The initial network model can be a deep learning model or a neural network model. During specific implementation, the restored image of the sample image is input into the initial network model to obtain an output result, a model loss value is calculated according to the output result and the super-resolution image of the sample image, and the step of determining the sample image based on a preset sample set is continuously executed until the initial network model is converged to obtain the super-resolution network model. In the training process of the model, the repaired image of the sample image is used as the input of the initial network model, the super-resolution image of the sample image is used as the target output, and the initial network model is continuously trained until the initial network model converges.
The super-resolution method of the image comprises the steps of firstly determining distortion parameters of an image to be processed; then according to the distortion parameter, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed; and then inputting the repaired image into a pre-trained super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed. According to the method, only the repaired image of the image to be processed can be input into the super-resolution network model to obtain the super-resolution image of the image to be processed, only one super-resolution network model needs to be trained, and the cost of model training is reduced.
The embodiment of the invention also provides another super-resolution method of the image, which is realized on the basis of the method of the embodiment; the method mainly describes a specific process (realized by the following step S406) of performing super-resolution processing on a repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to an image to be processed, as shown in fig. 4, the method comprises the following steps:
in step S402, a distortion parameter of the image to be processed is determined.
And S404, performing distortion restoration processing on the image to be processed according to the distortion parameters to obtain a restored image of the image to be processed after restoration.
And step S406, inputting the repaired image and the image to be processed into a pre-trained super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
During specific implementation, the image to be processed and the repaired image corresponding to the image to be processed can be input into a pre-trained super-resolution network model for super-resolution processing, and the super-resolution image corresponding to the image to be processed is output. The super-resolution network model is obtained by training through the following steps 20-23:
step 20, determining a sample image based on a preset sample set.
And step 21, determining the distortion parameters of the sample image.
And step 22, performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image.
In the specific implementation, the specific implementation manner of the steps 20-22 is the same as that of the steps 10-12, and is not described herein again.
And 23, inputting the repaired image of the sample image and the sample image into a preset initial network model to train the initial network model to obtain a super-resolution network model.
The initial network model can be a deep learning model or a neural network model. In specific implementation, the sample image and the restored image of the sample image are jointly input into the initial network model to obtain an output result, a model loss value is calculated according to the output result and the super-resolution image of the sample image, and the step of determining the sample image based on a preset sample set is continuously executed until the initial network model converges to obtain the super-resolution network model. In the training process of the model, the sample image and the repaired image of the sample image are used as the input of the initial network model, the super-resolution image of the sample image is used as the target output, and the initial network model is continuously trained until the initial network model converges.
The super-resolution network model is different from the super-resolution network model used in fig. 3 only in the input layer, and the structures of other network layers are the same.
The super-resolution method of the image comprises the steps of firstly determining distortion parameters of an image to be processed; then according to the distortion parameter, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed; and then inputting the image to be processed and the repaired image of the image to be processed into a pre-trained super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed. The method obtains the super-resolution image of the image to be processed based on the image to be processed and the repaired image of the image to be processed, can improve the precision of the super-resolution image, and only needs to train one super-resolution network model without training a plurality of network models for each distortion type, thereby reducing the cost of model training.
Corresponding to the embodiment of the super-resolution method for the image, the embodiment of the invention also provides a super-resolution device for the image, as shown in fig. 5, the device comprises:
and a distortion parameter determining module 50, configured to determine a distortion parameter of the image to be processed.
And the distortion repairing module 51 is configured to perform distortion repairing processing on the image to be processed according to the distortion parameters to obtain a repaired image after the image to be processed is repaired.
The super-resolution processing module 52 is configured to perform super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image.
The super-resolution device of the image firstly determines the distortion parameter of the image to be processed; then according to the distortion parameter, carrying out distortion restoration processing on the image to be processed to obtain a restored image of the image to be processed; then, performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of the preset sample image, the sample image is obtained according to restored image training of the sample image. The method comprises the steps of carrying out distortion restoration processing on an image to be processed according to distortion parameters of the image to be processed, and then carrying out super-resolution processing on the restored image of the image to be processed through a super-resolution network model to obtain a super-resolution image of the image to be processed; meanwhile, the method does not need manual intervention, can automatically obtain the super-resolution image of the image to be processed, and saves the labor cost.
Specifically, the distortion parameter includes a distortion type; the distortion parameter determination module 50 includes: the quality analysis unit is used for carrying out quality analysis on the image to be processed to obtain the distortion type of the image to be processed and the distortion intensity corresponding to the distortion type; and the distortion type determining unit is used for determining the final distortion type of the image to be processed according to the distortion intensity of each distortion type if the distortion types of the image to be processed comprise a plurality of types.
Further, the distortion type determination unit is configured to: determining the distortion type with the maximum distortion intensity as the final distortion type of the image to be processed; or determining the distortion type with the distortion intensity higher than a preset intensity threshold value as the final distortion type of the image to be processed.
Specifically, the distortion parameter includes a distortion type; the distortion repairing module 51 is configured to: and performing distortion restoration processing on the image to be processed by a distortion restoration mode corresponding to the distortion type of the image to be processed to obtain a restored image after restoration of the image to be processed.
Further, the super-resolution processing module 52 is configured to: and inputting the repaired image into a super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
Specifically, the apparatus further includes a first network training module, configured to: determining a sample image based on a preset sample set; determining a distortion parameter of the sample image; performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image; and inputting the repaired image of the sample image into a preset initial network model so as to train the initial network model and obtain a super-resolution network model.
Further, the super-resolution processing module 52 is configured to: and inputting the repaired image and the image to be processed into a super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
Specifically, the apparatus further includes a second network training module, configured to: determining a sample image based on a preset sample set; determining a distortion parameter of the sample image; performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image; and inputting the repaired image of the sample image and the sample image into a preset initial network model so as to train the initial network model and obtain a super-resolution network model.
The super-resolution device for images provided by the embodiment of the present invention has the same implementation principle and technical effect as the foregoing method embodiments, and for brief description, reference may be made to the corresponding contents in the foregoing method embodiments for the part where the apparatus embodiments are not mentioned.
The embodiment of the present invention also provides a server, which is shown in fig. 6 and includes a processor 101 and a memory 100, where the memory 100 stores machine executable instructions capable of being executed by the processor, and the processor 101 executes the machine executable instructions to implement the super resolution method for the image.
Further, the server shown in fig. 6 further includes a bus 102 and a communication interface 103, and the processor 101, the communication interface 103 and the memory 100 are connected through the bus 102.
The memory 100 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The processor 101 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 101. The processor 101 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and completes the steps of the method of the foregoing embodiment in combination with the hardware thereof.
Embodiments of the present invention also provide a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement a super-resolution method of the above-described image.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the apparatus and/or the electronic device described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (11)
1. A method for super-resolution of an image, the method comprising:
determining a distortion parameter of an image to be processed;
performing distortion restoration processing on the image to be processed according to the distortion parameters to obtain a restored image of the image to be processed after restoration;
performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of a preset sample image, the sample image is obtained according to restored image training of the sample image.
2. The method of claim 1, wherein the distortion parameters comprise a distortion type; the step of determining the distortion parameter of the image to be processed comprises the following steps:
performing quality analysis on the image to be processed to obtain a distortion type of the image to be processed and distortion intensity corresponding to the distortion type;
and if the distortion types of the image to be processed comprise a plurality of types, determining the final distortion type of the image to be processed according to the distortion strength of each type of distortion.
3. The method according to claim 2, wherein the step of determining a final distortion type of the image to be processed according to the distortion strength of each distortion type comprises:
determining the distortion type with the maximum distortion intensity as the final distortion type of the image to be processed;
or, determining the distortion type with the distortion intensity higher than a preset intensity threshold value as the final distortion type of the image to be processed.
4. The method of claim 1, wherein the distortion parameters comprise a distortion type; the step of performing distortion restoration processing on the image to be processed according to the distortion parameter to obtain a restored image after restoration of the image to be processed includes:
and performing distortion restoration processing on the image to be processed by a distortion restoration mode corresponding to the distortion type of the image to be processed to obtain a restored image after the image to be processed is restored.
5. The method according to claim 1, wherein the step of performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed comprises:
and inputting the repaired image into the super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
6. The method of claim 5, wherein the super-resolution network model is trained by:
determining a sample image based on a preset sample set;
determining a distortion parameter for the sample image;
performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image;
inputting the repaired image of the sample image into a preset initial network model so as to train the initial network model and obtain the super-resolution network model.
7. The method according to claim 1, wherein the step of performing super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed comprises:
and inputting the repaired image and the image to be processed into the super-resolution network model, and outputting a super-resolution image corresponding to the image to be processed.
8. The method of claim 7, wherein the super-resolution network model is trained by:
determining a sample image based on a preset sample set;
determining a distortion parameter for the sample image;
performing distortion restoration processing on the sample image according to the distortion parameters of the sample image to obtain a restored image of the sample image;
inputting the repaired image of the sample image and the sample image into a preset initial network model so as to train the initial network model and obtain the super-resolution network model.
9. An apparatus for super-resolution of an image, the apparatus comprising:
the distortion parameter determining module is used for determining the distortion parameter of the image to be processed;
the distortion restoration module is used for carrying out distortion restoration processing on the image to be processed according to the distortion parameters to obtain a restored image after the image to be processed is restored;
the super-resolution processing module is used for carrying out super-resolution processing on the repaired image through a pre-trained super-resolution network model to obtain a super-resolution image corresponding to the image to be processed; the super-resolution network model comprises the following steps: and after distortion restoration processing is carried out on the sample image based on the distortion parameters of a preset sample image, the sample image is obtained according to restored image training of the sample image.
10. A server comprising a processor and a memory, the memory storing machine executable instructions executable by the processor, the processor executing the machine executable instructions to implement the super resolution method of images of any one of claims 1 to 8.
11. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to implement a super resolution method of images as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010623605.1A CN111784580A (en) | 2020-06-30 | 2020-06-30 | Super-resolution method and device for image and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010623605.1A CN111784580A (en) | 2020-06-30 | 2020-06-30 | Super-resolution method and device for image and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111784580A true CN111784580A (en) | 2020-10-16 |
Family
ID=72760892
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010623605.1A Pending CN111784580A (en) | 2020-06-30 | 2020-06-30 | Super-resolution method and device for image and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111784580A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972021A (en) * | 2022-04-13 | 2022-08-30 | 北京字节跳动网络技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN116309160A (en) * | 2023-03-10 | 2023-06-23 | 北京百度网讯科技有限公司 | Image resolution restoration method, device, equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103475876A (en) * | 2013-08-27 | 2013-12-25 | 北京工业大学 | Learning-based low-bit-rate compression image super-resolution reconstruction method |
| CN107194879A (en) * | 2017-06-26 | 2017-09-22 | 司马大大（北京）智能系统有限公司 | Super resolution ratio reconstruction method, device and electronic equipment |
| WO2018119565A1 (en) * | 2016-12-26 | 2018-07-05 | 深圳大学 | Method and device for correcting and reconstructing a barrel distorted image |
| CN109215028A (en) * | 2018-11-06 | 2019-01-15 | 福州大学 | A kind of multiple-objection optimization image quality measure method based on convolutional neural networks |
| CN109285119A (en) * | 2018-10-23 | 2019-01-29 | 百度在线网络技术（北京）有限公司 | Super resolution image generation method and device |
| WO2019112084A1 (en) * | 2017-12-06 | 2019-06-13 | 한국과학기술원 | Method for removing compression distortion by using cnn |
| CN109919920A (en) * | 2019-02-25 | 2019-06-21 | 厦门大学 | The full reference of unified structure and non-reference picture quality appraisement method |
| US20190294961A1 (en) * | 2018-03-23 | 2019-09-26 | Abbyy Production Llc | Generative augmentation of image data |
| CN110782445A (en) * | 2019-10-25 | 2020-02-11 | 北京华捷艾米科技有限公司 | No-reference image quality evaluation method and system |
| CN111105357A (en) * | 2018-10-25 | 2020-05-05 | 杭州海康威视数字技术股份有限公司 | Distortion removing method and device for distorted image and electronic equipment |
-
2020
- 2020-06-30 CN CN202010623605.1A patent/CN111784580A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103475876A (en) * | 2013-08-27 | 2013-12-25 | 北京工业大学 | Learning-based low-bit-rate compression image super-resolution reconstruction method |
| WO2018119565A1 (en) * | 2016-12-26 | 2018-07-05 | 深圳大学 | Method and device for correcting and reconstructing a barrel distorted image |
| CN107194879A (en) * | 2017-06-26 | 2017-09-22 | 司马大大（北京）智能系统有限公司 | Super resolution ratio reconstruction method, device and electronic equipment |
| WO2019112084A1 (en) * | 2017-12-06 | 2019-06-13 | 한국과학기술원 | Method for removing compression distortion by using cnn |
| US20190294961A1 (en) * | 2018-03-23 | 2019-09-26 | Abbyy Production Llc | Generative augmentation of image data |
| CN109285119A (en) * | 2018-10-23 | 2019-01-29 | 百度在线网络技术（北京）有限公司 | Super resolution image generation method and device |
| CN111105357A (en) * | 2018-10-25 | 2020-05-05 | 杭州海康威视数字技术股份有限公司 | Distortion removing method and device for distorted image and electronic equipment |
| CN109215028A (en) * | 2018-11-06 | 2019-01-15 | 福州大学 | A kind of multiple-objection optimization image quality measure method based on convolutional neural networks |
| CN109919920A (en) * | 2019-02-25 | 2019-06-21 | 厦门大学 | The full reference of unified structure and non-reference picture quality appraisement method |
| CN110782445A (en) * | 2019-10-25 | 2020-02-11 | 北京华捷艾米科技有限公司 | No-reference image quality evaluation method and system |
Non-Patent Citations (4)
| Title |
|---|
| QINGLONG CHANG 等: "Deep learning based image Super-resolution for nonlinear lens distortions", 《NEUROCOMPUTING》, 31 January 2018 (2018-01-31), pages 969 - 982 * |
| 杨伟: "车载采集城市街景图像的超分辨率重建", 《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》, no. 3, 15 March 2018 (2018-03-15), pages 035 - 141 * |
| 杨蕊: "动态序列遥感图像超分辨率重建技术研究", 《中国博士学位论文全文数据库 工程科技Ⅱ辑》, no. 5, 15 May 2020 (2020-05-15), pages 028 - 8 * |
| 高志荣 等: "一种基于神经网络的人脸图像超分辨率重构算法", 《中南民族大学学报(自然科学版)》, no. 1, 15 March 2016 (2016-03-15), pages 114 - 118 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972021A (en) * | 2022-04-13 | 2022-08-30 | 北京字节跳动网络技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN116309160A (en) * | 2023-03-10 | 2023-06-23 | 北京百度网讯科技有限公司 | Image resolution restoration method, device, equipment and storage medium |
| CN116309160B (en) * | 2023-03-10 | 2024-04-12 | 北京百度网讯科技有限公司 | Image resolution restoration method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109872304B (en) | Image defect detection method and device, electronic device and storage medium | |
| CN111028171B (en) | Method, device and server for determining noise level of image | |
| US20190294931A1 (en) | Systems and Methods for Generative Ensemble Networks | |
| CN111784580A (en) | Super-resolution method and device for image and server | |
| CN110796624B (en) | Image generation method and device and electronic equipment | |
| CN110223244B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN112862753B (en) | Noise intensity estimation method and device and electronic equipment | |
| CN111696064B (en) | Image processing method, device, electronic equipment and computer readable medium | |
| CN112614066A (en) | Image restoration method and device and electronic equipment | |
| Zhang et al. | Reduced reference image quality assessment based on statistics of edge | |
| CN111415304A (en) | Underwater vision enhancement method and device based on cascade deep network | |
| CN111105376A (en) | Single-exposure high-dynamic-range image generation method based on double-branch neural network | |
| CN112801536A (en) | Image processing method and device and electronic equipment | |
| CN111031359B (en) | Video playing method and device, electronic equipment and computer readable storage medium | |
| CN111787300B (en) | VR video processing method and device and electronic equipment | |
| CN111524060B (en) | System, method, storage medium and device for blurring portrait background in real time | |
| Safonov et al. | Adaptive sharpening of photos | |
| CN112669240B (en) | High-definition image restoration method and device, electronic equipment and storage medium | |
| WO2021093718A1 (en) | Video processing method, video repair method, apparatus and device | |
| CN111612766A (en) | Image quality evaluation method and device and electronic equipment | |
| CN114494931A (en) | Intelligent classification processing method and system for video image faults | |
| CN113808021A (en) | Image processing method and device, image processing model training method and device, and electronic equipment | |
| CN112533024A (en) | Face video processing method and device and storage medium | |
| CN112348788A (en) | Image quality evaluation method and device, electronic equipment and storage medium | |
| CN114119377A (en) | Image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |