CN114626985A

CN114626985A - Image super-resolution method, image super-resolution model training method and device

Info

Publication number: CN114626985A
Application number: CN202210269560.1A
Authority: CN
Inventors: 磯部駿
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2022-03-18
Filing date: 2022-03-18
Publication date: 2022-06-14

Abstract

The disclosure relates to an image super-resolution method, an image super-resolution model training device and an image super-resolution model training medium. The image super-resolution method comprises the following steps: acquiring an original resolution image; inputting the original resolution image into a trained image super-resolution model to obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution; and taking the target resolution image as a super-resolution processing result of the original resolution image. The method and the device can improve the model precision of the image super-resolution model, and further improve the processing effect of image details.

Description

Image super-resolution method, image super-resolution model training method and device

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image super-resolution method, an image super-resolution model training device, an electronic device, and a storage medium.

Background

With the development of image processing technology, an image super-resolution processing technology for reconstructing an observed low-resolution image into a corresponding high-resolution image to improve the resolution of an original image appears, and the technology can meet the requirement of high-definition display of the image, so that the technology has important application value in the fields of monitoring equipment, satellite images, medical images and the like.

In the related art, currently, super-resolution processing of images is mainly implemented by using a pre-trained super-resolution model, a low-resolution picture is input into the super-resolution model, a corresponding high-resolution image is output by the super-resolution model, the model generally inputs the low-resolution image into the model during training, a prediction result of the super-resolution processing on the low-resolution image is output by the model, and a loss function is constructed by using the prediction result and a real high-resolution image, so that training of the model is implemented. However, image details are often in a missing state in a low-resolution image, so that the image details are not effectively processed by performing image super-resolution processing on a directly trained image super-resolution model.

Disclosure of Invention

The present disclosure provides an image super-resolution method, an image super-resolution model training method, an apparatus, an electronic device, and a storage medium, so as to at least solve the problem in the related art that image details are not well processed by image super-resolution processing. The technical scheme of the disclosure is as follows:

according to a first aspect of the embodiments of the present disclosure, there is provided an image super-resolution method, including:

acquiring an original resolution image;

inputting the original resolution image into a trained image super-resolution model to obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; wherein, the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution;

and taking the target resolution image as a super-resolution processing result of the original resolution image.

In an exemplary embodiment, the image super-resolution method further includes: acquiring an original resolution sample image and a target resolution sample image matched with the original resolution sample image; obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image; inputting the original resolution sample image into an image super-resolution model to be trained to obtain an intermediate resolution predicted image corresponding to the original resolution sample image and a target resolution predicted image; and training the image super-resolution model to be trained by using the difference between the intermediate-resolution predicted image and the intermediate-resolution sample image and the difference between the target-resolution predicted image and the target-resolution sample image to obtain the trained image super-resolution model.

In an exemplary embodiment, the training the image super-resolution model to be trained by using the difference between the intermediate-resolution prediction image and the intermediate-resolution sample image and the difference between the target-resolution prediction image and the target-resolution sample image to obtain a trained image super-resolution model includes: obtaining a first loss value based on the intermediate resolution sample image and the intermediate resolution predicted image, and obtaining a second loss value based on the target resolution sample image and the target resolution predicted image; and training the image super-resolution model to be trained by utilizing the first loss value and the second loss value to obtain the trained image super-resolution model.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; the method for inputting the original resolution sample image into the image super-resolution model to be trained to obtain the intermediate resolution predicted image corresponding to the original resolution sample image and the target resolution predicted image comprises the following steps: inputting the original resolution sample image into the first image super-resolution sub-model to obtain a first image characteristic; obtaining the intermediate-resolution prediction image based on the first image characteristic; inputting the first image characteristic into the second image super-resolution sub-model to obtain a second image characteristic; and obtaining the target resolution prediction image based on the second image characteristic.

In an exemplary embodiment, training the super-resolution image model to be trained by using the first loss value and the second loss value includes: and training the first image super-resolution sub-model by using the first loss value and the second loss value, and training the second image super-resolution sub-model by using the second loss value.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; the step of inputting the original resolution sample image into the first image super-resolution sub-model to obtain a first image characteristic comprises the following steps: acquiring a current first image super-resolution sub-model; under the condition that the current first image super-resolution sub-model is the first one of the plurality of first image super-resolution sub-models, inputting the original resolution sample image into the current first image super-resolution sub-model to obtain a first image characteristic corresponding to the current first image super-resolution sub-model; and under the condition that the current first image super-resolution sub-model is not the first image super-resolution sub-model, inputting the first image characteristic corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model into the current first image super-resolution sub-model to obtain the first image characteristic corresponding to the current first image super-resolution sub-model.

In an exemplary embodiment, the inputting the first image feature into the second image super-resolution sub-model to obtain a second image feature includes: and inputting the first image characteristic corresponding to the last first image super-resolution sub-model in the plurality of first image super-resolution sub-models into the second image super-resolution sub-model to obtain the second image characteristic.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; the obtaining the intermediate-resolution predicted image based on the first image feature includes: obtaining intermediate resolution predicted images respectively corresponding to each current first image super-resolution sub-model based on the first image characteristics corresponding to each current first image super-resolution sub-model; deriving a first loss value based on the intermediate-resolution sample image and the intermediate-resolution predicted image, comprising: and obtaining a first loss value corresponding to each current first image super-resolution sub-model respectively based on the intermediate-resolution predicted image corresponding to each current first image super-resolution sub-model respectively and the intermediate-resolution sample image matched with each current first image super-resolution sub-model respectively.

In an exemplary embodiment, the obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image includes: performing up-sampling processing on the original resolution sample image to obtain an up-sampled image corresponding to the original resolution sample image; acquiring a first influence weight of the up-sampling image for an intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image for the intermediate resolution sample image to be generated; and performing weighting processing on the up-sampling image and the target resolution sample image according to the first influence weight and the second influence weight to obtain the intermediate resolution sample image.

In an exemplary embodiment, the number of the intermediate resolution sample images to be generated is plural; the obtaining a first influence weight of the up-sampling image on an intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image on the intermediate resolution sample image to be generated includes: obtaining sample image sequence of a current intermediate resolution sample image to be generated in a plurality of intermediate resolution sample images to be generated; and determining a first influence weight of the upsampled image for the current intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image for the current intermediate resolution sample image to be generated according to the sample image sequence.

According to a second aspect of the embodiments of the present disclosure, there is provided an image super-resolution model training method, including:

acquiring an original resolution sample image and a target resolution sample image matched with the original resolution sample image, wherein the original resolution is lower than the target resolution;

obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image; wherein an intermediate resolution is between the original resolution and the target resolution;

inputting the original resolution sample image into an image super-resolution model to be trained to obtain an intermediate resolution predicted image corresponding to the original resolution sample image and a target resolution predicted image;

obtaining a first loss value based on the intermediate resolution sample image and the intermediate resolution predicted image, and obtaining a second loss value based on the target resolution sample image and the target resolution predicted image;

and training the image super-resolution model to be trained by using the first loss value and the second loss value to obtain the trained image super-resolution model.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; the method for inputting the original resolution sample image into the image super-resolution model to be trained to obtain the intermediate resolution predicted image corresponding to the original resolution sample image and the target resolution predicted image comprises the following steps: inputting the original resolution sample image into the first image super-resolution sub-model to obtain a first image characteristic; obtaining the intermediate-resolution prediction image based on the first image characteristic; inputting the first image characteristic into the second image super-resolution sub-model to obtain a second image characteristic; and obtaining the target resolution prediction image based on the second image characteristics.

In an exemplary embodiment, the training the super-resolution image model to be trained by using the first loss value and the second loss value includes: and training the first image super-resolution sub-model by using the first loss value and the second loss value, and training the second image super-resolution sub-model by using the second loss value.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; the obtaining the intermediate-resolution predicted image based on the first image feature includes: obtaining intermediate resolution prediction images respectively corresponding to the current first image super-resolution submodels based on the first image characteristics corresponding to the current first image super-resolution submodels; deriving a first loss value based on the intermediate-resolution sample image and the intermediate-resolution predicted image, comprising: and obtaining a first loss value corresponding to each current first image super-resolution sub-model based on the intermediate resolution prediction image corresponding to each current first image super-resolution sub-model and the intermediate resolution sample image matched with each current first image super-resolution sub-model.

According to a third aspect of the embodiments of the present disclosure, there is provided an image super-resolution device including:

an original image acquisition unit configured to perform acquisition of an original resolution image;

a target image acquisition unit configured to input the original resolution image into a trained image super-resolution model to obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; wherein, the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution;

a processing result acquisition unit configured to perform super-resolution processing of the target resolution image as the original resolution image.

In an exemplary embodiment, the image super-resolution device further includes: a super-resolution model training unit configured to perform acquiring an original resolution sample image and a target resolution sample image matched with the original resolution sample image; obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image; inputting the original resolution sample image into an image super-resolution model to be trained to obtain an intermediate resolution predicted image corresponding to the original resolution sample image and a target resolution predicted image; and training the image super-resolution model to be trained by utilizing the difference between the intermediate-resolution predicted image and the intermediate-resolution sample image and the difference between the target-resolution predicted image and the target-resolution sample image to obtain the trained image super-resolution model.

In an exemplary embodiment, the super-resolution model training unit is further configured to perform deriving a first loss value based on the intermediate resolution sample image and the intermediate resolution prediction image, and deriving a second loss value based on the target resolution sample image and the target resolution prediction image; and training the image super-resolution model to be trained by utilizing the first loss value and the second loss value to obtain the trained image super-resolution model.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; the super-resolution model training unit is further configured to input the original resolution sample image into the first image super-resolution sub-model to obtain a first image feature; obtaining the intermediate-resolution prediction image based on the first image characteristic; inputting the first image characteristic into the second image super-resolution sub-model to obtain a second image characteristic; and obtaining the target resolution prediction image based on the second image characteristics.

In an exemplary embodiment, the super-resolution model training unit is further configured to perform training the first image super-resolution sub-model using the first loss value and the second loss value, and training the second image super-resolution sub-model using the second loss value.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; the super-resolution model training unit is further configured to execute acquiring a current first image super-resolution sub-model; under the condition that the current first image super-resolution sub-model is the first one of the plurality of first image super-resolution sub-models, inputting the original resolution sample image into the current first image super-resolution sub-model to obtain a first image characteristic corresponding to the current first image super-resolution sub-model; and under the condition that the current first image super-resolution sub-model is not the first image super-resolution sub-model, inputting the first image characteristic corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model into the current first image super-resolution sub-model to obtain the first image characteristic corresponding to the current first image super-resolution sub-model.

In an exemplary embodiment, the super-resolution model training unit is further configured to perform inputting a first image feature corresponding to a last one of the plurality of first image super-resolution submodels into the second image super-resolution submodel, so as to obtain the second image feature.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; the super-resolution model training unit is further configured to execute first image characteristics corresponding to each current first image super-resolution sub-model based on the first image characteristics to obtain intermediate resolution prediction images corresponding to each current first image super-resolution sub-model; and the system is configured to execute an intermediate resolution prediction image respectively corresponding to each current first image super-resolution sub model and an intermediate resolution sample image respectively matched with each current first image super-resolution sub model, so as to obtain a first loss value respectively corresponding to each current first image super-resolution sub model.

In an exemplary embodiment, the super-resolution model training unit is further configured to perform an upsampling process on the original resolution sample image, so as to obtain an upsampled image corresponding to the original resolution sample image; acquiring a first influence weight of the up-sampling image for an intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image for the intermediate resolution sample image to be generated; and performing weighting processing on the up-sampling image and the target resolution sample image according to the first influence weight and the second influence weight to obtain the intermediate resolution sample image.

In an exemplary embodiment, the number of the intermediate resolution sample images to be generated is plural; the super-resolution model training unit is further configured to perform sample image ordering of the current intermediate resolution sample image to be generated in a plurality of intermediate resolution sample images to be generated; and determining a first influence weight of the upsampled image for the current intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image for the current intermediate resolution sample image to be generated according to the sample image sequence.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an image super-resolution model training apparatus, including:

a sample image acquisition unit configured to perform acquisition of an original resolution sample image and a target resolution sample image matched with the original resolution sample image, wherein the original resolution is lower than the target resolution;

an intermediate sample obtaining unit configured to obtain an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image; wherein an intermediate resolution is between the original resolution and the target resolution;

a prediction image obtaining unit configured to perform input of the original resolution sample image into an image super-resolution model to be trained, to obtain an intermediate resolution prediction image corresponding to the original resolution sample image, and a target resolution prediction image;

a model loss obtaining unit configured to obtain a first loss value based on the intermediate resolution sample image and the intermediate resolution prediction image, and obtain a second loss value based on the target resolution sample image and the target resolution prediction image;

and the model training unit is configured to train the image super-resolution model to be trained by using the first loss value and the second loss value to obtain a trained image super-resolution model.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; the prediction image acquisition unit is further configured to input the original resolution sample image into the first image super-resolution sub-model to obtain a first image characteristic; obtaining the intermediate-resolution predicted image based on the first image characteristic; inputting the first image characteristic into the second image super-resolution sub-model to obtain a second image characteristic; and obtaining the target resolution prediction image based on the second image characteristic.

In an exemplary embodiment, the model training unit is further configured to perform training the first image super resolution sub-model using the first loss value and the second loss value, and training the second image super resolution sub-model using the second loss value.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; the prediction image acquisition unit is further configured to execute acquisition of a current first image super-resolution sub-model; under the condition that the current first image super-resolution sub-model is the first one of the plurality of first image super-resolution sub-models, inputting the original resolution sample image into the current first image super-resolution sub-model to obtain a first image characteristic corresponding to the current first image super-resolution sub-model; and under the condition that the current first image super-resolution sub-model is not the first image super-resolution sub-model, inputting the first image characteristic corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model into the current first image super-resolution sub-model to obtain the first image characteristic corresponding to the current first image super-resolution sub-model.

In an exemplary embodiment, the prediction image obtaining unit is further configured to perform inputting the first image feature corresponding to the last one of the plurality of first image super-resolution sub-models into the second image super-resolution sub-model, so as to obtain the second image feature.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; the prediction image acquisition unit is further configured to execute first image characteristics corresponding to each current first image super-resolution sub-model to obtain intermediate resolution prediction images respectively corresponding to each current first image super-resolution sub-model; the model loss obtaining unit is further configured to execute an intermediate resolution prediction image corresponding to each current first image super-resolution sub-model and an intermediate resolution sample image matched with each current first image super-resolution sub-model respectively, so as to obtain a first loss value corresponding to each current first image super-resolution sub-model respectively.

According to a fifth aspect of embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image super-resolution method according to any embodiment of the first aspect or the image super-resolution model training method according to any embodiment of the second aspect.

According to a sixth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions, when executed by a processor of an electronic device, enable the electronic device to perform the image super-resolution method according to any one of the embodiments of the first aspect, or the image super-resolution model training method according to any one of the embodiments of the second aspect.

According to a seventh aspect of embodiments of the present disclosure, there is provided a computer program product, which includes instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the image super-resolution method according to any one of the embodiments of the first aspect, or the image super-resolution model training method according to any one of the embodiments of the second aspect.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:

obtaining an original resolution image; inputting the original resolution image into a trained image super-resolution model to obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution; and taking the target resolution image as a super-resolution processing result of the original resolution image. When the image super-resolution model is trained, besides the original resolution sample image and the target resolution sample image, the intermediate resolution sample image with the resolution between the original resolution and the target resolution is further introduced for model training, so that the model precision of the image super-resolution model is improved, and the processing effect of image details is further improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.

FIG. 1 is a flow diagram illustrating a method for super-resolution of an image according to an exemplary embodiment.

FIG. 2 is a flow diagram illustrating a training image super-resolution model according to an exemplary embodiment.

Fig. 3 is a flow diagram illustrating the derivation of a predictive image by an image super-resolution model according to an exemplary embodiment.

FIG. 4 is a flowchart illustrating obtaining a first image feature according to an example embodiment.

FIG. 5 is a flow diagram illustrating obtaining an intermediate resolution sample image according to an exemplary embodiment.

FIG. 6 is a flowchart illustrating a method of image super-resolution model training according to an exemplary embodiment.

FIG. 7 is a flowchart illustrating a method of training a video super-resolution model according to an exemplary embodiment.

Fig. 8 is a schematic diagram illustrating a serial configuration of residual modules in accordance with an exemplary embodiment.

Fig. 9 is a block diagram illustrating an image super-resolution device according to an exemplary embodiment.

FIG. 10 is a block diagram illustrating an image super-resolution model training apparatus according to an exemplary embodiment.

FIG. 11 is a block diagram illustrating an electronic device in accordance with an example embodiment.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in the present disclosure are both information and data that are authorized by the user or sufficiently authorized by various parties.

Fig. 1 is a flowchart illustrating an image super-resolution method according to an exemplary embodiment, where the image super-resolution method is used in a terminal, as shown in fig. 1, and includes the following steps.

In step S101, an original resolution image is acquired.

The original resolution image refers to an original image which needs to be subjected to image super-resolution processing, and the resolution of the image is the original resolution. Specifically, when image super-resolution processing is required for an image at the original resolution, a user may trigger a processing request for performing the image super-resolution processing to his terminal, and the terminal may respond to the request to obtain the image at the original resolution for which the image super-resolution processing is required.

In step S102, inputting the original resolution image into the trained image super-resolution model to obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution.

The target resolution image refers to an image super-resolution model which is trained, and the image at the target resolution is output after the super-resolution processing is performed on the original resolution image, wherein the target resolution is higher than the original resolution. The image super-resolution model refers to a model for performing super-resolution processing on an image, the model can increase the resolution of the image from an original resolution to a target resolution, and the model can be trained from a sample image at the original resolution, i.e., an original resolution sample image, an image at the target resolution corresponding to the sample image, i.e., a target resolution sample image, and a sample image at an intermediate resolution, i.e., an intermediate resolution sample image, obtained from the original resolution sample image and the target resolution sample image, wherein the intermediate resolution is a resolution between the original resolution and the target resolution.

For example, the original resolution may be resolution a, and the target resolution may be resolution B, wherein the resolution A is less than the resolution B, when the training of the image super-resolution model is carried out, the terminal can acquire an image A1 of the sample image 1 corresponding to the resolution A as an original resolution image and can also acquire an image B1 of the image corresponding to the resolution B as a target resolution image, and the terminal may, based on the sample image a1 and the image B1, obtain the image C1 of which image 1 corresponds to the resolution C as an intermediate-resolution sample image, and the resolution C is between the resolution a and the resolution B, and when training of the image super-resolution model is performed, the training of the image super-resolution model can be performed using image a1, image B1, and image C1 instead of using only image a1 and image B1.

After the image super-resolution model is trained, the original resolution image which needs to be subjected to image super-resolution processing can be further input into the image super-resolution model which is trained, and the image super-resolution model outputs the original resolution image to obtain a target resolution image corresponding to the original resolution image.

In step S103, the target resolution image is used as a super-resolution processing result of the original resolution image.

Finally, the terminal may further use the target resolution image of the image super-resolution model as a processing result of performing super-resolution processing on the original resolution image in step S102.

In the image super-resolution method, an original resolution image is obtained; inputting the original resolution image into a trained image super-resolution model to obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution; and taking the target resolution image as a super-resolution processing result of the original resolution image. When the image super-resolution model is trained, besides the original resolution sample image and the target resolution sample image, the intermediate resolution sample image with the resolution between the original resolution and the target resolution is further introduced for model training, so that the model precision of the image super-resolution model is improved, and the processing effect of image details is further improved.

In an exemplary embodiment, as shown in fig. 2, the image super-resolution method may further include:

in step S201, an original resolution sample image and a target resolution sample image matching the original resolution sample image are acquired.

The original resolution sample image is a sample image at an original resolution, and the target resolution sample image is a sample image at a target resolution. Specifically, the terminal may acquire a certain image as a sample image, and take an image of the sample image corresponding to the original resolution as an original resolution sample image, and take an image of the sample image corresponding to the target resolution as a target resolution sample image.

In step S202, an intermediate resolution sample image is obtained from the original resolution sample image and the target resolution sample image.

The intermediate resolution sample image refers to a sample image at an intermediate resolution, and the resolution of the intermediate resolution sample image is greater than the resolution of the original resolution sample image and less than the resolution of the target resolution sample image. Specifically, after obtaining the original resolution sample image and the target resolution sample image corresponding to a certain sample image, the terminal may further obtain an intermediate resolution sample image corresponding to the intermediate resolution of the sample image based on the original resolution sample image and the target resolution sample image.

In step S203, the original resolution sample image is input into the image super-resolution model to be trained, and an intermediate resolution prediction image corresponding to the original resolution sample image and a target resolution prediction image are obtained.

The model can output and obtain a corresponding predicted image at the intermediate resolution and a predicted image at the target resolution based on an input original resolution sample image. The intermediate resolution prediction image refers to a prediction image which is obtained by outputting the image super-resolution model to be trained and is at the intermediate resolution and corresponding to the input original resolution sample image, and the target resolution prediction image refers to a prediction image which is obtained by training the model and is at the target resolution and corresponding to the input original resolution sample image. Specifically, the terminal may input the original resolution sample image acquired in step S201 into an image super-resolution model that needs to be trained, and output a prediction image of the original resolution sample image corresponding to an intermediate resolution, that is, an intermediate resolution prediction image, and a prediction image of the original resolution sample image corresponding to a target resolution, that is, a target resolution prediction image.

In step S204, the image super-resolution model to be trained is trained by using the difference between the intermediate-resolution prediction image and the intermediate-resolution sample image and the difference between the target-resolution prediction image and the target-resolution sample image, so as to obtain a trained image super-resolution model.

Finally, the terminal may further calculate an image difference between the intermediate resolution predicted image output by the image super-resolution model and the intermediate resolution sample image obtained in step S202, and calculate an image difference between the target resolution predicted image output by the image super-resolution model and the target resolution sample image obtained in step S201, so as to perform model training using the above difference, thereby obtaining a trained image super-resolution model.

In this embodiment, the terminal may acquire an original resolution sample image at an original resolution and a target resolution sample image at a target resolution corresponding to the original resolution sample image, and may obtain an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image, and further train the image super-resolution model by using the original resolution sample image, the intermediate resolution sample image, and the target resolution sample image, so that the model accuracy of the trained image super-resolution model may be improved.

Further, step S204 may further include: obtaining a first loss value based on the intermediate resolution sample image and the intermediate resolution predicted image, and obtaining a second loss value based on the target resolution sample image and the target resolution predicted image; and training the image super-resolution model to be trained by utilizing the first loss value and the second loss value to obtain the trained image super-resolution model.

The first loss value refers to a loss value calculated by a first loss function constructed in advance from the intermediate-resolution sample image and the intermediate-resolution prediction image, and the second loss value is a loss value calculated by a second loss function constructed in advance from the target-resolution sample image and the target-resolution prediction image, and the first loss function and the second loss function may have the same expression form or different expression forms.

Specifically, after the terminal obtains the intermediate resolution sample image, the intermediate resolution prediction image, the target resolution sample image, and the target resolution prediction image, a first loss value corresponding to the trained image super-resolution model may be obtained according to the intermediate resolution sample image and the intermediate resolution prediction image, and a second loss value corresponding to the model may be obtained according to the target resolution sample image and the target resolution prediction image, so that training of the image super-resolution model may be implemented based on the first loss value and the second loss value, and the model parameters of the image super-resolution model may be updated, thereby obtaining the trained image super-resolution model.

In this embodiment, the terminal may obtain the first loss value based on the intermediate resolution sample image and the intermediate resolution prediction image, and obtain the second loss value based on the target resolution sample image and the target resolution prediction image, so that the training of the image super-resolution model may be implemented by using the first loss value and the second loss value, and the accuracy of the trained image super-resolution model may be further improved.

Further, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; as shown in fig. 3, step S203 may further include:

in step S301, the original resolution sample image is input into a first image super-resolution sub-model to obtain a first image feature.

In this embodiment, the image super-resolution model to be trained may be composed of two sub-network models, which are a first image super-resolution model and a second image super-resolution sub-model, respectively, where the first image super-resolution sub-model is used to obtain an intermediate resolution sample image, and the second image super-resolution sub-model is used to obtain a target resolution prediction image. In this embodiment, the terminal may input the original resolution sample image into a first image super-resolution sub-model in the image super-resolution model to be trained, and obtain the corresponding first image feature through the first image super-resolution sub-model.

In step S302, an intermediate resolution prediction image is obtained based on the first image feature.

After the first image feature is obtained in step S301, the terminal may further use the extracted first image feature to obtain a corresponding intermediate resolution predicted image, for example, the first image feature may be input to an upper sampling layer to perform pixel rearrangement, so as to obtain a predicted image corresponding to the first image feature as the intermediate resolution predicted image.

In step S303, the first image feature is input into the second image super-resolution sub-model to obtain a second image feature.

And the second image feature refers to an image feature extracted by the second image super-resolution sub-model, and after the first image feature is obtained by outputting the first image super-resolution sub-model in step S301, the output first image feature can be further used as an input of the second image super-resolution sub-model, and further feature extraction is performed by the second image super-resolution sub-model, so as to obtain a corresponding second image feature.

In step S304, a target resolution prediction image is obtained based on the second image feature.

After the second image feature is obtained in step S303, the terminal may further use the extracted second image feature to obtain a corresponding predicted image with the target resolution, for example, the second image feature may be input to an upper sampling layer to perform pixel rearrangement, so as to obtain a predicted image corresponding to the second image feature as the predicted image with the target resolution.

In this embodiment, the image super-resolution model may be composed of two sub-network models, respectively a first image super-resolution sub-model and a second image super-resolution sub-model, wherein the first image super-resolution sub-model may obtain a first image feature based on an input original resolution sample image and may further obtain a corresponding intermediate resolution prediction image based on the first image feature, and the second image super-resolution sub-model may further perform feature extraction based on the first image feature output by the first image super-resolution sub-model to obtain a second image feature, thereby obtaining a target resolution prediction image using the second image feature, obtaining the intermediate resolution prediction image by dividing the image super-resolution model into the first image super-resolution sub-model and the second image super-resolution sub-model and using output results of the sub-models, and the target resolution predicted image, the accuracy of the obtained intermediate resolution predicted image and the target resolution predicted image can be improved.

In addition, the training of the image super-resolution model to be trained by using the first loss value and the second loss value may further include: and training the first image super-resolution sub-model by using the first loss value and the second loss value, and training the second image super-resolution sub-model by using the second loss value.

In this embodiment, after obtaining the first loss value and the second loss value, the first loss value and the second loss value may be used to train the first image super-resolution sub-model and the second image super-resolution sub-model, respectively. The terminal can perform gradient update on the first image super-resolution sub-model by using the first loss value and the second loss value to realize the training of the first image super-resolution sub-model, and the second image super-resolution sub-model is obtained by training on the second loss value, or performs gradient update on the second image super-resolution sub-model by using the second loss value to realize the training of the second image super-resolution sub-model.

In this embodiment, after the first loss value and the second loss value are obtained, the first image super-resolution sub-model may be trained based on the first loss value and the second loss value, and the second image super-resolution sub-model may be trained based on the second loss value, so that the model accuracy of the trained first image super-resolution sub-model and the trained second image super-resolution sub-model may be improved.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; as shown in fig. 4, step S301 may further include:

in step S401, a current first image super-resolution sub-model is acquired.

In this embodiment, the number of the first image super-resolution submodels for outputting the intermediate-resolution prediction image may be plural, and for example, may include: the first image super-resolution submodel 1, the first image super-resolution submodel 2 and the first image super-resolution submodel 3 are respectively used for outputting 3 kinds of intermediate resolution predicted images with different resolutions, wherein the first image super-resolution submodel 1 is used for outputting the intermediate resolution predicted image with the resolution of 1, the first image super-resolution submodel 2 is used for outputting the intermediate resolution predicted image with the resolution of 2, the first image super-resolution submodel 3 is used for outputting the intermediate resolution predicted image with the resolution of 3, and the current first image super-resolution submodel is any one of the first image super-resolution submodels, can be the first image super-resolution submodel 1, and can also be the first image super-resolution submodel 2 or the first image super-resolution submodel 3.

In step S402, when the current first image super-resolution sub-model is the first one of the plurality of first image super-resolution sub-models, the original resolution sample image is input into the current first image super-resolution sub-model, and the first image feature corresponding to the current first image super-resolution sub-model is obtained.

In this embodiment, the plurality of first image super-resolution submodels have a fixed network connection order, for example, the first image super-resolution submodel 1 is connected to the first image super-resolution submodel 2, and the first image super-resolution submodel 2 is connected to the first image super-resolution submodel 3, so that the terminal can determine the connection order of each first image super-resolution submodel. If the current first image super-resolution sub-model obtained in step S401 is the first image super-resolution sub-model, for example, the first image super-resolution sub-model 1, the terminal may input the original resolution sample image to the current first image super-resolution sub-model, so as to obtain the first image feature corresponding to the current first image super-resolution sub-model. That is, the original resolution sample image may be input into the first image super-resolution sub-model 1, and the corresponding first image feature 1 may be obtained by the first image super-resolution sub-model 1.

In step S403, when the current first image super-resolution sub-model is not the first image super-resolution sub-model, the first image feature corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model is input into the current first image super-resolution sub-model, so as to obtain the first image feature corresponding to the current first image super-resolution sub-model.

If the current first image super-resolution sub-model obtained in step S401 is not the first image super-resolution sub-model, the first image feature corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model may be further input into the current first image super-resolution sub-model, so as to obtain the first image feature corresponding to the current first image super-resolution sub-model.

For example, if the current first image super-resolution sub-model is the first image super-resolution sub-model 2, the terminal may input the first image features 1 obtained by the first image super-resolution sub-model 1 into the first image super-resolution sub-model 2, so that the corresponding first image features 2 are obtained by the first image super-resolution sub-model 2, and if the current first image super-resolution sub-model is the first image super-resolution sub-model 3, the terminal may input the first image features 2 obtained by the first image super-resolution sub-model 2 into the first image super-resolution sub-model 3, so that the corresponding first image features 3 are obtained by the first image super-resolution sub-model 3.

In this embodiment, the number of the first image super-resolution submodels may be multiple, in this embodiment, multiple corresponding first image features may be extracted and obtained through multiple first image super-resolution submodels, and if the first image feature is not the first image super-resolution submodel, the first image feature output by the previous first image super-resolution submodel may be used as an input to obtain a corresponding first image feature.

Further, step S303 may further include: and inputting the first image characteristic corresponding to the last first image super-resolution sub-model in the plurality of first image super-resolution sub-models into the second image super-resolution sub-model to obtain a second image characteristic.

Specifically, if the number of the first image super-resolution submodels is multiple, when the first image features are input to the second image super-resolution submodel, the first image super-resolution submodel with the last connection order may be determined from the multiple first image super-resolution submodels, and the first image features extracted by the first image super-resolution submodel may be input to the second image super-resolution submodel, so that the corresponding second image features may be obtained by the second image super-resolution submodel.

For example, the plurality of first image super-resolution submodels may include: the terminal comprises a first image super-resolution sub-model 1, a first image super-resolution sub-model 2 and a first image super-resolution sub-model 3, wherein the connection sequence of the first image super-resolution sub-model 3 is the last one, and then the terminal can take the first image features 3 extracted by the first image super-resolution sub-model 3 as the input of the second image super-resolution sub-model, so that the second image features are obtained.

In this embodiment, when the number of the first image super-resolution submodels is multiple, the first image feature extracted by the last first image super-resolution submodel may be used as the input of the second image super-resolution submodel, so that the accuracy of the extracted second image feature may be further improved.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; step S302 may further include: obtaining intermediate resolution predicted images respectively corresponding to the current first image super-resolution submodels based on the first image characteristics corresponding to the current first image super-resolution submodels; deriving the first loss value based on the intermediate resolution sample image and the intermediate resolution predicted image may further comprise: and obtaining a first loss value corresponding to each current first image super-resolution sub-model based on the intermediate resolution prediction image corresponding to each current first image super-resolution sub-model and the intermediate resolution sample image matched with each current first image super-resolution sub-model.

In this embodiment, the number of the intermediate resolution sample images obtained by the terminal by obtaining the original resolution sample image and the target resolution sample image is plural, and the intermediate resolution sample images respectively correspond to different intermediate resolutions of the sample images, and each intermediate resolution sample image is also matched with each first image super-resolution sub-model. For example, the first image super-resolution sub-model may include: the first image super-resolution submodel 1, the first image super-resolution submodel 2, and the first image super-resolution submodel 3, the intermediate resolution sample image obtained by the terminal may also include: an intermediate resolution sample image 1, an intermediate resolution sample image 2, and an intermediate resolution sample image 3. Meanwhile, the number of the intermediate resolution prediction images obtained by the terminal through the first image super-resolution submodel can be multiple, that is, the intermediate resolution prediction images corresponding to the first image super-resolution submodels are obtained by respectively using the first image features corresponding to the first image super-resolution submodels. Then, the terminal may further obtain a first loss value corresponding to each first image super-resolution sub-model according to the intermediate-resolution prediction image corresponding to each first image super-resolution sub-model and the intermediate-resolution sample image matched with each first image super-resolution sub-model.

For example, the first image super-resolution sub-model may include: the system comprises a first image super-resolution sub-model 1, a first image super-resolution sub-model 2 and a first image super-resolution sub-model 3, wherein the first image super-resolution sub-model is matched with an intermediate resolution sample image 1, an intermediate resolution sample image 2 and an intermediate resolution sample image 3 respectively. After obtaining the first image features corresponding to each first image super-resolution sub-model, the terminal can further obtain intermediate resolution predicted images corresponding to the first image features, that is, an intermediate resolution predicted image 1, an intermediate resolution predicted image 2 and an intermediate resolution predicted image 3. The terminal may further calculate a first loss value of each of the intermediate resolution prediction image 1 and the intermediate resolution sample image 1 as a first loss value corresponding to the first image super-resolution sub-model 1, and calculate a first loss value of each of the intermediate resolution prediction image 2 and the intermediate resolution sample image 2 as a first loss value corresponding to the first image super-resolution sub-model 2, and calculate a first loss value of each of the intermediate resolution prediction image 3 and the intermediate resolution sample image 3 as a first loss value corresponding to the first image super-resolution sub-model 3.

In this embodiment, by setting the intermediate resolution sample image matched with each first image super-resolution sub-model, the first loss value corresponding to each first image super-resolution sub-model can be calculated, and by the above method, the training for the plurality of first image super-resolution sub-models can be realized, so that the precision of the trained image super-resolution model is further improved.

In an exemplary embodiment, as shown in fig. 5, step S202 may further include:

in step S501, an up-sampling process is performed on the original resolution sample image, and an up-sampled image corresponding to the original resolution sample image is obtained.

In this embodiment, since the image size of the original resolution sample image is different from the image size of the target resolution sample image, in the process of acquiring the intermediate resolution sample image, the original resolution sample image may be enlarged to the image size of the target resolution sample image through the upsampling process, so as to obtain the upsampled image.

In step S502, a first influence weight of the up-sampled image on the intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image on the intermediate resolution sample image to be generated are obtained.

The intermediate resolution sample image to be generated refers to an intermediate resolution sample image that needs to be obtained, the first influence weight refers to an influence weight of an up-sampling image on the intermediate resolution sample image, the second influence weight refers to an influence weight of a target resolution sample image on the intermediate resolution sample image, and the first influence weight and the second influence weight may be preset. Specifically, the terminal may obtain a first influence weight and a second influence weight which are set in advance, so as to obtain an influence weight of the up-sampled image on the intermediate resolution sample image to be generated and an influence weight of the target resolution sample image on the intermediate resolution sample image to be generated, respectively.

In step S503, the up-sampled image and the target resolution sample image are weighted according to the first influence weight and the second influence weight, and an intermediate resolution sample image is obtained.

Finally, after the first and second influence weights are obtained, the up-sampled image and the target resolution sample image may be weighted based on the first and second influence weights, and an intermediate resolution sample image may be output.

In this embodiment, an upsampled image corresponding to the original resolution sample image may be obtained in an upsampling manner, and the upsampled image and the target resolution sample image may be subjected to weighting processing based on a first influence weight of the upsampled image on the intermediate resolution sample image and a second influence weight of the target resolution sample image on the intermediate resolution sample image, so that an accurate intermediate resolution sample image may be obtained.

Further, the number of the intermediate resolution sample images to be generated is plural; step S502 may further include: obtaining sample image sequencing of a current intermediate resolution sample image to be generated in a plurality of intermediate resolution sample images to be generated; and determining a first influence weight of the up-sampling image for the current intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image for the current intermediate resolution sample image to be generated according to the sample image sequencing.

In this embodiment, the number of the intermediate resolution sample images to be generated may be multiple, each of the intermediate resolution sample images corresponds to a different intermediate resolution, the current intermediate resolution sample image to be generated refers to any one of the intermediate resolution sample images to be generated, the sample image sorting refers to the current intermediate resolution sample image to be generated, and the sorting in the multiple current intermediate resolution sample images to be generated may be, for example, sorting according to a size relationship of the intermediate resolutions. In this embodiment, the upsampled image has different first influence weights for different generated intermediate resolution sample images, and the target resolution sample image may also have different second influence weights for different generated intermediate resolution sample images, and the first influence weights and the second influence weights may be adapted to the intermediate resolution sample images in a plurality of current intermediate resolution sample images to be generated. The terminal obtains the sample image ranking corresponding to each sample image to be generated with the current intermediate resolution, and then obtains each first influence weight and each second influence weight which are adaptive to each other based on the sample image ranking.

For example, for an intermediate resolution sample image to be generated 1, the terminal may be preset with a corresponding first influence weight as the first influence weight 1 and a corresponding second influence weight as the second influence weight 1, and for an intermediate resolution sample image to be generated 2, the terminal may be preset with a corresponding first influence weight as the first influence weight 2 and a corresponding second influence weight as the second influence weight 2, so that the terminal may sort, based on sample images of each current intermediate resolution sample image to be generated in a plurality of intermediate resolution sample images to be generated, so as to obtain a first influence weight and a second influence weight matched with each current intermediate resolution sample image to be generated.

In this embodiment, the terminal may determine the corresponding first influence weight and second influence weight based on the order of each intermediate-resolution sample image to be generated in the plurality of intermediate-resolution sample images to be generated, so that the plurality of generated intermediate-resolution sample images to be generated may have different intermediate resolutions, and the generation efficiency of the plurality of intermediate-resolution sample images may be further improved.

Fig. 6 is a flowchart illustrating an image super-resolution model training method according to an exemplary embodiment, where the image super-resolution model training method is used in a terminal, as shown in fig. 6, and includes the following steps.

In step S601, an original resolution sample image and a target resolution sample image matched with the original resolution sample image are acquired, wherein the original resolution is lower than the target resolution.

The original resolution sample image refers to a sample image at an original resolution, and the target resolution sample image refers to an image at a target resolution corresponding to the sample image, wherein the original resolution is lower than the target resolution. In this embodiment, the terminal may acquire an image of a certain sample image corresponding to the original resolution as an original resolution sample image, and may acquire an image of the sample image corresponding to the target resolution as a target resolution sample image.

In step S602, an intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; wherein the intermediate resolution is between the original resolution and the target resolution.

The intermediate resolution sample image refers to an image corresponding to the sample image and having an intermediate resolution, and the resolution of the intermediate resolution sample image is greater than that of the original resolution sample image and less than that of the target resolution sample image. Specifically, after obtaining the original resolution sample image and the target resolution sample image corresponding to a certain sample image, the terminal may further obtain an intermediate resolution sample image corresponding to the intermediate resolution of the sample image based on the original resolution sample image and the target resolution sample image.

In step S603, the original resolution sample image is input into the image super-resolution model to be trained, and an intermediate resolution prediction image corresponding to the original resolution sample image and a target resolution prediction image are obtained.

The model can output and obtain a corresponding predicted image at the intermediate resolution and a predicted image at the target resolution based on the input original resolution sample image. The intermediate resolution prediction image refers to a prediction image which is obtained by outputting the image super-resolution model to be trained and is at the intermediate resolution and corresponding to the input original resolution sample image, and the target resolution prediction image refers to a prediction image which is obtained by training the model and is at the target resolution and corresponding to the input original resolution sample image. Specifically, the terminal may input the original resolution sample image acquired in step S601 into an image super-resolution model that needs to be trained, and output by the model a predicted image in which the original resolution sample image corresponds to an intermediate resolution, that is, an intermediate resolution predicted image, and a predicted image in which the original resolution sample image corresponds to a target resolution, that is, a target resolution predicted image.

In step S604, a first loss value is obtained based on the intermediate-resolution sample image and the intermediate-resolution prediction image, and a second loss value is obtained based on the target-resolution sample image and the target-resolution prediction image.

The first loss value refers to a loss value calculated from the intermediate-resolution sample image and the intermediate-resolution prediction image by a first loss function constructed in advance, and the second loss value refers to a loss value calculated from the target-resolution sample image and the target-resolution prediction image by a second loss function constructed in advance. Specifically, after the terminal obtains the intermediate resolution sample image, the intermediate resolution prediction image, the target resolution sample image, and the target resolution prediction image, a first loss value corresponding to the trained image super-resolution model may be obtained according to the intermediate resolution sample image and the intermediate resolution prediction image, and a second loss value corresponding to the model may be obtained according to the target resolution sample image and the target resolution prediction image.

In step S605, the image super-resolution model to be trained is trained by using the first loss value and the second loss value, so as to obtain a trained image super-resolution model.

Finally, the terminal may further implement training of the image super-resolution model based on the first loss value and the second loss value obtained in step S604 to update the model parameters of the image super-resolution model, so as to obtain the trained image super-resolution model for performing super-resolution processing on the image.

In the image super-resolution model training method, an original resolution sample image and a target resolution sample image matched with the original resolution sample image are obtained, wherein the original resolution is lower than the target resolution; obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image; wherein the intermediate resolution is between the original resolution and the target resolution; inputting the original resolution sample image into an image super-resolution model to be trained to obtain an intermediate resolution predicted image corresponding to the original resolution sample image and a target resolution predicted image; obtaining a first loss value based on the intermediate resolution sample image and the intermediate resolution predicted image, and obtaining a second loss value based on the target resolution sample image and the target resolution predicted image; and training the image super-resolution model to be trained by using the first loss value and the second loss value to obtain the trained image super-resolution model. When the image super-resolution model is trained, besides the original resolution sample image and the target resolution sample image, the intermediate resolution sample image with the resolution between the original resolution and the target resolution is further introduced for model training, so that the model precision of the image super-resolution model is improved, and the processing effect of image details is further improved.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; step S603 may further include: inputting an original resolution sample image into a first image super-resolution sub-model to obtain a first image characteristic; obtaining an intermediate-resolution predicted image based on the first image characteristics; inputting the first image characteristics into the second image super-resolution sub-model to obtain second image characteristics; and obtaining the target resolution prediction image based on the second image characteristics.

In this embodiment, the image super-resolution model to be trained may be composed of two sub-network models, which are a first image super-resolution model and a second image super-resolution sub-model, respectively, where the first image super-resolution sub-model is used to obtain an intermediate resolution sample image, and the second image super-resolution sub-model is used to obtain a target resolution prediction image.

In this embodiment, the terminal may input the original resolution sample image into a first image super-resolution sub-model in the image super-resolution model to be trained, and obtain a corresponding first image feature through the first image super-resolution sub-model, and then may further obtain a corresponding intermediate resolution predicted image by using the extracted first image feature, for example, the first image feature may be input to an upper sampling layer to perform pixel rearrangement, so as to obtain a predicted image corresponding to the first image feature, which is used as the intermediate resolution predicted image.

The second image feature refers to an image feature extracted by the second image super-resolution sub-model, after the first image feature is obtained by outputting the first image feature by the first image super-resolution sub-model, the output first image feature can be further used as input of the second image super-resolution sub-model, further feature extraction is performed by the second image super-resolution sub-model, so that the corresponding second image feature is obtained, and further, the extracted second image feature is used for obtaining a corresponding target resolution prediction image, for example, the second image feature is input to an upper sampling layer for pixel rearrangement, so that a prediction image corresponding to the second image feature is obtained and used as the target resolution prediction image.

In this embodiment, the image super-resolution model may be composed of two sub-network models, respectively a first image super-resolution sub-model and a second image super-resolution sub-model, wherein the first image super-resolution sub-model may obtain first image features based on the input original resolution sample image and may further obtain a corresponding intermediate resolution predicted image based on the first image features, and the second image super-resolution sub-model may further extract features based on the first image features output by the first image super-resolution sub-model to obtain second image features, so as to obtain a target resolution predicted image by using the second image features, the image super-resolution model is divided into the first image super-resolution sub-model and the second image super-resolution sub-model, and the intermediate resolution predicted images are respectively obtained by using the output results of the sub-models, and the target resolution predicted image, the accuracy of the obtained intermediate resolution predicted image and the target resolution predicted image can be improved.

In an exemplary embodiment, step S605 may further include: and training the first image super-resolution submodel by using the first loss value and the second loss value, and training the second image super-resolution submodel by using the second loss value.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; inputting the original resolution sample image into a first image super-resolution sub-model to obtain a first image feature, which may further include: acquiring a current first image super-resolution sub-model; under the condition that the current first image super-resolution sub-model is the first one of the plurality of first image super-resolution sub-models, inputting the original resolution sample image into the current first image super-resolution sub-model to obtain a first image characteristic corresponding to the current first image super-resolution sub-model; and under the condition that the current first image super-resolution sub-model is not the first image super-resolution sub-model, inputting the first image characteristic corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model into the current first image super-resolution sub-model to obtain the first image characteristic corresponding to the current first image super-resolution sub-model.

In this embodiment, the number of the first image super-resolution submodels for outputting the intermediate-resolution prediction image may be plural, and the plural first image super-resolution submodels are respectively used for outputting the intermediate-resolution prediction images of 3 different resolutions, and have a fixed network connection order. If the obtained current first image super-resolution sub-model is the first image super-resolution sub-model, the terminal can input the original resolution sample image to the current first image super-resolution sub-model, so that the first image characteristic corresponding to the current first image super-resolution sub-model is obtained. And if the obtained current first image super-resolution sub-model is not the first image super-resolution sub-model, the first image characteristics corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model can be further input into the current first image super-resolution sub-model, so that the first image characteristics corresponding to the current first image super-resolution sub-model are obtained.

In an exemplary embodiment, inputting the first image feature into a second image super-resolution sub-model to obtain a second image feature includes: and inputting the first image characteristic corresponding to the last first image super-resolution sub-model in the plurality of first image super-resolution sub-models into the second image super-resolution sub-model to obtain a second image characteristic.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; deriving the intermediate resolution predicted image based on the first image feature may further comprise: obtaining intermediate resolution predicted images respectively corresponding to the current first image super-resolution submodels based on the first image characteristics corresponding to the current first image super-resolution submodels; step S604 may further include: and obtaining a first loss value corresponding to each current first image super-resolution sub-model based on the intermediate resolution prediction image corresponding to each current first image super-resolution sub-model and the intermediate resolution sample image matched with each current first image super-resolution sub-model.

In this embodiment, the number of the intermediate resolution sample images obtained by the terminal by obtaining the original resolution sample image and the target resolution sample image is plural, and the intermediate resolution sample images respectively correspond to different intermediate resolutions of the sample images, and each intermediate resolution sample image is also matched with each first image super-resolution sub-model. Meanwhile, the number of the intermediate resolution prediction images obtained by the terminal through the first image super-resolution submodel can be multiple, that is, the intermediate resolution prediction images corresponding to the first image super-resolution submodels are obtained by respectively using the first image features corresponding to the first image super-resolution submodels. Then, the terminal may further obtain a first loss value corresponding to each first image super-resolution sub-model according to the intermediate-resolution prediction image corresponding to each first image super-resolution sub-model and the intermediate-resolution sample image matched with each first image super-resolution sub-model.

In an exemplary embodiment, a training method of a video super-resolution model is provided, which reduces the difficulty of each reconstruction by introducing a progressive loss function, thereby reducing the size of a super-resolution space. Under the progressive solution space optimization, the video super-resolution network can obtain better results under the approximate same scale. The specific flow of the scheme can be shown in fig. 7, and comprises the following steps:

the video super-resolution network is first split into two sub-networks. Specifically, the video super-resolution network is a serial structure of N layers of residual modules, each residual module comprises two 2D convolutions, an activation function ReLU, and a structure diagram of fig. 8.

Where each 2D convolutional layer employs a 3x3 convolutional kernel. This embodiment will split the N residual modules into two sub-networks with depth N/2, and each network update is constrained by a different target image. For the first sub-network, the constraint goal is to interpolate the up-sampled low-resolution image and weight it with the true high-resolution image by a weight of 0.5. The weighted result will be lost in some detail than the true high resolution image, but better than the direct low resolution image. For super-resolution, different regions have different degrees of difficulty. Therefore, the flat area does not need to be particularly clear in detail, and the detail is expected to be clear for the difficult area, so that the network can pay more attention to the reconstruction of the difficult area during optimization. The output of the first subnetwork is fed into the second subnetwork, and the obtained result is supervised by a real high resolution image. It can be seen that at the first optimization of the network, the solution space is significantly smaller than the solution space directly with the true high resolution, and then progressively expands into a larger solution space. I.e. it is equivalent to split a difficult problem into two steps, first solving the first problem and then solving the next problem.

In addition, the optimization goal of the first sub-network is a weighting of 0.5, and the weighting can be changed according to different training scenarios. The number of second step-ups may vary. For example, as the network grows in size, more incremental steps may be used. If more passes are used, the weighted weight becomes larger. For example, when 3 passes are used, the weight of the first pass is suggested to be 0.3, the second pass is suggested to be 0.6, and the third pass is a true high resolution image.

The embodiment provides a method for constructing a super-resolution model by using a progressive loss function, which can enable a neural network to obtain a better result under the condition of approximately unchanged scale. In the embodiment, the original optimization problem is decomposed into multiple optimization processes, and different optimization processes are supervised by different optimization targets, so that the network can pass through the process from easy to difficult during learning, and the super-resolution processing effect can be improved.

It should be understood that although the various steps in the flowcharts of fig. a-Y are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least some of the steps in fig. a-Y may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps or stages.

It is understood that the same/similar parts between the embodiments of the method described above in this specification can be referred to each other, and each embodiment focuses on the differences from the other embodiments, and it is sufficient that the relevant points are referred to the descriptions of the other method embodiments.

Fig. 9 is a block diagram illustrating an image super-resolution device according to an exemplary embodiment. Referring to fig. 9, the apparatus includes an original image acquisition unit 901, a target image acquisition unit 902, and a processing result acquisition unit 903.

An original image acquisition unit 901 configured to perform acquisition of an original resolution image;

a target image obtaining unit 902, configured to input the original resolution image into the trained image super-resolution model, and obtain a target resolution image matched with the original resolution image; the trained image super-resolution model is obtained by training based on an original resolution sample image, a target resolution sample image matched with the original resolution sample image and an intermediate resolution sample image; the original resolution is lower than the target resolution, and the intermediate resolution sample image is obtained according to the original resolution sample image and the target resolution sample image; the intermediate resolution is between the original resolution and the target resolution;

a processing result acquisition unit 903 configured to perform super-resolution processing of the target resolution image as the original resolution image.

In an exemplary embodiment, the image super-resolution device further includes: a super-resolution model training unit configured to perform acquiring an original resolution sample image and a target resolution sample image matched with the original resolution sample image; obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image; inputting the original resolution sample image into an image super-resolution model to be trained to obtain an intermediate resolution predicted image corresponding to the original resolution sample image and a target resolution predicted image; and training the image super-resolution model to be trained by utilizing the difference between the intermediate-resolution prediction image and the intermediate-resolution sample image and the difference between the target-resolution prediction image and the target-resolution sample image to obtain the trained image super-resolution model.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; the super-resolution model training unit is further configured to input the original resolution sample image into a first image super-resolution sub-model to obtain a first image characteristic; obtaining an intermediate-resolution predicted image based on the first image characteristics; inputting the first image characteristics into a second image super-resolution sub-model to obtain second image characteristics; and obtaining a target resolution prediction image based on the second image characteristics.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; a super-resolution model training unit, further configured to execute acquiring a current first image super-resolution sub-model; under the condition that the current first image super-resolution sub-model is the first one of the multiple first image super-resolution sub-models, inputting the original resolution sample image into the current first image super-resolution sub-model to obtain a first image characteristic corresponding to the current first image super-resolution sub-model; and under the condition that the current first image super-resolution sub-model is not the first image super-resolution sub-model, inputting the first image characteristic corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model into the current first image super-resolution sub-model to obtain the first image characteristic corresponding to the current first image super-resolution sub-model.

In an exemplary embodiment, the super-resolution model training unit is further configured to perform input of the first image feature corresponding to the last one of the plurality of first image super-resolution submodels into the second image super-resolution submodel, resulting in the second image feature.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; the super-resolution model training unit is further configured to execute first image characteristics corresponding to each current first image super-resolution sub-model, and obtain intermediate resolution prediction images corresponding to each current first image super-resolution sub-model; and the system is configured to execute an intermediate resolution prediction image respectively corresponding to each current first image super-resolution sub-model and an intermediate resolution sample image respectively matched with each current first image super-resolution sub-model, so as to obtain a first loss value respectively corresponding to each current first image super-resolution sub-model.

In an exemplary embodiment, the super-resolution model training unit is further configured to perform an upsampling process on the original resolution sample image, resulting in an upsampled image corresponding to the original resolution sample image; acquiring a first influence weight of an up-sampling image for an intermediate resolution sample image to be generated and a second influence weight of a target resolution sample image for the intermediate resolution sample image to be generated; and weighting the up-sampling image and the target resolution sample image according to the first influence weight and the second influence weight to obtain an intermediate resolution sample image.

In an exemplary embodiment, the number of intermediate resolution sample images to be generated is plural; a super-resolution model training unit further configured to perform obtaining a sample image ordering of a current intermediate resolution sample image to be generated among a plurality of intermediate resolution sample images to be generated; according to the sample image sorting, determining a first influence weight of the up-sampling image for the current intermediate resolution sample image to be generated and a second influence weight of the target resolution sample image for the current intermediate resolution sample image to be generated.

FIG. 10 is a block diagram illustrating an image super-resolution model training apparatus according to an exemplary embodiment. Referring to fig. 10, the apparatus includes a sample image acquisition unit 1001, an intermediate sample acquisition unit 1002, a prediction image acquisition unit 1003, a model loss acquisition unit 1004, and a model training unit 1005.

A sample image acquisition unit 1001 configured to perform acquisition of an original resolution sample image and a target resolution sample image matched with the original resolution sample image, wherein the original resolution is lower than the target resolution;

an intermediate sample acquiring unit 1002 configured to perform obtaining an intermediate resolution sample image from an original resolution sample image and a target resolution sample image; wherein the intermediate resolution is between the original resolution and the target resolution;

a prediction image obtaining unit 1003 configured to perform input of an original resolution sample image into an image super-resolution model to be trained, obtain an intermediate resolution prediction image corresponding to the original resolution sample image, and obtain a target resolution prediction image;

a model loss obtaining unit 1004 configured to perform deriving a first loss value based on the intermediate-resolution sample image and the intermediate-resolution prediction image, and deriving a second loss value based on the target-resolution sample image and the target-resolution prediction image;

a model training unit 1005 configured to perform training of the image super-resolution model to be trained by using the first loss value and the second loss value, resulting in a trained image super-resolution model.

In an exemplary embodiment, the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model; a prediction image obtaining unit 1003 further configured to perform inputting the original resolution sample image into a first image super-resolution sub-model, resulting in a first image feature; obtaining an intermediate-resolution predicted image based on the first image characteristics; inputting the first image characteristics into a second image super-resolution sub-model to obtain second image characteristics; and obtaining the target resolution prediction image based on the second image characteristics.

In an exemplary embodiment, the model training unit 1005 is further configured to perform training the first image super-resolution sub-model using the first loss value and the second loss value, and training the second image super-resolution sub-model using the second loss value.

In an exemplary embodiment, the number of the first image super-resolution submodels is plural; a prediction image obtaining unit 1003 further configured to perform obtaining a current first image super-resolution sub-model; under the condition that the current first image super-resolution sub-model is the first one of the multiple first image super-resolution sub-models, inputting the original resolution sample image into the current first image super-resolution sub-model to obtain a first image characteristic corresponding to the current first image super-resolution sub-model; and under the condition that the current first image super-resolution sub-model is not the first image super-resolution sub-model, inputting the first image characteristic corresponding to the last first image super-resolution sub-model of the current first image super-resolution sub-model into the current first image super-resolution sub-model to obtain the first image characteristic corresponding to the current first image super-resolution sub-model.

In an exemplary embodiment, the predicted image obtaining unit 1003 is further configured to perform inputting the first image feature corresponding to the last one of the plurality of first image super-resolution sub-models into the second image super-resolution sub-model, and obtain the second image feature.

In an exemplary embodiment, the number of the intermediate resolution sample images is multiple, and each intermediate resolution sample image is respectively matched with each current first image super-resolution sub-model; a prediction image obtaining unit 1003 further configured to perform obtaining an intermediate resolution prediction image corresponding to each current first image super-resolution sub-model based on the first image feature corresponding to each current first image super-resolution sub-model; the model loss obtaining unit 1004 is further configured to execute an intermediate resolution prediction image corresponding to each current first image super-resolution sub-model and an intermediate resolution sample image respectively matched with each current first image super-resolution sub-model, so as to obtain a first loss value corresponding to each current first image super-resolution sub-model.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

FIG. 11 is a block diagram illustrating an electronic device 1100 for image super-resolution or for image super-resolution model training, according to an example embodiment. For example, the electronic device 1100 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a gaming console, a tablet device, a medical device, a fitness device, a personal digital assistant, and so forth.

Referring to fig. 11, electronic device 1100 may include one or more of the following components: processing component 1102, memory 1104, power component 1106, multimedia component 1108, audio component 1110, input/output (I/O) interface 1112, sensor component 1114, and communications component 1116.

The processing component 1102 generally controls the overall operation of the electronic device 1100, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 1102 may include one or more processors 1120 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 1102 may include one or more modules that facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

The memory 1104 is configured to store various types of data to support operations at the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 1104 may be implemented by any type or combination of volatile or non-volatile storage devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

The power supply component 1106 provides power to the various components of the electronic device 1100. The power components 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 1100.

The multimedia component 1108 includes a screen that provides an output interface between the electronic device 1100 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 1100 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 1110 is configured to output and/or input audio signals. For example, the audio component 1110 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 1100 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, audio component 1110 further includes a speaker for outputting audio signals.

The I/O interface 1112 provides an interface between the processing component 1102 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 1114 includes one or more sensors for providing various aspects of state assessment for the electronic device 1100. For example, the sensor assembly 1114 may detect an open/closed state of the electronic device 1100, the relative positioning of components, such as a display and keypad of the electronic device 1100, the sensor assembly 1114 may also detect a change in position of the electronic device 1100 or components of the electronic device 1100, the presence or absence of user contact with the electronic device 1100, orientation or acceleration/deceleration of the device 1100, and a change in temperature of the electronic device 1100. The sensor assembly 1114 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1116 is configured to facilitate wired or wireless communication between the electronic device 1100 and other devices. The electronic device 1100 may access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1116 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 1100 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components for performing the above-described methods.

In an exemplary embodiment, a computer-readable storage medium comprising instructions, such as the memory 1104 comprising instructions, executable by the processor 1120 of the electronic device 1100 to perform the method described above is also provided. For example, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product is also provided, which includes instructions executable by the processor 1120 of the electronic device 1100 to perform the above-described method.

It should be noted that the descriptions of the above-mentioned apparatus, the electronic device, the computer-readable storage medium, the computer program product, and the like according to the method embodiments may also include other embodiments, and specific implementations may refer to the descriptions of the related method embodiments, which are not described in detail herein.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image super-resolution method, comprising:

acquiring an original resolution image;

2. The method of claim 1, further comprising:

acquiring an original resolution sample image and a target resolution sample image matched with the original resolution sample image;

obtaining an intermediate resolution sample image according to the original resolution sample image and the target resolution sample image;

and training the image super-resolution model to be trained by utilizing the difference between the intermediate-resolution predicted image and the intermediate-resolution sample image and the difference between the target-resolution predicted image and the target-resolution sample image to obtain the trained image super-resolution model.

3. The method according to claim 2, wherein the training the image super-resolution model to be trained by using the difference between the intermediate-resolution prediction image and the intermediate-resolution sample image and the difference between the target-resolution prediction image and the target-resolution sample image to obtain a trained image super-resolution model comprises:

and training the image super-resolution model to be trained by utilizing the first loss value and the second loss value to obtain the trained image super-resolution model.

4. The method of claim 3, wherein the image super-resolution model to be trained comprises a first image super-resolution sub-model and a second image super-resolution sub-model;

the method for inputting the original resolution sample image into the image super-resolution model to be trained to obtain the intermediate resolution predicted image corresponding to the original resolution sample image and the target resolution predicted image comprises the following steps:

inputting the original resolution sample image into the first image super-resolution sub-model to obtain a first image characteristic;

obtaining the intermediate-resolution prediction image based on the first image characteristic;

inputting the first image characteristic into the second image super-resolution sub-model to obtain a second image characteristic;

and obtaining the target resolution prediction image based on the second image characteristics.

5. An image super-resolution model training method is characterized by comprising the following steps:

6. An image super-resolution device, comprising:

7. An image super-resolution model training device is characterized by comprising:

8. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the image super-resolution method of any one of claims 1 to 4 or the image super-resolution model training method of claim 5.

9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the image super-resolution method of any one of claims 1 to 4, or the image super-resolution model training method of claim 5.

10. A computer program product comprising instructions therein, which when executed by a processor of an electronic device, enable the electronic device to perform the image super-resolution method of any one of claims 1 to 4, or the image super-resolution model training method of claim 5.