WO2020187042A1 - Image processing method, device and apparatus, and computer readable medium - Google Patents

Abstract

Disclosed are an image processing method and device, comprising: receiving an input image, and processing the input image to determine a first image feature of a first size, a second image feature of a second size and a third image feature of a third size, the first size being less than the second size and the second size being less than the third size; compensating the second image feature by using the first image feature to generate a compensated second image feature of the second size; compensating the third image feature by using the compensated second image feature to generate a compensated third image feature of the third size; and determining an output image based on the compensated second image feature and the compensated third image feature.

Description

Image processing method, device, equipment and computer readable medium

Cross-reference of related literature

This disclosure claims the priority of the Chinese patent application No. 201910209661.8 filed on March 19, 2019, and the content of the above-mentioned Chinese patent application is quoted here in full as a part of this application.

Technical field

The present disclosure relates to the field of image processing, and in particular to a method, device, device, and computer-readable medium for image processing.

Background technique

In the existing image processing method, a recursive manner is used to implement processing for multiple image features of different sizes associated with the input image. However, in image processing algorithms that use recursion, since the results generated by each level of recursion need to be saved for subsequent use, it will occupy a larger memory space.

In addition, in the existing image processing methods that use recursion, high-resolution image information will be transferred to low-resolution image information, which leads to a more complex parameter optimization environment, which makes the results of image processing worse. .

Summary of the invention

In view of the above problems, the present disclosure provides a new image processing method, which can realize the processing of multiple image features of different sizes without using a recursive structure.

According to an aspect of the present disclosure, an image processing method is provided, including: receiving an input image, processing the input image to determine a first image feature of a first size, a second image feature of a second size, and a third image feature. Size of the third image feature, wherein the first size is smaller than the second size, and the second size is smaller than the third size; the first image feature is used to compensate the second image feature to Generating a compensated second image feature of a second size; using the compensated second image feature to compensate the third image feature to generate a compensated third image feature of a third size; and based on the The compensated second image feature or the compensated third image feature determines an output image.

In some embodiments, using the first image feature to compensate the second image feature to generate a compensated second image feature of the second size includes: down-sampling the second image feature to Obtain the down-sampled second image feature of the first size; perform a de-superposition operation on the down-sampled second image feature and the first image feature to generate the first compensated image feature of the first size; The compensation image feature is up-sampled to obtain the up-sampled first compensation image feature of the second size; the up-sampled first compensation image feature and the second image feature are superimposed to generate the second size The second image feature after compensation.

In some embodiments, performing a de-superimposition operation on the down-sampled second image feature and the first image feature includes: performing a subtraction on the down-sampled second image feature and corresponding elements in the first image feature Operation; or perform a convolution operation on the combination of the down-sampled second image feature and the first image feature.

In some embodiments, performing an overlay operation on the up-sampled first compensated image feature and the second image feature includes: performing the up-sampled first compensated image feature and corresponding elements in the second image feature Addition operation.

In some embodiments, using the compensated second image feature to compensate the third image feature to generate the compensated third image feature includes: down-sampling the third image feature to obtain The down-sampled third image feature of the second size; performing a de-superimposition operation on the down-sampled third image feature and the compensated second image feature to generate a second compensated image feature of the second size; The second compensated image feature is up-sampled to obtain an up-sampled second compensated image feature of a third size; an overlay operation is performed on the third image feature and the up-sampled second compensated image feature to generate The compensated third image feature of the third size.

In some embodiments, determining the output image based on the compensated second image feature or the compensated third image feature includes: down-sampling the compensated second image feature to obtain the first size The down-sampling of the compensated second image feature; the down-sampling of the compensated second image feature and the first image feature to perform a de-superimposition operation to generate a third compensated image feature of the first size; Up-sampling the third compensated image feature to obtain an up-sampled third compensated image feature of a second size; executes on the compensated second image feature and the up-sampled third compensated image feature Superimposing operation to generate a further compensated second image feature of the second size; down-sampling the compensated third image feature to obtain the down-sampled compensated third image feature of the second size; The down-sampled compensated third image feature and the further compensated second image feature perform a de-overlapping operation to generate a fourth compensated image feature of the second size; up-sampling the fourth compensated image feature, To obtain the up-sampled fourth compensated image feature of the third size; perform a superposition operation on the compensated third image feature and the up-sampled fourth compensated image feature to generate a further compensated second image feature of the third size Three image features; and generating an output image based on the further compensated second image feature or the further compensated third image feature.

In some embodiments, processing the input image to determine a first image feature of a first size, a second image feature of a second size, and a third image feature of a third size associated with the input image includes : Determine a first input image of a first size, a second input image of a second size, and a third input image of a third size according to the input image, respectively, for the first input image, the second input image and Processing the third input image to determine a first image feature of a first size, a first input image feature of a second size, and a second input image feature of a third size; up-sampling the first image feature, And perform an overlay operation on the first input image feature and the up-sampled first image feature to obtain a second image feature of a second size; up-sampling the second image feature, and the up-sampled The second image feature and the second input image feature perform an overlapping operation to obtain a third image feature of a third size.

In some embodiments, the input image has a first size, and determining the first input image of the first size, the second input image of the second size, and the third input image of the third size according to the input image includes: The input image is determined to be the first input image of the first size; the first input image of the first size is up-sampled to generate the second input image of the second size; the second input image of the second size is up-sampled To generate a third input image of a third size.

In some embodiments, determining the output image based on the compensated second image feature or the compensated third image feature includes: using the first image feature to compensate the compensated second image feature , To generate a further compensated second image feature; use the further compensated second image feature to compensate the compensated third image feature to generate a further compensated third image feature; and based on the further compensation The second image feature or the further compensated third image feature generates an output image.

In some embodiments, using the first image feature to compensate the compensated second image feature to generate a further compensated second image feature includes: down-sampling the compensated second image feature , To obtain the down-sampled and compensated second image feature of the first size; perform a de-superposition operation on the down-sampled, compensated second image feature and the first image feature to generate the first size of the first image feature Three compensated image features; up-sampling the third compensated image feature to obtain the up-sampled third compensated image feature of the second size; the up-sampled third compensated image feature and the compensated first The two image features are superimposed to generate a second image feature of the second size that is further compensated.

In some embodiments, using the further compensated second image feature to compensate the compensated third image feature to generate the further compensated third image feature includes: compensating the compensated third image feature Down-sampling is performed to obtain the down-sampled compensated third image feature of the second size; the down-sampling compensated third image feature and the further compensated second image feature are de-superposed to Generate a fourth compensated image feature of the second size; up-sample the fourth compensated image feature to obtain an up-sampled fourth compensated image feature of the third size; compare the compensated third image feature and The up-sampled fourth compensated image feature performs a superposition operation to generate a further compensated third image feature of a third size.

In some embodiments, the second size is M times the first size, the third size is M times the second size, and M is an integer greater than one.

According to another aspect of the present disclosure, an image processing device is provided, including: a receiving module configured to receive an input image; an image feature processing module configured to process the input image to determine a first image of a first size Feature, a second image feature of a second size, and a third image feature of a third size, wherein the first size is smaller than the second size, and the second size is smaller than the third size; an image feature compensation module, It includes: a first compensation unit configured to compensate the second image feature using the first image feature of the first size to generate a compensated second image feature of the second size; and a second compensation unit, Configured to use the compensated second image feature to compensate the third image feature to generate a compensated third image feature of a third size; and an output module configured to be based on the compensated second The image feature or the compensated third image feature determines the output image.

In some embodiments, the first compensation unit is further configured to: use a down-sampling sub-unit to down-sample the second image feature to obtain a down-sampled second image feature of the first size; The sub-unit performs a de-superimposition operation on the down-sampled second image feature and the first image feature of the first size to generate the first compensated image feature of the first size; the up-sampling sub-unit is used to perform the de-superposition operation on the first compensated image feature Up-sampling is performed to obtain the up-sampled first compensated image feature of the second size; and the overlay subunit is used to perform an overlay operation on the up-sampled first compensated image feature and the second image feature to generate a second The size of the compensated second image feature.

In some embodiments, the de-overlap subunit is further configured to: perform a subtraction operation on the down-sampled second image feature and the corresponding element in the first image feature of the first size; or perform a subtraction operation on the down-sampled The combination of the second image feature and the first image feature of the first size performs a convolution operation.

In some embodiments, the superimposing subunit is further configured to perform an addition operation on corresponding elements in the up-sampled first compensated image feature and the second image feature.

In some embodiments, the second compensation unit is further configured to: use a down-sampling subunit to down-sample the third image feature to obtain a down-sampled third image feature of the second size; The sub-unit performs a de-superimposition operation on the down-sampled third image feature and the compensated second image feature to generate a second compensated image feature of the second size; using an up-sampling sub-unit to perform a de-overlapping operation on the second compensated image The feature is up-sampled to obtain the up-sampled second compensated image feature of the third size; the overlay subunit is used to perform the overlay operation on the third image feature and the up-sampled second compensated image feature to generate a third The third image feature after size compensation.

In some embodiments, the output module is further configured to: use a downsampling subunit to downsample the compensated second image feature to obtain a downsampled compensated second image feature of the first size Use the de-superimposition subunit to perform a de-superimposition operation on the down-sampled compensated second image feature and the first image feature to generate the third compensated image feature of the first size; use the up-sampling sub-unit to The third compensated image feature is up-sampled to obtain the up-sampled third compensated image feature of the second size; the superimposed subunit is used to perform the up-sampling on the compensated second image feature and the up-sampled third compensated image The feature performs a superposition operation to generate a further compensated second image feature of the second size; the down-sampling subunit is used to down-sample the compensated third image feature to obtain the down-sampled compensated second size The third image feature; using the de-overlap subunit to perform a de-overlap operation on the down-sampled compensated third image feature and the further compensated second image feature to generate a fourth compensated image feature of the second size; Up-sampling the fourth compensated image feature using an up-sampling subunit to obtain an up-sampled fourth compensated image feature of a third size; using an overlay sub-unit to up-sample the compensated third image feature The latter fourth compensated image feature performs a superposition operation to generate a further compensated third image feature of a third size; and an output image is generated based on the further compensated second image feature or the further compensated third image feature.

In some embodiments, the image feature processing module is further configured to determine a first input image of a first size, a second input image of a second size, and a third input image of a third size according to the input image, respectively The first input image, the second input image, and the third input image are processed to determine a first image feature of a first size, a first input image feature of a second size, and a second input image of a third size. Input image features; up-sampling the first image feature, and performing an overlay operation on the first input image feature and the up-sampled first image feature to obtain a second image feature of a second size; The second image feature is up-sampled, and the up-sampled second image feature and the second input image feature are superimposed to obtain a third image feature of a third size.

In some embodiments, the input image has a second size, and determining the first input image of the first size, the second input image of the second size, and the third input image of the third size according to the input image includes: The input image of the second size is down-sampled to generate a first input image of the first size; the input image is determined as a second input image of the second size; the input image of the second size is up-sampled To generate a third input image of a third size.

In some embodiments, the image processing device includes a cascaded N-level image feature compensation module, wherein the i+1 level image feature compensation module is configured to compensate the i-th level image feature using the first image feature of the first size The compensated second image feature generated by the module is compensated to obtain a further compensated second image feature, and the compensated third image feature generated by the i-th level image feature is compensated by the further compensated second image feature Performing compensation to obtain a further compensated third image feature, where N is an integer greater than 1, and 1≤i<N; and the output module is further configured to: based on the compensated image feature generated by the Nth level image feature compensation unit The second image feature or the compensated third image feature generated by the Nth level image feature compensation unit determines the output image.

In some embodiments, the i+1-th level image feature compensation module is further configured to: down-sample the compensated second image feature to obtain the down-sampled compensated second image feature of the first size Image features; performing a de-superimposition operation on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size; up-sampling the third compensated image feature , To obtain the up-sampled third compensated image feature of the second size; perform a superposition operation on the up-sampled third compensated image feature and the compensated second image feature to generate a further compensated image of the second size The second image feature.

In some embodiments, the i+1th level image feature compensation module is further configured to: down-sample the compensated third image feature to obtain the down-sampled compensated third image feature of the second size Image features; performing a de-overlapping operation on the down-sampled compensated third image feature and the further compensated second image feature to generate a fourth compensated image feature of the second size; on the fourth compensated image feature Up-sampling is performed to obtain the up-sampled fourth compensated image feature of the third size; the superimposed operation is performed on the compensated third image feature and the up-sampled fourth compensated image feature to generate the third size Third image feature for further compensation.

In some embodiments, the second size is M times the first size, the third size is M times the second size, and M is an integer greater than one.

According to another aspect of the present disclosure, there is also provided an image processing device including a processor and a memory, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes the Image processing method.

According to another aspect of the present disclosure, there is also provided a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to execute the image processing method described above.

Using the technical solution provided by the present disclosure, by sequentially compensating image features of different sizes associated with the input image, the processing of multiple image features of different sizes of the input image can be achieved without recursion, thereby reducing Memory consumption during image processing. In addition, with the technical solution provided by the present disclosure, since the image features of lower resolution are always used to compensate for the image features of higher resolution, the image information of higher resolution will not affect the image information of lower resolution. This may affect the parameter optimization process of the image processing method and improve the image quality of the final output image.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative work, other drawings can be obtained from these drawings. The following drawings are not deliberately scaled and drawn according to actual size and proportions, and the focus is to show the gist of the present disclosure.

Fig. 1 shows a schematic flowchart of an image processing method according to an embodiment of the present disclosure;

Figure 2 shows a schematic diagram of the principle of generating super-resolution images using the back projection method;

Fig. 3 shows a schematic block diagram of an image processing device according to an embodiment of the present disclosure;

Fig. 4 shows a schematic block diagram of another image processing apparatus according to an embodiment of the present disclosure;

Fig. 5 shows an example of an image feature processing module according to an embodiment of the present disclosure;

FIG. 6 shows an exemplary network structure of an image processing apparatus according to an embodiment of the present disclosure;

Fig. 7 shows a schematic structural diagram of an image feature compensation unit according to an embodiment of the present disclosure;

FIG. 8A shows a schematic diagram of the working principle of the superposition subunit according to the present disclosure;

FIG. 8B shows a schematic diagram of the working principle of the de-superimposition subunit according to the present disclosure; and

FIG. 9 shows an architecture diagram of a computing device according to an embodiment of the present disclosure.

detailed description

To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Unless otherwise defined, the technical terms or scientific terms used in the present disclosure shall have the ordinary meanings understood by those with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity, or importance, but are only used to distinguish different components. Similarly, "including" or "including" and other similar words mean that the elements or items appearing in front of the word cover the elements or items listed after the word and their equivalents, without excluding other elements or items. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections or signal connections, whether direct or indirect.

A convolutional network for image processing can use images as input and output, where the convolutional network can include at least one convolutional layer. And the convolutional layer included in the convolutional network is used to process the image features associated with the input image. After the parameters in the convolutional network are determined through training, the convolutional network can be used to implement image processing. For example, a trained convolutional network can be used to generate a super-resolution image based on the input image or perform deblurring, denoising, coloring, and dehazing on the input image. The original high-resolution clear image can be used to implement the training of the convolutional network.

Fig. 1 shows a schematic flowchart 100 of an image processing method according to an embodiment of the present disclosure. In step S102, an input image can be received.

In step S104, the received input image may be processed to determine a first image feature having a first size, a second image feature having a second size, and a third image feature having a third size associated with the input image. Image feature, wherein the first size is smaller than the second size, and the second size is smaller than the third size. The image size mentioned here may be a size in pixels. In this case, the larger the size of the image or image feature means the higher its resolution.

The size mentioned here refers to the length or width of an image or image feature. In some embodiments, the second size is M times the first size, the third size is M times the second size, and M may be an integer greater than 1. For example, M can be equal to 2, 3, 4, etc. In the present disclosure, M=2 is taken as an example to describe the principle of the present disclosure, that is, the second size is twice the first size, and the third size is twice the second size to describe the principle involved in the present disclosure. For example, assuming that the size of the first image feature is 16*16, the size of the second image feature is 32*32, and the size of the third image feature is 64*64. It can be understood that M is not limited to being an integer, and can also be implemented as any number greater than 1.

In some embodiments, other input images different in size from the input image may be generated based on the received input image to obtain image features of other sizes required in the subsequent image processing process. For example, by up-sampling or down-sampling, a first input image with a first size, a second input image with a second size, and a third input image with a third size can be determined from the input image. The technical solution provided in the present disclosure does not limit the size of the input image, and the input image can be processed in the manner of up-sampling and down-sampling to meet the input requirements of the image processing device provided in the present disclosure.

In some embodiments, a first image feature having a first size, a second image feature having a second size, and a third image feature having a third size may be determined based on the first input image, the second input image, and the third input image. Image characteristics.

In the embodiments of the present disclosure, the "image feature" refers to the output result of processing the image using a trained analysis network. For example, the analysis network can be implemented as a convolutional network. The image features obtained by convolving the image can represent higher-order information in the image, such as semantic information in the image. By using the trained neural network to further process the image features, it is possible to generate predefined processing results for the input image.

The first image feature of the first size can be used to compensate the second image feature and the third image feature of higher resolution.

In some embodiments, in step S104, a first input image having a first size, a second input image having a second size, and a third input image having a third size may be determined according to the input image.

For example, if the input image has a first size, the input image can be determined as the first input image, and the input image can be upsampled once to generate a second input image with the second size, and the input image can be upsampled twice To generate a third input image with a third size. For another example, if the input image has a second size, the input image may be determined as the second input image, and the input image may be down-sampled once to generate a first input image with the first size, and the input image may be up-sampled once To generate a third input image with a third size. For another example, if the input image has a third size, the input image can be determined as the third input image, and the input image can be down-sampled once to generate a second input image with the second size. Down-sampling to generate and have a first input image of the first size. By analogy, the technical solution provided in the present disclosure does not limit the size of the input image, and the input image can be processed in the manner of up-sampling and down-sampling to meet the input requirements of the image processing device provided in the present disclosure.

The first input image, the second input image, and the third input image are respectively processed to determine a first image feature having a first size, a first input image feature having a second size, and a third input image. The size of the second input image feature.

Then, the first image feature may be up-sampled, and the first input image feature and the up-sampled first image feature may be superimposed to obtain a second image feature of a second size. Further, the second image feature may be up-sampled, and the up-sampled second image feature and the second input image feature may be superimposed to obtain a third image feature.

In step S106, the first image feature may be used to compensate the second image feature to generate a compensated second image feature of the second size.

Fig. 2 shows a schematic diagram of the principle of generating a super-resolution image using the back projection method. As shown in FIG. 2, the block 210 with an up arrow indicates an up-sampling operation, and the block 220 with a down arrow indicates a down-sampling operation. The circle 230 including the plus sign represents the superimposition operation, and the circle 240 including the plus sign with the minus sign represents the de-superimposition operation.

If a larger-sized high-resolution image is to be generated based on the low-resolution image LR, the low-resolution image LR may be up-sampled by the up-sampling unit 210 to increase the size of the low-resolution image LR. However, it is understandable that the quality of a high-resolution image obtained by only one upsampling is not high. In order to improve the quality of high-resolution images, high-resolution images can be compensated by back projection.

For example, the down-sampling unit 220 may be used to down-sample the high-resolution image generated by upsampling the low-resolution image LR once, and the de-overlapping unit 240 may be used to determine the difference between the down-sampled high-resolution image and the original low-resolution image. The difference image between images, such a difference image can be used to represent the difference between the high-resolution image and the original low-resolution image. The difference image determined in this way can be used to compensate for the high-resolution image. For example, the above-mentioned difference image can be up-sampled to the same size as the high-resolution image, and the up-sampled difference image can be superimposed with the high-resolution image to generate a high-resolution image that is closer to the content information of the original low-resolution image. Resolution image.

In some embodiments, in step S106, the down-sampling subunit may be used to down-sample the second image feature to obtain the down-sampled second image feature of the first size. Then, the de-superimposition subunit may be used to perform a de-superimposition operation on the down-sampled second image feature and the first image feature to generate the first compensated image feature of the first size. Further, an up-sampling subunit may be used to up-sample the first compensated image feature to obtain the up-sampled first compensated image feature of the second size. Further, the superimposing subunit may be used to perform a superimposing operation on the up-sampled first compensated image feature and the second image feature to generate a compensated second image feature of the second size.

Wherein, the de-superimposition subunit may be used to generate difference information between two image features, and it may be configured to perform a subtraction operation on the down-sampled second image feature and corresponding elements in the first image feature. Or the de-superimposition subunit may be configured to perform a convolution operation on the combination of the down-sampled second image feature and the first image feature, that is, use the trained convolution layer to generate the down-sampled second image feature and the first image The difference between the features to achieve the above-mentioned de-overlay operation. The superimposition subunit can be used to superimpose information between two image features. For example, it can be configured to perform a convolution operation on the first compensated image feature and the second image feature after upsampling, or it can be configured to upsample The corresponding elements in the latter first compensated image feature and the second image feature perform an addition operation to implement the above-mentioned superimposition operation.

In step S108, the compensated second image feature may be used to compensate the third image feature of the third size to generate the third image feature after compensation of the third size.

In some embodiments, step S908 may further include: using a down-sampling subunit to down-sample the third image feature to obtain a down-sampled third image feature of the second size. Then, the de-superimposition subunit may be used to perform a de-superimposition operation on the down-sampled third image feature and the compensated second image feature to generate a second compensated image feature of the second size. Further, the up-sampling subunit may be used to up-sample the second compensated image feature to obtain the up-sampled second compensated image feature of the third size. Further, the superimposing subunit may be used to perform a superimposing operation on the third image feature and the up-sampled second compensated image feature to generate a third-sized compensated third image feature.

Wherein, the de-superimposition subunit may be configured to perform a subtraction operation on corresponding elements in the down-sampled third image feature and the compensated second image feature. For example, the value of the element in the second image feature after compensation can be used to subtract the value of the corresponding element in the third image feature after downsampling. In some embodiments, the de-overlap subunit may be configured to perform a convolution operation on the combination of the down-sampled third image feature and the compensated second image feature, that is, use the trained convolution layer to down-sample The combination of the latter third image feature and the compensated second image feature is convolved to generate the difference between the down-sampled third image feature and the compensated second image feature. For example, the down-sampled second image feature and the first image feature can be spliced together to form a feature with a larger size. By performing convolution processing on the feature with a larger size, a new image feature with the same size as the down-sampled second image feature and the first image feature can be obtained. The new image feature obtained by convolution using the above method can represent the difference between the down-sampled second image feature and the first image feature. The specific process of the above-mentioned convolution processing is described below in conjunction with FIG. 8B.

The superimposition subunit can be used to superimpose information between two image features, for example, it can be configured to perform a convolution operation on the third image feature and the up-sampled second compensated image feature or to perform a convolution operation on the third image feature and up-sampling The corresponding element in the latter second compensated image feature performs an addition operation. The specific process of the above-mentioned convolution processing is described below in conjunction with FIG. 8A.

In step S110, the output image may be determined based on the compensated second image feature or the compensated third image feature. In some embodiments, a synthesis network may be used to synthesize the compensated second image features output in step S906, so as to generate an output image with a second size. Therefore, the image processing method provided by the present disclosure can process an input image of a first size to generate a 2x magnified output image, and can also process an input image of a second size to generate an image-enhanced output with a constant size. image. In other embodiments, a synthesis network may be used to synthesize the compensated third image feature output in step S908, so as to generate an output image with a third size. Therefore, by using the image processing method provided by the present disclosure, the input image of the first size can be processed to generate a 4 times magnified output image, and the input image of the second size can also be processed to generate a 2 times magnified output image. The input image of the third size can be processed to generate an image-enhanced output image of the same size. Since the image processing method provided by the present disclosure can process multiple image features of different sizes, those skilled in the art can select output image features of different sizes as needed to obtain the final output image.

In some embodiments, in step S110, the first image feature may be used to compensate the compensated second image feature to generate a further compensated second image feature. Then, the further compensated second image feature may be used to compensate the compensated third image feature to generate a further compensated third image feature. Further, an output image may be generated based on the further compensated second image feature or the further compensated third image feature.

In some embodiments, using the first image feature to compensate the compensated second image feature to generate a further compensated second image feature may include: downloading the compensated second image feature Sampling to obtain the down-sampled compensated second image feature of the first size. Then, a de-superimposition operation may be performed on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size. Further, the third compensated image feature may be up-sampled to obtain the up-sampled third compensated image feature of the second size. Further, a superposition operation is performed on the up-sampled third compensated image feature and the compensated second image feature of the second size to generate a further compensated second image feature of the second size.

Using the further compensated second image feature to compensate the compensated third image feature to generate the further compensated third image feature may include: down-sampling the compensated second image feature to Obtain the down-sampled compensated second image feature of the first size. Then, a de-superimposition operation may be performed on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size. Then, the third compensated image feature may be up-sampled to obtain the up-sampled third compensated image feature of the second size. Further, a superposition operation may be performed on the up-sampled third compensated image feature and the compensated second image feature of the second size to generate a further compensated second image feature of the second size.

With the image processing method provided by the present disclosure, the image feature compensation module can be used to compensate two or more high-resolution image features without using a complex recursive structure. In addition, by sequentially compensating image features of different sizes in the order of resolution from low to high, it can be ensured that only low-resolution information is transferred to high-resolution image features, and high-resolution information will not be Transfer to low-resolution image features, thereby reducing the complexity of image processing methods.

Fig. 3 shows a schematic block diagram of an image processing device according to an embodiment of the present disclosure. As shown in FIG. 3, the image processing apparatus 300 may include a receiving module 310, an image feature processing module 320, an image feature compensation module 330, and an output module 340.

The receiving module 310 may be configured to receive input images. In some embodiments, pictures stored in the database can be retrieved as input images. In other embodiments, an image can be collected as an input image by an image collection device (for example, a camera, a video camera), etc.

The image feature processing module 320 may be configured to process the input image received by the receiving module 310 to determine a first image feature having a first size, a second image feature having a second size, and a first image feature associated with the input image. A three-size third image feature, wherein the first size is smaller than the second size, and the second size is smaller than the third size. In some embodiments, the second size is M times the first size, the third size is M times the second size, and M is an integer greater than one.

In some embodiments, the receiving module 310 may generate other input images of different sizes based on the received input image, thereby generating image features of different sizes required in the subsequent image processing process. In some embodiments, the received input image may be up-sampled or down-sampled to obtain a first input image of a first size, a second input image of a second size, and a third input image of a third size.

In some embodiments, a first image feature having a first size, a second image feature having a second size, and a third image feature having a third size may be determined based on the first input image, the second input image, and the third input image. Image characteristics.

The image feature compensation module 330 may be configured to use the first image feature of the first size to compensate the second image feature and the third image feature of higher resolution. As shown in FIG. 3, the image feature compensation module 330 may include a first compensation unit 331 and a second compensation unit 332. The first compensation unit 331 may be configured to use the first image feature to compensate the second image feature to generate a compensated second image feature of the second size. The second compensation unit 332 may be configured to use the compensated second image feature to compensate the third image feature to generate a compensated third image feature of a third size. In some embodiments, the first compensation unit 331 and the second compensation unit 332 may be implemented as the same structure.

In some embodiments, the first compensation unit 331 and the second compensation unit 332 may use the principle of back-projection to perform the compensation operation.

The image feature compensation module 330 shown in FIG. 3 only includes two compensation units, that is, the image feature compensation module shown in FIG. 3 can compensate two different sizes of larger image features. However, the content of the present disclosure does not stop there. It is understandable that those skilled in the art can set more levels of compensation units in the image feature compensation module according to actual conditions, so that image features of more sizes can be compensated.

In some embodiments, the first compensation unit 331 may be further configured to: use a down-sampling subunit to down-sample the second image feature to obtain the down-sampled second image feature of the first size; The sub-unit performs a de-superposition operation on the down-sampled second image feature and the first image feature to generate the first compensated image feature of the first size; the up-sampling sub-unit is used to up-sample the first compensated image feature to obtain The first compensated image feature after upsampling of the second size; and the superposition operation is performed on the upsampled first compensated image feature and the second image feature by the superimposing subunit to generate the compensated first image feature of the second size 2. Image features.

Wherein, the de-superimposition subunit can be used to generate difference information between two image features, and it can be configured to perform a subtraction operation on the down-sampled second image feature and corresponding elements in the first image feature; or configured as a pair The combination of the down-sampled second image feature and the first image feature performs a convolution operation, that is, the trained convolution layer is used to generate the difference between the down-sampled second image feature and the first image feature. For example, the down-sampled second image feature and the first image feature can be spliced together to form a feature with a larger size. By performing convolution processing on the feature with a larger size, a new image feature with the same size as the down-sampled second image feature and the first image feature can be obtained. The new image feature obtained by convolution using the above method can represent the difference between the down-sampled second image feature and the first image feature.

The superimposition subunit can be used to superimpose information between two image features. For example, it can be configured to perform a convolution operation on the first compensated image feature and the second image feature after upsampling, or it can be configured to upsample The corresponding elements in the latter first compensated image feature and the second image feature perform an addition operation.

Then, the second compensation unit 332 may be configured to use a down-sampling sub-unit to down-sample the third image feature to obtain a down-sampled third image feature of the second size; The third image feature and the compensated second image feature perform a de-overlapping operation to generate a second compensated image feature of a second size; the second compensated image feature is upsampled by an upsampling subunit to obtain the first A three-size up-sampled second compensated image feature; and using an overlay subunit to perform an overlay operation on the third image feature and the up-sampled second compensated image feature to generate a third-size compensated third Image characteristics.

Wherein, the de-overlap subunit may be configured to perform a subtraction operation on corresponding elements in the down-sampled third image feature and the compensated second image feature. For example, the value of the element in the second image feature after compensation can be used to subtract the value of the corresponding element in the third image feature after downsampling. In some embodiments, the de-overlap subunit may be configured to perform a convolution operation on the combination of the down-sampled third image feature and the compensated second image feature, that is, use the trained convolutional layer to generate down-samples The difference between the post-third image feature and the compensated second image feature.

The superimposition subunit can be used to superimpose information between two image features, for example, it can be configured to perform a convolution operation on the third image feature and the up-sampled second compensated image feature or to perform a convolution operation on the third image feature and up-sampling The corresponding element in the latter second compensated image feature performs an addition operation.

The output module 340 may be configured to determine an output image based on the compensated second image feature or the compensated third image feature. In some embodiments, a synthesis network may be used to synthesize the compensated second image features output by the image feature compensation module 330, so as to generate an output image with a second size. Therefore, the image processing method provided by the present disclosure can process an input image of a first size to generate a 2x magnified output image, and can also process an input image of a second size to generate an image-enhanced output with a constant size. image. Alternatively, a synthesis network may be used to synthesize the compensated third image feature output by the image feature compensation module 330, so as to generate an output image with a third size. Therefore, the image processing device provided by the present disclosure can process an input image of a first size to generate a 4 times magnified output image, and can also process an input image of a second size to generate a 2 times magnified output image. The input image of the third size can be processed to generate an image-enhanced output image of the same size. Since the image processing method provided by the present disclosure can process multiple image features of different sizes, those skilled in the art can select output image features of different sizes as needed to obtain the final output image.

Among them, the synthesis network can be implemented as a convolutional network. The synthesis network can be used to synthesize image features into images.

The image processing method provided by the present disclosure can perform processing of super-resolution, image enhancement, deblurring, denoising, dehazing, and coloring on the input image.

Taking super-resolution processing as an example, the input image may be a low-resolution image with a first size. The input image can be up-sampled to the second size and the third size by up-sampling the input image at least once. Using the image processing method described above, the input image features of the first size, the second size and the third size can be analyzed, and the input image features of the second size and the third size can be compared by using the input image features of the first size. Perform compensation processing to obtain the compensated second-size and third-size image features, and use the compensated second-size or third-size image features to synthesize to obtain a second-size or third-size super-resolution image .

Taking image enhancement processing as an example, the input image may be a high-resolution image with a third size. The input image can be down-sampled into the first size and the second size by down-sampling the input image at least once. Using the method described above, the input image features of the first size, the second size, and the third size are obtained through analysis, and the input image features of the second size and the third size are analyzed by using the input image features of the first size. The compensation process can obtain the image features of the second size and the third size after compensation. Using the compensated image features of the third size, an enhanced image of the third size can be synthesized. If the input image is of the second size, the compensated image feature of the second size may be used to synthesize to obtain an enhanced image of the second size.

With the image processing device provided by the present disclosure, the image feature compensation module can be used to compensate two or more image features with different resolutions without using a complex recursive structure. In addition, by sequentially compensating image features of different sizes in the order of resolution from low to high, it can be ensured that only low-resolution information is transferred to high-resolution image features, and high-resolution information will not be Transfer to low-resolution image features, thereby reducing the complexity of the image processing device.

Fig. 4 shows a schematic block diagram of another image processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 4, the image processing device 400 may include a receiving module 410, an image feature processing module 420, N image feature compensation modules 430-1 to 430-N cascaded, and an output module 440. Among them, the receiving module 410, the image feature processing module 420, and the output module 440 can be implemented as the receiving module 310, the image feature processing module 320, and the output module 340 shown in FIG. 3, and details are not described herein again.

Each of the cascaded N image feature compensation modules 430-1 to 430-N may be implemented as the image feature compensation module 330 shown in FIG. 3. Wherein, each image feature compensation module can use the first image feature of the first size to compensate the image feature larger than the first size. Wherein, for the i-th level image feature compensation module, the first image feature of the first size can be used to compensate the second image feature of the second size and the third image feature of the third size. Then, the compensated second image feature and the compensated third image feature output by the i-th level image feature compensation module can be input to the i+1 level image feature compensation module. Therefore, the second image feature input to the i+1 level image feature compensation module is the compensated second image feature output by the i level image feature compensation module, and the third image feature input to the i+1 level image feature compensation module Is the compensated third image feature output by the i-th level image feature compensation module. Therefore, the i+1 level image feature compensation module may be configured to use the first image feature to compensate the compensated second image feature generated by the i level image feature compensation module to obtain a further compensated second image feature, and The further compensated second image feature is used to compensate the compensated third image feature generated by the i-th level image feature compensation module to obtain the further compensated third image feature.

In some embodiments, using the first image feature to compensate the compensated second image feature to generate a further compensated second image feature may include: downloading the compensated second image feature Sampling to obtain the down-sampled compensated second image feature of the first size. Then, a de-superimposition operation may be performed on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size. Further, the third compensated image feature may be up-sampled to obtain the up-sampled third compensated image feature of the second size. Further, a superposition operation is performed on the up-sampled third compensated image feature and the compensated second image feature of the second size to generate a further compensated second image feature of the second size.

Using the further compensated second image feature to compensate the compensated third image feature to generate the further compensated third image feature may include down-sampling the compensated second image feature to obtain The compensated second image feature after downsampling of the first size. Then, a de-superimposition operation may be performed on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size. Then, the third compensated image feature may be up-sampled to obtain the up-sampled third compensated image feature of the second size. Further, a superposition operation may be performed on the up-sampled third compensated image feature and the compensated second image feature of the second size to generate a further compensated second image feature of the second size.

As shown in FIG. 4, in the image processing device 400, the output module 440 may be configured to be based on the compensated second image feature generated by the Nth level image feature compensation unit and the compensated first image feature generated by the Nth level image feature compensation unit. Three image features determine the output image. For example, the output module 440 may use a synthesis network to synthesize the compensated second image feature generated by the Nth level image feature compensation unit into a second size output image, or use a synthesis network to synthesize the compensation generated by the Nth level image feature compensation unit The latter third image feature is synthesized into an output image of the third size.

Using the image processing device shown in FIG. 4, the high-resolution image features can be compensated multiple times by the multi-level feature compensation unit, so that images with better quality can be output.

Fig. 5 shows an example of an image feature processing module according to an embodiment of the present disclosure. As shown in FIG. 5, the image feature processing module 520 may include analysis networks 521-1, 521-2, and 521-3, and a first image feature processing unit 522, a second image feature processing unit 523, and a third image feature processing unit 524 .

As shown in FIG. 5, the image feature processing module 520 may be used to process the received input image. In some embodiments, the first image feature corresponding to the first input image may be determined based on the first input image of the first size, the second input image of the second size, and the third input image of the third size determined by the input module , The second input image feature corresponding to the second input image and the third input image feature corresponding to the third input image.

Then, the analysis networks 521-1, 521-2, 521-3 can be used to process the first input image, the second input image, and the third input image, respectively, to obtain the first image feature corresponding to the first input image and the corresponding The first input image feature in the second input image and the third input image feature corresponding to the third input image.

As shown in FIG. 5, the first image feature processing unit 522, the second image feature processing unit 523, and the third image feature processing unit 524 can be used to perform processing on the first input image feature, the second input image feature, and the third input image feature. Process to determine the first image feature, the second image feature, and the third image feature.

The first image feature processing unit 522 may be configured to up-sample the first image feature, and output the up-sampled first image feature to the second image feature processing unit. Further, the first image feature processing unit 522 may also be configured to output the first image feature to an image feature compensation module connected to the image feature processing module 520.

The second image feature processing unit 523 may be configured to perform a superposition operation on the second input image feature and the up-sampled first image feature to obtain a second image feature of a second size. Further, the second image feature processing unit 523 may also be configured to output the second image feature to the image feature compensation module connected to the image feature processing module 520 and the third image feature processing unit 524.

The third image feature processing unit 524 may be configured to up-sample the second image feature, and perform an overlay operation on the up-sampled second image feature and the third input image feature to obtain the third image feature. Further, the third image feature processing unit 524 may also be configured to output the third image feature to the image feature compensation module connected to the image feature processing module 520.

Fig. 6 shows an exemplary network structure of an image processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 6, the image processing device 600 may include an input module (not shown), an image feature processing module 620, cascaded three-level image feature compensation modules 630-1, 630-2, and 630-3, and an output module 640.

As shown in FIG. 6, the image feature processing module 620 may include an up-sampling sub-unit 611. In some embodiments, the image feature processing module 620 may further include a downsampling subunit (not shown). Up-sampling and down-sampling the input image using the up-sampling sub-unit and the down-sampling sub-unit, the first input image of the first size, the second input image of the second size, and the third input of the third size can be determined based on the input image image.

The image feature processing module 620 may further include analysis networks 621-1, 621-2, 621-3 for processing the first input image, the second input image, and the third input image, respectively. The image feature processing module 620 may further include a first image feature processing unit 622, a second image feature processing unit 623, and a third image feature processing unit 624. The image feature processing module 620 may be configured to process the first input image, the second input image, and the third input image to determine the first image feature, the second image feature, and the third image feature. The image feature processing module 620 may be implemented in the form of the image feature processing module 520, which will not be repeated here.

The cascaded three-level image feature compensation modules 630-1, 630-2, and 630-3 may be the same. The following takes the image feature compensation module 630-1 as an example to explain the principle of the present disclosure.

As shown in FIG. 6, the image feature compensation module 630-1 may be formed by a plurality of image feature compensation units 631. Although the image feature compensation module 630-1 in FIG. 6 includes only three image feature compensation units, those skilled in the art can understand that the image feature compensation module may include more or less images in accordance with the principles of the present disclosure. The feature compensation unit, for example, two image feature compensation units or more than four image feature compensation units.

Fig. 7 shows a schematic structural diagram of an image feature compensation unit according to an embodiment of the present disclosure. As shown in FIG. 7, the image feature processing unit 631 may include 3 input terminals, 3 output terminals, and up-sampling sub-unit 710, superimposing sub-unit 720, down-sampling sub-unit 730, and de-superimposing unit 740. In some embodiments, the image feature processing unit can perform the following operations: use the upsampling subunit 710 to perform upsampling on input 1; use the superposition subunit 720 to perform superposition operations on the upsampled input 1 and input 2; use downsampling The sub-unit 730 performs down-sampling on the input 3; and the de-superimposing unit 740 performs a de-superimposing operation on the down-sampled input 3 and the image features output by the superimposing sub-unit.

In some embodiments, the upsampling sub-unit 710 may be implemented as a convolutional network including a normalization layer and a strided convolution layer. The down-sampling subunit 730 may be implemented as a convolutional network including a normalization layer and a transposed convolutional layer (strided transposed convolution). In other embodiments, the upsampling subunit 710 may also be implemented as conventional upsampling, such as linear interpolation, bicubic interpolation, Lanczos interpolation, and so on.

In some embodiments, the superimposition subunit 720 may be implemented as a convolutional network including a convolutional layer. For example, two image features to be superimposed can be combined into a feature with a larger size and input into a superposition subunit in the form of a convolutional network for processing. The output of the convolutional network is configured to have the same image feature size as the image feature size to be superimposed. The trained convolutional network can output the result of image information superimposed with two image features. In other embodiments, the superimposing subunit 720 may also be configured to directly add the values of the corresponding elements of the two image features to be superimposed, so as to realize the information superimposition of the two image features. For example, FIG. 8A shows a schematic diagram of the working principle of the superposition subunit according to the present disclosure. In order to superimpose the image feature 810 and the image feature 820, the image feature 810 and the image feature 820 can be combined into a feature with a larger size and input into a convolutional network. The convolutional network can output the superimposed image feature 830.

In some embodiments, the de-superimposition sub-unit 740 may be implemented as a convolutional network including a convolutional layer. For example, two image features to be processed can be combined into one feature with a larger size and input into a de-superimposition subunit in the form of a convolutional network for processing. The output of the convolutional network is configured to have the same image feature size as the image feature size to be processed. The trained convolutional network can output the result that represents the difference information of the two image features. In other embodiments, the de-overlap subunit 740 may also be configured to directly subtract the values of the corresponding elements of the two image features to be processed to determine the difference information between the two image features. For example, FIG. 8B shows a schematic diagram of the working principle of the de-superimposition subunit according to the present disclosure. In order to determine the difference information between the image feature 840 and the image feature 850, the image feature 840 and the image feature 850 can be combined into a feature with a larger size and input into a convolutional network. The convolutional network can output the image feature 860 representing the difference.

Continuing to refer to FIG. 7, since the image feature compensation unit shown in FIG. 7 includes 3 input terminals and 3 output terminals, when 3 image features are input to the image feature compensation unit, the image feature compensation unit can realize the aforementioned functions . If one or two of the inputs are missing, the image feature compensation unit will skip the corresponding operation. For example, when only input 1 and input 2 are input to the image feature compensation unit, the image feature compensation unit will omit the operation of the down-sampling sub-unit 730 and the operation of the de-superimposition sub-unit 740, and directly output the result of the superimposition sub-unit 720 as output 1. , Output 2 and output 3. For another example, when only input 2 is input to the image feature compensation unit, the image feature compensation unit will not perform any operation and directly output input 2 as output 1, output 2, and output 3. For another example, when only input 2 and input 3 are input to the image feature compensation unit, the image feature compensation unit will omit the operation of the up-sampling sub-unit 710 and the overlap operation for input 1 and input 2, and directly input 2 and the down-sampled input 3 perform de-superposition operation. And you can directly use input 2 and output as output 2 and output 1.

Using the above principles, the first image feature processing unit 622, the second image feature processing unit 623, and the third image feature processing unit 624 in the image feature processing module in FIG. 6 can also be implemented as the image features shown in FIG. The form of the compensation unit. Among them, the arrows shown in FIG. 6 representing the units 622, 623, 624, and 631 represent the input and output directions of the processing unit. It can be seen that the first image feature processing unit 622 operates according to only input 2, the second image feature processing unit 623 operates according to only input 1 and input 2, and the third image feature processing unit 624 operates according to only input 2. There are input 1 and input 2 modes for operation.

Therefore, the image processing apparatus according to the embodiment of the present disclosure shown in FIG. 4 can be realized by using the network structure shown in FIG. 6. It can be understood that using the network structure shown in FIG. 6, the present disclosure does not limit the size of the input image and the output image. Regardless of whether the input is a low-resolution small-size image or a high-resolution large-size image, the network structure shown in FIG. 6 can be used to process image features of different sizes associated with the input image.

The network structure shown in FIG. 6 can be trained by using training sets determined for different purposes. For example, the network 600 can be trained using high-resolution original images to determine the network 600 for generating super-resolution images based on low-resolution images. For another example, the network 600 can be trained using high-definition original images to determine the network 600 for generating a clear image based on the blurred image. For another example, the network 600 can be trained using color original images to determine the network 600 used to color the grayscale image.

In some embodiments, the network 600 may be used to process the sample images used for training, and compare the difference between the image output by the network 600 and the real image. For example, at least one of the L1 regular term and the L2 regular term between the output image of the network 600 and the real image may be determined as the loss function of the network, and the parameters in the network 600 may be adjusted to minimize the loss function. For example, the parameters of the convolution kernel in the up-sampling sub-unit, the down-sampling sub-unit superimposing sub-unit, and the de-superimposing sub-unit implemented as a convolutional network can be adjusted to minimize the loss function of the network 600.

In addition, the method or device according to the embodiment of the present application can also be implemented with the aid of the architecture of the computing device shown in FIG. 9. Figure 9 shows the architecture of the computing device. As shown in FIG. 9, the computing device 1000 may include a bus 910, one or more processors (CPU) 920, a read only memory (ROM) 930, a random access memory (RAM) 940, a communication port 950 connected to a network, Input/output components 960, hard disk 970, etc. The storage device in the computing device 900, such as the ROM 930 or the hard disk 970, can store various data or files used in the processing and/or communication of the method for locating an electronic device provided in this application, and program instructions executed by the CPU. The computing device 900 may also include a user interface 980. Of course, the architecture shown in FIG. 9 is only exemplary. When implementing different devices, one or more components of the computing device shown in FIG. 9 may be omitted according to actual needs.

The embodiments of the present application can also be implemented as a computer-readable storage medium. The computer-readable storage medium according to the embodiment of the present application stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to the embodiments of the present application described with reference to the above drawings can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and/or non-volatile memory. The volatile memory may include random access memory (RAM) and/or cache memory (cache), for example. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

Those skilled in the art can understand that the content disclosed in this application can have many variations and improvements. For example, the various devices or components described above can be implemented by hardware, or can be implemented by software, firmware, or a combination of some or all of the three.

In addition, as shown in the present application and claims, unless the context clearly suggests exceptional circumstances, the words "a", "an", "an" and/or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "including" and "including" only suggest that the clearly identified steps and elements are included, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

In addition, although this application makes various references to certain units in the system according to the embodiments of this application, any number of different units can be used and run on the client and/or server. The units are merely illustrative, and different units may be used for different aspects of the systems and methods.

In addition, a flowchart is used in this application to illustrate the operations performed by the system according to the embodiment of the application. It should be understood that the preceding or following operations are not necessarily performed exactly in order. Instead, the various steps can be processed in reverse order or simultaneously. At the same time, you can also add other operations to these processes, or remove a step or several operations from these processes.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in ordinary dictionaries should be interpreted as having meanings consistent with their meanings in the context of related technologies, and should not be interpreted in idealized or extremely formalized meanings, unless explicitly stated here. So defined.

The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all these modifications are intended to be included in the scope of the present invention defined by the claims. It should be understood that the above is an illustration of the present invention and should not be considered as limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included in the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims (15)

1. An image processing method, including:

   Receive the input image,

   The input image is processed to determine a first image feature of a first size, a second image feature of a second size, and a third image feature of a third size, wherein the first size is smaller than the second size, so The second size is smaller than the third size;

   Using the first image feature to compensate the second image feature to generate a compensated second image feature of a second size;

   Compensate the third image feature by using the compensated second image feature to generate a compensated third image feature of a third size; and

   The output image is determined based on the compensated second image feature or the compensated third image feature.
2. 8. The image processing method according to claim 1, wherein using the first image feature to compensate the second image feature to generate a compensated second image feature of the second size comprises:

   Down-sampling the second image feature to obtain the down-sampled second image feature of the first size;

   Performing a de-superimposition operation on the down-sampled second image feature and the first image feature to generate a first compensated image feature of the first size;

   Up-sampling the first compensated image feature to obtain an up-sampled first compensated image feature of a second size;

   A superposition operation is performed on the up-sampled first compensated image feature and the second image feature to generate a compensated second image feature of a second size.
3. 3. The image processing method of claim 2, wherein performing a de-superimposition operation on the down-sampled second image feature and the first image feature comprises:

   Perform a subtraction operation on the down-sampled second image feature and the corresponding element in the first image feature; or

   Perform a convolution operation on the combination of the down-sampled second image feature and the first image feature.
4. The image processing method of claim 2 or 3, wherein performing an overlay operation on the up-sampled first compensated image feature and the second image feature comprises: performing the up-sampled first compensated image feature Perform an addition operation with the corresponding element in the second image feature.
5. 3. The image processing method according to claim 2 or 3, wherein using the compensated second image feature to compensate the third image feature to generate the compensated third image feature comprises:

   Down-sampling the third image feature to obtain the down-sampled third image feature of the second size;

   Performing a de-superimposition operation on the down-sampled third image feature and the compensated second image feature to generate a second compensated image feature of a second size;

   Up-sampling the second compensated image feature to obtain an up-sampled second compensated image feature of a third size;

   A superposition operation is performed on the third image feature and the up-sampled second compensated image feature to generate a compensated third image feature of a third size.
6. 8. The image processing method of claim 5, wherein determining the output image based on the compensated second image feature or the compensated third image feature comprises:

   Down-sampling the compensated second image feature to obtain the down-sampled compensated second image feature of the first size;

   Performing a de-superimposition operation on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size;

   Up-sampling the third compensated image feature to obtain an up-sampled third compensated image feature of the second size;

   Performing a superposition operation on the compensated second image feature and the up-sampled third compensated image feature to generate a further compensated second image feature of a second size;

   Down-sampling the compensated third image feature to obtain the down-sampled compensated third image feature of the second size;

   Performing a de-superimposition operation on the down-sampled compensated third image feature and the further compensated second image feature to generate a fourth compensated image feature of the second size;

   Up-sampling the fourth compensated image feature to obtain an up-sampled fourth compensated image feature of a third size;

   Performing an overlay operation on the compensated third image feature and the up-sampled fourth compensated image feature to generate a further compensated third image feature of a third size; and

   An output image is generated based on the further compensated second image feature or the further compensated third image feature.
7. 7. The image processing method according to any one of claims 1 to 6, processing the input image to determine a first image feature of a first size and a second image feature of a second size associated with the input image And the third image features of the third size include:

   Determining a first input image of a first size, a second input image of a second size, and a third input image of a third size according to the input image,

   The first input image, the second input image, and the third input image are respectively processed to determine the first image feature of the first size, the first input image feature of the second size, and the first input image of the third size. 2. Input image characteristics;

   Up-sampling the first image feature, and performing an overlay operation on the first input image feature and the up-sampled first image feature to obtain a second image feature of a second size;

   Up-sampling the second image feature, and performing an overlay operation on the up-sampled second image feature and the second input image feature to obtain a third image feature of a third size.
8. 8. The image processing method of claim 7, wherein the input image has a first size,

   The first input image of the first size is determined according to the input image, the second input image of the second size and the third input image of the third size include:

   Determining the input image as a first input image of a first size;

   Up-sampling the first input image of the first size to generate a second input image of the second size;

   The second input image of the second size is up-sampled to generate a third input image of the third size.
9. 5. The image processing method of claim 1, wherein determining the output image based on the compensated second image feature or the compensated third image feature comprises:

   Using the first image feature to compensate the compensated second image feature to generate a further compensated second image feature;

   Using the further compensated second image feature to compensate the compensated third image feature to generate a further compensated third image feature; and

   An output image is generated based on the further compensated second image feature or the further compensated third image feature.
10. 9. The image processing method of claim 9, wherein using the first image feature to compensate the compensated second image feature to generate a further compensated second image feature comprises:

    Down-sampling the compensated second image feature to obtain the down-sampled compensated second image feature of the first size;

    Performing a de-superimposition operation on the down-sampled compensated second image feature and the first image feature to generate a third compensated image feature of the first size;

    Up-sampling the third compensated image feature to obtain an up-sampled third compensated image feature of the second size;

    A superposition operation is performed on the up-sampled third compensated image feature and the compensated second image feature to generate a further compensated second image feature of the second size.
11. 10. The image processing method according to claim 10, wherein using the further compensated second image feature to compensate the compensated third image feature to generate the further compensated third image feature comprises:

    Down-sampling the compensated third image feature to obtain the down-sampled compensated third image feature of the second size;

    Performing a de-superimposition operation on the down-sampled compensated third image feature and the further compensated second image feature to generate a fourth compensated image feature of the second size;

    Up-sampling the fourth compensated image feature to obtain an up-sampled fourth compensated image feature of a third size;

    A superposition operation is performed on the compensated third image feature and the up-sampled fourth compensated image feature to generate a further compensated third image feature of a third size.
12. 5. The image processing method according to claim 1, wherein the second size is M times the first size, the third size is M times the second size, and M is an integer greater than 1.
13. An image processing device including:

    The receiving module is configured to receive input images;

    An image feature processing module configured to process the input image to determine a first image feature of a first size, a second image feature of a second size, and a third image feature of a third size, wherein the first size is smaller than The second size, the second size being smaller than the third size;

    Image feature compensation module, including:

    A first compensation unit configured to use the first image feature to compensate the second image feature to generate a compensated second image feature of a second size; and

    The second compensation unit is configured to compensate the third image feature by using the compensated second image feature to generate a compensated third image feature of a third size; and

    The output module is configured to determine an output image based on the compensated second image feature or the compensated third image feature.
14. An image processing device, comprising a processor and a memory, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes the image processing according to any one of claims 1-12 method.
15. A computer-readable storage medium having instructions stored thereon, which when executed by a processor, causes the processor to execute the image processing method according to any one of claims 1-12.

Images