CN112581362A

CN112581362A - Image processing method and device for adjusting image details

Info

Publication number: CN112581362A
Application number: CN201910920550.8A
Authority: CN
Inventors: 江鹤
Original assignee: Medtronic Inc
Current assignee: Medtronic Inc
Priority date: 2019-09-27
Filing date: 2019-09-27
Publication date: 2021-03-30

Abstract

The present disclosure relates to an image processing method, apparatus, computer system, and computer readable medium for adjusting image details. In particular, an image processing method includes receiving an original image as an input to a first of a plurality of cascaded residual generators. The method further comprises obtaining, via each of the plurality of cascaded residual generators, a respective residual representation to derive a plurality of residual representations. The method further includes performing a weighted summation of the original image and the plurality of residual representations to obtain a detail adjusted image. The image processing method is suitable for real-time application scenes, and can adaptively obtain the detail adjustment image with better performance.

Description

Image processing method and device for adjusting image details

Technical Field

The present disclosure relates generally to image processing, and more particularly to methods and apparatus for processing images to adjust image details, change image resolution.

Background

With the development and popularization of electronic technology, a large number of images are generated every day. In the fields of medical imaging, surveillance and security applications, there is often a wide demand for high resolution images with higher pixel density, as it provides more detail. However, on the one hand, it is difficult to generate a large number of high-resolution images due to the resolution limitation of the manufacturing process of the optical image sensor and its related devices and the influence of shot noise; on the other hand, it is often necessary to convert a high-resolution image into a low-resolution image for storage and transmission. Therefore, it is necessary to restore a high-resolution image from a low-resolution image that is relatively easily available by an image processing technique. In the related art, such an image processing technique is often referred to as image super-resolution. As a low-level computer vision task, image super-resolution is widely applied to various electronic products such as smart phones, televisions, wearable devices and the like.

Existing super-resolution methods can be roughly divided into two categories: enhanced edges and learning-based methods. Edge-enhanced super-resolution methods are typically based on interpolating low-resolution images to obtain high-resolution images without the need for training samples. The super-resolution method based on learning obtains an image high-frequency information model from a training sample set through machine learning, and then predicts a test low-resolution image to obtain a high-resolution image. In the edge enhancement method, some kernel-based interpolation algorithms, including exposure fusion methods such as bilateral filtering and guided filtering, often face the problem of low visual quality of the reconstructed image, for example, blur, jaggy, pseudo-gradient edge or artifact exists. While other optimization-based methods and tone mapping methods can generate vivid images with abundant details, the computation speed is low due to the complex computation structure, and even the practical application is difficult. In addition, some existing resolution enhancement methods need to adjust algorithm parameters according to specific application situations, and are not flexible and convenient to apply.

Disclosure of Invention

The following presents a simplified summary of one or more embodiments in order to provide a basic understanding of one or more aspects of the disclosure. This section does not set forth the embodiments, is not intended to recite key contents or key elements of the embodiments, or to delineate any scope of the embodiments or the claims. The sole purpose of this section is to present some concepts of the embodiments in a form that will facilitate a more detailed description of the embodiments that follow. It is to be understood that the following more detailed description includes further or alternative embodiments beyond those described in this section.

Embodiments described herein include image processing methods, systems, apparatuses, and computer-readable media for adjusting image details.

In one embodiment, an image processing method is provided. The method includes receiving an original image as an input to a first of a plurality of cascaded residual generators. The method further comprises obtaining, via each of the plurality of cascaded residual generators, a respective residual representation to derive a plurality of residual representations. The method further includes performing a weighted summation of the original image and the plurality of residual representations to obtain a detail adjusted image.

In some embodiments, at least a portion of the output of each residual generator is linear with respect to the gradient of the input of that residual generator.

In some embodiments, each residual generator comprises a cascade of upsamplers and downsamplers and subtractors. The cascaded upsamplers and downsamplers are configured to upsample and downsample an input signal in any sequential order to obtain a recovered signal. The subtractor is configured to subtract the restored signal from the input signal to obtain a corresponding residual representation. The upsampling multiple of the upsampler and the downsampling multiple of the downsampler are equal to make the resolution of the restored signal equal in magnitude to the resolution of the input signal. The upsampling multiple of the upsampler is set such that a structural similarity preservation threshold for the input signal is satisfied.

In some embodiments, for the plurality of cascaded residual generators, the weight coefficients assigned to the respective residual representation derived from each residual generator are inversely proportional to the order of the residual generators.

In some embodiments, for a plurality of cascaded residual generators 1,2, … …, N, the residual representation obtained for the kth residual generator is assigned a weight factor proportional to the inverse of the factorial of k, k being an integer between 1 and N.

In some embodiments, the weighted summation of the original image and the plurality of residual representations to obtain the detail-adjusted image comprises: assigning a respective plurality of first weight coefficients to the plurality of residual representations; for each of the plurality of residual representations, dividing the residual representation into a plurality of non-overlapping block groups and assigning a respective plurality of second weight coefficients to the plurality of non-overlapping block groups, wherein each second weight coefficient is directly proportional to the image information entropy of the respective block group and the sum of the plurality of second weight coefficients is 1; performing a weighted summation of all block groups in the plurality of residual representations to obtain an estimated detail representation of the original image, wherein a weight coefficient for each block group is a product of a respective first weight coefficient and a respective second weight coefficient; and deriving a detail-adjusted image based on a linear calculation of the original image and the estimated detail representation.

In some embodiments, the weighted summation of the original image and the plurality of residual representations to obtain the detail-adjusted image comprises: weighted summing the plurality of residual representations to obtain an estimated detail representation of the original image; and deriving a detail-adjusted image based on the linear calculations of the original image and the estimated detail representation.

In another embodiment, a computer system is provided. The computer system includes a processor and a storage medium. A storage medium is coupled to the processor and has instructions stored thereon. The instructions, when executed by the processor, cause the computer system to perform operations comprising: receiving an original image as an input to a first of a plurality of cascaded residual generators; obtaining, via each of the plurality of cascaded residual generators, a respective residual representation to derive a plurality of residual representations; and performing weighted summation on the original image and the plurality of residual representations to obtain a detail adjustment image.

In another embodiment, an image processing apparatus is provided. The image processing apparatus includes a plurality of cascaded residual generators and synthesizers. A first residual generator of the plurality of cascaded residual generators receives an original image as an input. Each of the plurality of cascaded residual generators generates a respective residual representation to obtain a plurality of residual representations. A synthesizer is configured to perform a weighted summation of the original image and the plurality of residual representations to synthesize a detail adjusted image.

In another embodiment, a non-transitory computer readable medium is provided. The non-transitory computer readable medium has instructions stored thereon. The instructions, when executed by the processor, cause the processor to: receiving an original image as an input to a first of a plurality of cascaded residual generators; obtaining, via each of the plurality of cascaded residual generators, a respective residual representation to derive a plurality of residual representations; the original image and the plurality of residual representations are weighted and summed to obtain a detail adjusted image.

Other embodiments and various non-limiting examples, scenarios and implementations are described in more detail below. The following description and the annexed drawings set forth in detail certain illustrative embodiments of the disclosure. These embodiments are indicative, however, of but a few of the various ways in which the principles of the disclosure may be employed. Other advantages and novel features of the described embodiments of the disclosure will become apparent from the following detailed description of the disclosure when considered in conjunction with the drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments of the present disclosure and together with the description, serve to explain the principles of the disclosure. For convenience of description, the dimensions of the various features shown in the drawings are not necessarily drawn to scale. Meanwhile, in the drawings, the same or similar reference numerals and letters denote the same or similar items in the embodiments of the present disclosure. Thus, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

The present disclosure may be more clearly understood from the following detailed description, taken with reference to the accompanying drawings, in which:

FIG. 1 shows a block diagram of a system for adjusting image details in accordance with one or more embodiments of the present disclosure.

Fig. 2 shows a schematic block diagram of a residual generator in a system according to one or more embodiments of the present disclosure.

FIG. 3 shows a flowchart of a method for adjusting image details in accordance with one or more embodiments of the present disclosure.

Fig. 4 shows a flowchart of a method for obtaining a residual representation according to one or more embodiments of the present disclosure.

Fig. 5 illustrates a diagram of image patch-based weighted fusion in accordance with one or more embodiments of the present disclosure.

FIG. 6 illustrates a block diagram of an example non-limiting environment, including a computer, operable to implement image detail adjustment in accordance with one or more embodiments of the present disclosure.

Detailed Description

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure or its application. Unless specifically stated otherwise, the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments should also be construed as merely illustrative, and not limiting the scope of the present disclosure.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

As described above, the image super-resolution method in the prior art has the problems of low reconstructed image quality, low calculation speed, poor adaptive capability or poor robustness. There is a need for a new method of adjusting image resolution that solves one or more of the above-mentioned problems of the prior art.

The present disclosure relates to a network-based image processing method, system, apparatus, and computer-readable medium for adjusting image details. The original image and the estimated detail representation can be finally subjected to linear calculation to obtain an image with adjusted details by using a multilayer cascaded residual generator to obtain detail features of different levels of the original image and performing weighted fusion to obtain the estimated detail representation. The method is simple in calculation structure and high in calculation speed, and can be applied to real-time scenes. Moreover, the image after detail enhancement has better edge holding capacity and detail holding capacity. Furthermore, the resulting features of the different levels of detail are statistically substantially orthogonal. It can be seen that the decomposition of the original image by the image processing method according to the present disclosure is complete and pure, neither repetition nor omission, and the information of the original image is maximally retained with the least amount of information.

In the present disclosure, the terms "include", "including" and similar words are to be interpreted openly as "including but not limited to". The term "based on" should be read as "based at least in part on. The term "an embodiment" or "one embodiment" should be read as "at least one embodiment". The term "another embodiment" should be read as "at least one other embodiment".

One or more embodiments of the present disclosure will now be described in detail with reference to fig. 1. Fig. 1 shows a block diagram of a system 100 for adjusting image details in accordance with one or more embodiments of the present disclosure. The system 100 may include a plurality (1 to N, N being an integer greater than 1) of residual generators 102, summers 104, and combiners 106. The residual generators 1 to N may be cascaded in series, i.e. the output of a previous residual generator is connected to the input of a subsequent residual generator. It should be noted that in the description of the embodiments of the present disclosure, unless clearly defined or limited otherwise, terms such as "connected" should be interpreted broadly, and it may be directly connected or indirectly connected through an intermediate medium, for example.

Each residual generator 102 may be configured to fit the input signal and to find an error, i.e., a residual, between the input signal and the fitted signal. The system 100 may receive an original image as an input to a first of a plurality of cascaded residual generators 1 to N. Residual generators 1 through N may produce respective residual representations 1 through N. The N residual representations may be given respective weights at summer 104 and summed in a weighted manner to obtain an estimated detail representation. The merger 106 may derive a detail adjusted image based on the original image and the estimated detail representation.

The original image may be represented as a two-dimensional array of pixel values (including a one-dimensional array, which may be considered as a special form of a two-dimensional array) in which each element corresponds to a pixel point. The original image may not be limited to a black-and-white image or a color image. Accordingly, the pixel values may be luminance/grayscale values or (R, G, B) or (C, M, Y, K) tuples corresponding to the pixel points. The image resolution reflects the information content of the image and is characterized by the number of pixel points in a unit size. For a given physical size, a low resolution original image corresponds to a smaller two-dimensional array dimension than a high resolution image.

For a given single image f (x), where x is a pixel and f (x) is the corresponding pixel value, it can be considered to consist of a base layer b (x) and a detail layer d (x), i.e. f (x) b (x) + d (x). Existing super-resolution methods mostly try to obtain the base layer by using different a priori assumptions and optimization methods, and in practice, it is often the approximation b' (x) of the base layer that is obtained. The image e (x) after detail adjustment based on the original single image can be modeled as formula (1), where α is the adjustment coefficient. It can be seen that the detail adjustment image e (x) means that the detail layer in the original image is enhanced or weakened relative to the base layer. When detail enhancement is performed, the adjustment coefficient α is usually 2.

e(x)＝b′(x)+α×(f(x)-b′(x)) (1)

However, the applicant has appreciated that equation (1) may be rewritten as equation (2) below, where t ═ α -1 may represent a detail adjustment coefficient (also referred to as a scaling coefficient) with respect to the original image. For the original image f (x), the network Net may be used to fit the details of f (x), i.e., the approximation of the detail layer d '(x) ═ f (x) -b' (x) ═ Net (f (x)). Also, Net (f (x)) can be further designed to

Where N is the number of network layers, ω_kIs the weight of the k-th layer in the network, f_k(x) For the pixel values of the kth layer in the network (which may be referred to as the kth layer image for convenience), k ≦ 1 ≦ N and k is an integer.

As shown in FIG. 1, each residual generator 102 of the cascade may be used to obtain a respective residual representation, corresponding to a respective f_k(x) In that respect By weighted summation of the residual representations, the summer 104 obtains, i.e., an estimated detail layer d' (x). And then, carrying out scaling of a scaling coefficient on the estimated detail representation, and combining the scaled detail representation with the original image to obtain a detail adjustment image e (x). Based on the existingCompared with a super-resolution method of an optimization or neural network, the system for obtaining the detail adjustment image according to the embodiment of the disclosure has a simple computing structure, is suitable for real-time application scenes, does not need to carry out a large amount of different parameter training and adjustment on various application scenes, and has self-adaptability.

Although summer 104 and combiner 106 are shown as separate components in fig. 1, it is understood that in some embodiments summer 104 and combiner 106 may be integrated into a single summer to derive the detail adjusted image directly based on the original image and the respective residual representations. In the case of integration into a single summer, the weighting coefficient of the original image may be 1, and the corresponding weighting coefficient of each residual representation may be the product of the illustrated weighting coefficients 1 to N and the scaling coefficient. In other embodiments, the weight coefficients of the original image may be other values, and the corresponding weight coefficients of each residual representation are adjusted accordingly. At this time, the detail layer and the base layer are both adjusted in the detail-adjusted image compared with the original image.

As previously described, the residual generator 102 may include a fitter and a subtractor. For the kth of the cascaded residual generators, the fitter may be based on the input signal f_k-1(x) Obtaining a fitting signal f'_k-1(x)＝Fit(f_k-1(x) Fit) where Fit represents the fitting function and the subtractor can obtain the error f of the input signal and the fitted signal_k(x)＝f_k-1(x)-Fit(f_k-1(x) The error is the output of the kth residual generator.

In some embodiments, residual generators 1 to N may be identical, i.e. may comprise identical fitters. In other embodiments, residual generators 1 through N may each be implemented using a different fitter.

In some embodiments, the output of the residual generator 102 may be at least partially linear with respect to its input. In other words, at least a portion of the output of the residual generator 102 may be linear with respect to its input. In particular, the fitter in the residual generator may be at least partially linear. I.e. the Fit function is at least partially linear. Residual generation for concatenationThe kth residual generator in the synthesizer, at least partially linear meaning f_k(x) And f_k-1(x) Are at least partially linearly related. Then f_k(x) May be expressed as a x f_k-1(x) + b, wherein a and b are constants. It can further be seen that for f_k(x) With the input signal f_k-1(x) Satisfies a linear relationship. The specific derivation is shown in the following formula (3).

Wherein x₀As a reference pixel, a pixel of a picture,

denotes the gradient, x-x₀λ is a very small value, ρ_kAnd gamma_kIs a constant.

In some embodiments, each of the cascaded plurality of residual generators 1 to N may be at least partially linear. Then it can be further derived from equation (3),

wherein

Is a constant.

It can be seen that where each of the cascaded plurality of residual generators 1 to N is at least partially linear, the output of the kth residual generator is at least partially linear with the k-steps of the original input image.

At this time, the detail layer estimated in equation (2) can be further expressed as

In the aspect of mathematics, the method for improving the stability of the artificial teeth,the base layer b (x) of the original image is sufficiently smooth. When the number of layers of the cascaded residual generators is greater than a certain threshold, for those layer residual generators that exceed the threshold, their respective step (equal to the layer number) gradient for the original input image can be considered to be approximately equal to the respective step gradient for the detail layer of the original input image, i.e., it is assumed that the layer residual generators for which the respective step gradient for the original input image is greater than the layer number are approximately equal to the respective step gradient for

Where k is greater than a certain threshold epsilon. That is, as the number of cascaded residual generator layers increases, the corresponding residual generator extracts deeper information (higher frequency information) of a detail layer of the original input image. And the detail layer information of different levels (different frequencies) extracted by each residual error generator is subjected to weighted fusion, so that the detail layer of the original image can be accurately reconstructed.

Further, it can be proven by testing that the respective residual representations obtained by the respective residual generators are almost orthogonal to each other. This means that there is no or little information intersection and repetition between residual representations. Little redundant information is utilized in reconstructing the detail layer of the original image.

When gradient operation is carried out on the detail layer of the original image, the higher the gradient order is, the smaller the occupation ratio of the detail information of the corresponding level in the detail layer is. In some embodiments, for more accurate reconstruction of the detail layer of the original image, for a plurality of cascaded residual generators, the weight coefficients assigned to the residual representation derived from each residual generator may be inversely proportional to the order of the residual generators. I.e. the higher the number of layers, the higher the residual generator f_k(x) Corresponding to the weight coefficient ω_kThe smaller. However, it should be recognized that in some application-specific scenarios, it may be desirable for the composite image to have a particular distortion relative to the original image. If desired, in other embodiments, the weight coefficients assigned to the residual representations derived from each residual generator may represent other functions relative to the order of the residual generators, rather than being inversely proportional. Thus, the composite image is equivalent to applying a window function to the frequency domain information of the original image. For example, the weight coefficients may be constant for each residual generator, or may be constant for each residual generatorIn a proportional order with respect to the respective residual generators. Those skilled in the art can design other window functions to design the weighting coefficients according to the actual application requirements.

The taylor series expansion of the detail layer d (x) of the original image is compared with the estimated detail layer d' (x) expressed as formula (5), which is advantageous for determining the weight coefficient ω_kIs preferred. In some embodiments, for cascaded residual generators 1 to N, the weight coefficient ω assigned to the kth residual generator_kProportional to the inverse of the factorial of k. At this time, since the estimated detail layer d' (x) is closer to the true detail layer d (x) of the original image, better image reconstruction quality can be obtained. For example, when the detail adjustment image enhances the detail, the subjective and objective evaluation results (including Structural Similarity (SSIM) index, signal retention capability, edge retention capability, etc.) of the detail adjustment image relative to the original image are further improved.

In some embodiments, the fitter in the residual generator 102 comprises an upsampler and a downsampler. Fig. 2 illustrates a schematic block diagram of a residual generator including an upsampler, a downsampler, and a subtractor according to some embodiments of the present disclosure. The residual generator 202 in fig. 2 corresponds to the residual generator 102 in fig. 1 and may include a fitter 212 and a subtractor 218. Fitter 212 may include a cascade of upsamplers 214 and downsamplers 216.

The upsampler 214 is used to upsample (also referred to as amplify or interpolate) an input signal (e.g., an image). For example, for an original image with size M × N, s times up-sampling is performed to obtain an image with size (M × s) × (N × s), where one pixel becomes s × s pixels. The upsampler 214 may be implemented by interpolation, i.e., a suitable interpolation algorithm is used to insert new elements between pixels based on the original image pixels.

The downsampler 216 is used to downsample (also referred to as down-scale or down-sample) an input signal (e.g., an image). For example, an original image with size M × N is sampled s times to obtain an image with size (M/s) × (N/s), where s × s pixels become one pixel. The down sampler 214 may also be implemented by an interpolation method, that is, replacing a plurality of pixels with one pixel based on the original image pixels.

The upsampling multiple of the upsampler and the downsampling multiple of the downsampler may be equal such that the resolution (i.e., size) of the input image remains unchanged after the upsampler and the downsampler are cascaded. The order in which the upsampler and the downsampler are cascaded is not limited. Although a down-sampler is shown in fig. 2 connected to the output of the up-sampler, it should be understood that in other embodiments the input signal may be received by the down-sampler, while the up-sampler is connected to the output of the down-sampler. The upsamplers and downsamplers may employ one of a variety of existing interpolation algorithms, including, but not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos (Lanczos) interpolation, and the like. In some embodiments, the downsampler may be the inverse of the upsampler. However, due to the non-ideality of the interpolation algorithm, there is always an error between the fitting signal (also called a recovered signal) obtained after the up-sampling and down-sampling and the original input signal. A subtractor 218 may be used to calculate this error, resulting in a residual representation.

In some embodiments, the upsampling multiple of the upsampler may be set not to exceed the structural similarity preservation threshold, so that the self-similarity of the image blocks between the input image and the upsampled image (i.e., the upsampled image) can be preserved during the upsampling process. Accordingly, the downsampling multiple of the downsampler may also be set not to exceed the structural similarity preservation threshold. In some embodiments, the structural similarity maintenance threshold is 1.33. I.e. the upsampling multiple and the downsampling multiple do not exceed 1.33.

As described above, in some embodiments, the residual generator may be at least partially linear. Accordingly, the upsampler and the downsampler may be at least partially linear. Preferably, the upsampler and the downsampler are bilinear samplers. Accordingly, the up-sampling multiple of the up-sampler and the down-sampling multiple of the down-sampler are preferably 1.25 to maintain structural similarity between the original image and the scaled image.

Returning to fig. 1, it is now discussed how the number of cascaded residual generators, i.e. the value of N, is set. The more the number of cascaded residual generators is, that is, the more the number of layers of the network is, the more the detail information extracted from the original image is, the more accurate the estimation detail obtained finally is. In the test verification, the larger the SSIM value is along with the increase of the value of N, the better the quality of the detail adjustment image is. But the computation time also increases greatly with increasing N. Therefore, in practice, the number of residual generators may be set appropriately so as to satisfy both the calculation speed requirement and the requirements of subjective and objective image evaluation indices (e.g., SSIM value, signal retention capability, edge retention capability, etc.). Preferably, the number of cascaded residual generators may be equal to or greater than 20.

In some embodiments, the residual generator 102, adder 104, synthesizer 106 in fig. 1, and fitter 212 (including upsampler 214 and downsampler 216), subtractor 218 in fig. 2 may be executed by a general-purpose or special-purpose processor in the form of program instructions stored on a non-transitory computer-readable medium to implement the respective functions. In other embodiments, the residual generator 102, adder 104, synthesizer 106 of fig. 1, and fitter 212 (including upsampler 214 and downsampler 216), subtractor 218 of fig. 2 may be implemented as programmable logic circuits implementing specific functions. In other embodiments, residual generator 102, adder 104, synthesizer 106 of fig. 1, and fitter 212 (including upsampler 214 and downsampler 216) and subtractor 218 of fig. 2 may be implemented as application specific integrated circuits. The components of fig. 1 and 2, which are based on hardware implementations, are generally capable of operating with high throughput and fewer computing resources than software-based implementations. In particular, when the residual generator 102 is implemented in hardware circuitry, cascaded multiple residual generators may be implemented by time-based iterative use of a single hardware circuit block, thereby further reducing computational resources.

FIG. 3 shows a flowchart of a method for adjusting image details in accordance with one or more embodiments of the present disclosure. The method may be performed by the components of the system shown in fig. 1. For the sake of brevity, only the main steps of the method will be described herein. What is referred to herein may be incorporated as a supplement to the embodiment shown in fig. 1. Other steps or other features not referred to herein may be referred to as described with respect to fig. 1.

In step S302, an original image is input to a first one of a plurality of cascaded residual generators. The residual generator may be configured to fit the input signal and to find an error between the input signal and the fitted signal as a residual. Cascaded residual generators means that the output of a previous residual generator can be connected to the input of a subsequent residual generator.

In step S304, a respective residual representation is obtained via each of a plurality of cascaded residual generators. In some embodiments, the output of each residual generator may be at least partially linear with respect to the gradient of its input. That is, part or all of the respective residual representation output by each residual generator may be a linear representation of the gradient with its respective input signal (i.e. the output of the previous residual generator or the original image). Accordingly, as previously described, for N cascaded residual generators, the output of the kth residual generator is at least partially linear with the k-steps of the original input image.

In some embodiments, the residual generator may include cascaded upsamplers and downsamplers and subtractors. Fig. 4 illustrates a method flow for obtaining respective residual representations with a residual generator, according to some embodiments of the present disclosure. In step S402, a restored signal may be obtained from an input signal by a cascade of up-sampling and down-sampling. The order of the cascaded upsampling and downsampling is not limited. The upsampling multiple of the upsampling and the downsampling multiple of the downsampling may be equal such that the image resolution of the restored signal is equal to the image resolution of the input signal. In some embodiments, the downsampling may be the inverse of the upsampling. Preferably, the upsampling and downsampling may be linear samples, e.g. bilinear samples.

In step S404, the input signal is subtracted from the restored signal to obtain a residual representation. The residual representation carries part of the detail information in the input image signal and can thus be used to reconstruct image details as well as to synthesize a detail adjusted image.

Returning to fig. 3, in step S306, the original image and the plurality of residual representations are weighted and summed to obtain a detail adjusted image. The detail adjustment image is an image in which the detail layer in the original image is adjusted with respect to the base layer. In some embodiments, step S306 may include: the method further comprises the steps of weighted summing the plurality of residual representations to obtain an estimated detail representation, and obtaining a detail adjusted image based on the original image and the estimated detail representation. The detail-adjusted image may be derived based on a linear calculation of the original image and the estimated detail representation. In some embodiments, the weight coefficients assigned to the plurality of residual representations are inversely proportional to the order of the corresponding residual generators in the cascade. In a further embodiment, the weight coefficients assigned to the plurality of residual representations are proportional to the inverse of the factorization of the order of the corresponding residual generators in the cascade.

In a further implementation, differences in entropy of information between different groups of blocks of an image in each residual representation may be taken into account when assigning weight coefficients to the obtained plurality of respective residual representations. For example, for any obtained residual representation, as with the input image, which is also embodied as a two-dimensional array of images, different regions in the image contain different amounts of information. There may be some areas where the amount of information is large and other areas where the amount of information is very small. Accordingly, each residual representation may be divided into a plurality of block groups, an image information entropy of each block group is calculated, and weights are assigned to the respective block groups according to the magnitude of the image information entropy of each block group.

Fig. 5 shows a weighted fusion diagram based on image patches according to an embodiment of the present disclosure. As previously mentioned, the residual generated by the kth residual generator represents f_k(x) The first weight coefficient allocated is ω_k. In a further embodiment, the residual represents f_k(x) Can be divided into non-overlapping blocks 1 to M, M being an integer greater than 1. For the ith block group f_k(x_i) I is an integer between 1 and M, and the assigned weight coefficient is a first weight coefficient omega_kAnd a second weight coefficient W_iProduct of, i.e. w'_k＝ω_k×W_iWherein the second weight coefficient W_iTo the ithThe entropy of the image information of the block groups is proportional and the weighting factor W is applied to the M block groups divided_iThe sum is 1. In some embodiments, the ratio of the entropy of the image information for the ith block group to the sum of the entropy of the image information for all M block groups, i.e. the ratio

Wherein

E_i＝-∑_μp(x_μ)log₂(p(x_μ))。

E_iEntropy of image information, p (x), representing a block group i_μ) Is the probability that a pixel x of value μ occurs in block group i. At this time, the aforementioned formula (2) can be expressed as

It can be seen that the estimated detail representation can be obtained by weighted summation of all block groups in all N residual representations, with the weight coefficient being the first weight coefficient ω_kAnd a second weight coefficient W_iThe product of (a).

It is worth noting that while the method, system, apparatus and medium for adjusting image details according to the present disclosure were introduced above with super-resolution techniques that improve image resolution, the method, system, apparatus and medium according to the present disclosure may also be applied to other computer vision tasks, for example, for speeding up the efficiency of image searches.

FIG. 6 illustrates a block diagram of an example non-limiting environment, including a computer, operable to implement image detail adjustment in accordance with one or more embodiments of the present disclosure. The following discussion is intended to provide a general, brief description of a suitable computing environment. Environment 600 may implement one or more embodiments described herein. Computer 602 may include a system processor 604, an image processor 606, a system memory 608, interfaces 610, I/O interfaces 614, a network adapter 616, and a bus 618. In addition, the computer 602 may also be connected to an external image capturing apparatus 620 and an image display apparatus 622 via a wired or wireless network through the network adapter 616. Image capture device 620 and image display device 622 may also be integrated as part of computer 602. The image capture device 620 may be used to capture one or more raw images for input to the image processor 606 for processing. Image display 622 may be used to present one or more detail adjustment images. The image processor 606 may include a residual generator and adder, etc., as described in fig. 1 and 2. In some embodiments, the image processor 606 may be a separate processor from the system processor 604 to increase the speed of operation. In other embodiments, the image processor 606 may be part of the system processor 604.

The system bus 618 can be any of several types of bus structures that can couple the components of the computer 602. The processing unit 604 can be any of various commercially available processors. Dual microprocessors and other multiprocessor architectures also can be employed as the processing unit.

The system memory 608 may be volatile or non-volatile memory. By way of illustration, and not limitation, nonvolatile memory can include ROM, programmable ROM (prom), electrically programmable ROM (eprom), electrically erasable prom (eeprom), or flash memory. Volatile memory can include RAM, which acts as external cache memory. By way of illustration and not limitation, RAM can take many forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DRRAM). The computer 602 also includes an internal Hard Disk Drive (HDD)612 that is connected through a hard disk drive interface 610. The system memory 608 and HDD 612 may store various data (including data for original images, residual representations, estimated detail representations, and detail adjusted images) and program instructions.

The computer 602 may operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer 624. The remote computer 624 can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node. The logical connections depicted include wired/wireless connectivity to a Local Area Network (LAN) and a Wide Area Network (WAN), and the like. In a networked environment, program modules depicted relative to the computer, such as residual error generators, adders, and the like, or portions thereof, may be stored in the memory/storage devices of the remote computer.

Those skilled in the art will recognize that the methods and systems of the present disclosure may be implemented in a number of ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be implemented as a program recorded in a memory, the program comprising machine readable instructions for implementing a method according to the present disclosure when executed by a processor. Thus, the present disclosure also covers tangible and/or non-transitory computer-readable storage media storing programs for performing methods according to the present disclosure.

Computer readable storage media can be any available storage media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media may be implemented in connection with any method or technology for storage of information such as computer-readable instructions, program modules, structured data, or unstructured data. Tangible and/or non-transitory computer-readable storage media may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, other magnetic storage devices, and/or other media which can be used to store the desired information.

Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the foregoing examples are for purposes of illustration only and are not intended to limit the scope of the present disclosure. It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An image processing method comprising:

receiving an original image as an input to a first of a plurality of cascaded residual generators;

obtaining, via each of the plurality of cascaded residual generators, a respective residual representation to derive a plurality of residual representations; and

and performing weighted summation on the original image and the residual representations to obtain a detail adjustment image.

2. The method of claim 1, wherein at least a portion of the output of each residual generator is linear with respect to the gradient of the input of that residual generator.

3. The method of claim 1, wherein each residual generator comprises:

a cascaded upsampler and downsampler configured to upsample and downsample an input signal in any sequential order to obtain a recovered signal; and

a subtractor configured to subtract the restored signal from the input signal to obtain the corresponding residual representation,

wherein the up-sampling multiple of the up-sampler and the down-sampling multiple of the down-sampler are equal to each other, so that the resolution of the restored signal is equal to the resolution of the input signal,

wherein the upsampling multiple of the upsampler is set such that a structural similarity preserving threshold for the input signal is met.

4. The method of claim 1, wherein, for the plurality of cascaded residual generators, the weight coefficients assigned to the respective residual representation derived from each residual generator are inversely proportional to the order of the residual generators.

5. The method of claim 4, wherein for a plurality of cascaded residual generators 1,2, … …, N, the residual representation obtained for the kth residual generator is assigned a weight factor proportional to the inverse of the factorial of k, k being an integer between 1 and N.

6. The method of claim 1, wherein the weighted summation of the original image and the plurality of residual representations to derive a detail-adjusted image comprises:

assigning a respective plurality of first weight coefficients to the plurality of residual representations;

for each of the plurality of residual representations:

dividing the residual representation into a plurality of non-overlapping groups of blocks; and

assigning a respective plurality of second weight coefficients to the plurality of non-overlapping block groups, each second weight coefficient being directly proportional to an image information entropy of the respective block group, and a sum of the plurality of second weight coefficients being 1;

performing a weighted summation of all block groups in the plurality of residual representations to obtain an estimated detail representation of the original image, wherein a weight coefficient for each block group is a product of a respective first weight coefficient and a respective second weight coefficient; and

deriving the detail adjusted image based on a linear calculation of the original image and the estimated detail representation.

7. The method of claim 1, wherein the weighted summation of the original image and the plurality of residual representations to derive a detail-adjusted image comprises:

performing a weighted summation on the plurality of residual representations to obtain an estimated detail representation of the original image; and

8. A computer system, comprising:

a processor; and

a storage medium coupled to a processor and having instructions stored thereon that, when executed by the processor, cause the computer system to:

9. The computer system of claim 8, wherein at least a portion of the output of each residual generator is linear with respect to the gradient of the input of the residual generator.

10. The computer system of claim 8, wherein, for the plurality of cascaded residual generators, the weight coefficients assigned to the respective residual representation derived from each residual generator are inversely proportional to the order of the residual generators.

11. The computer system of claim 10, wherein for a plurality of cascaded residual generators 1,2, … …, N, the residual representation obtained for the kth residual generator is assigned a first weight factor proportional to the inverse of the factorial of k, k being an integer between 1 and N.

12. The computer system of claim 8, wherein weighted summing the original image and the plurality of residual representations to obtain a detail adjusted image comprises: :

for each of the plurality of residual representations:

dividing the residual representation into a plurality of non-overlapping groups of blocks;

13. An image processing apparatus comprising:

a plurality of cascaded residual generators, a first of the plurality of cascaded residual generators receiving an original image as input, each of the plurality of cascaded residual generators generating a respective residual representation to obtain a plurality of residual representations; and

a combiner configured to perform a weighted summation of the original image and the plurality of residual representations to synthesize a detail-adjusted image.

14. Image processing apparatus according to claim 13, wherein at least a portion of the output of each residual generator is linear with respect to the gradient of the input of that residual generator.

15. The image processing apparatus according to claim 13, wherein, for the plurality of cascaded residual generators, the weight coefficients assigned to the respective residual representation derived from each residual generator are inversely proportional to the order of the residual generators.

16. The image processing device of claim 15, wherein the weighted summation of the original image and the plurality of residual representations further comprises:

for each of the plurality of residual representations:

17. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:

18. The computer readable medium of claim 17, wherein at least a portion of the output of each residual generator is linear with respect to the gradient of the input of that residual generator.

19. The computer readable medium of claim 17, wherein, for the plurality of cascaded residual generators, the weight coefficients assigned to the respective residual representation derived from each residual generator are inversely proportional to the order of the residual generators.

20. The computer readable medium of claim 17, wherein the weighted summation of the original image and the plurality of residual representations to derive a detail adjusted image comprises:

for each of the plurality of residual representations: