WO2014114635A1

WO2014114635A1 - Method and apparatus for performing super-resolution of single images

Info

Publication number: WO2014114635A1
Application number: PCT/EP2014/051117
Authority: WO
Inventors: Mehmet TÜRKAN; Dominique Thoreau; Philippe Guillotel
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2013-01-24
Filing date: 2014-01-21
Publication date: 2014-07-31
Anticipated expiration: 2015-07-24
Also published as: EP2948922A1

Abstract

A method for performing super-resolution of a single image comprises steps of downsampling (S₁) a LR input image by a first factor, separating the LR input image into overlapping blocks (x_i), and for each of the blocks (x_i) searching similar blocks in the downsampled LR images, determining blocks (c_i,₁,c_i,₂,c_i,₃) at corresponding positions in the LR input image, locally optimizing (S₄) the blocks by linear approximation to obtain locally optimized patches (z^k _a), downsampling (S₅ ) the locally optimized patches (z^k _a) by the first factor to obtain a downsampled locally optimized patch (z^k _a,d), comparing the downsampled locally optimized patch (z^k _a,d) with the input block (x ⁱ ₀) and optimizing the number and/or weights (w_i,1,w_i,2,w_i,3) of the determined blocks (c_i,₁,c_i,₂,c_i,₃), wherein a locally optimized patch (x¹ _i) is obtained, and repeating the above steps with different sampling factors, wherein for each repetition the locally optimized patch (x¹ _i, x² _i, x³ _i) of the previous iteration is used as input block. The locally optimized patch (x³ _i) of the last iteration are accumulated and averaged (S₈), and IBP (S₉) is performed to obtain a HR output image (S₁₀).

Description

METHOD AND APPARATUS FOR PERFORMING SUPER-RESOLUTION OF SINGLE IMAGES

Field of the invention

This invention relates to a method for performing super-resolution of single images, and to an apparatus for performing super-resolution of single images.

Background

Super-resolution (SR) is an inverse problem of recovering a high-resolution (HR) image from one or more low-resolution (LR) images. HR means high pixel density in an image, hence an HR image provides more clear and detailed contents to the viewer. A number of real-world applications utilizes SR algorithms, as SR provides a relatively low-cost software solution to this inverse problem when compared to high-cost hardware enhancements in optics and sensors technology.

In order to recover the unknown HR image, conventional SR methods (e.g., [1 ,2, 3]) fuse a set of observed LR images at sub-pixel shifts. If there are enough LR images, the missing high-frequency information can successfully be recovered in the HR image. However, an insufficient number of LR observations, registration (motion estimation) errors and unknown point spread function practically limit the applicability of these classical multi-image SR methods to small upscaling ratios which is typically less than two [4].

Exemplar-based methods have been developed to overcome limitations of classical multi-image SR. These methods assume that the missing high-frequency information is explicitly available in other natural images and it can successfully be retrieved from exemplar LR and HR patch pairs. In [5], LR and HR patch pairs are collected from other natural images, and low- and high-frequency relations of these patches are learned via a Markov network. This method has later been simplified in [6] with a sequence of estimations of HR patches by a nearest neighbor (NN) search from the database of exemplars, i.e. examples. A manifold learning based SR approach has been proposed in [7]. In this method, each LR patch is first reconstructed from its K nearest neighbors ( -NN) taken from the exemplar database. The HR patch is then hallucinated (or estimated) with the corresponding HR exemplar patches by considering geometric similarities of LR and HR patch spaces (manifolds). A sparse representations based SR method has been proposed in [8]. This method jointly learns two connected dictionaries by enforcing the similarity between sparse representations of exemplar LR and HR patch pairs. In this way, the sparse representation of a given LR patch over the learned LR dictionary can be applied with the learned HR dictionary to hallucinate the unknown HR patch.

A unified framework combining the classical multi-image and exemplar-based methods has been proposed in [9] by considering SR from a single LR image. This method first searches image patches in downscaled versions of a given LR image. When a NN is found for the current LR patch, its parent in the LR image is regarded as the HR patch and copied to the appropriate location in the

appropriate scale of the HR image. The obtained different scales of HR images have finally been generated coarse-to-fine until the desired resolution is reached. Another approach of SR from a single LR image has been described in [10] by considering a local self-similarity assumption on natural images. The given LR image is initially interpolated using a custom designed linear operator. After carrying out a strictly-localized NN search in the low-frequency band, the missing high-frequency information of the interpolated image is retrieved from the high- frequency band of the LR image. A similar SR technique has been proposed in [1 1 ], exploiting local cross-scale self-similarities to extrapolate the high-frequency band of the given LR image. This algorithm additionally imposes a total variation [12] regularization in its stages for a robust resolution enhancement.

Summary of the Invention The present invention provides an improved self-contained and exemplar-based framework for single-image SR relying on multi-scale and locally-optimized neighbor embeddings of LR and HR patches. Following the recurrence property of similar patches across different scales of an image, it uses only the observed LR image and its appropriately downscaled versions for SR. Given a LR patch, its size is gradually increased by relying on local geometric similarities of low- and high-resolution patch spaces under small scaling factors. In each step of the algorithm, the local geometry of the given patch and its K-NN patches is characterized using locally linear embedding (LLE) [13]. In order to assure the local image compatibility, the parameter K is locally optimized. The invention allows averaging overlaps between adjacent patches, which improves the smoothness of the target HR image. Finally, the global consistency between the output HR image and the input LR image is enforced by using an iterative back projection (IBP) [2] with a non-local means (NLM) [14] kernel. The proposed algorithm can easily be ported and parallelized on GPU processors. Moreover, an advantage confirmed by experimental results is that it produces more natural looking textures together with cleaner and sharper edges in the resulting HR images, compared to other known SR methods.

In one embodiment, the present invention provides a method for performing super-resolution of single images. In another embodiment, the present invention provides an apparatus for performing super-resolution of single images. In one embodiment, the invention provides a computer readable medium having executable instructions to cause a computer to perform said method for performing super-resolution of single images.

In principle, a method for performing super-resolution of a single image comprises steps of downsampling Si a LR input image by a first factor, separating the LR input image into overlapping blocks x, , and for each of the blocks x, searching similar blocks in the downsampled LR image, determining blocks Ci,i ,Ci,2,Ci,3 at corresponding positions in the LR input image, locally optimizing S₄ the blocks by linear approximation to obtain locally optimized patches , downsampling S₅ the locally optimized patches by the first factor to obtain a downsampled locally optimized patch z _d , comparing the downsampled locally optimized patch z _d with the input block x₀ ^l and optimizing the number and/or weights w, ,i ,Wi,2,Wi,3 of the determined blocks Ci,i ,Ci,2,Ci,3 , wherein a locally optimized patch is obtained, and repeating the above steps with different sampling factors, wherein for each repetition the locally optimized patch xf, xf , xf of the previous iteration is used as input block. The locally optimized patch xf of the last iteration are accumulated and averaged Ss , and iterative back projection Sg is performed to obtain a HR output image Sio .

In one embodiment, the full (or at least a more sophisticated) super-resolution processing is done only for blocks with high texture, while a simplified super- resolution processing is applied for blocks with low texture. In one embodiment, the super-resolution processing is done only for blocks with high texture, while it is skipped for blocks with low texture.

Further objects, features and advantages of the invention will become apparent from a consideration of the following description and the appended claims when taken in connection with the accompanying drawings.

Brief description of the drawings

Exemplary embodiments of the invention are described with reference to the accompanying drawings, which show in

Fig.1 how exemplar LR and HR patch pairs are collected directly from the input LR image and its appropriately downscaled version;

Fig.2 a flow-chart (part 1 ) of the method in a first embodiment of the invention; Fig.3 a flow-chart (part 2) of the method in the first embodiment of the invention;

Fig.4 a flow-chart (part 3) of the method in the first embodiment of the invention;

Fig.5 an overview of the method in one embodiment of the invention;

Fig.6 an apparatus according to one embodiment of the invention;

Fig.7 a comparison of experimental results for fine-detailed texture areas;

Fig.8 a comparison of experimental results for reducing jaggy and ringing artifacts;

Fig.9 experimental results showing sharper and cleaner edges as compared to results from [1 1 ]; Fig.10 experimental results showing sharper and cleaner edges as compared to results of bi-cubic interpolation;

Fig .1 1 a flow-chart of a method in one embodiment of the invention where highly textured blocks and lowly textured blocks are treated different; and

Fig.12 an apparatus according to one embodiment of the invention, where highly textured blocks and lowly textured blocks are treated different.

Detailed description of the invention In the following, the underlying ideas and main steps of the proposed SR algorithm are described, and the obtained experimental results are demonstrated.

First, single-image super-resolution is explained.

The aim of single-image SR is to recover a HR image which is consistent with the given LR image according to an image generation model. In general, the observed LR image X of size N x M is assumed to be a blurred and downscaled version of the unknown HR image Y (of size SN x SM), hence the image generation model can be written as

X = (Y * H)jS (1 )

where H is a low-pass (blur) kernel, and * and J,S represent convolution and downscaling (with a factor of S) operators respectively. The inverse problem, estimating Y for a given X, is highly ill-posed since there might be multiple HR solutions (under different initial assumptions) which produce the same LR observation X. In order to regularize the problem, a global energy function incorporating a prior on Y is generally formulated and minimized as

argmin llA^' - (Y * H) I s \\ + yR (Y) (2) y

where \\X— (Y * H) I is the data term with regard to eq. (1 ), R(Y) is an applicable regularization term based on a prior knowledge on Y and γ is a parameter controlling the contribution of the prior on the solution. As most of the natural images vary smoothly with a limited local activity, R(Y) is usually chosen to penalize the variation of Y. According to the invention, the above SR optimization is solved in two successive steps; first, by calculating an initial estimate Yo of the HR image to satisfy a patch- based local constraint through a prior R(Y) as

F₀ = arg min /?(F) (3) and then, by minimizing this data term iteratively by projecting Y₀ onto the solution space of (1 ) in order to satisfy the image generation model exactly, i.e.,

gmin || K - Ko ll i subject to (Y ^* H)js (4)

Next, local priors from optimized neighbor embeddings are described.

Consider a lexical ordering of image pixels and assume image patches as column vectors. Denote an LR patch (of size n*n) in X centered at pixel / as x, , and its spatially corresponding unknown HR patch (of size Sn χ Sn) in Y as y,, i = 1 ...NM. To recover the HR patch yj (for all i), the LR patch Xj (for all i) is iteratively upscaled by a small factor at each step t of the algorithm until the desired scaling factor S has been reached. In the following, details of the underlying ideas of the proposed local prior are given.

First, exemplar patch pairs from a single image are considered. The underlying main idea is based on the assumption that small image patches are very likely to repeat themselves within and across different scales of an image [9]. Hence, exemplar LR and HR patch pairs are collected directly from the given LR image and its appropriately downscaled version by matching LR patches (located in the downscaled image) and their spatially corresponding HR parents (suited in the input LR image). Fig.1 shows how exemplar LR and HR patch pairs are collected directly from the input LR image 10 and its appropriately downscaled version 1 1 . E.g. LR patch 13 and HR patch 15 are a (LR and HR) patch pair, and LR patch 14 and HR patch 15 are a patch pair. Each LR patch in the downscaled image 1 1 has a corresponding HR patch in the input image 10.

A fundamental observation that has been reported in [10] is that natural images can preserve their various important features only under small scaling factors. Therefore, the input LR image is downscaled with a sufficiently reasonable ratio in order to extract LR and HR patch pairs of greater relevance, and moreover, to maintain the local geometric similarities of LR and HR patch spaces (as explained further below). For this purpose, the size of a given LR patch is increased gradually, e.g., by p pixels in each dimension (p is admissibly smaller than n; e.g. p=1,2 or 3) at each step of the algorithm. In one embodiment, the size of a given LR patch is increased gradually by Ph pixels in horizontal dimension and by p_v pixels in vertical dimension (p_h not equal p_v) in at least one step of the algorithm. However, usually Ph equals p_v .

Now consider a given LR patch Xj of size n x n. At each step t > 0, expand the size incrementally by p-pixels to obtain x- of size (n + tp) x (n + tp) using x-^_1 of size (n + (t-1 )p) x (n + (t-1 )p). To do this, collect HR exemplars of sizes

(n + tp) x (n + tp) from the input LR image and their matching LR exemplars of sizes (n + (t-1 )p) x (n + (t-1 )p) from the downscaled LR image by a factor of s_t= (n+(t-^p) / (n+tp).

As the input LR image is downscaled by a non-integer value s_t , irregular grid points of the two images are related in order to ensure that the collected LR and HR patches are in phase. Because of the partial lack of translation invariance, we inevitably need to search and match exemplars in regular offsets (in both images), which will lead to a drastic reduction in the number of extracted exemplar pairs. In order to maximize the amount of the data present at a step t, multiple downscaled images O^are created by offsetting the input LR image X by 0, ... , n + (t-1)p - 1 pixels in each dimension, and then collect exemplar patches with regular offsets (depending on s_t) from image pairs [X, }v_q (V means "for all") while keeping the phases of LR and HR patch pairs always aligned together. Denote these pairs

Now the term "partial lack of translation invariance" is explained. As the LR image is downsampled with non-integer factors, it may happen that not all patches in the LR image and downsampled image correspond to each other because of the irregular grid relation. For example, when downsampling by ¾, patches need to be related (to collect exemplars) by shifting 4 pixels (offset) in the LR image and 3 pixels (offset) in the downsampled image. The patches in between these offsets are related to each other in sub-pixel precision. So, there is a partial lack of translation invariance, as there is no one-to-one correspondence between all kinds of translations of patches in both images. The two images are translation invariant just with suitable offsets. This is actually related with phase information of pixels. According to the invention, the reduction of the number of exemplars is compensated by creating different downsampled images with different phases.

Next, optimized neighbor embeddings are described.

As a local prior, each local HR patch should resemble its neighboring (similar) patches taken from the given exemplar set. For this purpose, the regularization term is formulated with a neighbor embedding optimization such that each HR patch xf v, at step t can optimally be hallucinate timated or modeled) as a weighted linear c

R(xf) = (5) where VIA,* denote

f x . The second term in eq. (5) represents a sum-to-one constraint on the weighting coefficients. This constraint enforces the approximation of xf to lie in the sub-space spanned by its K-NN, and the invariance to translations of the data point and its neighbors.

Similar to other manifold learning based SR methods, the invention benefits from the local geometric similarities of LR and HR patch spaces, and relates the intrinsic geometric properties of local LR and HR patch neighborhoods using the neighbor embedding assumption. Here, it is also reasonable to assume a stronger neighbor embedding relation as the collected exemplar patches are of greater relevance under small scaling factors (see above in Exemplar patch pairs from a single image). Hence, begin transforming eq. (5) into an approximate LR one, such that

To characterize the intrinsic properties of the LR neighborhood, eq. (6) can optimally be minimized and solved for the weighting coefficients w, = [Wi,i , . .. ,Wi,_K]^T using LLE as w, = (G^l) I (l^T G^¹!) where G, denotes the Gram matrix of K-NN patches {z_t¾ in reference to xf^-1 and 1 is the column vector of ones. Assuming that the local LR geometry has been preserved in the HR patch space, the HR embedding x\ can then be estimated by

In practice, transforming eq. (5) into eq. (6) does not necessarily secure the local neighborhood relation in both LR and HR patch spaces, i.e., HR exemplar patches

because of the noise and other reasons. As a result, the calculated weights {w_i>k} would not be the optimum for hallucinating the HR patch x\ in (7). In other words, the local prior R (x - ^-1) in (6) does not put any constraint on the estimated values of x - except the initial neighbor embedding assumption between LR and HR patch spaces. In order to improve the localized estimation of x\, we introduce a new regularization term by enforcing the hallucinated HR patch x\ to resemble the original LR correspondence xf according to the image generation model as

Rixt¹) = Rixt¹) + η ||x? - P^c [(Pxf * h) 1_§ί] \\ (8) where h is a low-pass kernel, P transforms stacked (vectorized) values into patches with respective sizes, P^c is the inverse operator of P and s_t is an appropriate scaling factor, i.e., s_t = Si x S2 X ... x s_t. For example, if S=2, p=1 , n=3, then three iterations are needed to upscale the LR patch for S=2 with p=1 :

Si = ¾, S2=4/5 S3=5/6, then s = s1 =¾, s₂ = Si x S2 =3/5 and

s₃ = Si x S2 x S3 = 3/6.

In order to minimize eq. (8), an iterative algorithm based on an optimization on the number of used NN patches is used. At any step t, the method proceeds as follows. For each given LR patch x - ^-1 , first a K-NN search in the given exemplar set of LR patches is made. Denote the obtained LR patches and their HR correspondences as {ζ^, ζ^} _ . Then vary the number κ of used NN patches from 1 to K and optimize R (xf^-1) for each distinct value of κ independently. As a result, K ossible estimations {x '^K}_{K i}of the HR signal x\ are obtained, i.e.

where f ,^* _fc]= org min # (xf^_1) and k = 1 κ , V K. We finally obtain an estimation of the HR patch x\ according to the local patch compatibility criterion by

The above patch-based optimization on x-^-1 to obtain x is iteratively applied until the desired scaling factor S has been reached, e.g., if τ iterations are required for an upscaling of S, then y_t = xf V^'\. Now, it is explained how to come from local prior to global smoothness. Since the optimized neighbor embedding of LR and HR patches have been calculated for each single pixel i in the LR image, we implicitly enforce as much as possible the overlaps between adjacent patches in the target HR image. In this manner, the pixel values in the HR grid have been estimated multiple times. These estimations are averaged in order to force all image patches to agree on the overlapped areas, hence to satisfy the smoothness constraint in the HR image. In other embodiments, other weighting strategies than uniform averaging can be used, but this means an increase in the computational complexity. In the following, it is explained how global consistency can be improved through back projection, in particular iterative back projection (IBP).

In order to enforce the global consistency between the recovered HR image Y and the input LR image X, equation (4) is solved by employing the known IBP algorithm. Denote Yo as the initial HR image as recovered above. The update equation for IBP is given as

Y_m = Y_m-1 + [(X - (Ym-i * H) Is)†S] * P (1 1 ) where Y_m is the estimated HR image at iteration m, m = 1 ...M,†S represents the upscaling (with a factor of S) operator and P is a back-projection kernel. Typically, P is chosen to be a fixed isotropic kernel (isotropic kernel is a 2D filter which is symmetric in all dimensions, or circularly symmetric). However, edges usually have anisotropic structures, and the content of the projected data is dynamic along the iterations of IBP. As a result, this kind of kernel often produces results with jaggy and ringing artifacts especially around edges. To guide IBP in a wise manner, an NLM-based kernel (non-local means [14]) is used instead, because it is more adapted to the content of the data to be processed. After M iterations of IBP, we take the result Y_M as our final estimate of the HR image. Figs.2-4 show an exemplary flow-chart according to one embodiment of the invention. Fig.5 shows an overview of one embodiment of the invention (for {S, p, n} = {2, 1 , 3}, exemplary). Reference signs correspond in all figures. While three iterations are shown (first iteration S1-S6, second iteration Si'-Se' and third iteration Si"-S6"), it is also possible to use more or less iterations, depending on a desired scaling factor and on a maximum scaling factor per iteration.

Some of the particularly important features of the invention are the following.

The size of an LR patch is gradually increased (e.g., 1 -pixel at a time or per iteration), until the required scale is reached.

The LR image is downscaled with non-integer factors and example patches are collected, taking the phase information into account.

The K value (for K-nearest neighbor or K-NN search) is optimized by checking the compatibility of the HR patch at each step.

Overlapping areas are simply averaged.

A back-projection method with non-local means based kernel is used.

In one embodiment, the super-resolution processing is simplified or skipped for low textured portions of the image.

As shown in Fig.5 and according to one embodiment of the invention, super- resolution of a single image is performed by downsampling Si a LR input image by a first small factor (e.g. ¾) several times with different phases to obtain several different downsampled LR images Φ^-Φ^, separating the LR input image into overlapping blocks (e.g. 3x3 pixels each) x,, for each of the blocks x, searching one or more similar (preferably equal, or substantially equal) blocks of same size bi,i,bi,2,bi,3 in one or more or each of the downsampled LR images, and

determining blocks Ci,i ,Ci,2,Ci,3 at corresponding positions in the LR input image (wherein the blocks in the LR image are upsampled by the reciprocal of the first small factor, e.g. 4/3 so that they have a size of e.g. 4x4 pixels; they are determined just by their position). These blocks are locally optimized S₄ by linear approximation as described above to obtain locally optimized patches and then downsampled S₅ by the same small factor as previously (e.g. ¾) to obtain

. This patch is then compared to the original patch x₀ ^l and optimized to minimize the SSD (sum of squared distances) by varying the number of used patches b,,i, b,,2, bi,3 , or the weights Wi,i ,Wi,2,Wi,₃ of the used patches bi,i,bi,2,bi,3, or both (one or more of) the patches bi,i,bi,2,bi,3 and (one or more of) their weights Wi,i ,Wi,2,Wi,₃.

A flow-chart of one embodiment of the invention is shown in Fig .1 1 . In this embodiment, blocks with higher texture are treated different from blocks with lower texture. Each block is determined to be either highly textured or lowly textured. Super-resolution of a single image is performed by downsampling Si a LR input image by a first small factor (e.g. ¾) several times with different phases to obtain several different downsampled LR images Φ^-Φ^, separating the LR input image into overlapping blocks (e.g. 3x3 pixels each) x,, and determining highly textured blocks and lowly textured blocks respectively among the overlapping blocks x,. The method further includes for each of the lowly textured blocks Xi performing a simplified upsampling method, and for each of the highly textured blocks x, searching one or more similar (preferably equal, or substantially equal) blocks of same size bi,i,bi,2,bi,3 in one or more or each of the downsampled LR images, and determining blocks Ci,i,Ci,2,Ci,3 at corresponding positions in the LR input image (wherein the blocks in the LR image are upsampled by the reciprocal of the first small factor, e.g. 4/3 so that they have a size of e.g. 4x4 pixels; they are determined just by their position). These blocks are locally optimized S₄ by linear approximation as described above to obtain locally optimized patches and then downsampled S₅ by the same small factor as previously (e.g. ¾) to obtain z _d . This patch is then compared to the original patch x₀ ^l and optimized to minimize the SSD (sum of squared distances) by varying the number of used patches b,,i, b,,2, b,,3 , or the weights Wi,i ,Wi,2,Wi,₃ of the used patches bi,i,bi,2,bi,3, or both (one or more of) the patches bi,i,bi,2,bi,3 and (one or more of) their weights

Wi,i ,Wi,2,Wi,₃.

For the lowly textured blocks, any simplified upsampling method can be used that reduces the computation effort. For example, linear, bilinear, cubic or bi-cubic interpolation can be used for the lowly textured blocks. In one embodiment, the upsampling method for lowly textured blocks is similar to the upsampling method of the highly textured blocks, except at least one of the following: for each of the lowly textured blocks x, , searching less blocks (than for the highly textured blocks) of same size bi,i,bi,2,bi,3 that are similar or substantially similar to the lowly textured block, searching in less of the downsampled LR images (than for the highly textured blocks), using simplified local optimization as compared with the local optimization for the highly textured blocks, and using less patches b,,i, b,,2, bi,3 or using a limited range of weights Wi,i ,Wi,2,Wi,₃ for the used patches bi,i,bi,2,bi,3, or both, in the optimization to minimize the SSD, or skip the optimization to minimize the SSD.

A description of a separation of the blocks into highly textured and lowly textured blocks is given below.

With the resulting locally optimized patch x\ , the above-described procedure is then repeated at least once. However, the resulting locally optimized patch x\ that serves as input has now more pixels (e.g. 4x4) than the patch xf that was used in the previous iteration, and therefore also the new LR image and its downscaled versions are larger, the downscaling factor is different (e.g. 4/5) and more downscaled versions Φ0 - 15 exist because more positions and thus more phases (e.g. 4x4=16 phases for a 4x4 pixel patch) of the patch are possible.

Finally, the patches are super-imposed and averaged Ss, as described above, and the IBP S9 is performed to obtain the HR output image S10. It is a super- resolution version of the LR input image.

The above-mentioned searching one or more similar blocks of same size i,i,bi,2,bi,3 in one or more or each of the downsampled LR images can use any commonly known search algorithm, e.g. as used in motion estimation (e.g. SSD).

In one embodiment, the accumulating and averaging Ss comprises, for each pixel, counting overlapping patches that contribute to the pixel, adding values of overlapping patches that contribute to the pixel and dividing the accumulated values by the number of overlapping patches.

Further, it is important to note the following.

In each iteration (first iteration S1-S6, second iteration Si'-Se' and third iteration Si"-S6"), there are several downscaled images (e.g. Φ^-Φ^ i^{n tne} fi^rst iteration S1-S6), since the input image is downscaled with non-integer values. These downscaled images correspond to different phases of pixels caused by this non- integer downsampling. These provide the "neighbors" of the neighbor embedding. In steps ββ,ββ',ββ" a simple norm calculation is performed. It may be SSD (sum of squared distance). Optimization can be done by minimizing this norm (SSD).

Exemplary upscaling factors shown in Fig.2-5 are 3:4, 4:5 and 5:6, i.e. one additional pixel in each dimension per iteration. However, as described above, it can be generalized to be p pixels, p is considerably smaller than the block size, e.g. not more than ½ of the block size. E.g. it can be 1 ,2,3,... depending on the block size (n) and computational sources. For example, for a 3x3 low-resolution patch, three iterations are used to make it 6x6 when p=1 . So, the patch size is increased by p pixels until the required scaling is reached. As another example, for a scaling factor of three, if p=2 and n=3, factors would be 3/5, 5/7, 7/9, i.e. two additional pixel in each dimension per iteration.

In step S₇, the algorithm simply stops if the required scaling has been reached and y, is assigned to the final result.

Non-local means kernel in step Sg is a well-known adaptive filter for denoising applications. In the above-described embodiment, the back-projection kernel was changed with the Non-local means kernel, as it gives better results than using a Gaussian filter. Back projection is generally well known, but here it is a part of the algorithm and the kernel is NLM-based. In another embodiment however, back- projection kernel may be used instead. Fig.6 shows, in one embodiment, an apparatus for performing super-resolution of a single image, comprising a separator SEPi for separating the LR input image into overlapping blocks x, , one or more (here three) cascaded upscaling units CAU1-3, a HR image assembling unit IAU, and an IBP unit IU. Each cascaded upscaling unit provides its output x[, x , x₃ ^l as input to the next upscaling unit. Each cascaded upscaling unit CAUi-3 comprises one or more first downsamplers DUi for downsampling a LR input image by a first factor several times with different phases to obtain several different downsampled LR images Φ^-Φ^, a first selector SEL1 for selecting a current block x₀ ^l , one or more first search units SUi for searching, for each current block x₀ ^l of all the blocks x, , one or more similar blocks of same size bi_,i ,bi_,2,bi,3 of a current block x₀ ^l in one or more or each of the downsampled LR images, one or more first position determining units PDU1 for determining blocks Ci_,i ,Ci_,2,Ci,3 at corresponding positions in the LR input image, wherein the blocks in the LR image are upsampled in one or more up-samplers UPS by the reciprocal of the first factor, a first optimizer OPT1 for locally

optimizing the blocks by linear approximation to obtain locally optimized patches z (as in step S₄ of the method described above), a second downsampler DU2 for downsampling the locally optimized patches by the first factor to obtain a down- sampled locally optimized patch z _d, a first comparator CUi for comparing the downsampled locally optimized patch z£_d with the input block x₀ ^l , and a first control unit CTR1 for optimizing the number and/or weights w, ,i ,Wi,2,Wi,3 of the determined blocks Ci,i ,Ci_,2,Ci,3 , wherein the Sum-of-Square-Distance between the downsampled locally optimized patch z£_d with the input block x₀ ^l is minimized and wherein a locally optimized patch x\ is obtained.

The first and second downsampler DUi .Dlb and up-samplers UPS use a different factor (and its reciprocal) per each of the one or more cascaded upscaling units (e.g. ¾ for downsampling and 4/3 for upsampling in the first upscaling unit CAUi , 4/5 for downsampling and 5/4 for upsampling in the second upscaling unit CAU2, and 5/6 for downsampling and 6/5 for upsampling in the third upscaling unit CAU3). The last of the cascaded upscaling units CAU3 provides its output to the HR image assembling unit IAU. The HR image assembling unit IAU, in one

embodiment, accumulates and averages the locally optimized patch xf of the last cascaded upscaling unit CAU3 and provides its output to the IBP unit IU, which provides the final HR output image.

In principle, an optimal number K of patches, i.e. a number K that is optimal for a current block to be upsampled, can be calculated in various ways. In one embodiment, an optimal number K of patches is obtained by downsampling the reconstructed (or upsampled) block, and computing an error between a current block and the downsampled/reconstructed block. Different numbers K of patches are tested, and the number K is selected that leads to a minimization of the error. In one embodiment, the error is the sum of absolute differences. In another embodiment, the error is the square distance or square difference. In the following, a separation of the blocks into highly textured and lowly textured blocks is described.

In one embodiment, the separation of the blocks into highly textured and lowly textured blocks is done according to the blocks' energy. In one embodiment, the gradient energy is used. The gradient energy can be determined, for example, as a sum of squared spatial differences of luminance values within the block, or as a sum of horizontal and vertical gradients within the block. The higher the gradient energy, the higher the texturization of the block. In one embodiment, the block's energy is determined as the difference between a minimum and a maximum luminance value within the block. Then, the separation of the blocks into highly textured and lowly textured blocks is done according to the difference between a minimum and a maximum luminance value within the block. The higher the difference, the higher the texturization of the block.

In one embodiment, a threshold is defined for determining whether a block is highly textured or lowly textured. If the block energy is below the threshold, a simplified upscaling is used. In one embodiment, a pre-defined fixed threshold is used. A typical value (for block energy calculated according to min-max luminance difference) would be 5, i.e. if luminance_max - luminance_mi_n > 5 , the block is considered as highly textured. In one embodiment, an adaptive threshold is used. Since the threshold selection has an impact on both processing power and image quality, it may be adaptive with respect to available processing power (with a lower threshold for more available processing power), with respect to required image quality (with a lower threshold for higher image quality), or both. Also other factors may be considered in threshold selection, e.g. image size, bit amount per pixel etc. For example, the threshold may be in a range of 3-7 instead of 5. Note that the threshold needs to be scaled according to the employed energy measure, such that a reasonable amount of blocks is classified in each of high and low texture.

Fig .1 1 shows a flow-chart of a method, in one embodiment of the invention, where highly textured blocks are treated different from lowly textured blocks. In this embodiment, a method for performing super-resolution of a single image comprises steps of downsampling Si a LR input image by a first factor several times with different phases to obtain several different downsampled LR images ΦΟ- Β , separating the LR input image into overlapping blocks x, , and for each of the blocks determining S1 .1 a measure of its texture or a degree of its texturization, and determining S1 .2 according to the measure of its texture whether the block has high texture or low texture. Then, for each of the blocks having high texture, the following steps are performed:

searching one or more similar blocks bi,i,bi,2,bi,3 of same size of a current block x₀ ^l in one or more or each of the downsampled LR images,

determining blocks Ci,i ,Ci,2,Ci,3 at corresponding positions in the LR input image, wherein the blocks in the LR image are upsampled by the reciprocal of the first factor, locally optimizing S₄ the blocks by linear approximation to obtain locally optimized patches z ,

downsampling S₅ the locally optimized patches by the first factor to obtain a downsampled locally optimized patch z _d ,

comparing the downsampled locally optimized patch z£_d with the input block x₀ ^l and optimizing the number and/or weights w,,i ,Wi,2,Wi,3 of the determined blocks Ci,i ,Ci,2,Ci,3 , wherein a Sum-of-Square-Distance between the downsampled locally optimized patch z _d with the input block x₀ ^l is minimized and wherein a locally optimized patch x\ is obtained. Further, for each of the blocks having low texture, simplified upsampling S1 .6 is performed. The following steps are then repeated: downsampling S'i several times with different phases to obtain several different downsampled LR images Φ0- 15 , searching, determining, locally optimizing S'₄, and downsampling S'₅ , wherein for the downsampling a different second factor is used, and wherein for each repetition the locally optimized patch x\, xf , xf of the previous iteration is used as input block. Finally, the locally optimized patch xf of the last iteration is accumulating and averaged Ss , and IBP is performed S9 , wherein a HR output image S10 is obtained.

In one embodiment of the method, the simplified upsampling is one of linear interpolation, bilinear interpolation, cubic interpolation and bi-cubic interpolation.

In one embodiment of the method, the simplified upsampling for blocks with low texture comprises steps of

searching one or more similar blocks bi,i,bi,2,bi,3 of same size of a current block x₀ ^l in one or more or each of the downsampled LR images, wherein less blocks are searched than for blocks with high texture, or the number of downsampled LR images in which the searching is done in less than for blocks with high texture; determining blocks Ci,i ,Ci,2,Ci,3 at corresponding positions in the LR input image, wherein the blocks in the LR image are upsampled by the reciprocal of the first factor, locally optimizing S₄ the blocks by linear approximation to obtain locally optimized patches z ,

comparing the downsampled locally optimized patch z£_d with the input block x₀ ^l and optimizing the number and/or weights w,,i ,Wi,2,Wi,3 of the determined blocks Ci,i ,Ci,2,Ci,3 , wherein the Sum-of-Square-Distance between the downsampled locally optimized patch z£_d with the input block x₀ ^l is minimized and wherein a locally optimized patch x\ is obtained. In one embodiment of the method, the step of determining S1 .1 for a block a measure of its texture or a degree of its texturization comprises measuring the block's energy according to a maximum luminance difference within the block, or according to a sum of luminance gradients within the block.

In one embodiment of the method, the step of determining S1 .2 for a block whether it has high texture or low texture comprises comparing an energy measure of the block with a threshold, and a block having an energy measure above the threshold is determined to have high texture and a block having an energy measure below the threshold is determined to have low texture.

Fig.12 shows an apparatus according to one embodiment of the invention, where highly textured blocks are treated in a different way than lowly textured blocks. Additionally to the above-described apparatus, the apparatus in this embodiment further comprises a block energy measure determining unit EM for determining a measure of the current block's texture or a degree of its texturization, and a comparison and decision unit ECD for comparing the measure of texture with a threshold so as to determine according to the measure of its texture whether the current block has high texture or low texture. As shown in Fig.12, the one or more first search units SUi , one or more first position determining units PDUi , first optimizer ΟΡΤΊ , second downsampler DU2 , first comparator CUi and first control unit CTRi are operative for only those current blocks that have high texture x₀ ^l _HT , as determined to the comparison and decision unit ECD. The blocks having low texture Q.LT C"^"® upsampled in a simplified upsampling unit SUP by using any simplified upsampling method, as described above.

In one embodiment, the simplified upsampling unit SUP comprises one or more secondary first search units SUi' for searching one or more similar blocks of same size bi,i ,bi,2,bi,3 of a current block x₀ ^l in one or more or each of the downsampled LR images, wherein less blocks are searched than for blocks with high texture, or the number of downsampled LR images in which the searching is done in less than for blocks with high texture, one or more secondary first position determining units PDUi' for determining blocks Ci_,i ,Ci,2,Ci,3 at corresponding positions in the LR input image, wherein the blocks in the LR image are upsampled by the reciprocal of the first factor, secondary first optimizer OPTY for locally optimizing S₄ the blocks by linear approximation to obtain locally optimized patches ,

secondary second downsampler DU2' for downsampling the locally optimized patches by the first factor to obtain a downsampled locally optimized patch z , , and secondary first comparator CUi' for comparing the downsampled locally optimized patch z _d with the input block x₀ ^l and optimizing the number and/or weights w, ,i ,Wi_,2,Wi,3 of the determined blocks Q,i ,Ci_,2,Ci,3 , wherein the Sum-of- Square-Distance between the downsampled locally optimized patch z£_d with the input block x₀ ^l is minimized and wherein a locally optimized patch

is obtained.

In one embodiment, the block energy measure determining unit EM for

determining for a block a measure of its texture or a degree of its texturization comprises means, e.g. a processor, for measuring or calculating the block's energy according to a maximum luminance difference within the block, or according to a sum of luminance gradients within the block.

In one embodiment, the comparison and decision unit ECD for determining for a block whether it has high texture or low texture comprises a comparison unit, e.g. comparator, for comparing an energy measure of the block with a threshold, and a determining unit, e.g. switch, for determining blocks having an energy measure above the threshold to have high texture and blocks having an energy measure below the threshold to have low texture. Advantages of the invention can also be shown in experimental results. For example, in one embodiment the base enhancement factors are set to S = 2 and 3. To obtain larger ratios, the method is run multiple times with appropriate base enhancements. In an embodiment, the method is directly applied on concatenated RGB values of pixels. A good compromise between LR and HR patch sizes is found when n = 3, i.e., LR patches x, of size 3 x 3 and HR patches y of size 6 x 6 for {S, p} = {2, 1} and of size 9 x 9 for {S, p} = {3, 2}. Therefore, τ = 3 with {si , s₂, s₃}s=2 = {3/4, 4/5, 5/6} and {si , s₂, s₃} s=s = {3/5, 5/7, 7/9}. Allowed are 3 x 2 or 2 x 3 overlaps between two adjacent LR patches, leading to 6x4 or 4x6 (when S = 2) and 9 x 6 or 6 x 9 (when S = 3) overlaps between HR patches. K is set to 8. For IBP, M = 5 and the NLM kernel is of size 7x7. The kernel coefficients are calculated with blocks of sizes 7x7 using an exponential parameter σ = 0,01 . Fig. 5 demonstrates a sketch of the complete SR algorithm for {S, p, n} = {2, 1 , 3}. Experimental results of the algorithm, compared with different SR methods, are illustrated in Fig.7-10 (for the reported results of [9,10], images courtesy of their project web-pages). Fig.7 demonstrates a comparison between our result (right) and a result in [10] (left) for the "koala" image. This is a good example showing that the method according to the invention can provide more natural looking synthesis results, especially for fine-detailed texture areas, e.g. in the koala's fur. Figs.8-10 compare more results on different images for an upscaling factor of four. Particularly, Fig.8 (upper: [9], lower: present invention) shows the efficiency of using the NLM kernel for reducing the jaggy and ringing artifacts, e.g., in the ridge of the chip. From Fig.9 (left: [1 1], right: present invention) and Fig.10 (upper: bi-cubic; lower: present invention), also sharper and cleaner edges can be observed, compared to the different other methods.

In one embodiment, the method for performing super-resolution of a single image comprises a step of determining for each of the overlapping blocks x, whether a current block has high or low texture, and simplifying or skipping the subsequent steps if a current block has low texture. Thus, super-resolution according to the above-described method is performed only for the highly textured blocks, while for the lowly-textured blocks a simplified method is used. This reduces the

computational burden and makes the super-resolution processing faster. The determining of blocks with high or low texture is done, in one embodiment, immediately after the step of separating the LR input image into overlapping blocks Xi. In another embodiment, the step of determining blocks with high or low texture is performed initially, before said step of downsampling Si a LR input image. In this case, the blocks need not necessarily coincide with the overlapping blocks Xi, into which the image is separated later. For example, when highly or lowly textured regions are initially determined, there is a step of determining for each of the overlapping blocks x, whether or not it belongs to a highly or lowly textured region. As described above, each block has either high texture or low texture, so that determining one type of blocks implicitly yields also the other type.

In conclusion, a self-contained SR method based on optimized neighbor embed- ding of patch pairs is described. While making use of the recurrence property (i.e. self-similarity) of similar patches across different scales of an image, the given patch is gradually expanded, relying on geometric similarities of LR and HR patch spaces at small scaling ratios. Together with an adaptive IBP, the invention provides more natural looking textures and sharper edges with less artifacts when compared to other known methods. The invention is applicable e.g. for a complete GPU-oriented and totally localized implementation. As an advantage, it is faster or achieves higher performance respectively.

While there has been shown, described, and pointed out fundamental novel features of the present invention as applied to preferred embodiments thereof, it will be understood that various omissions and substitutions and changes in the apparatus and method described, in the form and details of the devices disclosed, and in their operation, may be made by those skilled in the art without departing from the spirit of the present invention. It is expressly intended that all

combinations of those elements that perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Substitutions of elements from one described embodiment to another are also fully intended and contemplated. For example, the terms "block" and "patch" are used synonymously. The same holds for "sampling" and "scaling".

It will be understood that the present invention has been described purely by way of example, and modifications of detail can be made without departing from the scope of the invention. Each feature disclosed in the description and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination. Features may, where appropriate be implemented in hardware, software, or a combination of the two. Connections may, where applicable, be implemented as wireless connections or wired, not necessarily direct or dedicated, connections. Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

Cited References

[I] M. Irani and S. Peleg, "Improving resolution by image registration," CVGIP: Graphical Models Image Process., vol. 53, no. 3, pp. 231-239, 1991 .

[2] M. Irani and S. Peleg, "Motion analysis for image enhancement: Resolution, occlusion, and transparency," J. Vis. Comm. Image Repres., vol. 4, pp. 324-335, 1993.

[3] S. Farsiu, M.D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super resolution," IEEE Trans. Image Process., vol. 13, no. 10, pp. 1327-1344, 2004.

[4] Z. Lin and H.-Y. Shum, "Fundamental limits of reconstruction- based superresolution algorithms under local translation," IEEE Trans. Pattern Anal. Mach. Intel!., vol. 26, no. 1 , pp. 83-97, 2004.

[5] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, "Learning low-level vision," Int. J.

Comp. Vis., vol. 40, no. 1 , pp. 25-47, 2000.

[6] W. T. Freeman, T. R. Jones, and E. C. Pasztor, "Example-based super-resolution,"

IEEE Comp. Graph. Appl., vol. 22, no. 2, pp. 56-65, 2002.

[7] H. Chang, D.-Y. Yeung, and Y. Xiong, "Super-resolution through neighbor

embedding," in IEEE Comp. Soc. Conf. Comp. Vis. Pattern Recogn., 2004, pp. 275-282.

[8] J. Yang, J. Wright, T. Huang, and Y. Ma, "Image super- resolution via sparse representation," IEEE Trans. Image Process., vol. 19, pp. 2861-2873, 2010.

[9] D. Glasner, S. Bagon, and M. Irani, "Super-resolution from a single image," in IEEE Int. Conf. Comp. Vis., 2009, pp. 349- 356.

[10] G. Freedman and R. Fattal, "Image and video upscaling from local self-examples,"

ACM Trans. Graph., vol. 28, no. 3, pp. 1-10, 2010.

[I I ] J. Salvador, A. Kochale, and M. Borsum, "Robust super- resolution for interactive video navigation," in IEEE Int. Conf. Consumer Electronics, 2012, pp. 333-334.

[12] L.I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms," Physica D: Nonlinear Phenomena, vol. 60, no. 14, pp. 259-268, 1992.

[13] S. Roweis and L. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, pp. 2323-2326, 2000.

[14] A. Buades, B. Coll, and J.-M. Morel, "A non-local algorithm for image denoising," in IEEE Comp. Soc. Conf. Comp. Vis. Pattern Recognition, 2005, pp. 60-65.

Claims

1 . A method for performing super-resolution of a single low-resolution input

image, comprising steps of

- downsampling (Si) the low-resolution input image by a first factor two or more times with different phases to obtain several different downsampled low-resolution images (Φο-Φ^);

- separating the low-resolution input image into overlapping blocks (x,), and for each of the blocks (x,) searching, in one or more or each of the down- sampled low-resolution images, one or more blocks (bi,i,bi,2,bi,3 ) having same size as a current block (x₀ ^l ) and being similar to the current block;

- determining blocks (c,,i ,Cj,2,Cj,3) at corresponding positions in the low- resolution input image, wherein the blocks in the low-resolution image are upsampled by the reciprocal of the first factor;

- locally optimizing (S₄) the blocks by linear approximation to obtain locally optimized patches (z );

- downsampling (S₅ ) the locally optimized patches (z ) by the first factor to obtain a downsampled locally optimized patch (z£_d) ;

- comparing the downsampled locally optimized patch (z£_d) with the input block (x₀ ^l ) and optimizing the number and/or weights (w, ,i ,Wi,2,Wi,3) of the determined blocks (Ci,i,Ci,2,Ci,3), wherein the Sum-of-Square-Distance between the downsampled locally optimized patch (z£_d) with the input block (x₀ ^l ) is minimized and wherein a locally optimized patch (xf) is obtained;

- repeating the steps of downsampling (S'i) two or more times with different phases to obtain several different downsampled low-resolution images

(Φ0-Φ15 ), searching, determining, locally optimizing (S'₄), downsampling (S'5 ), wherein for the downsampling a different second factor is used, wherein for each repetition the locally optimized patch (xf , xf , xf ) of the previous iteration is used as input block;

- accumulating and averaging (Ss) the locally optimized patch (xf ) of the last iteration; and - performing iterative back projection (Sg), wherein a high-resolution output image (S10) is obtained.

2. Method according to claim 1 , wherein the first factor and the second factor are non-integers.

3. Method according to claim 1 or 2, wherein the accumulating and averaging (Ss) comprises, for each pixel, counting overlapping patches that contribute to the pixel, adding values of overlapping patches that contribute to the pixel and dividing the accumulated values by the number of overlapping patches.

4. Method according to one of the claims 1 -3, wherein the iterative back

projection (Sg) uses a Non-local means kernel.

5. Method according to one of the claims 1 -3, further comprising, for each of the blocks (Xi) resulting from said step of separating the low-resolution input image into overlapping blocks (x,), steps of

- determining (S1 .1 ) a measure of its texture or a degree of its texturization; and

- determining (S1 .2) according to the measure of its texture whether the block has high texture or low texture;

wherein said steps of searching one or more similar blocks of same size, determining blocks (c,,i ,Cj,2,Cj,3) at corresponding positions in the low-resolution input image, locally optimizing (S₄) the blocks by linear approximation, down- sampling (S₅ ) the locally optimized patches by the first factor and comparing the downsampled locally optimized patch (z£_d) with the input block are performed only for each of the blocks having high texture, and wherein a simplified upsampling (S1 .6) is performed for each of the blocks having low texture.

6. Method according to claim 5, wherein the simplified upsampling is one of linear interpolation, bilinear interpolation, cubic interpolation and bi-cubic interpolation.

7. Method according to claim 5, wherein the simplified upsampling for blocks with low texture comprises steps of

- searching one or more similar blocks of same size (bi,i,bi,2,bi,3 ) of a current block (x₀ ^l ) in one or more or each of the downsampled low-resolution images, wherein less blocks are searched than for blocks with high texture, or the number of downsampled low-resolution images in which the searching is done in less than for blocks with high texture;

- comparing the downsampled locally optimized patch (z£_d) with the input block (x₀ ^l ) and optimizing the number and/or weights (w, ,i ,Wi,2,Wi,3) of the determined blocks (Ci,i ,Ci,2,Ci,3), wherein the Sum-of-Square-Distance between the downsampled locally optimized patch (z£_d) with the input block (x₀ ^l ) is minimized and wherein a locally optimized patch (x ) is obtained.

8. Method according to one of the claims 5-7, wherein the step of determining

(51 .1 ) for a block a measure of its texture or a degree of its texturization comprises measuring the block's energy according to a maximum luminance difference within the block, or according to a sum of luminance gradients within the block.

9. Method according to one of the claims 5-8, wherein the step of determining

(51 .2) for a block whether it has high texture or low texture comprises comparing an energy measure of the block with a threshold, and a block having an energy measure above the threshold is determined to have high texture and a block having an energy measure below the threshold is determined to have low texture.

10. An apparatus for perfornning super-resolution of a single low-resolution input image, comprising a separator (SEP) for separating the low-resolution input image into overlapping blocks (x,), one or more cascaded upscaling units (CAU1 -3), a high-resolution image assembling unit (IAU), and an iterative back projection unit (IU),

wherein each cascaded upscaling unit provides its output {x[, x , x₃ ^l ) as input to the next upscaling unit,

and wherein each cascaded upscaling unit (CAU) comprises

- one or more first downsamplers (DUi) for downsampling a low-resolution input image by a first factor several times with different phases to obtain several different downsampled low-resolution images (<& Q -<& Q );

- first selector (SEI_i) for selecting a current block (x₀ ^l );

- one or more first search units (SUi) for searching, for each current block (x₀ ^l ) of the blocks (x,), one or more similar blocks of same size (bi,i,bi,2,bi,3 ) of a current block (x₀ ^l ) in one or more or each of the downsampled low- resolution images;

- one or more first position determining units (PDUi) for determining blocks (Ci,i,Ci_,2,Ci,3) at corresponding positions in the low-resolution input image, wherein the blocks in the low-resolution image are upsampled in one or more upsamplers (UPS) by the reciprocal of the first factor;

- first optimizer (ΟΡΤΊ) for locally optimizing (S₄) the blocks by linear

approximation to obtain locally optimized patches (z );

- second downsampler (DU2) for downsampling (S₅) the locally optimized patches (z ) by the first factor to obtain a downsampled locally optimized patch (z£_d);

- first comparator (CUi) for comparing the downsampled locally optimized patch (z _d) with the input block (x₀ ^l ); and

- first control unit (CTR1) for optimizing the number and/or weights (w, _,1 , w_{i 2}, Wi,₃) of the determined blocks (Ci,i ,Ci,2,Ci,3), wherein the Sum-of-Square- Distance between the downsampled locally optimized patch (z _d) with the input block (x₀ ^l ) is minimized and wherein a locally optimized patch (x ) is obtained; wherein the first and second downsampler (DUi.Dl ) and upsamplers (UPS) use a different factor per each of the one or more cascaded upscaling units, and

wherein the last of the cascaded upscaling units (CAU3) provides its output to the high-resolution image assembling unit (IAU), the high-resolution image assembling unit (IAU) accumulating and averaging (S9) the locally optimized patch (xf ) of the last cascaded upscaling unit (CAU3) and providing its output to the iterative back projection unit (IU) which provides a high-resolution image.

1 1 . Apparatus according to claim 10, wherein the simplified upsampling unit

performs one of linear interpolation, bilinear interpolation, cubic interpolation and bi-cubic interpolation.

12. The apparatus according to claim 10 or 1 1 , further comprising

- block energy measure determining unit (EM) for determining (S1 .1 ) a measure of the current block's texture or a degree of its texturization;

- comparison and decision unit (ECD) for comparing the measure of texture with a threshold so as to determine (S1 .2) according to the measure of its texture whether the current block has high texture or low texture; and

- simplified upsampling unit (SUP) for performing, for each of the blocks having low texture ( O.LT). simplified upsampling (S1 .6);

wherein said one or more first search units (SUi), one or more first position determining units (PDU₁), first optimizer (OPT₁), second downsampler (DU₂) , first comparator (CUi) and first control unit (CTR₁) are operative for only those current blocks that have high texture {X₀ ¹,HT) according to said comparison and decision unit (ECD).

13. Apparatus according to claim 12, wherein the simplified upsampling unit

(SUP) comprises

- one or more secondary first search units (SUi') for searching one or more similar blocks of same size (bi,i,bi,2,bi,3 ) of a current block (x₀ ^l ) in one or more or each of the downsampled LR images, wherein less blocks are searched than for blocks with high texture, or the number of downsampled low-resolution images in which the searching is done in less than for blocks with high texture;

- one or more secondary first position determining units (PDUi') for

determining blocks (c,,i ,Cj,2,Cj,3) at corresponding positions in the LR input image, wherein the blocks in the low-resolution image are upsampled by the reciprocal of the first factor;

- secondary first optimizer (OPTY) for locally optimizing (S₄) the blocks by linear approximation to obtain locally optimized patches (z );

- secondary second downsampler (DU2') for downsampling (S₅ ) the locally optimized patches (z ) by the first factor to obtain a downsampled locally optimized patch (z£_d);

- secondary first comparator (CUi') for comparing the downsampled locally optimized patch (z _d) with the input block (x₀ ^l ) and optimizing the number and/or weights (Wi,i ,Wi,2,Wi,₃) of the determined blocks (c,,i ,Ci,2,Ci,3>, wherein the Sum-of-Square-Distance between the downsampled locally optimized patch (z _d) with the input block (x₀ ^l ) is minimized and wherein a locally optimized patch (x ) is obtained.

14. Apparatus according to one of the claims 12-13, wherein the block energy measure determining unit (EM) for determining for a block a measure of its texture or a degree of its texturization comprises means for measuring or calculating the block's energy according to a maximum luminance difference within the block, or according to a sum of luminance gradients within the block.

15. Apparatus according to one of the claims 12-14, wherein the comparison and decision unit (ECD) for determining for a block whether it has high texture or low texture comprises a comparison unit for comparing an energy measure of the block with a threshold, and a determining unit for determining blocks having an energy measure above the threshold to have high texture and blocks having an energy measure below the threshold to have low texture.