WO2013167758A1 - Micro lens array and imaging apparatus - Google Patents

Abstract

A micro-lens array for a light-field camera where the centres of the micro-lenses are displaced versus a regular lattice. The small displacements are defined by the parameters of the light-field camera and optionally by user defined parameters, and arranged in a particular fashion in order to provide an improved distribution of a reconstructed image.

Description

MICRO LENS ARRAY AND IMAGING APPARATUS

The invention relates to light-field camera and to a micro-lens array for light-field camera.

Light-Field cameras record 4D (four dimensional) light-field data which can be transformed into various reconstructed images like re-focused images with freely selected focal distance that is the depth of the image plane which is in focus. A re- focused image is built by projecting the various 4D light-field pixels into a 2D (two dimensional) image. Unfortunately the resolution of a re-focused image varies with the focalization distance.

For example, the publication US 2010/0265381 A1 , "Imaging Device" proposes an imaging device with a micro-lens array where the micro-lenses are displaced from an equidistant arrangement to a non-linear arrangement according to the height of the image on the imaging element. The pitch between the micro-lenses changes from the centre to periphery of the micro-lens array. The displacement provides an optical correction in order to compensate the image displacement implied by the geometric distortion of the main lens. The publication provides however no solution for improving the resolution of reconstructed images when the focal distance changes.

Particular aspects of light-field cameras will first be exposed. Light-Field cameras design.

We consider light-field cameras which record a 4D light-field on a single sensor like a 2D regular array of pixels. Such light-Field cameras can be for instance: 1 ) a plenoptic camera comprising a main lens, an array of lenses and a sensor 12; or

2) a multi-camera array comprising an array of lenses and a single sensor, but without main lens.

The array of lenses is often a micro-device, which is commonly named a micro-lens array. Figure 1 illustrates a plenoptic camera 1 with three major elements: the main lens 10, the micro-lens array 1 1 and the sensor 12. Figure 2 illustrates a multi-camera array 2 with two major elements: the micro-lens array 1 1 and the single sensor 12.

Optionally spacer or spacer material may be located between the micro-lens array around each lens and the sensor to prevent light from one lens overlapping with the light of other lenses at the sensor side.

It is worth noting that the multi-camera array can be considered as a particular case of plenoptic cameras where the main lens has an infinite focal length. Indeed, a lens with an infinite focal length has no impact on the rays of light. The present invention is applicable to plenoptic cameras as well as multi camera arrays.

4D Light-Field data

Figure 4 illustrates the image which is recorded at the sensor. The sensor of a light- field camera records an image of the scene which is made of a collection of 2D micro-images, also called small images, arranged within a 2D image. Each small image is produced by a lens from the array of lenses. Each small image is represented by a circle, the shape of that small image being function of the shape of the micro-lens. A pixel of the sensor is located by its coordinates (x,y) . p \s the distance in pixels between two centres of contiguous micro lens images. The micro- lenses are chosen such as p is larger than a pixel width. A micro-lens image is referenced by its coordinates . Some pixels might not receive any light from any micro-lens; those pixels are discarded. Indeed, the space between the micro-lenses can be masked to prevent photons falling outside of a lens (if the micro-lenses are square or another close packed shape, no masking is needed). However most of the pixels receive the light from one micro-lens. The pixels are associated with four coordinates (x,y) and . The centre of the micro-lens image on the sensor is labelled (χ^.,γ^) . Figure 4 illustrates the first micro-lens image (0,0) centred on

(x_{0 0},j_{0 0}) . The pixels of the sensor 12 are arranged in a regular rectangular lattice.

The micro-lenses are arranged in a regular rectangular lattice. The pixels lattice and the micro-lenses lattice are relatively rotated by Θ . The coordinate (χ^,γ^) can be written in function of the 4 parameters: p , Θ and (x_0;0,j¾₀) :

Figure 4 also illustrates how an object, represented by the black squares 3, in the scene is simultaneously visible on numerous micro-lens images. The distance w between two consecutive imaging points of the same object 3 on the sensor is known as the disparity. The disparity depends on the physical distance between the camera and the object, w converges to p as the object becomes closer to the camera. Depending on the light-field camera design, w is either larger or smaller than p (if d is respectively larger or smaller than / , (see Figure 3 and next sub-section about the geometrical property of the light-field camera). Figure 4 illustrates a case where w is smaller than p . An important characteristic is the number r of consecutive lenses through which an object is imaged, r is in units of i . It is estimated by considering the cumulated disparity on consecutive lenses: wr - pr < p . One obtains the following characteristic for the number of replications, r :

Where \_a\ denotes the ceiling value of a . This equation is an estimation which assumes that the micro-lens images are squared with no left-over space (i.e. Close packed or abutting), r is given for one dimension, an object is therefore visible in r² micro-lens images considering the 2D grid of micro-lens. Without the ceiling function, r would be a non-integer value, r is in fact an average approximation. In practice, an object can be seen r or r + 1 times depending on rounding effect. Geometrical property of the light-field camera

The previous section introduced w the disparity of a given observed object, and p the distance between two consecutive micro-lens images. Both distances are defined in pixel units. They are converted into physical distances (meters) W and P by multiplying respectively w and p by the pixels size δ of the sensor: W = 5w and Ρ = δρ . The distances W and can be computed knowing the characteristics of the plenoptic camera. Figure 3 gives a schematic view of the plenoptic camera with the following elements:

• The main lens 10 is an ideal thin lens with a focal distance F .

• The micro lens array 1 1 is made of micro-lenses having a focal distance / .

The pitch of the micro-lenses is φ . The micro-lens array is located at the fix distance D from the main lens. The micro-lenses might have any shape like circular or squared. The diameter of the shape is smaller or equal to φ . One can consider the particular case where the micro-lenses are pinholes. In this case the following equation remains valid with f = d .

• The sensor 12 is made of a squared lattice of pixels having each a physical size of δ . δ is in unit of meter per pixel. The sensor is located at the fix distance d from the micro-lens array.

• The object (not visible in the figure 3) is located at the distance z of the main lens. This object is focused by the main lens at a distance z' from the main lens. The disparity of the object between two consecutive lens is equal to W . The distance between 2 micro-lens image centres is P .

Following the mathematics of thin lenses we have:

11-_L

z z' ^~ F

From the Thales law we can derive that:

D - z' D - z'+d

φ W

Mixing the 2 previous equations the following equation is easily demonstrated:

This equation gives the relation between the physical object located at distance z from the main lens and the disparity W of the corresponding views of that object. This relation is build using geometrical considerations and does not assume that the object is in focus at the sensor side. The focal length / of the micro-lenses and other properties such as the lens apertures allow determining if the micro-lens images observed on the sensor are in focus. In practice, one typically tunes the distance D and d once for all using the relation: l ₊1_ j_

D - z' d f

The micro-lens images observed on the sensor of an object located at distance z from the main lens appears in focus so long as the circle of confusion is smaller than the pixel size δ . In practice the range [z_m, z_M ] of distances z which allows observing in focus micro-images is large and can be optimized depending on the focal length / , the apertures of the main lens and the micro-lenses, the distances D and d : for instance one can tune the micro-lens camera to have a range of z from 1 meter to infinity [l,∞] .

Also from the Thales law one derives P :

_ D + d

^{6 ~} D (7)

Ρ = φε

The ratio e defines the enlargement between the micro-lens pitch and the micro-lens images pitch projected at the sensor side.

Variation of the disparity

The light-field camera being designed, the values D , d , / and F are tuned and fixed. The disparity W varies with the object distance. One can note particular values of W :

• W foe ^{is tne} disparity for an object at distance z_focus such that the micro-lens images are exactly in focus, it corresponds to equation (6) . Mixing equations (4) and (6) one obtains: W_f = φ— (8) is the disparity for an object located at distance

lens. According to equation (5) one obtains:

The variation of disparity is an important property of the light-field camera. The ratio W_aF / W_f0CUS is a good indicator of the variation of disparity. Indeed the micro-lens images of objects located at z_focus are sharp and the light field camera is designed to observed objects around z_focus which are also in focus. The ratio is computed with equations (8) and (9):

The ratio is very close to one. In practice the variations of disparity is typically within few percent around W_focus . The present inventor has further brought to light the following aspects. Image refocusing method

A major interest of the light-field cameras is the ability to compute 2D images where the focal distance is freely adjustable. To compute a 2D image out of the 4D light- field, the small images observed on the sensor are zoomed, shifted and summed. A given pixel (x,y) of the sensor associated with the micro-lens is projected into a 2D image according to the following equation:

Where (Χ, Υ) is the coordinate of the projected pixel on the 2D refocused image. The coordinate (X, Y) is not necessarily integer. The pixel value at location (x,y) is projected into the 2D refocused image using common image interpolation technique. Parameter s controls the size of the 2D refocused image, and g controls the plane which is in focus (the plane perpendicular to the optical axis, for which the 2D image is in focus) as well as the zoom performed on the small images. The output image is s² times the sensor image size. In this formulation the size of the re-focused image is independent from the parameter g , and the small images are zoomed by sg .

The previous equation can be reformulated due to the regularity of position of the centres of the micro-lens images.

[X = sgx + sp(l - g)(cos0 - i - sin Θ^■ j) + s(l - g)x_{0 0}

[Y = sgy + sp{\ - g)(sin 0 · / + cos# · j) + s{\ - g)y_0fi

The parameter g can be expressed as function of p and w . It is computed by simple geometry. It corresponds to the zoom that must be performed on the micro- lens images, using their centres as reference, such that the various zoomed views of a same objects get superposed. One deduces the following relation: g = -^- (13) p— w This relation is used to select the distance z of the objects in focus in the projected image. The value of g can be negative depending on the light-field camera design. A negative value means that the micro-lens images need to be inverted before being summed. One notices that r =

.

Including this last relation into equation (12) one rewrites the projection equation:

The last formulation has the great advantage to simplify the computation of the projected coordinate by splitting the pixel coordinates (x,y) and the lens coordinates (i ) of the 4 D light-field.

Sampling property of the refocused image

The different pixels of the light-field image are projected into the re-focused image according to the above described method and define a set of projected coordinates (X,Y) into the grid of the refocused image. It has been recognized by the present inventors that the distribution of the set of projected coordinates is an important property which can be used to characterise the resolution of the refocused image, and in particular, the regularity or homogeneity of the distribution. As will be explained later the present invention addresses this homogeneity.

It is not trivial to characterise the homogeneity of the projected 4D light-field pixels into the 2D re-focused image. To study this property one initially considers a simple projection equation assuming that the rotation angle Θ is zero, and the coordinate of the first micro-lens centre (x_0;0,j¾₀) is equal to (0,0) . We will return to consider nonzero values for Θ later. One therefore obtains the following simplified projection equation with u = sg :

X = ux - uwi = ^SP (x - wi)

p - w

Y = uy - uwj = ^SP (y - wj)

p - w

This set of equation shows a simple relation between the 4 dimensions x,y,i,j and the projected coordinates (X, Y) . The value u = sg is a constant independent of w if s = k/g where k is any constant. In this condition, the size of the re-focused image is function of w and is equal to kl g times the size of the original image. Figure 5 illustrates the 1 D projected coordinate X for a particular settings: 5 = 0.5 , g = l. il, w = 151.05 , « = 3.83 and p = 173.67 . The x-axis shows the projected coordinates , the y-axis indicates the micro-lens coordinates i of the projected pixels. One notices that 8 micro-lenses contribute to the observed projected coordinates , which in this case is equal to r + 1 . The distribution of the projected points X is not homogeneous since the values h and H representing respectively the minimum and the maximum sampling steps between 2 consecutive projected coordinates are substantially different from each other. In this example, the projected coordinates are nearly superposed, clustered in groups of eight. Figure 6 illustrates the same view with the same settings except that w = 151.25 . The distribution of the projected points X is homogeneous, the projected points being distributed with equal spacing along the axis X . One notices this ideal case where h = H = u/{w) . Where {a} denotes the fractional part of a . {¾>} plays a major role in the maximum sampling steps between the projected coordinates. Several cases of h and H occur depending on {¾>}

With (w) = 0 : h = 0 and H = u . On average, r projected coordinates X overlap. The distance between 2 non-overlapped consecutive X is constant and equal to H . The projected coordinates define a perfect sampling with a constant sampling step equal to H = u .

With {w} = n I N where n and N are positive integers such as 0 < « < N < r . ln this case the number of overlapped projected coordinates X is equal, on average, to rgcd(n, N) / N where gcd(n, N) refers to the greatest common divisor between n and N . The projected coordinates define a perfect sampling with a constant sampling step equal to H = ugcd(n,N)/ N . The sampling step is smaller if N is a prime number. Indeed, if N is not a prime number, the number of overlapped coordinates increase as well as the sampling step. The perfect sampling of the projected coordinates X with the smallest sampling step is obtained for {w} = n / r and gcd(n,r) = l . ή = η / Ν where n and N are positive integers such as 0 < n < N and In this case there are no overlapped projected coordinates^ . But the sampling defined by the projected coordinates is not perfect: h = u gcd(n, N)/ N and H = u - h . The projected pixels are clustered, in other word some projected pixels are sampled with a small sampling step equal to h , where other pixels are sampled with a larger sampling step H : the projected coordinates appear clustered.

H is a good indicator to estimate the resolution of the re-focused image. Figure 7 illustrates the normalized sampling step as a function of {w} for a conventional light field camera characterized by /? = 173.67 and w_/OCMi « 150 . This function is built from the 3 cases described above: points surrounded by the black circles depict the first case ( h = 0 and (w} = 0 ); points surrounded by the empty circles depict the second case (all the possible regular grids) ; other points lying on the dark line segments correspond to the third case with all possible n values and any N > r (all possible irregular grids). The best possible resolution is given by h = H lu = \lr = 111 .

Problem of light-field cameras

The projection of the 4D light-field pixels defines a set of projected points having a distribution which depends on the selected focal distance (i.e. The object plane which it is desired to be in focus). As explained above, the resolution of the re-focused image highly depends on the distribution of the projected coordinates (X, Y) . The resolution can be estimated by the maximum sampling stepH . Unfortunately, H depends on {w} and varies from values of u to u lr . Variations of H make the resolution of the re-focused image vary. The present inventors have recognized that the distribution characterizes the resolution of the projected image. It is an object or at least one aspect of the invention to mitigate sampling variation in reconstructed images and therefore to obtain a more constant resolution of the reconstructed images for any selected focal distance.

The present invention discloses a micro-lens array, suitable for use in a light-field camera for example, where the centres of the micro-lenses are slightly displaced versus a regular lattice. The small displacements are preferably defined by characteristic parameters of the light-field camera and in certain embodiments can advantageously be arranged in order to provide an optimal distribution of the projected image.

In accordance with one aspect of the present invention there is provided a micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein the micro-lens array comprises a plurality of micro-lens sub-sets, each sub-set comprising a two dimensional array of (Q) micro lenses, and wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a common pattern, the common pattern defining different displacements for each micro-lens of the subsets.

The common pattern defines a displacement model whereby each micro-lens of a sub-set is located according to the common pattern but differently located with respect to the regular lattice. The pattern, in certain embodiments, defines a number of possible displacements and hence positions, for each micro-lenses. Although it is simpler for the actual displacements of micro-lenses in different sub sets to be the same, embodiments of the invention allow different subsets of the plurality to have different displacements, while still adhering to the same, common pattern or model of displacements. However, the pattern or model is such that even with a certain degree of flexibility provided for each lens displacement, the relative displacements between micro-lenses of a subset adhere to a controlled relationship, and such relationship is observed similarly across subsets of the plurality.

Thanks to these characteristics, the resolution of an image reconstructed from the micro-images is improved over conventional light-field cameras. In particular a good diversity of sampling is obtained while variations of resolution are avoided. Thus a more regular resolution is obtained for any focalization distance. Typically, all subsets of the micro-lens array share a common pattern. However, the plurality of subsets need not encompass the whole micro-lens array. It could be envisaged for example that a first common pattern could apply to a first plurality of subsets, and a second common pattern could apply to a second plurality. Advantageously, the common pattern defines each displacement as a function of the position of each micro-lens within the sub-set. The common pattern may further define each displacement as a function of the number (Q) of micro-lenses in each sub-set. The obtained dispositions of the micro-lenses provide for an advantageous distribution for the projected image and reduce the super-position or the clustering of pixels in a reconstructed image, which permits to obtain a more constant resolution for any focalization distance.

According to one embodiment, the common pattern defines displacements in integer multiples of unit displacement vectors. The vectors are preferably orthogonal vectors. These features allow reducing the super-position or the clustering of pixels while reconstructing an image from the micro-images. The magnitude (τ ) of the unit displacement vectors is advantageously a function of the focal distance of micro- lenses.

In another embodiment, the magnitude of the unit displacement vectors is a function of the number (Q) of micro-lenses in each sub-set. The multiple of the unit vectors for each micro-lens may further be a function of the position of the micro-lens within the sub-set. The resulting dispositions of the micro-lenses provide for an advantageous distribution for the projected image and reduce the super-position or the clustering of pixels in a reconstructed image.

In a particular embodiment, the sub-sets comprise a square array of NxN = Q micro- lenses. Furthermore, the common pattern defines a plurality of possible displacements for each micro-lens, each of said plurality being equivalent in modulo N. In a further embodiment, the displacement of at least one micro-lens in each subset is zero. This allows providing a common reference in each micro-lens sub-set so that the computation of a reconstructed image is simplified. Furthermore, it allows the relative position between the micro-lenses to be precisely obtained. This feature also simplifies the fabrication of the micro-lens array. Advantageously, the common pattern and the associated displacements are independent of the location of the sub-set within the micro-lens array.

The micro-lens array as set out above may be embodied in an imaging device including a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a microimage.

In accordance with another aspect of the present invention there is provided a micro- lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein the micro-lens array comprises a plurality of micro-lens sub-sets, each sub-set comprising an array of NxN micro lenses, each micro-lens of the subset having a focal distance f, wherein micro-lenses of each subset are displaced relative to the regular lattice according to a displacement pattern, said displacement pattern defining the displacement of each micro-lens as an integer multiple of unit vectors, said unit vectors having a magnitude τ wherein τ is a function of f/N.

These features allow reducing the super-position or the clustering of pixels when reconstructing an image from the micro-images. Advantageously, the magnitude has a fixed value representing characteristics of the imaging device.

According to a further aspect, the present invention also relates to an imaging device comprising the disclosed micro-lens array.

In embodiments of the invention, the displacement (if any) of each micro-lens is of the order of 1/1000 of the micro-lens pitch. Embodiments may therefore have displacements in the range of approximately 0-5μηη, or approximately .0-1 Ομηη for example. Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which:

Figure 1 is a schematic view of a light-field camera;

Figure 2 is a schematic view of a particular light-field camera;

Figure 3 is a detailed view of a light-field camera made of perfect lenses;

Figure 4 is a schematic view of the 4D light-field data recorded by the 2D image sensor of a light-field camera;

Figure 5 is an illustration of the coordinates of the projected 4D Light-Field pixels into the 2D projected image in a normal case;

Figure 6 is an illustration of the coordinates of the projected 4D Light-Field pixels into the 2D projected image in a case associated with a particular disparity;

Figure 7 illustrates a normalized maximum sampling step of the projected/ reconstructed image from 4D light-field data for a conventional light-field camera; Figure 8 is a schematic view of a micro-lens array according to an embodiment of the invention with displaced micro-lenses for a super-resolution factor N=2;

Figure 9 is a schematic view of the a micro-lens array according to an embodiment of the invention with displaced micro-lenses for a super-resolution factor N=3;

Figure 10 illustrates a normalized maximum sampling step of the projected/ reconstructed from 4D light-field data obtained with the micro-lens array of figure 8 with displaced micro-lenses for a super-resolution factor of N=2;

Figure 11 illustrates a normalized maximum sampling step of the projected/ reconstructed from 4D light-field data obtained with the micro-lens array of figure 8 with displaced micro-lenses for a super-resolution factor of N=3.

Figure 12 shows positioning of a micro-lens array relative to a sensor array for a rotation angle Θ with a freely selected position of the first micro-lens.

Figure 13 shows a schematic view of a micro-lens array with displaced micro-lenses for a super-resolution factor N=2 and a rotation angle Θ = arctan(l/2)

Figure 14 shows a schematic view of a micro-lens array with displaced micro-lenses for a super-resolution factor N=3 and a rotation angle Θ = arctan(l/2)

Figure 15 shows a schematic view of a micro-lens array with displaced micro-lenses for a super-resolution factor N=2 and a rotation angle Θ = arctan(l) Figure 16 shows a schematic view of a micro-lens array with displaced micro-lenses for a super-resolution factor N=3 and a rotation angle Θ = arctan(l)

Figure 17 shows sampling of 4D light-field pixels on2D re-focus image depending on the disparity

Figure 18 illustrates a first lens array design having multiple groups of sub-sets.

Figure 19 illustrates a second lens array design having multiple groups of sub-sets. Figure 20 illustrates a third lens array design having multiple groups of sub-sets. Figure 21 shows a general purpose processing device

Figure 22 shows a process for projecting a 2D image from a 4D light field

Figure 23 shows schematically an image capture device

Micro-lens array with displaced micro-lenses

The pixels of the 4D light-field image are projected into a re-focus image. As described above, the maximum sampling step of the projected coordinates depends on {w} the fractional part of the disparity. The variations of sampling step are due to the superposition or clustering of the projected coordinates for certain values of {w} as illustrated in figure 5.

To decrease the superposition or the clustering of the projected coordinates (X, Y) , the micro-lens images are shifted as compared to a regular array, so as to reduce or prevent overlapping or clustering of projected pixels. In other words, in embodiments of the invention the centre of a given micro-lens is shifted by the given shift

(Δ.(ζ^',/),Δ₇. (ζ^',7)) so that the modified projected coordinates (Χ', Υ') of this new light- field camera would become:

(Δ.(ζ^',/),Δ₇. (ζ^',7)) are shifts given in unit of distance between the micro-lens centres, or the micro-lens image centres. The motivation of moving the micro-lenses is to have a perfect and constant sampling of the projected coordinates (Χ', Υ') for any w = + n / N where N is a selected positive integer smaller or equal to r , and n is any integer such as n e [o, N[ .ln conventional light-field, the sampling step is a function of n for a given TV . If the sampling step is made independent of n , then TV acts as a super-resolution factor. Equation (16) becomes:

X" = —X' = N(^j -LwJi)- ⁱⁱ-A_i ^.(ⁱ,7)(NL^wJ ^{+ I})

N ⁽¹⁷⁾

Y" = —T = Niy-lwlj nj-A_jii iNlwj + n)

u

(X",Y") are normalized projected coordinates such that (X",Y") are integers for a perfect sampling of the projected coordinates (Χ',Υ'). For a perfect sampling A.(z^',y^')(NLvJ+/?) and

must also be integers respectively equal to k(i,j) and . These constraints give us the following values for (A.(z,y),A₇ (z,y)) :

The displacement of the micro-lens images depends on w. In other words, for a given micro-lens displacement (A.(z,y),A₇(z,y)), the shift of the corresponding micro- lens image depends on the disparity w. The previous equations can be approximated by taking into consideration the two considerations: 1) w» N; and 2) the variations of w are small, w can be considered constant and equal to w_focus.

Indeed, it has been shown (cf. equation (10)) that the ratio W^IW^, which is equal to w_aF lw_focus, is typically very close to 1. In this condition, equation (18) can be approximated by:

W focu, TV W focus TV ₍₁₉₎

W ^rr focus T ^JV^V W ^rr focus T ^JV^V

The second line of the previous equation is given knowing that w_f0CUS =W_f0CUSl δ , where δ is the physical size of a pixel. The approximation of

(/,/)) does not depend on the disparity w. Thus, by using this approximation, it is possible to build a micro-lens array with an irregular grid of micro-lenses such as the projected coordinates (Χ',Υ') do not overlap or cluster as it happens for the projected coordinates (X,Y) of conventional light-field imaging devices. Condition for an optimal micro-lens displacement for optimal homogeneity

A remaining question is how to define the 2 functions k(i,j) and to have optimum micro-lens displacements such that the projected coordinate (X",Y") have a minimum clustering, and a perfect sampling when w =

+ nl N . Equation (17) can be simplified considering equation (19) and w » N :

To obtain a perfect sampling the set of projected coordinates (X",Y") defined by the various lens coordinates must have all possible integer values whatever n , and also the number of contiguous lenses to obtain the perfect sampling must be minimum and equal to N (considering one dimension). This constraint can be reformulated by taking into consideration modular arithmetic modulo N :

X" ≡ N(x-[_w )-ni-k(i,j) ≡ -ni-k{i,j) (modN)

Y" ≡ N(y-[w]j)-nj-l(i,j) ≡ -nj-l(i,j) (modN) ⁽²¹⁾ k(i,j) and are 2 periodic functions, one period is defined with e [0,N[² , into [0,N[. One is searching k(i,j) and such that for any given ne[o,N[, all the set of projected coordinates ( "modN,7"modN) defined by

(z^'modN, /modN) e [o,N[² is equal to Σ^^Σ^¹ _*^ where 8_ab is the Dirac function located at (a,b) with (a,b) being integer numbers.

Theoretical solution

To solve equation (21) the following linear solutions are considered:

Equation (21) becomes:

\γ"≡ -nj-Ci-Ej-L (modN)

For the second member of the previous equation, one derives the value of Ci , which also appear in the first member after multiplying it by C :

(CX"≡ -Cni-CAi-CBj-CK (modN)

I Ci≡ -Y"-nj-Ej-L (modN)

Replacing the second member into the first member (24) becomes:

X"≡-j(n² +n(A +

+ A)-CK (modN) (25)

The set of projected coordinates located at ( "modN,7"modN) must cover all coordinates Σ^^Σ^¹ _*^ for (z^'modN, y^'modN) e [0,N[² whatever we[o,N[. With equation (25) one deduces that "(modN) must have any possible values whatever 7"(modN). Thus the second order polynomial function m(n) = n² + n(A + E)+AE-BC must verify: gcd(m(n) mod N,N) = 1 VR G [θ, N[

gcd(n² +n(A + E)+AE-BC(modN),N)=l Vne[0,N[ ^

The NxN micro-lenses defines a sub-set of micro-lenses. For a given sub-set the values A,B,C,E,K,L are freely selected according to the previous equation. The parameters A,B,C,E,K,L may take different values in the different sub-sets. Parameters K,L define which if any micro-lens from a given subset is not displaced with respect to the regular lattice.

Experimental solution

Many values A,B,C,E verify equation (26). The special case: A = 0, B = T, C = \, E = l, K = 0 and L = 0 is detailed in this section. The proposed solution has the following form:

T is a free parameter which has been experimentally determined for various values N . The experimentation consists in testing various values of e[o, N[ such that the constraint gcd(m(n)modN, N)= l is respected for any « e [o, N[ . The following table indicates the smallest value of T according to that constraint:

It follows that the periodic functions k(i,j) and are fully characterized and thus the shifts (Δ^/),Δ_; (/,/)) of the micro-lens image versus the regular grid are also fully characterized. The shifts are given in unit of To convert the shifts into physical unit at the micro-lens side, the shifts must be multiplied by φ . The physical shifts (¾(z^",y^"),A; (/,/)) at the micro-lens side are computed easily by combining equation (8) and (19):

The physical shifts can be decomposed in the increment τ = ·^/^_Ν which is multiplied by the integers values given by k(i,j) and to obtain the physical shifts. The increment τ is independent from the characteristics of the main lens. Thus the main lens can be replaced by any optical system which delivers a focus images located perpendicularly to the main optical axis at location z' (as illustrated in Figure 3). This invention applies to many light-field cameras such as: an array of cameras (as illustrated in Figure 2); a plenoptic camera (as illustrated in Figure 1 ); a plenoptic camera where the main lens is a zoom which delivers zoomed images in focus at location z' .

The design of the micro-lens array is therefore defined by:

• The focal distance / of the micro-lenses.

• The average pitch φ between consecutive lenses.

• The distance d between the micro-lens array and the sensor.

• The pixel size δ of the sensor.

• The super-resolution factor N which is freely selected between [l,r] .

• The micro-lens centres ( ,, ,) are located following the equation:

It should be recalled that the functions k(i, and are defined modulo N : thus the centres ( ,, ,) are valid as well as whatever a being an

integer. Consequently the displacements can be negative.

The micro-lens array is designed according to the previous settings. If the size of the micro-lenses is equal to the pitch φ , then the micro-lenses might have a very small overlaps due to displacement the micro-lenses versus the squared lattice. This issue is solved by designing micro-lenses such that the micro-lens size is smaller than _ /<5(N - l^/^ ghgpg ₀f the micro-lenses can be circular, rectangular or any shape without any modification of the previous equations. The number of micro-lens to be designed in the micro-lens array is defined such that

is equal to the physical size of the sensor. The micro-lens array being designed, it is located at distance d from the sensor. It is interesting to note that the above demonstration remains valid whatever the coordinates of the first lens are and whatever the angular position between the micro-lens array and the lattice of pixels is.

Micro-lens array design

An imaging device including the above proposed arrangement as well as a micro- lens array will now be described. The following values are chosen for the different parameters:

Symbols Values Comments

F 70mm Main focal distance

f 2mm Micro-lens focal distance

d 2.3mm Distance between the micro-lens array

and the sensor

Φ 1 mm Micro-lens pitch

δ 0.004mm/pixel Physical size of pixel from the sensor

5000mm Object is located at 5 meters from the

main lens

z" 70.994mm Distance between the main lens of the

focus plan of object z_focus

D 86.327mm Distance between the main lens and the

micro-lens array such that images on

sensor is in focus.

D - z' 15.33mm Distance between: the focus plan of the

object z observed through the main

lens, and the micro-lens array.

e 1 .0266 Enlargement

P 1 . 0266mm Pitch in physical unit of the micro-lens

images projected on the sensor

P 256.66pixel Pitch in pixel unit of the micro-lens

images projected on the sensor

w_f 1 .15mm Disparity in physical unit observed on

the sensor of the object located at distance z from the main lens.

287.5pixel Disparity in pixel unit observed on the

sensor of the object located at distance

f^{r0m tne ma}'ⁿ '^enS- r 8 Averaged number of replications for an

object located at distance z from the

main lens. Figure 8 illustrates a case where the super-resolution factor is chosen to beN = 2 which also corresponds to the size of the N by N sub-set. In this case the increment

T = fi/j_N is equal to τ = \ .Ί4μηι . Figure 8 shows the displacements of the micro- lenses versus a regular lattice. The regular lattice is defined by the equidistant dashed lines 0,1 ,2,3 extending in both directions i and j . The directions i and j are preferably perpendicular. In a conventional micro-lens array the centres of the micro- lenses are located at the intersections of the lines defining the regular lattice to form a regular grid of equidistant micro-lenses. According to an embodiment of the present invention the micro-lenses are arranged in the following way on the array 1 1 in accordance with formula 29. Blocks of N by N , N being the super-resolution factor having here the value 2, micro-lenses indicated by the bold dashed squares form a subsets of micro-lenses 200, 220, 240, 202, 222, 242 .... The blocks or micro-lens sub-sets are replicated in and j directions such that the micro-lens subsets are adjacently disposed in the two directions: subsets 200,220, 240... and subsets 202,222,242... are adjacently disposed in direction , while subsets 200,202... and subsets 220,222... are adjacently disposed in direction j . The micro-lens array is therefore formed of a plurality of micro-lens sub-sets disposed in a tiling form. In the present example the micro-lenses of each sub-set are all identical displaced in view of simplifying calculation of a reconstructed image. However, different arrangements can be given to the micro-lens sub-sets over the micro-lens array. This can be done for example by choosing different values for A,B,C,E in different micro-lens sub-sets.

The bold arrows indicate the displacement as a shift vector of the micro-lens centres. The amount of displacement is given by a multiple of a fixed increment τ in accordance with formula 29 and the table at the end of this paragraph. The arrows displayed in the figure have been artificially zoomed for illustration purpose.

A plurality of micro-lenses are shifted with respect to the regular lattice: in the illustration micro-lenses C10, C30, C12, C32 ... are shifted by τ in the direction j . Micro-lenses C1 1 , C31 ... are shifted by τ in the direction . Micro-lenses C01 , C21 ... are shifted by τ in the directions i and j . It follows that micro-lenses are set out of regular alignment in a particular way that reduces the superposition or the clustering of pixels in a reconstructed image. Preferably each micro-lens in each micro-lens sub-set is displaced by a different shift vector to increase the resolution of a reconstructed image. Each shift vector has a shift magnitude and a direction.

Optionally at least one micro-lens in each subset is not displaced with respect to the regular lattice: in the illustration the centre of micro-lenses COO, C20, C02, C22... belonging respectively to subsets 200,220,202,222... are located on the intersection lines of the regular lattice, here at the top left corner of each subset (K = L = 0 ). In practice, the micro-lenses which are not displaced can be any of the lenses of a subset. The position of a micro-lens which is not displaced can vary from one sub-set to another. Preferably one micro-lens is not displaced in each of the sub-set and the other micro-lenses of the sub-set have relative displacements with respect to the undisplaced micro-lens. The displaced micro-lenses in each sub-set have displacements which are defined relative to an un-displaced micro-lens.

As described above, the displacements are determined such that the superposition or the clustering of pixels in a reconstructed image is decreased (see also figure 10).

The values k(i,j) , and ( ,, ,) for the first sub-set of 2x2 micro-lenses illustrated in figure 8 are given in the following table:

The micro-lens array illustrated in Figures 8 and 9 is made unitarily of glass or synthetic glass. Possible processes for forming the micro-lenses on a glass plate includes lithography and/or etching and/or melting techniques.

Figure 9 is similar to figure 8 but illustrates a case with a super-resolution factor of N = 3 . In this case the increment is τ = 1 Α6μιη . The micro-lenses are arranged in a similar way as in figure 8 except that the subsets of micro-lenses are composed of 3 by 3 micro-lenses. The subsets 300,330,303,333 of micro-lenses are adjacently disposed in the two directions i and j . The displacement of the micro-lenses versus the regular lattice is similar to figure 8.

A plurality of micro-lenses are shifted with respect to the regular lattice: micro-lenses C10... are shifted by τ in the direction j . Micro-lenses C12... are shifted by 2τ in the direction . Micro-lenses C01 , C31 ... are shifted by τ in the direction and j. Micro-lenses C20... are shifted by 2τ in the direction j. Micro-lenses C12... are shifted by 2τ in the direction . Micro-lenses C02, C32... are shifted by 2τ in the direction and j . Micro-lenses C1 1 ... are shifted by τ in direction and by 2τ in direction j . Micro-lenses C22... are shifted byr in direction j and by 2τ in direction i . Optionally at least one micro-lens in a sub-set is not displaced with respect to the regular lattice: in the illustration the center of micro-lenses COO, C30... belonging respectively to subsets 300,330... are located on the intersections of the regular lattice, preferably at the top left corner of each subset (K = L = 0 ). In practice, the micro-lenses which are not displaced can be any of the lenses of a sub-set. The position of the micro-lens which is not displaced can vary from one sub-set to another. Preferably one micro-lens is not displaced in each of the sub-set and the other micro-lenses of the sub-set have relative displacements with respect to the undisplaced micro-lens. The displaced micro-lenses in each sub-set have displacements which are defined relative to the un-displaced micro-lens.

As described above, the displacements are determined such that the superposition or the clustering of pixels in a reconstructed image is reduced (see also figure 1 1 ).

The values k(i,j) , and ( ,, ,) for the first sub-set of 3x3 micro-lenses illustrated in figure 9 are given in the following table:

It is important to note that the relative positions of the micro-lenses in each N by N size sub-set (N being the number of micro-lenses in each directions i , j ) is defined modulo N. Thus all displacements modulo N will also be solutions. This means that if a given displacement (μ,-, μ,-) is a solution, then the displacement (μ_ί + αΝ, μ_] + ΰΝ) with a and b integers is also solution.

Resolution of the projected image

The resolution of the projected image can be estimated by computing its maximum sampling step H as for the conventional light-field camera made of a lens array arranged following a square lattice as presented in figure 7.

The projected coordinates (Χ',Υ') , obtained by the proposed micro-lens array with a super-resolution factor N , defines a set of points in the 2D projected/reconstructed. The set of points according to the proposed micro-lens array is characterized by the maximum sampling step H' . The values of H' have a simple expression for projected coordinates obtained with a disparity having the form: {w} = nl(NM) with n and M being positive integers such as 0 < n < M <

. The maximum sampling step is equal to H'= ugcd(n,M)/(NM) . The largest value of H' is u /N , one recalls that the largest value H obtained for the conventional square lattice micro-lens array is equal to u . Figure 10 illustrates the normalized ΗΊη values with the super-resolution factor N = 2 as a function of the fractional part of the disparity {w} . The corresponding characteristic parameters of the light-field camera are the one given above. The dashed line recalls the normalized H/u values obtained with a conventional light- field camera equipped with a regular square lattice micro-lens array. Similarly, Figure 1 1 illustrates the normalized ΗΊη values with the super-resolution factor N = 3 .

One can observe in figures 9 and 10 that the resolution of the reconstructed image varies less than a conventional (dash lines) with regular square lattice. Therefore, a more regular resolution is obtained with the proposed micro-lens array. The regularity of the resolution increases with the value of the super-resolution factor N .

The present invention also applies for light-field cameras made of an array of lenses and one sensor as illustrated in figure 2. The array of lenses is designed with the equation (29).

In the above we have assumed that the rotation angle Θ between the pixel array and the micoro lens array is zero. We now extend the teaching to consider cases where Θ is non-zero.

Assuming the first micro-lens centre is equal to (0,0) , and considering the rotation angle Θ between the pixel lattice and the micro-lens array, equation (14) can be rewritten as:

X = ux - ww(cos0 · i - sin Θ^■ j)

Y = uy - uw(s 0 - i + cosO - j) In matrix formulation equation (30) becomes:

(31)

To decrease the super-position or the clustering of the projected coordinates (X, Y) , the main idea is to shift the micro-lens images such as the projected pixels never overlap or get less clustered.

In other words, the centre of a given micro-lens should be shifted by the given shift (Δ^ ,/ Δ . ( ,/)) in respectively orientations i and j such as the modified

projected coordinates (X, Y ) of this new light-field camera would become:

(32)

A_t(i, j) indicates the shift amplitude of the lens in the i orientation; A_j (i, j)

indicates the shift amplitude of the lens in the j orientation. Orientations i and j are characterized by the micro-lens array. In the previous equation the shifts

A_t(i, j) and A^i ) are multiplied by a rotation matrix. To simplify the expression, A_t(i, j) and A^i ) are replaced by A_x(i, j) and A_y (i, j) which indicate the shift amplitude of the lens in the orientation x and respectively y of the pixel

sensor. Equation (32) becomes:

That equation can be modified slightly by setting w = cw' with c being a real constant such as: a = c cos6 and b = c sin# . w' is the modified disparity. The equation

becomes.

smaller or equal to r , and n is any integer such as « e [o,N[ . N acts as a super- resolution factor between the resolution of the micro-lens images and the resolution of the re-focused image computed for modified disparity W . Equation (34) becomes:

( ", 7") are normalized projected coordinates such that ( ", 7") are integers for a perfect sampling of the projected coordinates ( ',Ρ) . For a perfect sampling

NwA_x(i,j) and NwA_y(i,j) must also be integers respectively equal to k_N(i,j) and l_N(i,j) . Unfortunately NwA_x(i,j) and NwA_y(i,j) varies with the disparity w ; but it has been shown in equation (10) that the ratio W_aF IW_f0CUS , which is equal to ^waF I ^w focus · ^{is verv} close to 1 ; thus the variation of w are small and w can be

replaced by w_focus . The shifts A_x(i,j) and A are characterized by:

1 k_N(i ) 1 l_N(i>j)

w focus N w N

(36) δ k_N(i,j)

focus N focus N

Also for a perfect sampling with ( ", 7") being integers, a and b should also be integers. Since Θ = arctan(b/a) , only specific rotation angles are compatible with a perfect sampling. The following equations assume that a and b are integers. A remaining question is how to define the 2 functions k_N(i,j) and l_N(i,j) to have optimum micro-lens displacements such that the projected coordinate ( ", 7") have a minimum clustering, and a perfect sampling for the modified disparities having the form w'=

To obtain a perfect sampling the set of projected coordinates ( ", 7") defined by the various lens coordinates must have all possible integer values whatever n , also the number of contiguous lenses to obtain the perfect sampling must be minimum and equal to N (considering one dimension). This constraint can be reformulated by taking into consideration modular arithmetic modulo N :

k_N(i,j) and l_N(i,j) are 2 periodic functions: one period is defined with e [o, N[² , into [o, N[ (in other words these two functions are defined modulo N ). We wish to find k_N(i,j) and l_N(i,j) such that for any given n e [o, N[ , all the set of projected coordinates ( "modN, F'modN) defined by (/mod N, /mod N) e [o, N[² is equal to

where δ_α is the Dirac function located at (α,β) with (α,β) being integer numbers. To solve e uation (38) the following linear solutions are considered:

Equation (38) becomes:

The determinant s(n) of the previous system of equations given in matrix formulation is equal to: s(n) = (a² + b² )n² + (oA - bB + bC + aE)n + AE - BC (41) To have a solution, equation (40) must have a determinant which is not null modulo N and also prime with N for any n e [o, N[ . Note that there are no conditions on K and L since they correspond only to translations. The condition on the determinant s(n) is: gcd(s(n) modN, N) = l V« e [θ, N[

(42) gcd((a² + b² i² + (aA - bB + bC + aE)n + AE -BC, N) = \ Vn e [θ, N[

The Nx N micro-lenses define a sub-set of micro-lenses. The parameters

A,B,C,E,K,L should be identical for all sub-sets. If the parameters vary from one sub-set to the next, the distribution of the ( ", 7") coordinates might be not perfect. It is important to recall that the condition given by equation (42) applies only for certain angles Θ defined such that tan# = bl a . The displacements A,B,C,E,K,L are optimal to ensure a perfect sampling for modified disparities w'= \_w']+ n/ N . By definition w = cW , thus the perfect sampling is ensured only for certain disparities w = c ([w'J+ n /N). With c = al cos0 = b/sin Θ or c = V a + b . Between two values of w associated with a perfect sampling, the distribution of the projected coordinates ( ", 7") are not perfect but quite close to perfect. The distribution of the projected coordinates evolves a little with w in a complex manner. The variations of the distribution are smaller with small values of c . Advantageous choice of c are c = 1 , c = l and c = V corresponding to the following angles:

Experimental solution

Many values A,B, C, E verify equation (42), to compute these values one selects 2 integers a and b which define the angle of rotation θ , and a super-resolution factor N . The set of 4 values A,B, C, E are defined modulo N , the total number of set is equal to N⁴ . To find one set of A,B, C, E values with verify equation (42), the N⁴ set of values are tested sequentially until one set verify the equation (42).

Experimentations show that from the N⁴ set of A,B, C, E values between 2% to 50% verify the constraint given by equation (42) for any N , a and b . The following table indicates a set of A,B, C, E values which verifies equation (42) for various a , b and N :

The periodic functions k_N(i,j) and l_N(i,j) are fully characterized and thus the shifts (A_x(i,j),A_y(i,j)) of the micro-lens image versus the regular grid is also fully characterized. The shifts are given in units of . To convert the shifts into physical unit at the micro-lens side, the shifts are multiplied by φ . The physical shifts

(A_y(i,j),A_x(i,j)) at the micro-lens side are computed easily by combining equation (8) and (36):

The physical shifts ca which is

multiplied by the integers values given by k_N(i,j) and l_N(i,j) .

The design of the micro-lens array is defined by:

• The focal distance / of the micro-lenses.

• The average pitch φ between consecutive lenses.

• The distance d between the micro-lens array and the sensor.

• The pixel size δ of the sensor.

• The super-resolution factor N which is freely selected between [l,r] .

• 2 integers a and b (one of which is non-null) which characterize the angle Θ = arctan(b/a) between the micro-lens array and the pixel array. Preferably, a and b are chosen such that gcd(a,b) = l and c = /a² + b² is as small as possible.

• The micro-lens centres ( ,, ,) are located following the equation: f δ

μ, = O^'cos0 -7 sin #) +—— k_N(i,j)

d N

f δ

μ . = φ(ism θ +jcosθ) +—— l_N(i,j)

d N

The micro-lens centres ( ,, ,) are given in meters in the sensor reference frame (following x and y axis). It should be recalled that the functions k_N(i,j) and l_N(i,j) are defined modulo N : thus the centres ( ,, ,) are valid as well as μ_ί + αΑ^δ/_] ,μ . + α_] ^δ/ whatever being integers. The micro-lens array is designed according to the previous settings. If the size of the micro-lenses is equal to the pitch φ , then the micro-lenses might have a very small overlaps due to the displacement of the micro-lenses versus the squared lattice. This issue is solved by designing micro-lenses such that the micro-lens size is smaller than φ - ^(^{Ν ~} ^ j_N■ The shape of the micro-lenses can be circular, rectangular or any shape without any modification of the previous equations.

The position of the first micro-lens (θ,θ) is located at (x_{0 0}, y_{0 0} ) in the sensor coordinate frame. This position is freely selected with no impact on the super- resolved projected coordinates. Figure 12 illustrates the micro-lens array rotated by Θ = 26.56° (corresponding to the case a = 2 and b = 1 ) relative to the sensor array. The position (x_{0 0}, y_{0 0} ) may lie outside the sensor array. Consequently some lens coordinates may be negative.

Examples according to the above design criteria are set out below.

Symbols Values Comments

F 70mm Main focal distance f 2mm Micro-lens focal distance d 2.3mm Distance between the micro-lens array and the

sensor

Φ 1 mm Micro-lens pitch δ 0.004mm/pixel Physical size of pixel from the sensor

5000mm Object is located at 5 meters from the main

lens

z" 70.994mm Distance between the main lens of the focus

plan of object z_focus

D 86.327mm Distance between the main lens and the micro- lens array such that images on sensor is in

focus. D - z' 15.33mm Distance between: the focus plan of the object

z observed through the main lens, and the

micro-lens array.

e 1.0266 Enlargement

P 1. 0266mm Pitch in physical unit of the micro-lens images

projected on the sensor

P 256.66pixel Pitch in pixel unit of the micro-lens images

projected on the sensor

w_f 1.15mm Disparity in physical unit observed on the

sensor of the object located at distance z_focus

from the main lens.

287.5pixel Disparity in pixel unit observed on the sensor of

the object located at distance z from the main lens.

r 8 Averaged number of replications for an object

located at distance z_focus from the main lens.

Super-resolution factor N = 2 ; rotation angle Θ = 26.56°

In case the super-resolution factor N = 2 , the increment τ = "^ ^y is equal to τ = 1.Ί4μηι The rotation angle is Θ = 26.56° with a = 2 and b = 1 . The values k₂(i,j) , l₂(i,j) and ( ,-, , ) for the first sub-set of 2 x 2 micro-lenses are given in the following table:

Figure 13 illustrates the displacement of the micro-lenses versus the regular squared lattice. The bold arrows indicate the displacement of the micro-lens centres by τ in the direction indicated by the arrow. The sub-set of 2 x 2 micro-lenses indicated by the bold dashed square, is replicated in and j directions. It is worth noting that the arrows displayed in the figure have been artificially zoomed for illustration purpose. Super-resolution factor N = 3 ; rotation angle Θ = 26.56°

In the case with the super-resolution factor N = 3 , τ = ΙΛ6μιη . The rotation angle is Θ = 26.56° with a = 2 and b = 1. The values k₃(i,j) , l₃(i,j) and ( ,, ,) for the first subset of 3x3 micro-lenses are given in the following table:

Figure 14 illustrates the displacement of the micro-lenses versus the regular squared lattice. Similar considerations as figure 13 apply to figure 14.

Super-resolution factor N = 2; rotation angle Θ = 45°

In case the super-resolution factor N = 2, the increment τ = ^/^_N is equal to τ = . The rotation angle is Θ = 45° with a = 1 and b = \. The values k₂(i,j) , l₂(i,j) and ( ,, ,) for the first sub-set of 2x2 micro-lenses are given in the following table:

Figure 15 illustrates the displacement of the micro-lenses versus the regular squared lattice. Similar considerations as figure 13 apply to figure 15. Super-resolution factor N = 3 ; rotation angle Θ = 45°

In the case with the super-resolution factor N = 3 , τ

. The rotation angle is Θ = 26.56° with a = 1 et b = 1 . The values k₃(i,j) , l₃(i,j) and ( , , , ) for the first subset of 3 x 3 micro-lenses are given in the following table:

Figure 16 illustrates the displacement of the micro-lenses versus the regular squared lattice. Similar considerations as figure 13 apply to figure 16. The proposed invention also applies for a light-field camera made of an array of lenses and one sensor. The array of lenses is designed with the equation (44). For a homogeneous sampling of the projected-coordinate, the rotation angle Θ between the pixel lattice and the micro-lens array should be accurate to within [- 1/ N_x,l/ N_x] where N_x is the sensor width. The translation between the pixel lattice and the micro- lens array defined by (x_{0 0}, y_{0 0}) can be set to any values, without detriment.

As has been described, a light-field camera or imaging apparatus allows capture of 4D light-field images. The 4D light-field pixels can be projected into various 2D re- focussed images with freely selected distance of focalization. A light-field camera using a shifted micro-lens array allows 2D re-focus images to be produced in which the projected 4D light-field pixels are regularly spaced with reduced or no crowding or overlapping. The 2D re-focus images have an almost perfect sampling which is super-resolved compared to the sampling of the 4D light-field. This property affords the 2D re-focussed images higher resolution, which is almost invariant with the distance of focalization compared to micro-lens arrays having regular spacing.

The shifts of the micro-lens centres have been computed for a given distance of focalization z_shift corresponding to a disparity W_shift (in the previous proposed micro- lens designs, z_shift has been selected to z_focus ) . The micro-lens shifts are calculated for projecting the 4D light-field pixels regularly with minimal or no crowding or overlapping into 2D re-focus images. This advantage applies for focalization distances z close to z_sMft . As the focalization distance z diverges from z_shift , the regularity of the projected 4D light-field pixels degrades. This is due to the

approximation of equation (19) where the variations of disparity W are assumed negligible.

Figure 17 illustrates how the regularity of the projected 4D light-field pixels reduces as the disparity W of the re-focus images diverges from W_shift . In this example the micro-lens array is made of sub-sets of size N = 3 . The increment associated to the sub-set is computed for w_shift = 90.0 . The sampling of the projected 4D light-field pixels are almost ideal and super-resolved for disparities w = 90.0 and w = 90.33 . The sampling of the projected 4D light-field pixels is less regular and less super-resolved for disparity w = 67.0 and w = 67.33 . For these 2 disparities, the shifted micro-lens array does allow sampling with a super-resolved factor of N = 3 .

In equation (28), the physical shifts ^~ _i{i, j),^~ _J {i, j) versus a regular grid are computed for a disparity W_shift = W_focus . The shifts are ideal for disparities W close to

W_focus and ensure a substantially regular spacing of the projected 4D light-field pixels.

Unfortunately, this property is degraded as the disparity W of the re-focusing diverges from the disparity W_focm for which the shifts have been computed. Selecting

W_shift = W_focus is an optimal choice since W_focus corresponds to an object at distance ^z foam which is observed exactly in focus on the micro-lens images. Objects at distance z around z_focus are also observed in focus on the micro-lens images depending on the depth-of-field of the main- and micro-lenses.

It therefore follows that the homogeneity of the projected 4D light-field pixels is almost ideal for object at distance z_shift corresponding to a disparity W_shift , and that the homogeneity of the 4D light-field pixels projected into a 2D re-focus image is reduced as the distance of focal ization z differs from z_shift . A shifted micro-lens array is described below, such that the homogeneity of the 4D light-field pixels projected into a 2D re-focus image is substantially ideal for a wider range of distances of

focal ization.

A lens array, or camera design described herein defines the notion of sub-set made of N x N micro-lenses which permits an almost constant super-resolution sampling of N . Within the sub-set of micro-lenses, the shifts are fully characterized by multiples or coefficients of unit vectors in orthogonal directions, and a magnitude, relative to a regular grid. The coefficients are characterized by the functions k_N(i,j) and l_N(i,j) where is a micro-lens coordinate. Following equation (19), the increment z_shift is characterized by: (45)

<5

τ shift d

1 + N

(46)

^{Z '}shift

D

A given increment z_shift must be selected and corresponds to a single disparity W_shift and distance z_shift (in the previous section the increment z_shift was defined by ^Wshift = ^w focus )■ ^ln equation (46), the focal length / of the micro-lenses does not

appear because this equation is not dependent upon whether or not the micro-lens images are in focus. Actually, the micro-lens images are exactly in focus for objects located at z_focus . Objects located at distance z are in focus if they are inside the depth-of-field of the system main- and micro-lenses. The size of the depth-of-field depends on the focal lengths and apertures of the main- and micro-lenses. This aspect does not impact the proposed designs which yield substantially regular super- resolved sampling of 2D re-focused images independently of the focus of the micro- lens images.

Here we propose to define a super-sub-set made of M x M sub-sets. Each sub-set (a,b) where (a,b) e [o, p within the super-sub-set is associated to a distinct

increment z(a,b) . M² increments z(a,b) must be defined with their M²

corresponding distances of focal ization z(a,b) and disparity W(a,b) . Within a sub-set (a,b) a single increment z(a,b) is used. The super-sub-set is then replicated on the complete micro-lens array. It characterizes the coefficients and magnitudes of the displacements in orthogonal directions of each micro-lens versus a regular grid.

The design of the micro-lens array is defined by:

• The focal distance / of the micro-lenses.

• The average pitch φ between consecutive lenses.

• The distance d between the micro-lens array and the sensor.

• The pixel size δ of the sensor. • The size M xM of the super-sub-set and the corresponding M² increments (a,b) with their associated M² distances of focal ization z(a,b) .

• The super-resolution factor N which is freely selected between [l,r/ ] .

• The micro-lens centres ( ,, ,) are located following the equation:

Where |_/7NJ is an i

L N. ( ) is equivalent to (a,b) the coordinate of the sub-set within the

'N

super-sub-sets. It should be recalled that the functions k_N(i,j) and l_N(i,j) are defined modulo N : thus the centres ( ,, ,) are valid as well as μ_ί + (M) whatever

being integers. Consequently the displacements versus the regular grid can be negative.

A super-sub-set is defined with M² various increments z{a,b) . These increments are associated to M² distances of focal ization z(a,b) according to equation (46) For the various sub-sets which share the same increments z(a,b) , the corresponding 4D light-field pixels are projected into a 2D re-focus image with substantially regular super-resolved sampling if the distance of focal ization z is chosen to be close to z(a,b) .

2D refocus images can be computed by projecting 4D light-field pixels which belong only to the sub-sets having the same increments (a,b) . In other words, a 2D refocus image with a given focalization distance z is computed with only a fraction of the 4D light-field pixels since only 1 4D light-field pixel is used out of M² on average. The sub-sets which are selected to project the 4D light-field pixels into a 2D re-focus image with the distance of focalization z are the ones with z(a,b) nearest to the desired value of z . This results in a 2D image with substantially regular super- resolved sampling.

The M² distances of focalization z(a,b) are chosen according to the range of desired focalization distances required to compute re-focus images, and can be freely selected as required.

An example of camera with a first design is described in this sub-section.

Symbols Values Comments

F 70mm Main focal distance f 2mm Micro-lens focal distance d 2.3mm Distance between the micro-lens array and the

sensor

Φ 1 mm Micro-lens pitch δ 0.004mm/pixel Physical size of pixel from the sensor

5000mm Object is located at 5 meters from the main

lens

z" 70.994mm Distance between the main lens of the focus

plan of object z_focus

D 86.327mm Distance between the main lens and the micro- lens array such that images on sensor is in focus.

D - z' 15.33mm Distance between: the focus plan of the object

z observed through the main lens, and the micro-lens array.

e 1.0266 Enlargement

P 1. 0266mm Pitch in physical unit of the micro-lens images

projected on the sensor

P 256.66pixel Pitch in pixel unit of the micro-lens images

projected on the sensor w_f 1.15mm Disparity in physical unit observed on the

sensor of the object located at distance z_focus

from the main lens.

287.5pixel Disparity in pixel unit observed on the sensor of

the object located at distance z from the main

lens.

r 8 Averaged number of replications for an object

located at distance z_focus from the main lens.

The super-resolution sampling factor is defined to N = 3 . With M = 2 , 4 distinct sub-sets are defined. The 4 increments (a,b) are defined such that they correspond to various distance of focalization: z(0,0) = 5000mm , z(0,l) = 2000mm , z(l,0) = 1000mm , and z(l,l) = 5000mm . In this configuration, the sub-sets (0,0) and (1,1) are shifted with the same increments, and therefore signal to noise at this distance of focalization can be improved, however it is equally possible to have 4, or M² different values. From equation (46) the 4 increments and their corresponding distances of focalization are computed and listed in the following table:

The values k₃(i,j) , l₃(i,j) and ( , , , ) for the first sub-set of 6 x 6 micro-lenses are given in the following table:

Figure 18 illustrates the displacement of the micro-lenses versus the regular squared lattice. The bold arrows indicate the displacement of the micro-lens centres by (a,b) in the direction indicated by the arrow. The super-sub-set, is replicated in and j directions. It is worth noting that the arrows displayed in that figure have been artificially zoomed for illustration purpose (also the size of the increments might not reflect exactly the increments listed in the above table).

In an alternative configuration, the sub-sets of Nx N micro-lenses are interleaved whereas in the previous design they are juxtaposed to form a super-sub-set. The interleaving extrends a sub-set of Nx N across NM x NM micro-lenses, and the distance between 2 consecutive micro-lens of a sub-set is no longer one micro-lens, but becomes M micro-lenses. Figure 14 illustrates the interleaving of the sub-sets with N = 3 and M = 2 : one interleaved sub-set of Nx N micro-lenses is illustrated with bold circles; the coefficient-set made up of M x M consecutive micro-lenses groups micro-lens of the sub-sets which share the same coefficient (same k_N(i,j) and l_N(i,j) values).

To ensure a regular super-resolved sampling of the projected 4D light-field pixels into a 2D re-focus image, the orientations k_N(i,j) and l_N(i,j) of the interleaved sub-sets needs to be redefined. Let k'_N(i',f) and l'_N(i',f) being the shift coefficients defined for the interleaved sub-sets. The shifts defined by k'_N(i',f) and l'_N(i',f) with

i'= fined by:

≡ CV+Ef+V modN

J_

(modN) (48)

M M

i J_ (modN)

^~M M

To determine the parameters k'_N(i',f) and l'_N(i',f) Equation (48) is expressed modulo N (The same argument was used to convert equation (20) into equation

(21)). Equation (48) becomes:

Y" ≡ N(y-[w\j-)-nj- _N(r,f) ≡ -nj-V_N(i f) (modN)

This new equation combines the lens coordinates with the modified lens

coordinates (', ). Knowing that (a,b) = (imodM ,jmodM) , for simplification are written as:

Where (a,b) is the increment associated to an interleaved sub-set of micro-lenses. A given modified coordinate (', ) relates to MxM micro-lenses which defined a coefficient-set of micro-lenses. Equation (49) becomes X" ≡ -nMf-na-k'_N(i',f) ≡ nMf-Ai'-Bf-K'-na (modN)

7" ≡ -nMf-nb-V_Nii f) ≡ nMf-Ci'-Ef-U-nb (modN)

This last equation is similar to equation (23). The terms (-na-nb) represent shifts which are constant for a given n and (a,b). Similarly to equation (23), this last equation is solved using the following constraint which governs the parameters k'_N{i',f) and _N{i',f):

gcd( ' (n) modN, N) = 1 V« e [θ, N[

gcd(nM² + nM(A'+E') + A'E'-B'C(modN),N) = l V« e [0,N[ With s'(n) = n²M² +nM(A + E)+AE-BC .

Many values A',B',C',E',K',L' verify equation (52). The special case: Ά=0, Β'=Τ',

C'= 1 , E'= 1 , K'= 0 and V= 0 is detailed in this section. The proposed solution has the following form:

is a free parameter which has been experimentally determined for various values N and M . The experimentation consists in testing various values of T'e [o,N[ such that the constraint gcd(s'(n)modN,N)=l is respected for any n e [o,N[. The following table indicates the smallest value of according to that constraint for: M = 1,

M = 2, M = 3, M = 4, M = 6, M = 7, M = 8 and = 9:

In the case of M = 5 and M = 10 the following table indicates the smallest value of according to the constraint: The periodic fu

haracterized thus the shifts (A (i,j),A'_j(i,j)) of the micro-lens image relative to the regular grid is also fully characterized. The shifts are given in units of .

The design of the micro-lens array is defined by:

• The focal distance / of the micro-lenses.

• The average pitch φ between consecutive lenses.

• The distance d between the micro-lens array and the sensor.

• The pixel size δ of the sensor.

• The size MxM of the super-sub-set and the corresponding M² increments (a,b) with their associated M² distances of focal ization z(a,b).

• The super-resolution factor N which is freely selected between [l,r/ ].

• The micro-lens centres ,, ,) are located following the equation:

Where i(M) is equal to i modulo M . (z^'mod ,7^'mod ) is the coordinate within the sub-set. It should be recalled that the functions k'_N(i,j) and V_N(i,j) are defined modulo N : thus the centres ( ,, ,) are valid as well as μ_ί + Na ^(i mod M,j mod M)k' ₁ + Να τ(ϊ mod M,j mod M)r

M M M M

whatever (a.ctj being integers. Consequently the displacements versus the regular grid can be negative. A camera with the second design is described in this sub-section.

The super-resolution sampling factor is defined to N = 3 . With M = 2 , 4 distinct sub- sets are defined. The 4 increments (a,b) are defined such that they correspond to various distance of focal ization: z(0,0) = 5000mm , z(0,l) = 2000mm , z(l,0) = 1000mm , and z(l,l) = 5000mm . The increments can be all different, or may be duplicated in some instances. From equation (46) the 4 increments and their corresponding distance of focal ization are computed and listed in the following table:

The values k , l and ( , , , ) for the first sub-set of 6 x 6 micro-lenses are given in the following table:

Figure 19 illustrates the displacement of the micro-lenses versus the regular squared lattice. The bold arrows indicate the displacement of the micro-lens centres by (a,b) in the direction indicated by the arrow. The super-sub-set, is replicated in and j directions. It is worth noting that the arrows displayed in that figure have been artificially zoomed for illustration purpose (also the size of the increments might not reflect exactly the increments listed in the tables above). For clarity, increments z(a,b) have been indicated only for selected micro-lenses.

The two designs presented above consider that all micro-lenses have the same focal length / ; the M² types of sub-sets of N x N micro-lenses are organized with M² increments (a,b) corresponding to M² distances of focal ization z(a,b) for which 2D re-focus images have a substantially perfect super-resolved sampling. There is now described a third variation where the sub-sets (a,b) are interleaved (similarly to the second design) and the micro-lenses have differing, or non-uniform, focal lengths f(a, b) .

For a given distance of focal ization z(a,b) , a focal length f(a, b) is desired such that the micro-lenses produce an image exactly in focus. f(a, b) is easily computed by merging equations (3) and (6):

This third design offers the advantage of providing micro-lens images which are exactly on focus for the M² distances of focal ization z(a,b) .

The third design is similar (orientations and increments) to the second design except that the focal lengths of the micro-lenses vary from one interleaved sub-set of micro- lenses to the next (as for the increments).

A camera with the third design is described in this sub-section Symbols Values Comments

F 70mm Main focal distance f Micro-lens focal length varies depending on the

sub-set of micro-lenses

d 2.3mm Distance between the micro-lens array and the

sensor

Φ 1 mm Micro-lens pitch δ 0.004mm/pixel Physical size of pixel from the sensor

5000mm Object is located at 5 meters from the main

lens

z" 70.994mm Distance between the main lens of the focus

plan of object z_focus

D 86.327mm Distance between the main lens and the micro- lens array such that images on sensor is in

focus.

D - z' 15.33mm Distance between: the focus plan of the object

z observed through the main lens, and the

micro-lens array.

e 1.0266 Enlargement

P 1. 0266mm Pitch in physical unit of the micro-lens images

projected on the sensor

P 256.66pixel Pitch in pixel unit of the micro-lens images

projected on the sensor

w_f 1.15mm Disparity in physical unit observed on the

sensor of the object located at distance z_focus

from the main lens.

287.5pixel Disparity in pixel unit observed on the sensor of

the object located at distance z from the main

lens.

r 8 Averaged number of replications for an object

located at distance z_focus from the main lens.

The super-resolution sampling factor is defined as N = 3 . With M = 2 , 4 distinct subsets are defined. The 4 increments (a,b) are defined such that they correspond to various distance of focal ization: z(0,0) = 1000mm , z(0,l) = 2000mm , z(l,0) = 4000mm , and z(l,l) = 8000mm . From equation (46) the 4 increments r(a,b)and their corresponding distances of focal ization z(a,b) are computed. Also with the 4 distances of focal ization the 4 focal lengths f(a,b) are computed from equation (55):

The values £'₃ (z^', y) , /'₃ (z^', y) and ( ,-, ,) for the first sub-set of 6 x 6 micro-lenses are identical to those calculated previously. Figure 20 illustrates the displacement of the micro-lenses versus the regular squared lattice. The bold arrows indicate the displacement of the micro-lens centres by (a,b) in the direction indicated by the arrow. The super-sub-set, is replicated in and j directions. It is worth noting that the arrows displayed in that figure have been artificially zoomed for illustration purpose (also the size of the increments might not reflect exactly the increments listed in table 1 1 ). In this design the f(a,b) of the micro-lenses varies with the increments z(a,b) . For clarity, the focal f(a,b) and increments z(a,b) have been indicated only for selected micro-lenses.

Fig. 21 illustrates a hardware configuration of a processing device which may be used in conjunction with the present invention. The processor may be embodied as a PC for example. Referring to Fig. 21 , a central processing unit (CPU) 2101 executes a program stored in a program read only memory (ROM) in a ROM 2102, and a program such as an operating system (OS) and an application loaded from a storage unit 2107 such as a hard disk, to a random access memory (RAM) 2103.

The RAM 2103 is a main memory of the CPU 2101 , and functions as, for example, a work area. An operation input may be provided for example from an input unit 2104 such as a keyboard or a pointing device (for example, a mouse, a touch pad, a touch panel, and a trackball). A display unit 2105 controls a display of a display device. A communication interface 2106 controls communication between the processing device and another apparatus, which may be connected via a network for example.

The processing device of Figure 21 can be used to perform the calculations and processing of respective processes described herein, by executing the program stored in a computer readable storage medium. For example, parameter values for main and micro-lens focal distance and array sensor spacing and pixel density, and super-resolution factor, can be input and/or stored. A program or programs operating according to the equations set out above, such as equation 26 and equation 29 for example can then be used to calculate and output the positions and/or displacements of the microlens centres of an array.

The output could be numerical data in the form of coordinates for example, or could be graphical data in the form of a mask for a photolithographic microlens array manufacturing process.

Figure 22 is a flow chart illustrating the process of obtaining a 2D image from a captured 4D light field, which process can for example be implemented on the processing device of Figure 21 .

In step 2201 , projection parameters are obtained, for example from a user input. An example of such parameters are values s and g in equation 1 1 . It could also be envisaged that only a single input corresponding to a plane to be in focus need be input, and other values could be set to default values, or calculates as required. Subsequently in step 2202, a projection mapping form a coordinate system of a 4D light field (eg pixel (x,y) of a sensor associated with the micro-lens ) to a 2D image coordinate system. This can be performed using equation 1 1 for example.

Finally, in step 2203, pixels values are projected according to the mapping

determined in step 2202, resulting in the desired 2D image. Projected coordinates may be non-integer and interpolations are therefore required to accumulate the 4D light-field value into the re-focus image in some cases. While 4D light-field pixels are projected into the re-focus image, a weight-map records the number of accumulated projected pixels. The weight-map also records the interpolation performed on the non-integer coordinates. Once all 4D light-field pixels are projected into the re- focused image and the weight-map, the re-focus image is divided by the weight-map so that each re-focused pixel received the same average contribution. The resulting 2D image can be output on a display, or stored and/or transmitted to another device for example.

The process of Figure 22 can be carried out on any image processing device suitably adapted, and in addition to a general processing device as illustrated in Figure 21 , such as a PC, an imaging device such as that illustrated in Figure 23 can be employed.

Referring to Figure 23, a communication control unit 2301 is a unit which allows communication with an information processing apparatus and controls wired or wireless communication with the information processing apparatus. A calculation unit (central processing unit (CPU)) 2302 performs control of the overall imaging apparatus based on an inputted signal and a program. A signal processing unit 2303 performs processing such as compression coding, contour enhancement, 2D projection, and noise elimination of a shot moving image.

An optical unit 2304 includes a lens, an autofocus driving motor, a zoom driving motor, and the like, and in embodiments of the present invention also includes a microlens array. A primary storage unit (DRAM) 2305 is used as a temporary storage area or the like of the CPU 2302. A secondary storage unit 2306 is a non-volatile storage unit such as a flash memory and stores various parameters. An operation member 2307 can include for example a cursor key, set/execute button, a menu button, and the like, and a user can use the operation member 107 for calling up a menu and selecting and deciding various settings in the imaging apparatus. A display unit 2308 is a member which displays image data and a graphical user interface (GUI), and a liquid crystal device (LCD), for example, is used for the display member 2308. The CPU 2302 functions as a display control unit which sets contents to be displayed on the display member. A read only memory (ROM) 2309 stores a control program to be loaded on the CPU 2302. A removable storage medium 2310 is a removable medium such as a memory card and stores data It will be understood that the present invention has been described above purely by way of example, and modification of detail can be made within the scope of the invention. Each feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination.

Claims

1 . A micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein: the micro-lens array comprises a plurality of micro-lens subsets, each sub-set comprising a two dimensional array of (Q) micro lenses, wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a common pattern, the common pattern defining different displacements for each micro-lens of the sub-sets.

2. The micro-lens array of claim 1 wherein said common pattern defines each displacement as a function of the position of each micro-lens within the sub-set.

3. The micro-lens array of claims 1 or 2 wherein said common pattern defines each displacement as a function of the number (Q) of micro-lenses in each sub-set.

4. The micro-lens array of any of the preceding claims wherein said common pattern defines displacements in integer multiples of unit displacement vectors.

5. The micro-lens array of claim 4 wherein the magnitude (τ ) of said unit displacement vectors is a function of focal distance of micro-lenses.

6. The micro-lens array of claims 4 or 5 wherein the magnitude of said unit displacement vectors is a function of the number (Q) of micro-lenses in each sub-set.

7. The micro-lens array of any of claims 4 to 6 wherein the multiple of said unit vectors for each micro-lens is a function of the position of the micro-lens within the sub-set.

8. The micro-lens array of any of the preceding claims wherein said sub-sets comprise a square array of NxN = Q micro-lenses.

9. The micro-lens array of claim 8 wherein said common pattern defines a plurality of possible displacements for each micro-lens, each of said plurality being equivalent in modulo N.

10. The micro-lens array of any of the preceding claims wherein the displacement of at least one micro-lens in each sub-set is zero.

1 1 . The micro-lens array of any of the preceding claims wherein the common pattern and the displacements are independent of the location of the sub-set in the micro- lens array. -lens array according to Claim 4, wherein said integer multiples (k, I)

where NxN defines the size of the sub-set in number of micro-lenses, and the values A, B, C, E being determined as a solution of the equation: gcd((n² + n(A + E) + AE -

N), N) = 1 V« e [θ, N[ .

13. An imaging device comprising a micro-lens array according to any of claims 1 to 12 and a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a micro-image.

14. An imaging device comprising a micro-lens array according to any of claims 1 to 1 1 and a photo-sensor having an array of pixels, wherein the axes of the pixel array have an angular displacement Θ relative to the axes of the micro-lens array,

15. The imaging device of Claim 14, wherein said common pattern defines each displacement as a function of the angular displacement Θ.

16. The imaging device of Claim 14 or Claim 15, wherein said common pattern defines displacements in integer multiples of unit displacement vectors, wherein said integer multiples (k, I) are given by

≡ Ci + Ej + L modN where NxN defines the size of the sub-set in number of micro-lenses, and the values A,B,C,E being determined as a solution of the equation: gcd((a² + b² )n² + (oA - bB + bC + aE)n + AE - BC, N) = 1 V« e [θ, N[ where tan# = bl a

17. The imaging device of any one of claims 13 to 16, wherein said common pattern defines displacements in integer multiples of unit displacement vectors and wherein the magnitude (τ ) of said unit displacement vectors is given by τ = ^ ^y where / is the micro-lens focal distance, 5 is the physical size of a sensor pixel, d is the distance between the micro-lens array and the sensor and NxN defines the size of the sub-set in number of micro-lenses.

18 A micro-lens array according to Claim 1 , wherein said plurality of micro-lens subsets is a first plurality of sub sets, and wherein said common pattern is a first common pattern, and further comprising at least a second plurality of subsets, the microlenses of each of the second plurality of sub-sets being displaced according to a second common pattern, said second common pattern being different from said first common pattern.

19. A micro-lens array according to Claim 18, wherein there are MxM pluralities of subsets, each sub-set of each plurality of subsets comprising a square array of NxN =

Q micro-lenses

20. A micro-lens array according to Claim 19 Wherein the micro-lenses of the array are arranged in super-sub-sets, each super-sub set comprising a regular array of MNxMN microlenses.

21 . A micro-lens array according to Claim 19 or Claim 20, wherein the micro-lenses of each individual sub-set in a super-sub-set are arranged adjacent to one another.

22. A micro-lens array according to Claim 19 or Claim 20, wherein the microlenses of each individual subset in a super-sub-set are non-adjacent, and are interleaved with microlenses of other sub-sets of the super-sub-set.

23. The micro-lens array of any one of Claims 18 to 22, wherein said common patterns define displacements in integer multiples of unit displacement vectors, and wherein the magnitude (τ ) of said unit displacement vectors is different for said first and second common patterns.

24. The micro-lens array of any one of Claims 18 to 23, wherein the integer multiples of said unit vectors for each micro-lens are a function of the position (/^',/) of the micro-lens within its respective sub-set, and wherein said integer multiples are the same for said first and second common patterns .

25. The micro-lens array of any one of Claims 18 to 24, wherein the magnitude (τ ) of the unit displacement vectors for at least one plurality of sub-sets is defined according to a user selected object focalisation distance.

26. An imaging device comprising a micro-lens array according to any of claims 18 to 24 and a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a micro-image.

27. A micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relative to a regular lattice, wherein:

the micro-lens array comprises a plurality of micro-lens sub-sets, each sub-set comprising an array of NxN micro lenses, each micro-lens of the subset having a focal distance / , wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a displacement pattern,

said displacement pattern defining the displacement of each micro-lens as integer multiples (k, I) of unit vectors, said unit vectors having a magnitude τ wherein the magnitude τ is a function of / IN.

28. The micro-lens array of claim 27, wherein the displacement of each micro-lens is defined as a function of the position of the micro-lens within the sub-set, and ■ I*- I 1 1 w ■ H ) ≡ Ai + Bj + K (modN) wherein said integer multiples (k, I) are given by <^ , '

≡ Ci + Ej + L (modN) the values A,B,C, E being determined as a solution of the equation: gcd((n² + n(A + E) + AE -

1 V« e [θ, N[ .

29. An imaging device comprising a micro-lens array according to claim 27 or claim 28, and a photo-sensor having an array of pixels, arranged so that each micro-lens projects an image of a scene on an associated region of the photo-sensor forming a micro-image, said sensor and said micro-lens array having a separation d _t and pixels of said photo sensor having size δ wherein ^τ -

30. A micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein: the micro-lens array comprises MxM groups of micro-lens sub-sets, each subset comprising an array of NxN micro lenses, said microlenses arranged in super-sub-sets comprsing an array of MNxMN microlenses wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a displacement pattern, said displacement pattern defining the displacement of each micro-lens as a function of the position (/^',/) of the microlens in its respective sub-set.

31 . The micro-lens array of Claim 30, wherein the micro-lenses of each individual sub-set in a super-sub-set are arranged adjacent to one another.

32. The micro-lens array of Claim 30, wherein the microlenses of each individual subset in a super-sub-set are non-adjacent, and are interleaved with microlenses of other sub-sets of the super-sub-set.

33. The micro-lens array of any one of claims 30 to 32, wherein said common pattern defines displacements in integer multiples of unit displacement vectors.

34. The micro-lens array of claim 33 wherein the magnitude (τ ) of said unit displacement vectors is different for micro-lens subsets of different groups.

35. The micro-lens array of claim 33 wherein the magnitude (τ ) of the unit displacement vectors for at least one group of sub-sets is defined according to a user selected object focal isation distance

36. The micro-lens array of any one of claims 33 to 35, wherein the integer multiples of said unit vectors for each micro-lens are a function of the position of the micro-lens within its respective sub-set, and wherein said integer multiples are the same for micro-lens subsets of different groups.

37. An imaging device comprising a micro-lens array according to any of claims 30 to 36 and a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a micro-image.