GB2505955A

GB2505955A - Micro lens array with a colour filter set and imaging apparatus suitable for a light-field colour camera

Info

Publication number: GB2505955A
Application number: GB1216631.0A
Authority: GB
Inventors: Benoit Vandame
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 2012-09-18
Filing date: 2012-09-18
Publication date: 2014-03-19
Anticipated expiration: 2032-09-18
Also published as: GB2505955B; GB201216631D0

Abstract

A micro-lens array is provided where the centres of the micro-lenses are slightly displaced versus a regular lattice. The small displacements may be defined by the characteristic parameters of the light-field camera. The array includes a colour filter with a repetitive pattern of filters mounted in relation to the array. The array is divided into elementary groups 1301 and super-sub-sets 1302 which comprise a two-dimensional array of NxN elementary groups. The elementary groups are displaced as a whole relative to a regular lattice pattern according to a common displacement pattern. The common displacement pattern operates such that a colour sampling step is close to or equal to the sampling step of projected pixel coordinates of a refocused image generated from a set of images obtained through the micro-lens array.

Description

MICRO LENS ARRAY WITH A COLOUR FILTER SET AND IMAGING

APPARATUS

The invention relates to light-field camera and to a micro-lens array mounted with a colour-filter-array for light-field camera integrating a sensor.

Light-Field cameras record 4D (four dimensional) light-field data which can be transformed into various reconstructed images like re-focused images with freely selected focal distance that is the depth of the image plane which is in focus. A re-focused image is built by projecting the various 4D light-field pixels into a 2D (two dimensional) image. Unfortunately the resolution of a re-focused image varies with the focal distance.

Also, colour light-field cameras wherein the lens array is mounted with a colour-filter-array (CFA) such that a lens let only photons with a given colour component defined by the CPA to pass through. The filters of the CFA are arranged according to a pattern which is replicated cyclically on top of all lenses. The CFA pattern is replicated cyclically on top of the lens array. As one lens only let one colour component pass through thus, pixels of the sensor records only one colour component depending on the lens from which it records photons. Light-field cameras made of monochrome sensor and a lens array mounted with a CFA are considered. The projection of the 4D light-field pixels defmes a set of projected points having special colours defined by the CFA pattern. Unfortunately the distribution of the colour components is clustered and not optimal. In the optimal case, the distribution of the colour component is such that a given projected coordinate receive the contribution all colour components of the CFA.

In some embodiments, the filter may be integrated, for example, directly in the micro-lens, While these filters still form an array of filters, they may be called a colour filter set which cover both the typical CEA and other embodiments of the filters. In all embodiments, a filter is associated with each micro-lens and is arranged according to a CFA pattern. In the following, the wording CFA will be used to design indifferently all these embodiments of the filters.

For example, the publication US 2010/0265381A1, "Imaging Device" proposes an imaging device with a micro-lens array where the micro-lenses are displaced from an equidistant arrangement to a non-linear arrangement according to the height of the image on the imaging element. The pitch between the micro-lenses changes from the centre to periphery of the micro-lens array. The displacement provides an optical correction in order to compensate the image displacement implied by the geometric distortion of the main lens. The publication provides however no solution for improving the resolution of reconstructed images when the focal distance changes.

The present invention has been devised to address one or more of the foregoing concerns. It describes an array of micro-lenses mounted with a CFA where the centers of the lenses are slightly displaced versus a square lattice. The small displacements are defined by the characteristics of the light-field camera in order to provide an optimal distribution of the projected image and also to provide an optimal distribution of colour component. Consequently, the resolution of the re-focused image is better and more

constant than for conventional light-field camera.

According to a first aspect of the invention there is provided a micro-lens array for an imaging device comprising micro-lenses, wherein the micro-lens array comprises a colour filter set mounted in relation with said micro-lens array, each filter of which being associated to one micro-lens of the micro-lens array, the colour filter set being arranged as a repetitive pattern of filters; a plurality of micro-lens elementary groups, each elementary group fitting the pattern of the colour filter set; a plurality of micro-lens super-sub-sets, each super-sub-set comprising a two dimensional array of NxN elementary groups, N being a super-resolution factor defined as an integer in the interval [2, r] where r is the number of consecutive micro-lens through which an object is imaged and wherein the elementary groups of each super-sub-set are displaced as a whole relative to a regular lattice according to a common displacement pattern and the common displacement pattern defining different displacements for each elementary group of the super-sub-sets such as to decrease the variation of the sampling step between the projected pixel coordinates of an image generated from the set of images obtained through the micro-lens array used in the imaging device by zooming, shifting and summing the small images formed by each micro-lens and such that the colour sampling step is close or equal to the sampling step of the said projected pixel coordinates. Accordingly an optimal distribution of a reconstructed image and an optimal distribution of colour are provided such that the de-mosaicing is almost performed. The resolution of the re-focused image is better than with a regular square lattice. The variation of resolution with the focalization distance is much smaller than with an array of lenses arranged according to a regular square lattice.

In an embodiment said displacement pattern defines each displacement as a function of the position (/,1) of each elementary group within the sub-set.

In an embodiment said displacement pattern defines each displacement as a function of the number NxN of elementary groups in each super-sub-set.

In an embodiment said disp'acement pattern defines displacements in integer multiples of unit displaccment vectors.

In an embodiment the magnitude r of said unit displacement vectors is a function of focal distance f of the micro-lenses, In an embodiment the magnitude of said unit displacement vectors is a function of the number N. In an embodiment the magnitude of said unit displacement vectors is a function off/N.

In an embodiment said displacement pattern defines a plurality of possible displacements for each elementary group, each of said plurality being equivalent in modulo N. In an embodiment the displacement of at least one elementary group in each super-sub-set is zero.

In an embodiment the displacement pattern and the displacements are independent of the location of the super-sub-set in the micro-lens array.

In an embodiment said integer multiples (k, 1) are given by IkO,J) Ai+Bj+K (modN) and the values A,B,C,E are determined as a Ll(1,I) Cz+Ej+L (modN) solution of the equation: gcd(n2M2 + nM(A'+E') + A' E'-B'C'(modN),N) = 1 Vn E [o, N[.

According to another aspect of the invention there is provided a imaging device comprising a micro-lens array according to any of claims 1 to 12 and a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a micro-image.

In an embodiment the magnitude of said unit displacement vectors is given by r = Where f is the micro-lens focal distance, S is the physical size of a sensor pixel, d is the distance between the micro-lens array and the sensor.

At least parts of the methods according to the invention may be computer implemented. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit', "module" or "system".

Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

Since the present invention can be implemented in software, the present invention can be embodied as computer readable code for provision to a programmable apparaflis on any suitable carrier medium. A tangible cather medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid state memory device and the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g a microwave or RF signal.

Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which: H Figure 1 is a schematic view of a light-field camera; Figure 2 is a schematic view of a particular light-field camera; Figure 3 is a detailed view of a light-field camera made of perfect lenses; Figure 4 is a schematic view of the 4D light-field data recorded by the 2D image sensor

of a light-field camera;

Figure 5 is an illustration of the coordinates of the projected 4D Light-Field pixels into the 2D projected image in a normal case; Figure 6 is an illustration of the coordinates of the projected 4D Light-Field pixels into the 2D projected image in a case associated with a particular disparity; Figure 7 illustrates a normalized maximum sampling step of the projectedl reconstructed image from 4D light-field data for a conventional light-field camera; Figure 8 is a schematic view of a micro-lens array according to an embodiment of the invention with displaced micro-lenses for a super-resolution factor N2; Figure 9 is a schematic view of the a micro-lens array according to an embodiment of the invention with displaced micro-lenses for a super-resolution factor N3; Figure 10 illustrates a normalized maximum sampling step of the projected! reconstructed from 4D light-field data obtained with the micro-lens array of figure 8 with displaced micro-lenses for a super-resolution factor of N=2; Figure 11 illustrates a normalized maximum sampling step of the projected! reconstructed from 4D light-field data obtained with the micro-lens array of figure 8 with displaced micro-lenses for a super-resolution factor of N3; Figure 12 illustrates the Bayer paftern used in a CFA in some embodiments of the invention; Figure 13 illustrates the position of the micro-lens in an embodiment of the invention.

Particular aspects of light-field cameras will first be exposed.

We consider light-field cameras which record a 4D light-field on a single sensor like a 2D regular array of pixels. Such light-Field cameras may be for instance a plenoptic camera comprising a main lens, an array of lenses and a sensor. They may also be a multi-camera array comprising an array of lenses and a sensor, but without main lens. The array of lenses is often a micro-device, which is commonly named a micro-lens array.

Figure 1 illustrates a plenoptic camera 1 with three major elements: the main lens 10, the micro-lens array 11 and the sensor 12. Figure 2 illustrates a multi-camera array 2 with two major elements: the micro-lens array 11 and the single sensor 12.

Optionally spacer or spacer material may be located between the micro-lens array around each lens and the sensor to prevent light from one lens overlapping with the light of other lenses at the sensor side. It is worth noting that the multi-camera array can be considered as a particular case of plenoptic cameras where the main lens has an infinite focal length. Indeed, a lens with an infinite focal length has no impact on the rays of light. The present invention is applicable to plenoptic cameras as well as multi camera arrays.

Figure 4 illustrates the image which is recorded at the sensor. The sensor of a light-field camera records an image of the scene which is made of a collection of 2D micro-images, also called small images, arranged within a 2D image. Each small image is produced by a lens from the array of lenses. Each small image is represented by a circle, the shape of that small image being function of the shape of the micro-lens. A pixel of the sensor is located by its coordinates (x, y) . p is the distance in pixels between two centres of contiguous micro lens images. The micro-lenses are chosen such as p is larger than a pixel width. A micro-lens image is referenced by its coordinates (i,j). Some pixels might not receive any light from any micro-lens; those pixels are discarded. Indeed, the space between the micro-lenses can be masked to prevent photons falling outside of a lens (if the micro-lenses are square or another close packed shape, no masking is needed). However most of the pixels receive the light from one micro-lens. The pixels are associated with four coordinates (x, y) and (i,j). The centre of the micro-lens image (i,j) on the sensor is labelled (x,1, Yl]). Figure 4 illustrates the first micro-lens image (0,0) centred on (x00,y0o). The pixels of the sensor 12 are arranged in a regular rectangular lattice. The micro-lenses are arranged in a regular rectangular lattice. The pixels lattice and the micro-lenses lattice are relatively rotated by 0. The coordinate Yt]) can be written in function of the 4 parameters: p, 0 and (x00, yo,o): f x,1 = pcosO.i-psinGi+Xo,o = psinOi+pcosGi+Yo,o Figure 4 also illustrates how an object, represented by the black squares 3, in the scene is simultaneously visible on numerous micro-lens images. The distance w between two consecutive imaging points of the same object 3 on the sensor is known as the disparity. The disparity depends on the physical distance between the camera and the object. w converges to p as the object becomes closer to the camera. Depending on the light-field camera design, w is either larger or smaller than p (if d is respectively larger or smaller than f, (see Figure 3 and related description about the geometrical property of the light-field camera). Figure 4 illustrates a case where w is smaller than p. An important characteristic is the number r of consecutive lenses through which an object is imaged. r is in units of /. It is estimated by considering the cumulated disparity on consecutive lenses: wr -pr <p. The following characteristic is obtained for the number of replications, r: r= (2) F L_wI Where LaJ denotes the ceiling value of a. This equation is an estimation which assumes that the micro-lens images are squared with no left-over space (i.e. Close packed or abutting). r is given for one dimension, an object is therefore visible in r2 micro-lens images considering the 2D grid of micro-lens. Without the ceiling function, r would be a non-integer value, r is in fact an average approximation. In practice, an object can be seen r or r + 1 times depending on rounding effect.

The previous section introduced w the disparity of a given observed object, and p the distance between two consecutive micro-lens images. Both distances are defined in pixel units. They are converted into physical distances (meters) W and I' by multiplying respectively w and p by the pixels size 8 of the sensor: W = & and P = Sp.

The distances W and P can be computed knowing the characteristics of the plenoptic camera. Figure 3 gives a schematic view of the plenoptic camera. The main lens 10 is an ideal thin lens with a focal distance F. The micro lens array 11 is made of micro-lenses having a focal distance f. The pitch of the micro-lenses is 0. The micro-lens array is located at the fix distance D from the main lens. The micro-lenses might have any shape like circular or squared. The diameter of the shape is smaller or equal to 0. The particular case where the micro-lenses are pinholes may be considered. In this case the following equation remains valid with f = d. The sensor 12 is made of a squared lattice of pixels having each a physical size of 8. 8 is in unit of meter per pixel. The sensor is located at the fix distance d from the micro-lens array. The object, not represented, is located at the distance z of the main lens. This object is focused by the main lens at a distance z' from the main lens. The disparity of the object between two consecutive lens is equal to W. The distance between 2 micro-lens image centres is P. Following the mathematics of thin lenses we have: (3) zz'F From the Thales law we can derive that: D-z'D-z'+d 4 0 -() Mixing the 2 previous equations the following equation is easily demonstrated: d z -P This equation gives the relation between the physical object located at distance z from the main lens and the disparity W of the corresponding views of that object.

This relation is build using geometrical considerations and does not assume that the object is in focus at the sensor side. The focal length / of the micro-lenses and other properties such as the lens apertures allow determining if the micro-lens images observed on the sensor are in focus. In practice, the distances D and d are typically tuned once for all using the relation: 1 11 +---(6) D-z' d f The micro-lens images observed on the sensor of an object located at distance z from the main lens appears in focus as long as the circle of confusion is smaller than the pixel size 8. In practice the range [zrn, z] of distances z which allows observing in focus micro-images is large and can be optimized depending on the focal length f, the apertures of the main lens and the micro-lenses, the distances D and d: for instance the micro-lens camera may be tuned to have a range of z from 1 meter to infinity [l,co].

Also from the Thales law P is derived: D+d e D (7) P=Øe Thc ratio e defines the enlargement between the micro-lens pitch and the micro-lens images pitch projected at the sensor side.

The light-field camera being designed, the values D, d, J and F are tuned and fixed. The disparity W varies with z, the object distance. Some particular values of W may be noted. W0 is the disparity for an object at distance z0 such that the micro-lens images are exactly in focus, it corresponds to equation (6) . Mixing equations (4) and (6) it is obtained: =0 (8) W is the disparity for an object located at distance z = aF from the main lens. According to equation (5) it is obtained: d (9) D_aF a-I The variation of disparity is an important property of the light-field camera. The ratio W I Wfi is a good indicator of the variation of disparity. Indeed the micro-lens images of objects located at are sharp and the light field camera is designed to observed objects around z1 which are also in focus. The ratio is computed with equations (8) and (9): 1+ 1 (10) d a-i The ratio is very close to one. In practice the variations of disparity is typically within few percentage around WfOCUS. The present inventor has further brought to light the following aspects.

A major interest of the light-field cameras is the ability to compute 2D images where the focal distance is freely adjustable. To compute a 2D image out of the 4D light-field, the small images observed on the sensor are zoomed, shifted and summed. A given pixel (x, y) of the sensor associated with the micro-lens (i, J) is projected into a 2D image according to the following equation: fX = s.(g(x-x,1)+x11) = s.(g(y-y,)+yj1) (ii) Where (X,Y) is the coordinate of the projected pixel on the 2D refocused image. The refocused image is therefore obtained by zooming, shifting and summing the small images formed by each micro-lens. The coordinate (X, Y) is not necessarily integer. The pixel value at location (x, y) is projected into the 2D refocused image using common image interpolation technique. Parameter s controls the size of the 2D refocused image, and g controls the plane which is in focus (the plane perpendicular to the optical axis, for which the 2D image is in focus) as well as the zoom performed on the small images. The output image is s2 times the sensor image size. In this formulation the size of the re-focused image is independent from the parameter g, and the small images are zoomed by sg.

The previous equation can be reformulated due to the regularity of position of the centres of the micro-lens images.

JX = sgx÷sp(1-g)(cos8i--sin9i)+s(l-g)xo,o 12 1 Y = s+sp(1_gsinO/+cosO.J)+S(lg)Yo The parameter g can be expressed as function of p and w. It is computed by simple geometry. It corresponds to the zoom that must be performed on the micro-lens images, using their centres as reference, such that the various zoomed views of a same objects get superposed. The following relation is deduced: (13) p-w This relation is used to select the distance z of the objects in focus in the projected image. The value of g can be negative depending on the light-field camera design. A negative valuc means that the micro-lens images need to be inverted before being sunimed. It may be noticed that r = g[j.

Including this last relation into equation (12) the projection equation may be rewritten: X = sgx-sgw(cosOi-sin8j)+-Xo,o 1 (14) Y = s'-sv(sinO'i+cosThj)+ Y00 p The last formulation has the great advantage to simplify the computation of the projected coordinate by splitting the pixel coordinates (x,y) and the lens coordinates

(i,j) of the 4D light-field.

The different pixels of the light-field image are projected into the re-focused image according to the above described method and define a set of projected coordinates (X,Y) into the grid of the refocused image. It has been recognized by the present inventors that the distribution of the set of projected coordinates is an important property which can be used to characterise the resolution of the refocused image, and in particular, the regularity or homogeneity of the distribution. As will be explained later the present invention addresses this homogeneity.

It is not trivial to characterize the homogeneity of the projected 4D light-field pixels into the 2D re-focused image. To study this property a simple projection equation is considered assuming that the rotation angle 8 is zero, and the coordinate of the first micro-lens centre (x00,y00) is equal to (0,0). This assumption does not impact the proposed study. The following simplified projection equation is obtained with u = sg: x = ux-uwi= (x-wi) (15) Y = uyuwj= P (y-wj) p-w This set of equation shows a simple relation between the 4 dimensions x,y,i,j and the projected coordinates (X, Y). The value u = sg is a constant independent of w if s = k / g where k is any constant. In this condition, the size of the re-focused image is ftinction of w and is equal to klg times the size of the original image.

Figure 5 illustrates the ID projected coordinate X for a paiticular settings: s=0.5, g=7.677, w=151.05, u=3.83 and p=173.67. The x-axis shows the projected coordinates X, the y-axis indicates the micro-lens coordinates i of the projected pixels. It is to be noted that 8 micro-lenses contribute to the observed projected coordinates X, which in this case is equal to r +1. The distribution of the projected points X is not homogeneous since the values h and H representing respectively the minimum and the maximum sampling steps between 2 consecutive projected coordinates X are substantially different from each other. In this example, the projected coordinates are nearly supcrposed, clustered in groups of eight.

Figure 6 illustrates the same view th the same settings except that w = 151.25.

The distribution of the projected points X is homogeneous, the projected points being distributed with equal spacing along the axis X. This ideal case is to be noted where /i = H = u/{w}. And where {a} denotes the fractional part of a. {w} plays a major role in the maximum sampling steps between the projected coordinates.

Several cases of h and H occur depending on {w}.

In a first case, we have {w} = 0: h = 0 and H = u. On average, r projected coordinates X overlap. The distance between 2 non-overlapped consecutive X is constant and equal to H. The projected coordinates define a perfect sampling with a constant sampling step equal to H = u.

In a second case, we have {w} = n / N where n and N are positive integers such as 0 <n <N «= r. In this case the number of overlapped projected coordinates X is equal, on average, to r gcd(n, N) / N where gcd(n, N) refers to the greatest common divisor between ii and N. The projected coordinates define a perfect sampling with a constant sampling step equal to H = u gcd(n, N) IN. The sampling step is smaller if N is a prime number. Indeed, if N is not a prime number, the number of overlapped coordinates increase as well as the sampling step. The perfect sampling of the projected coordinates X with the smallest sampling step is obtained for {w} = n / r and gcd(n,r)=1.

In a third case, we have {w} = n / N where n and N are positive integers such as 0 < n < N and N> r. In this case there are no overlapped projected coordinates X. But the sampling defined by the projected coordinates is not perfect: Ii = u gcd(n, N) / N and H = is -h. The projected pixels are clustered, in other word some projected pixels are sampled with a small sampling step equal to h, where other pixels are sampled with a larger sampling step H: the projected coordinates appear clustered.

K is a good indicator to estimate the resolution of the re-focused image. Figure 7 illustrates the normalized sampling step as a function of {w} for a conventional light field camera characterized by p = 173.67 and Wfocus 150. This function is built from thc 3 cases described above: points surrounded by the black circles depict the first case ( h = 0 and {w} = 0); points surrounded by the empty circles depict the second case (all the possible regular grids) ; other points lying on the dark line segments correspond to the third case with all possible n values and any N > r (all possible irregular grids).

The best possible resolution is given by h = H/u = hr = 1/7.

The projection of the 4D light-field pixels defines a set of projected points having a distribution which depends on the selected focal distance (i.e. The object plane which it is desired to be in focus). As explained above, the resolution of the re-focused image highly depends on the distribution of the projected coordinates (X, Y). The resolution can be estimated by the maximum sampling step H. Unfortunately, II depends on (w} and varies from values of is to u / r. Variations of H make the resolution of the re-focused image vary. The present inventors have recognized that the distribution characterizes the resolution of the projected image. It is an object or at least one aspect of the invention to mitigate sampling variation in reconstructed images and therefore to obtain a more constant resolution of the reconstructed images for any selected focal distance.

In the present invention, a micro-lens array where the centres of the micro-lenses are slightly displaced versus a regular lattice is used. The small displacements are preferably defined by characteristic parameters of the light-field camera and in certain embodiments can advantageously be arranged in order to provide an optimal distribution of the projected image.

The pixels of the 4D light-field image are projected into a re-focus image. As described above, the maximum sampling step of the projected coordinates depends on {} the fractional part of the disparity. The variations of sampling step are due to the superposition or clustering of the projected coordinates for certain values of {w} as illustrated in figure 5.

To decrease the superposition or the clustering of the projected coordinates (X, Y), the micro-lens images are shifted as compared to a regular array, so as to reduce or prevent overlapping or clustering of projected pixels. In other words, in embodiments of the invention the centre of a given micro-lens (i,i) is shifted by the given shift (A, (I, J), A (I, j)) so that the modified projected coordinates (X', Y') of this new light-

field camera would become:

1X' = ux_uw.O+A?Q,j))_X-lnvAiU,f) = uy_uw.(j+AO,i))Y-mvA1@i) (16) (A,(i,j),A1(i,J)) are shifts given in unit of distance between the micro-lens centres, or the micro-lens image centies. The motivation of moving the micro-lenses is to have a perfect and constant sampling of the projected coordinates (X,Y) for any w = Lwi + n / N where N is a selected positive integer smaller or equal to T, and n is any integer such as n [o, N[ .In conventional light-field, the sampling step is a function of n for a given N. If the sampling step is made independent of n, then N acts as a super-resolution factor. Equation (16) becomes: = = N(x_Lwj)_ni-a1(i,j)(NwJ+fl) u (17) F' = = N(y_4)_nj_A1O,i)(NLWj+fl) (x',Y") are noalized projected coordinates such that (X",Y") are integers for a perfect sampling of the projected coordinates (X', Y'). For a perfect sampling A1 (i, j)(Atwj + n) and A1 (1, j)(N{wj + must also be integers respectively equal to Hf,]) and l(i, j). These constraints give us the following values for (A1 (/, j), A, (i, I)): A (i, I) = k3(i,j) A (1,1) = N@,J) (18) 2 Nw Nw The displacement of the micro-lens images depends on w. In other words, for a given micro-lens displacement (A1 (I, j), A1 (i, j)) the shift of the corresponding micro-lens image depends on the disparity w. The previous equations can be approximated by taking into consideration the two considerations: 1) w>> N; and 2) the variations of w are small. w can be considered constant and equal to w10. Indeed, it has been shown (cf equation (10)) that the ratio W / W10,., which is equal to w / is typically very close to I. In this condition, equation (18) can be approximated by: A.(i,j) I kOi) A 0,]) 1 lO,j) AT Wf0ç112 N (19) 8 kQ,j) _____ foczc focus 1 5 The second line of the previous equation is given knowing that w10 = / 8, where 8 is the physical size of a pixel. The approximation of (A,(i,j),A,Ø,])) does not depend on the disparity w. Thus, by using this approximation, it is possible to build a micro-lens array with an irregular grid of micro-lenses such as the projected coordinates (X',Y') do not overlap or cluster as it happens for the projected coordinates (X, Y) of conventional light-field imaging devices.

A remaining question is how to define the 2 functions kNO,j) and lN('f) to have optimum micro-lens displacements such that the projected coordinate (X", Y") have a minimum clustering, and a perfect sampling when w = {wJ + n / N. Equation (17) can be simplified considering equation (19) and w >> N: = X' N(xLw)_nik@,j) (20) = Y' = N(y-Lwjj)-nil(i,j) To obtain a perfect sampling the set of projected coordinates (X", F') defined by the various lens coordinates (i, J) must have all possible integer values whatever n, and also the number of contiguous lenses to obtain the perfect sampling must be minimum and equal to N (considering one dimension). This constraint can be reformulated by taking into consideration modular arithmetic modulo N JX" N(x-Lwji)_ni-kN(i,j) -ni--k(i,J) (modN) N(y-Lw)-nf-1A@,j) -nj-1N@,J) (modN) kN(i,j) and 1,(i,j) are 2 periodic functions, one period is defined with (i,j) [o, iv[2, into [o, N[, these two functions are defined modulo N. kN (i, J) and N (i,j) are searched, such that for any given n c [o, N[, all the set of projected coordinates (x' mod N, V mod iv) defined by (i mod N, j mod N) E [o, N[2 is equal to H hN-I. . . bab where 8a.b is the Dirac function located at (a, b) with (a, b) being integer numbers.

To solve equation (21) the following linear solutions are considered: JkJ'i) Ai+B]+K (modiV) (22) tlN(t,j) Ci+Ej+L (modN) Equation (21) becomes: JX" -ni-Ai-Bj-K (modN) (23) Y" -nj-Ci-Ej-L (modN) For the second member of the previous equation, the value of Ci is derived, which also appear in the first member after multiplying it by C: 1cx" Cni-CAi-CBj-CK (modN) Ci-_Yu_nj_Ej_L (modN) (24) Replacing the second member into the first member (24) becomes: X_j(n2 +n(A+E)+AE_BC)(Y'+LXn+A)-CK (modN) (25) The set of projected coordinates located at (x" mod N, Y' mod N) must cover b-N-i b=N-I all coordinates aO b=O 8a,h br (z mod N, j mod N) c [o, N[ whatever n e [0, N[.

With equation (25) it may be deduced that X"(mod N) must have any possible values whatever Y"(mod N). Thus the second order polynomial function m(n)= n2 +n(A +E)+ AE-BC must verify: gcd(m(n)modN,N)1 vn[0,N[ gcd(n2 +n(A+ E)÷ AE_BC(modN),N)-1 Vn e[0,N[ The N x N micro-lenses defines a sub-set of micro-lenses. For a given sub-set the values A, B, C, E, K, L are freely selected according to the previous equation. The parameters A, B, C,E, K, L may take different values in the different sub-sets.

Parameters K, L define which if any micro-lens from a given subset is not displaced with respect to the regular lattice.

On an experimental point of view, many values A, B, C, E verify equation (26).

The special case: A=O, B=T, C=l, E=l, K=0 and L=O is detailed in this section. The proposed solution has the following form: JkJi,J) Tj (modN) (27 t1N@,j) i+j (modN) T is a free parameter which has been experimentally determined for various values N. The experimentation consists in testing various values of T c [o, N[ such that the constraint gcd(tn(n)modN, N) = 1 is respected for any n C [o, N[. The following table indicates the smallest value of T according to that constraint:

N TN TN TN TNT

1 0 6 1 11 3 16 4 21 1 2 1 7 1 12 3 17 1 22 3 3 1 8 1 13 1 18 1 23 1 41 91141193241 3 15 1 20325 3 It follows ihat the periodic functions kN(i,j) and IN(4f) are frilly characterized and thus the shifts (A, (i, J), A, (i, j)) of the micro-lens image versus the regular grid are also fully characterized. The shifts are given in unit of (I, j) To convert the shifts into physical unit at the micro-lens side, the shifts must be multiplied by 0. The physical shifts (A, (i, J)' A (1.])) at the micro-lens side are computed easily by combining equation (8) and (19): = f..kN(i,j) A(i,j) = i-.P_.l0,J) (28) The physical shifts can be decomposed in the increment V = dlv which is multiplied by the integers values given by kiO,i) and lOJ) to obtain the physical shifts.

The design of the micro-lens array is therefore defined by: * The focal distance f of the micro-lenses.

* The average pitch 0 between consecutive lenses.

* The distance d between the micro-lens array and the sensor.

* The pixel size 8 of the sensor.

* The super-resolution factor N which is freely selected between [i, r].

* The micro-lens centres (p,, are located following the equation: /1, = d (29) p1 = j.ø+-.-.IN(i,j) It should be recalled that the functions k(i,j) and IN(j'f) are defined modulo N: thus the centers (p1, p1) are valid as well as + a,p1 + a f) whatever a being an integer. Consequently the displacements can be negative.

The micro-lens array is designed according to the previous settings. If the size of the micro-lenses is equal to the pitch 0, then the micro-lenses might have a very small overlaps due to displacement the micro-lenses versus the squared lattice. This issue is solved by designing micro-lenses such that the micro-lens size is smaller than 0 --1,3N, The shape of the micro-lenses can be circular, rectangular or any shape without any modification of the previous equations. The number of micro-lens (i, 4 to be designed in the micro-lens array is defined such that (i, Jq) is equal to the physical size of the sensor. The micro-lens array being designed, it is located at distance d from the sensor. It is interesting to note that the above demonstration remains valid whatever the coordinates of the first lens.

These displacements define a pattern of displacements such that the variation of the sampling step between the projected coordinates is decreased. Therefore, as the resolution can be estimated by the maximum sampling step H, this allows to improve the resolution of the projected images, and this whatever the focus chosen for the projection and the corresponding disparity.

An imaging device including the above proposed arrangement as well as a micro-lens array will now be described. The following values are chosen for the different parameters: Symbols Values Comments F 70mm Main focal distance f 2mm Micro-lens focal distance d 2.3mm Distance between the micro-lens array and the sensor 0 1mm Micro-lens pitch S 0.004mm/pixel Physical size of pixel from the sensor 2focuc 5000mm Object is located at 5 meters from the main lens 1 70.994mm Distance between the main lens of the focus plan of object D 86.327mm Distance between the main lens and the micro-lens array such that images on sensor is in focus.

15.33mm Distance between: the focus plan of the object z observed through the main lens, and the micro-lens array, 1.0266 Enlargement P 1. 0266mm Pitch in physical unit of the micro-lens images projected on the sensor P 256.6opixel Pitch in pixel unit of the micro-lens images projected on the sensor 1.15mm Disparity in physical unit observed on the sensor of the object located at distance z from the main lens.

Wfflts 287.Spixel Disparity in pixel unit observed on the sensor of the object located at distance from the main lens.

-r 8 Averaged number of replications for an object located at distance z from the main lens.

Figure 8 illustrates a case where the super-resolution factor is chosen to be N = 2 which also corresponds to the size of the N by N sub-set. In this case the increment, corresponding to the magnitude of unit displacement vectors, V = is equal to r =l.74pm. On the figures 8 and 9, the value CQ,j) refers to the center of micro-lens having the coordinates (1,]).

As described above, the displacements are determined in integer multiples of the unit displacement vectors which extend in directions i andj, such that the superposition or the clustering of pixels in a reconstructed image is decreased (see also figure 10).

The values k, (i, J), 2 (i, f) and (p, for the first sub-set of 2x2 micro-lenses illustrated in figure 8 are given in the following table: I I k2(i,j) 12(i,j) 0 0 0 0 0 1 0 0 1 0 0 1 1 1 r Ø+r

_____ _______ -

Figure 8 illustrates the displacement of the micro-lenses versus the regular squared lattice. The bold arrows indicate the displacement of the micro-lens centres by r in the direction indicated by the arrow. The bloc of 2 x 2 micro-lenses indicated by the bold dashed square, is replicated in i and j directions. It is worth noting that the arrows displayed in that figure have been artificially zoomed for illustration purpose.

Figure 9 is similar to figure 8 but illustrates a case with a super-resolution factor of N = 3. In this ce the increment is r = 1.l6ym The values k3,j), 13(i,J) and (p,p1) for the first sub-set of 3x3 micro-lenses illustrated in figure 9 are given in the following table: I k3(z,J) 3@'J) _ 2 2 2 1 2Ø+2r 2Ø+r The resolution of the projected image can be estimated by computing its maximum sampling step H as for the conventional light-field camera made of a lens array arranged following a square lattice as presented in figure 7.

The projected coordinates (X', Y'), obtained by the proposed micro-lens array with a super-resolution factor N, defines a set of points in the 2D projected/reconstructed. The set of points according to the proposed micro-lens array is characterized by the maximum sampling step II'. The values of H' have a simple expression for projected coordinates obtained with a disparity having the form: {w} = n /(Z'/iVI) with n and M being positive integers such as 0 «= n cM «= Lr / NJ. The maximum sampling step is equal to H'= ugcd(n,M)/(NM) $ The largest value of H' is it / N, it is recalled that the largest value H obtained for the conventional square lattice micro-lens array is equal to U. Figure 10 illustrates the normalized H/u values with the super-resolution factor N = 2 as a function of the fractional part of the disparity (w}. The corresponding characteristic parameters of the light-field camera are the one given above. The dashed line recalls the normalized H/u values obtained with a conventional light-field camera equipped with a regular square lattice micro-lens array. Similarly, Figure 11 illustrates the nomrnlized H / it values with the super-resolution factor N = 3. It may be noted on these figures, that the sampling step between the projected pixel coordinates is nearly constant regarding the disparity value w. The sampling step curve regarding the disparity has been flattened when compared to the curve obtained with a conventional light-field camera. Therefore the worst cases corresponding to w values of 0, 0.5 and I have been eliminated and the sampling step becomes nearly constant for any fractional part of disparity. The sampling step is substantially constant and maximum for disparities having values 0, 1/3, 2/3 and I (Figure 11).

On can observe in figures 10 and 11 that the resolution of the reconstructed image varies less than a conventional (dash lines) with regular square lattice. Therefore, a more regular resolution is obtained with the proposed micro-lens array. The regularity of the resolution increases with the value of the supcr-resolution factor N. This can also be applied for light-field cameras made of an array of lenses and one sensor as illustrated in figure 2. The array of lenses is designed with the equation (29).

For a homogeneous sampling of the projected-coordinate, the rotation angle 6 between the pixel lattice and the micro-lens array is set to 0 or at least within {-1/N,l/Nj where N is the sensor width. The translation between the pixel lattice and the micro-lens array defined by (x00, y00) can be set to any values, and does not impact the advantages of the proposed invention.

The lens array made of shifted lens centres in the previous section is designed for monochrome sensor and for micro-lens array where the lenses let all possible photons passing through, they are colour neutral, In practice, colour components are often captured by a Colour Filter Array (CPA) which defines a repetitive pattern of M x M colour elements. One denotes (a,b) e [0,M[2 being the coordinate of a colour component within the CFA matrix. Typically, he CPA is cyclically mounted on top of the micro-lens array. But in some embodiments, the CFA may be an)where on the light path, as long as one filter is associated with one micro-lens. The lens (i, J) of the sensor is by definition associated to the colour coordinate (a, b) = (i mod M, j mod M).

A well-known CFA is the Bayer pattern which defines a 2 x 2 set of filters equal to Red, Green, Green and Blue as illustrated in Figure 12. A large variety of colour components (or more generally filters) can be used for the CFA mounted on the micro-lens array, among other on notices the following filters: neutral density filters to extend the dynamic range of the re-focus image; polarization filters to compute re-focus images with selectable polarization angle. CFA are mostly used to estimate colours with monochrome sensor. The filter of the CFA can be mounted directly on top of a given micro-lens, or it can be set anywhere between the given micro-lens and the pixels which receive the photons passing through that given lens. The goal of the CFA is that a given micro-lens image (1,1) receives only photons passing through the filter coordinate (a,b) = (imodM,jmod M) of the CFA.

With a CFA mounted on the micro-lens array, a given pixel records only one colour component since a pixel receives photons from one micro-lens only. For a given pixel (x,y), 4%J -1 colour components are missing. The missing colours cannot be estimated with the nearby pixels around (x, y) because all these pixels receive the same filtered photons from the same micro-lens (i, J). By construction, the 4D light-field pixel (x,y,i,j) is associated to the colour coordinate (a,b) = (imod M,jmod Al).

The process of estimating the missing colour components is called de-mosaicing and is famous for CFA mounted on sensor. In the case of micro-lens array mounted with a CFA, the de-mosaicing cannot be performed at the level of the 4D light-field records by the sensor. Nevertheless, de-mosaicing has a meaning when computing the re-focus images. Indeed, 4D light-field pixels are intended to be merged to form for instance re-focus images. In this context, 4D light-field pixels from different micro-lenses are interleaved or overlapped, thus allowing an estimation of re-focus images where re-focus pixels are computed with Al2 colour components.

In the context of the proposed invention, a light-field camera with a CFA mounted on top of the micro-lens anay and a monochrome sensor is considered. Image re-focusing makes the coordinates of 4D light-field pixels to interleave or to cluster.

Depending on the disparity, several 4D light-field pixels are projected at the same coordinate of the re-focus image. These 4D light-field pixels, which overlap, might have the same colour or distinct colours depending on the CFA attached to the micro-lenses.

The 4D light-field pixels are projected into the refocus image according to equation (20) which takes into consideration: the micro-lens shifts given by kN (i, J) and 1N (i, J) the set of disparities equal to w = Lwi + ii / N such that the projected coordinates are integers. In equation (20) it is assumed that the micro-lens array is not rotated with respect to the pixel matrix, and the re-focus image is computed for the projected coordinate (x' , y") which are integer for the set of disparity w = [wj + n / N. To obtain a re-focus image with a natural de-mosaicing, at least M2 4D light-field pixels must be projected into the same re-focus image coordinate which is possible if the replication factor is large. If the replication factor r »= NM then one object is visible at least on (NM micro-lenses which cover M2 sub-sets of Nx N micro-lenses.

An object visible at the integer 4D light-field coordinate (x,y,i,j) is considered.

On the micro-lens (i + N,j) this object is visible at the 4D light-field coordinate (x+Nw,y,i+N,j). The shift from x to x+Nw-_x+NLwJ+fl is independent to the lens shifts kN(i,j) and 1k('J) because lens (i,j) and lens (i + N,j) are associated respectively to the lens shifts kN (if, j) and kN (i + N,]) which are equal because kN (1,]) is defined modulo N. More generally, one considers the M2 4D light-field pixels (x+,nNw,y+mNw,i+mN,j+mN) where the same object is visible, with (rn,m)E [0,M[2. These 4D light-field pixels are projected into the same re-focus image coordinate (x',Y"). This can be demonstrated easily by replacing the 4D light-field pixel (x,y,i,i) by (x+mNw,y+myNw,i+mxN,]+myN) in equation (20). By 27 K definition, one increment of (m, m) corresponds to an increment of (N, N) lenses in the 4D light-field image; in other words the coordinate (m,m9) designates a sub-set of NxN lenses with m =Li/NJ and in = Li/NJ.

The M2 4D light-field pixels are projected at the same coordinate (x" , Yj. A natural de-mosaicing is performed if the colour coordinates of the M2 4D light-field pixels are all distinct. The M2 colour coordinates (a,b) of the projected coordinates are given by the following equation: i+m,N (modM) lb j+inN (modAl) (30) The 4f 2 colour coordinates (a,b) are all distinct if gcd(N,M)=1 (M is prime with N). This condition depends on the size M of the CFA and the super-resolution factor N. For instance if N = 3 and M = 2 (as a Bayer CFA) gcd(2,3) = 1, then the Al2 4D light-field which are projected at the same coordinate (x",Y") are observed through Al2 distinct micro-lenses which are covered by the M2 distinct colour components of the CFA. The de-mosaicing is automatic and perfect for any disparity w = LwJ+ n/3. The normalized sampling step of the projected pixel is equal to if/u = 1, and the sampling step of the colour coordinate is also equal to [f/u = 1.

Unfortunately, this advantage does not occur for instance if N = 2 and M = 2.

This case behaves badly because the Al2 4D light-field pixels, which are projected at the same coordinate (x',Y"), have in fact the same colour coordinate. In other word, the re-focus image benefits from the shifted micro-lens array because the projected coordinates are regularly spaced with a normalized sampling step equal to if/u = 1 independently to w = Hi + ii / N as demonstrated above. But the normalized sampling step of a given colour component is equal to H'/u = N = 2. The re-focus image nceds to be de-mosaiced by common de-mosaicing algorithms because the projection of the 4D light-field pixels into the re-focus pixels does not permit an automatic de-mosaicing.

This external de-mosaicing is just an estimation which cannot be as perfect as the natural de-mosaicing mentioned above.

The resolution of the re-focused image depends on the distribution of the projected coordinates (x",v"), and also on the distribution of the colour coordinate associated to 4D light-field pixels. With the shifted micro-lens array described previously, the distribution of the projected coordinates is quite constant with few variations depending on the disparity w. But the distribution of the colour coordinates has a large sampling step if gcd(N,M) != 1. In case gcd(N,M) = 1, the shifted micro-lens array described above offers a re-focus image where the projected colour coordinates have a small sampling step equal to the sampling step of the projected coordinates independently to their colour coordinate. One aspect of this invention is to propose a shifted micro-lens array design such that the projected 4D light-field pixels are regularly spaced with a sampling step close or equal to the sampling step of the colour coordinate such that the colour resolution of the re-focus image is good.

The proposed shifted micro-lens design is defined by modifying (20): XCFA" = = N(x_LwJO_fh1N[LUjL±jJ ii MM (31 2F4 = = N(y -[wJj)- -i' The shifts defined by k'M (i',j') and i'M (i',j) with 1= i/Mj and j' Li/Mi are parametric functions defined by: Jk'w(iJ') A'i'+B'j'+K' (modN) 1 iv(ij) C'i'+E'j'+L' (modN) AI[J+B'Lj+K' (modN) (32) i[ijHJ (modN) To determine the parameters k' (i,j') and 1TM (i, j') Equation (32) is expressed modulo N, according to the same mechanism that was used to convert equation (20) into equation (21)), Equation (32) becomes: JXCFA" N(x-[wJi)-ni-k'(i,f) -ni-k'(i',j') (modN) 1 N(y-wjj)-nj-l'NQ',f) -nj-1'(f,j') (modN) (33) This new equation mixes the lens coordinate (i, J) with the modified lens coordinate (i', j'). Knowing that (a, b) = (imod M, j mod U), for simplification (i, f) are written as: I M +imodM = Mi'+a (34) = JLzi+modM = Mj'+b Where (a,b) is the colour coordinate of the CFA pattern. A given modified coordinate (i', J') encompass M x M micro-lenses which defined a CPA-group of micro-lenses. (a,b) is the coordinate within a CPA-group. Equation (33) becomes -nMi'-na-k'(i',j) nMi'-Ai'--Bj'--K'-na (modN) 35 1 rFA -nMj-nb-l'N(i',J') nMj-Ci-Ej'-L-nb (modN) ( This last equation is quite similar to equation (23). The tems (-na,-nb) represents some shifts which are constant for a given ii and (a,b). similarly to equation (23), this last equation is solved with the following constraint which applies on the parameters of k' (1,!') and 1' (i',j'): gcd(s' (n) modN, N) =1 Vn e [o, N{ gcd(n2M2 + nM(A+E')+ AK-B' C'(modN), N) = i Vn E [o, N[ (36) With s' (n) = n2M2 + nM(A + E)+ AE -BC. This constraint states that for a given n and a given colour coordinate (a, b), the projected coordinates (x", r') modulo N occupies any positions within [0, N[ or in other words a regular sampling step equal to 1. The sampling step is independent of (cz,b) which is the colour coordinate. Therefore the set of projected coordinates (x", Y") define a regular sampling for all distinct colour coordinates. A given projected coordinate (X,Yh1) receives M2 4D light-field pixels with all distinct colour coordinate, thus the de-mosaicing of the re-focus image is naturally perfonned whatever M the size of the CFA pattern, The shifts k'ji,j') and iJ1,1) applies to NxN CFA-groups of MxM lenses, within a CFA-group all micro-lenses are shifted by the same value. One defines the notion of super-sub-set made of NM x NM lenses. A super-sub-set includes N x N CFA-groups of MxM lenses. The functions k'(i,j) and l'NQ',j') are defined by 6 parameters A', B' ,C', ±7, K', P. To ensure that the (a, b) take all possible values within [0,M[2 for a given super-sub-set, it is mandato that the Nx N functions k'N (1',;) and 1'N (i',j') (defined for the Nx N CFA-group) share the same 6 parameters A', B',C',E', K', V. Preferentially, the parameters A', B',C',E', K', L' are identical for all the super-sub-set which covers the micro-lens anay.

The previous equation considers the disparities equal to w = Lwi + ii / N. More generally, the projection equation of the proposed array of micro-lens is defined by: IXCFA' = ux -uw + A', (1, j)) = XCPA -uwA'1 (i, J) = uy -uw + A'1 (i, I)) = cpA -uwA'1 (i, J) (37) Where: A' (if) -k'N [LM[bi) A'1 (if) (LiLi) Wjoc,a (38) A', (J) -W k'w(Lj[j) A' (i,j) -IN Many values A',B',C',E',K',L' verify equation (36). The special case: A'=O, B'= T', C'= 1, E'= I, K'= 0 and L'= 0 is detailed in this section. The proposed solution has the following form: fkN(i',J) Ti' (mod N) 1 l(lJ) i'+j' (modN) (39) T' is a free parameter which has been experimentally detenined for various values N and M. The experimentation consists in testing various values of T' [o, N[ such that the constraint gcd(s' (n)mod N, N) = 1 is respected for any n e [o, N[. The following table indicates the smallest value of F according to that constraint for: M=I, M=2, M=3, M=4, M=6, M=7, M=8 andM=9: N rN r NJT' NIT' N r 1 0 6 1 11 3 16 1 21 1 2 1 7 1 12 1 17 1 223 3 1 8 1 13 1 18 1 23 1 4 1 9 1 14 1 19 3 24 1 3 10 3 1514 20j3 25 3 In the case of M = 5 and M = 10 the following table indicates the smallest value of T' according to the constraint: N r N r N r' N r N H 1 0 6 1 11 3 16 1 21 1 21 7 1 12 1 17 1 22 3 3 1 8 1 13118 1 23 1 4 1 9 114 1 19 3 24 1 1 10 1 15 1 20 1 25 1 I M = 2 is the size of the Bayer pattern which is the most common CFA. M = corresponds to the case where no CFA is used (CFA has the size of 1 by 1 pixel), this case is described in the previous section. Other values of M > 2 are given to illustrate the genericity of the proposed solution.

The periodic functions k'QiIMIUj/Mj), N Qi1MILj/iV are fully characterized thus the shifts (A', (/,j),A'1 (i,/)) of the micro-lens image versus the regular grid is also fully characterized. The shifts are given in unit of (1, j). To convert the shifts into physical unit at the micro-lens side, the shifts must be multiplied by 0.

The physical shifts (A (i, J) A'1 (/, j)) at the micro-lens side are computed easily by --combining equation (8) and (38): A', (i,j) = fk' [LJLJJ A'1,j = "M (LiLJJ (40) The physical shifts can be decomposed in the increment r = which is multiplied by the integers values given by k'N [i/Mj,{J/MJ) and (Ij/MJ,LjIMj The design of the micro-lens array is defined by: * The focal distance f of the micro-lenses, * The average pitch 0 bctween consecutive lenses.

* The distance ci between the micro-lens array and the sensor.

* The pixel size 8 of the sensor.

* The super-resolution factor N * The size of the Colour Filter Array M (N and M are selected such that r»=MN) * The micro-lens centres,u', ,p'1) are located following the equation: pt = dN LMJLM = It should he recalled that the functions k' Qi/MJ.L//Mj and 1\T ( / A'tJ, Li / tj) are defined modulo N thus the centres (,u'1.p'1) are valid as well as: (42) Whatever a and a1 being integers. Consequently the displacements can be negative and/or positive.

The micro-lens array is designed according to the previous settings. If the size of the micro-lenses is equal to the pitch 0, then the micro-lenses might have a very small overlaps due to displacement the micro-lenses versus the squared lattice. This issue is solved by designing micro-lenses such that the micro-lens size is smaller than 0 -(fS(N -1))/dN. The shape of the micro-lenses can be circular, rectangular or any shape without any modification of the previous equations. The number of micro-lens (i, j) to be designed in the micro-lens array is defined such that (ir, JØe) is equal to the physical size of the sensor. The micro-lens array being designed, is located at distance d from the sensor.

There is no other constraint to position the micro-lens array versus the main optical axis: the main optical axis crosses the micro-lens array at any location within the micro-lens array.

A particular embodiment is now described in relation with Figure 13 for the sake of example. The following dimensions are used: I Symbols Values Comments F 70mm Main focal distance 2mm Micro-lens focal distance d 23mm Distance between the micro-lens array and the sensor 0 1mm Micro-lens pitch e 0,004mm/pixel Physical size of pixel from the sensor z 5000mm Object is located at 5 meters from the main lens focus 70.994mm Distance between the main lens of the focus plan of object 2focuc D 86.327mm Distance between the main lens and the micro-lens array such that images on sensor is in focus.

D -15.33mm Distance between: the focus plan of the object z observed through the main lens, and the micro-lens array.

e 1.0266 Enlargement p 1. 0266mm Pitch in physical unit of the micro-lens images projected on the sensor p 256.66pixel Pitch in pixel unit of the micro-lens images projected on the sensor w 1.15mm Disparity in physical unit observed on the sensor of fiicus the object located at distance Zfi,cus from the main lens.

287.5pixel Disparity in pixel unit observed on the sensor of the A" object located at distance z from the main lens.

r Averaged number of replications for an object located at distance Zfocu from the main lens.

The case where N = 3 and M =2 (Bayer CFA) is considered. The shifts are characterized with the functions k'3Qi/2J,Lj/2j and l'3Qi/21L//2j). The increment, corresponding to the magnitude of the unit displacement vectors, r = f%JT is equal to r = l.74,wn. The values for the first super-sub-set of 6 x 6 micro-lenses are given in the following table. To compute,p'1) the parameters (a,., a1) from equation (42) have been tuned such that the displacements implies by k'3 JJ / 2j, [I /2j) and l'3Qi/2j[j/2j) are within]-3,0]. Thus and -N are positive as well as the displacements,p') given by S equation (41). ________ ________ ___ ? t'____ ___ ___ ____ o 1 0 2 1 2$ 0-i-v 4 1 4$ Ø-i-2r 1 1 0 0 3 1 3$ $+r 5 i 5$ Ø+2r o 2 V 2Ø+v 22 2$-fr 2Ø+2r 42 4Ø+r 2$ 1 2 Ø+r 2$-i-v 3 2 3$+r 2$+2v s 2 5Ø+r 2$ o 3 3Ø+v 2 3 2$+r 3$+2v 4 3 4g5+r 3$ 1 3 Ø+r 3$+r 3 3 3$+r 3Ø+2v 5 3 5q$+r 3$ o 4 2r 4$+2r 2 4 2Ø+2v 4$ 4 4 4$+2r 4Ø+r 1 4 çb+2r 4ç6+2r 3 4 3q$+2r 4$ 5 4 5$-t-2r 4Ø+r o 5 2r 5Ø+2r 2 2Ø+2r 50 4 5 4Ø+2r 5$+r 1 $+2r 5Ø+2v 3 3$+2r 50 5 5 5$+2r 5$+r Figure 13 illustrates the displacement of the micro-lenses versus the regular squared lattice for one super-sub-set. A complete micro-lens array is made by settings several super-sub-set in a regular manner. The thin horizontal or vertical dashed lines 0,1,2,3,4,5 illustrate the regular squared lattice. The dashed arrows, pointing the top-left corners of the CFA-groups, indicate the global displacement of a CFA-group versus the regular squared lattice. The bold arrows indicate the displacement in integer multiples of the unit displacement vectors of the micro-lenses versus the regular squared lattice (bold arrows are identical to dashed arrows). A detailed view of the shifts for lenses (2,2) (2,3) (3,2) and (3,3) versus the regular squared grid is given. It is worth noting that the arrows displayed in that figure have been artificially zoomed for illustration purpose.

In this example the micro-lens array is divided into elementary groups 1301 of ----MxM micro-lens, where M is the size of the CFA pattern. Actually, the CFA being mounted on the micro-lens array, it is to be noted that each elementary group fit exactly one pattern of the CFA, meaning that each micro-lens of an elementary group is associated with one colour filter. Then, the micro-lens are grouped to form super-sub-set 1302 of NMxNM micro-lens. These super-sub-sets are formed by NxN elementary groups of micro-lens. The elementary groups are displaced as a whole within the super-sub-set, meaning that each micro-lens belonging to a same elementary group are subject to the same displacement versus the regular squared lattice. The displacement of the elementary groups as a whole obeys to a pattern such that the projected 4D light-field pixels are regularly spaced with a sampling step close or equal to the sampling step of the colour coordinate such that the colour resolution of the re-focus image is good. The pattern is also such that the maximum sampling step H has fewer variation than it would have using a micro-lens array based on the regular lattice. Therefore, the resolution of the projected images is subject to fewer variations and is globally improved.

The proposed micro-lens design makes re-focus images to have a sampling which varies only a little with the disparity of the re-focus image. The proposed design with CFA mounted on micro-lenses, also makes the colour sampling step to be close or equal to the sampling step of the projected coordinates (whatever M the size of the CFA). Figure 10 and Figure 11 characterizes the sampling step of re-focused images obtained with the micro-lens design defined previously. These 2 figures also apply to the colour sampling step of the re-focus images computed with the proposed micro-lens design whatever M, Although the present invention has been described hereinabove with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications will be apparent to a skilled person in the art which lie within the scope of the present invention.

Many further modifications and variations will suggest themselves to those versed in the art upon making reference to the foregoing illustrative embodiments, which are given by way of example only and which are not intended to limit the scope H of the invention, that being determined solely by the appended claims, in particular the different features from different embodiments may be interchanged, where appropriate.

In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used.

Claims

CLAIMS1. A micro-lens array for an imaging device comprising micro-lenses, wherein the micro-lens array comprises: -a colour filter set mounted in relation with said micro-lens array, each filter of which being associated to one micro-lens of the micro-lens array, the colour filter set being arranged as a repetitive pattern of filters; -a plurality of micro-lens elementary groups, each elementary group fitting the pattern of the colour filter set; -a plurality of micro-lens super-sub-sets, each super-sub-set comprising a two dimensional array of NxN elementar y groups, N being a super-resolution factor defined as an integer in the interval [2, rJ where r is the number of consecutive micro-lens through which an object is imaged and wherein: -the elementary groups of each super-sub-set are displaced as a whole relative to a regular lattice according to a common displacement pattern; and -the common displacement pattern defining different displacements for each elementary group of the super-sub-sets such as to decrease the variation of the sampling step between the projected pixel coordinates of a refocused image generated from the set of images obtained through the micro-lens array used in the imaging device and such that the colour sampling step is close or equal to the sampling step of the said projected pixel coordinates.
2. The micro-lens array of claim 1 wherein said displacement pattern defines each displacement as a function of the position (i,)) of each elementary group within the sub-set.
3. The micro-lens array of claims 1 or 2 wherein said displacement pattern defines each displacement as a function of the number NXN of elementary groups in each super-sub-set.
4. The micro-lens array of any of the preceding claims wherein said displacement pattern defines displacements in integer multiples of unit displacement vectors.
5. The micro-lens array of claim 4 wherein the magnitude r of said unit displacement vectors is a function of focal distance f of the micro-lenses.
6. The micro-lens array of claims 4 or 5 wherein the magnitude of said unit displacement vectors is a function of the number N.
7. The micro-lens array of claim 6 wherein the magnitude of said unit displacement vectors is a function off/N.
8. The micro-lens array of claim 8 wherein said displacement pattern defines a plurality of possible displacements for each elementary group, each of said plurality being equivalent in modulo N.
9. The micro-lens array of any of the preceding claims wherein the displacement of at least one elementary group in each super-sub-set is zero.
10. The micro-lens array of any of the preceding claims wherein the displacement pattern and the displacements are independent of the location of the super-sub-set in the micro-lens array,
11. The micro-lens array according to Claim 4, wherein: -said integer multiples (k, 1) are given by Ik(i,j) Ai+Bj+K (modN) and where Ll(i,j) Ci+Ej+L (modN) -the values A, B, C, E being determined as a solution of the equation: gcd(n 2W + nM(A!+EI) + A' E'-B' C' (modN), N) =1 Vn E [o, N[.
12. An imaging device comprising a micro-lens array according to any of claims 1 to 12 and a photo-sensor having an array of pixels, each micro-lens projecting anHimage of a scene on an associated region of the photo-sensor forming a micro-image.
13. The imaging device of claim 13 as dependent upon claim 4, wherein the magnitude of said unit displacement vectors is given by T = where *f is the micro-lens focal distance, 8 is the physical size of a sensor pixel, d is the distance between the micro-lens array and the sensor.