ES1299472U - PLENOPTIC CAMERA FOR MOBILE DEVICES (Machine-translation by Google Translate, not legally binding) - Google Patents

Abstract

A plenoptic camera, characterized in that it comprises a single main lens (102), a single microlens array (104) and a single image sensor (108), wherein the plenoptic camera further comprises a first reflective element (510) configured to reflect light rays (601a) captured by the plenoptic chamber before reaching the image sensor (108); and wherein the primary lens (102) comprises a plurality of uncemented lens elements. (Machine-translation by Google Translate, not legally binding)

Description

DESCRIPTION

PLENOPTIC CAMERA FOR MOBILE DEVICES

field of invention

The present invention falls within the field of microlens arrays, optical systems incorporating microlens arrays, light field imaging, light field cameras, and plenoptic cameras.

state of the art

Plenoptic cameras are imaging devices capable of capturing not only spatial information but also angular information of a scene. This captured information is known as a light field, which can be represented as a four-dimensional tuple LF ( px,py, lx, ly ), where px and py select the direction of arrival of the rays at the sensor and lx, ly are the spatial position of these rays. A plenoptic chamber is usually made up of an array of microlenses placed in front of a sensor.

This system is equivalent to capturing the scene from several points of view (the so-called plenoptic views); therefore, a plenoptic chamber can be considered a multi-view system. Another system that can capture a field of light can be made up of a multi-camera array. Consequently, information about the depths of different objects (ie the distance between the object itself and the camera) in the scene is implicitly captured in the light field. This capability of plenoptic cameras leads to a wide number of applications with respect to depth mapping and 3D imaging.

In 2012, Lytro introduced the first commercially available single-array plenoptic chamber to the international market and, three years later, the Lytro Illum chamber. Since then, no other light field cameras have entered the consumer electronics market. The first Lytro plenoptic camera had mechanical dimensions along the optical axis of 12 cm, and the Lytro Illum camera had an objective lens (like DSLR cameras) of more than 12 cm and an overall size of approximately 20 cm. The improved optics of the Lytro Illum camera, with a dedicated five lens group varifocal objective lens, allowed the Illum camera to outperform that of the first Lytro camera. Following these two forays into consumer cameras, Lytro moved into a very different market: the film market, producing extremely large cameras in in which the length of the optical system can be dozens of centimeters, with 755-megapixel sensors and extremely heavy solutions. These cameras are not handheld cameras, but professional film cameras that have to be supported by tripods or heavy mechanical structures.

In addition to Lytro, Raytrix has also brought to market various products based on light field technology, geared towards industrial applications. These cameras are large cameras with large objective lenses that ensure good depth estimation performance.

In conclusion, light field cameras have shown good performance in terms of 3D imaging and depth detection. However, plenoptic cameras have never been brought to the mobile device market because they are really difficult to miniaturize. US 9,647,150-B2 discloses a manufacturing method for miniaturized plenoptic sensors. However, as already explained, the smallest plenoptic camera released on the consumer electronics market is the 12 cm Lytro camera.

Performance in plenoptic cameras depends on key optical design factors, such as focal length and f-number, where a large focal length or small f-number can dramatically improve camera performance. Although it is easy to find small f-numbers in smartphone lenses, it is very difficult to design and manufacture large focal lengths to meet the design rules of the smartphone market, due to the very small thicknesses of the modules that place difficult constraints on the MTTL (Mechanical Total Track Length) of the cameras.

In addition, today's smartphone market tends to reduce the dimensions of mini cameras more and more with each generation, increasing the difficulty of designing large focal lengths. Therefore, there is a need to introduce light field technology to the smartphone market with a significant increase in focal length, while meeting the mechanical constraints in terms of smartphone size.

Definitions:

- Plenoptic camera: A device capable of capturing not only the spatial position but also the direction of arrival of the incident light rays.

- Multiview system: System capable of capturing a scene from several points of view. A plenoptic chamber can be considered a multi-view system. Stereoscopic and multi-stereoscopic cameras are also considered systems multiview.

- Light field: LF four-dimensional structure ( px,py, lx, ly ) that contains the light information captured by the pixels (px,py) under the microlenses (lx, ly) in a plenoptic camera.

- Depth: distance between the plane of an object point in a scene and the main plane of the camera, both planes are perpendicular to the optical axis. - Plenoptic view: two-dimensional image formed by taking a subset of the light field structure by choosing a certain value ( px,py), always the same (px,py) for each of the microlenses ( lx,ly).

- Microlens Array (MLA): Array of small lenses (microlenses).

- Depth map: two-dimensional image in which the calculated depth values of the object world are added as an additional value to each pixel (x,y) of the two-dimensional image, composing depth = f ( x,y).

- Disparity: Distance between two (or more) projections of an object point in a camera.

- Baseline: Difference between the position of two (or more) cameras in a stereoscopic (or multistereoscopic) configuration.

- Folded optics: An optical system in which the optical path is folded through reflective elements such as prisms or mirrors, so that the thickness of the system is changed to achieve a certain thickness specification.

- OTTL (Optical Total Track Length): length of the optical path followed by light from the point where it enters the optical system and to the point where it reaches the sensor.

- MTTL (Mechanical Total Track Length): total length of the device required to include the mechanical parts of the optical system.

- Prism or mirror: refers to the optical component used to reflect light at a certain angle, bending the optical path of light.

Summary of the invention

In order to introduce light field technology in the smartphone market, a new concept of plenoptic camera is presented with this document, where a prism or mirror or other reflective element is used to bend the optical path of the lens, allowing lenses with large focal lengths to be designed without increasing the thickness of the lens.

A first aspect of the present invention refers to a plenoptic camera for mobile devices that comprises a main lens, an array of microlenses, an image sensor and a first reflective element (preferably a prism or a mirror) configured to reflect the light rays captured by the plenoptic camera before reaching the image sensor, in order to bend the optical path of the captured light through the camera before hitting the image sensor.

In one embodiment, the first reflective element is arranged to receive the captured light rays before reaching the main lens. In another embodiment, the first reflective element is arranged to receive the light rays already focused by the main lens. When only one reflective element is used, the optical axis of the main lens is preferably parallel to the image sensor surface (in this way, the optical path is folded 90 degrees or any other arbitrary angle).

In another embodiment, the plenoptic chamber comprises one or more additional reflective elements (preferably prisms or mirrors) configured to reflect light rays reflected by the first reflective element before reaching the image sensor. Therefore, additional reflective elements are inserted between the first reflective element and the image sensor, in order to further bend the optical path and help reduce the physical dimensions of the plenoptic chamber in a given axis.

The main lens may comprise a plurality of lens elements. In particular, the main lens may comprise a first set and a second set of lens elements, each set comprising one or more concentric lens elements. The physical arrangement of both sets of lens elements may be such that the optical axis of the first set of lens elements is perpendicular to the optical axis of the second set of lens elements and parallel to the image sensor. In one embodiment, the first reflective element is disposed between the first and second set of lens elements. In another embodiment, the first reflective element is arranged to receive the captured light rays before reaching the main lens, and the plenoptic chamber comprises a second reflective element disposed between the first set and the second set of lens elements, wherein The second reflective element is configured to reflect light rays reflected by the first reflective element and already focused by the first set of lens elements, before reaching the image sensor.

Another aspect of the present invention refers to a camera module for mobile devices that includes the plenoptic camera described above. This camera module can be, for example, a separate part, directly integrated into a smartphone (for example, inserted into the smartphone or connected to the rear casing of the smartphone) by means of coupling and electrical contacts. In the camera module, the components of the plenoptic camera are arranged in such a way that the thickness of the camera module is less than 10 mm.

A further aspect of the present invention relates to a mobile device, preferably a smartphone, comprising the plenoptic camera or camera module described above. In the mobile device, the image sensor of the plenoptic camera can be arranged in such a way that the perpendicular line of the image sensor is parallel to the rear side of the mobile device. In this way, the light path of the light rays captured by the camera is folded by the first reflective element (and, optionally, by additional reflective elements), which makes it possible to reduce the thickness of the mobile device. In the mobile device, the components of the plenoptic chamber are preferably arranged such that the thickness of the mobile device is less than 10 mm.

Brief description of the figures

A series of drawings, which help to better understand the invention and which are expressly related to embodiments of said invention, presented as non-limiting examples thereof, are very briefly described below.

Figure 1A represents a schematic side view of a plenoptic camera system with an image sensor, a microlens array and a field lens, according to the prior art. Figure 1B represents, in a front view, the microimages produced by the microlenses on the image sensor. Figure 1C shows the pixels that form a microimage of the image sensor.

Figure 2 illustrates the disparity between two projections of the same object point through two cameras separated from each other by a baseline b.

Figure 3 shows the error in depth calculations versus the actual distance of objects in the object world for different focal lengths in a plenoptic camera.

Figure 4 shows a typical camera module for smartphones. Figure 5A represents a plenoptic chamber according to the prior art, with a pure (unfolded) plenoptic configuration. Figures 5B and 5C show a plenoptic chamber according to two different embodiments of the present invention, with folded optics configuration.

Figures 6A-6D show four different plenoptic chamber embodiments in accordance with the present invention.

Figure 7 shows a schematic example of a plenoptic camera according to the present invention, installed inside a smartphone.

Figures 8A-8D show four other embodiments of plenoptic chamber devices with folded optics configurations.

Figure 9 shows an image sensor with its appropriate image circle. Figure 10A shows a 3D view of a plenoptic chamber with folded optics configuration. Figure 10B shows a 3D view of a plenoptic chamber with folded optics configuration where the lenses have been cut to reduce the thickness of the device in the Z axis.

Detailed description

Conventional cameras capture two-dimensional spatial information from light rays captured by the sensor. In addition, color information can also be captured using so-called Bayer pattern sensors or other color sensors. However, no information about the direction of arrival of lightning is recorded by a conventional camera. Plenoptic cameras have the ability to record 3D information about different objects. Basically, a plenoptic camera is equivalent to capturing the scene from multiple viewpoints (so-called plenoptic views that act as multiple cameras distributed around the equivalent aperture of the plenoptic camera).

Typically, a plenoptic camera 100 (see Figure 1A ) is made by placing an array of microlenses 104 between the main lens 102 and image sensor 108. Each of the microlenses 106 ( lx,ly ) is forming a small image. , known as a microimage (110a, 110b), of the main aperture on image sensor 108 (see Figures 1B and 1C ), such that each pixel ( px,py) of any microimage (110a, 110b) is capturing light rays 101 coming from a different part of the main aperture, each of the microimages below any microlens is an image of the main lens aperture, and each pixel at position px1,py1 to pxn,pyn in each microlens 106 integrates light coming from a given part of the aperture ( axn,ayn ) regardless of the position of the microlens. Light passing through the aperture at position ( axn,ayn) from different locations in the object world will collide with different microlenses but will always be integrated by the pixel ( pxn,pyn) under each camera microlens. Consequently, the coordinates ( px,py) of a pixel within a microimage determine the arrival direction of the captured rays at a given microlens and ( lx,ly) determine the two-dimensional spatial position. All this information is known as a light field and can be represented by a four-dimensional LF matrix ( px,py,lx,ly) or a five-dimensional LF matrix ( px,py, lx, ly, c ) if the information is considered colored (c).

As mentioned above, in some key respects a plenoptic camera behaves like a multi-stereo camera (since both are multi-view systems) with a reduced baseline between views. That is, multi-stereoscopic systems can also record the light field. The behavior of stereoscopic and multistereoscopic cameras has been extensively studied. Articles such as "Quantization Error in Stereo Imaging" [Rodríguez, JJ, and Aggarwal, JK Quantization Error in Stereo Imaging. In Computer Vision and Pattern Recognition, 1988. Proceedings CVPR'88, Computer Society Conference on (pages 153-158). IEEE] show how long focal lengths improve depth error estimation at relatively long distances in multi-view systems.

The depth estimation of a stereoscopic camera follows the equation:

where z is the depth point of interest, b is the baseline, f is the focal length of the cameras (if both cameras have the same focal length), and d is the disparity. The disparity d represents the difference in the position of two projections (or more projections in the case of a multistereoscopic system) of the same point in the world of objects, in the two (or more) cameras of a stereoscopic (multistereoscopic) system, As an example , figure 2 shows two cameras separated from each other by a baseline b , and how, when the light coming from point P in the world of objects passes through the two equivalent lenses c1 and c2 of the two cameras and reaches the sensors s1 and s2 of the two cameras at two different sensor positions, the disparity d is the distance between the two images pl and p 2 of the same point P on the two sensors s1 and s2.

From the above equation, the depth estimation error can be calculated as:

where Az represents the absolute depth error, and Ad represents the absolute disparity error.

A plenoptic chamber follows the same equation for the error produced in depth calculations. In this case, the baseline corresponds to the aperture size of the optical system ( ).

where f# = f /D (that is, the number f).

Therefore, the depth error Az produced in a plenoptic camera can be reduced by increasing the focal length f of the optical system while maintaining the f- number , reducing the f-number while maintaining the focal length f (i.e. , increasing D), or decreasing the f-number while increasing the f-focal length. Mobile phone lenses are commonly designed with small f-numbers and small focal lengths (due to the restrictive thickness requirements of the mobile phone industry). Starting from a commercially available design of a smartphone lens, which has a small f-number and a small focal length, Figure 3 shows how the depth estimation error decreases quadratically with increasing focal length when holding the number f. The error produced by the focal length ( fi), which is a small focal length commonly found in the mobile phone industry, is four times greater than the error produced by the focal length f ² ( f ² = 2fi), and nine times greater than the error produced by f ³ f = 3fi).

However, increasing the focal length usually means increasing the OTTL (Optical Total Track Length) of an optical system. Even if it depends on the particular optical design, the relationship between focal length and OTTL roughly follows the expression 1.1 ^OTTL < 1.3 in deployed configurations, therefore a

Increasing focal length implies an almost proportional increase in OTTL to keep the f-number constant, and therefore an increase in MTTL (Mechanical Total Track Length), which makes the camera module (like the module of camera 400 for smartphones depicted in figure 4 ) is thicker (ie, a large Sz).

Figure 4 shows a schematic diagram of a typical camera module 400 for mobile devices, such as smartphones, to be illustrative but not limiting. Important dimensions (Sx x Sy x Sz) have been highlighted. Typical dimensions of camera modules used in the mobile phone industry are as follows: 4 mm < Sz < 6.5 mm; 8mm < Sy < 10mm; 8 mm < Sz < 10 mm, where Sx, Sy, and Sz correspond to the width, height, and thickness of the chamber module 400, respectively (according to the X, Y, and Z axes of Figure 7).

The most critical dimension is Sz, which coincides with the MTz (Mechanical Track in z). This size Sz of the chamber module 400 must be smaller than the thickness Tz of the mobile device, as shown in Figure 7, and mobile phone manufacturers tend to move to smaller thicknesses with each new generation of phones. This means that cameras need to follow these trends if the object is to fit them into the mobile device. Camera modules with Sz thicknesses greater than 10mm would be resoundingly rejected by the market, which targets cameras with Sz that is close to 5 and 4mm.

Today's smartphone market trends call for reduced Sz thickness for mini cameras, forcing vendors to design lenses with greatly reduced f focal lengths to meet customer specifications. Miniaturized plenoptic cameras (such as those disclosed in US9647150B2), even if never commercially released by anyone else with a form factor similar to Figure 4, can have greatly improved performance if the focal length f is increased to values not commonly seen in conventional imaging lenses in the mini-camera industry. Therefore, it is imperative to increase the focal length of a specific plenoptic system without violating the design rules of the smartphone market (which require very small thicknesses) to improve depth error accuracy and take the mini plenoptic camera to the next level. depth/3D cameras for handheld devices.

A first approach to increasing the focal length f is to scale all components of the optical system, increasing all dimensions while maintaining the f number. This involves changing the main lenses, changing the microlenses and the sensor itself, in such a way that an increase in the dimensions of the OTTL and MTTL is also forced, safely exceeding the requirements of the smartphone market in terms of small thicknesses (Sz).

A second approach to increasing the focal length f could be to scale the main lens but keep the size of the sensor and microlenses. The focal length f of the plenoptic camera would increase but, because the size of the microlenses and the sensor are kept the same, the FOV (field of view) would be reduced due to the fact that the sensor is no longer capturing the full FOV of the optical system, but only a subset. And what is worse, in this case the OTTL and MTTL would also be increased, leading to an increase in the length of the main lens and making it difficult to use in mobile phone applications.

These approaches to increase the focal length f allow to improve the error in the depth calculations (these move the design point towards the lower curves in Figure 3), making the camera more accurate, with lower error rates for estimating depths of objects located further from the camera (ie, with longer distances in Figure 3). However, the resulting OTTL and MTTL are increased and do not fit the restrictive thickness specifications of current smartphone mini-cameras, or in other words, a module like in Figure 4 would have too large a thickness Sz to fit within a Mobile phone.

In this context, a prism or mirror is used in the present invention to fold the optical path of light, increasing the OTTL without increasing the thickness Sz of the camera module 400. Therefore, a plenoptic device is presented herein. novel with folded optic configurations.

Figures 5A-5C show various embodiments of a plenoptic chamber, showing the benefits of folded devices in terms of thickness. In all of these embodiments, the primary lens 102 is formed from a single lens element, or a pair or group of cemented lens elements. In these figures, the term OT refers to the optical track length and MT refers to the mechanical track length. The mechanical track length in the Z axis (MTz) represented in figure 7 is the critical dimension to consider when embedding the camera in a mobile phone because it corresponds to the thickness Tz of the device (or, in other words, making that the thickness Sz is as small as possible in the chamber module 400 of figure 4). The three embodiments of Figures 5A-5C have the same optical performance in terms of focal length f and f number but a different MTz.

Figure 5A depicts a typical plenoptic chamber 500a according to the prior art. The setup of this 500a plenoptic camera is designed with a small f-number and a large f- focal length in order to obtain good depth error accuracy. The optical axis 502 of the main lens 102 is perpendicular to the image sensor 108, through the center of the image sensor 108 (ie, the normal line 504 of the image sensor 108 at its center point is coincident with the optical axis 502). . However, this configuration has a large OTTLa = OTza, which implies a large MTTLa = MTza that does not fit within the typical dimensions of a smartphone.

Figure 5B shows a plenoptic chamber 500b in accordance with one embodiment of the present invention. The plenoptic camera 500b depicted in Figure 5B uses a folded optic that reduces the MTz while maintaining the same focal length (OTTL and f-number remain the same as in Figure 5A). In In this configuration, the optical path is bent using a reflective surface of a first reflective element 510, such as a prism or a mirror, thus the OTTLb has two components, OTzb and OTxb, but the OTTLb is the same as that used in the Figure 5A ( OTTL b = OTTL a = OTz a = OTz b + OTx b ). In the configuration shown in Fig. 5B, the optical axis 502 of the main lens 502 is parallel to the image sensor 108 (ie, the optical axis 502 and the normal line 504 of the image sensor are perpendicular).

Unlike the previous configuration, however, the MTz thickness of the camera module has been reduced enough to fit within the low thickness requirements of mini-camera specifications, while retaining the benefits of long focal lengths. for plenoptic chamber systems. Or, in other words, the plenoptic chambers 500a and 500b in Figures 5A and 5B offer the same optical performance and the same f-number; however, the thickness of the plenoptic chamber 500a in Figure 5A is greater than the thickness of the plenoptic chamber 500b in Figure 5B (MTza > MTzb) or, if implemented in a module as in Figure 4, the thickness Sz would be less for the embodiment shown in Figure 5B.

Figure 5C depicts a plenoptic chamber 500c in accordance with another embodiment of the present invention. This plenoptic chamber 500c has a configuration in which two reflective elements, a first reflective element 510 and a second reflective element 512, have been introduced to bend the optical path. The second reflective element 512 (such as a prism or a mirror) reflects light rays that have already been reflected by the first reflective element 510. Additional reflective elements may be used (eg, a third reflective element, a fourth reflective element , etc.) to further reflect the light rays reflected by the above reflective elements located along the optical path. The OTTL in Figure 5C has three components, OTz ¹ c, OTxc and OTz ² c, where their sum coincides with OTTLa ( OTTL C = OTTL a = OTz lc + OTx c + OTz 2c ) of Figure 5A, such that so that the focal length remains constant and the MTz has been drastically reduced (MTzc < MTzb < MTza). In the configuration shown in Figure 5C, the optical axis 502 of the main lens 102 and the normal line of the image sensor 108 at its center point are parallel but not coincident (ie, they are located at different heights), due to that the optical path has been folded twice along the way.

Figures 6A-6D show various embodiments of plenoptic chamber devices (600a, 600b, 600c, 600d) with folded optics configuration, in order to to be illustrative, but not limiting, wherein the primary lens 102 is comprised of a plurality of uncemented lens groups or lens elements. The plenoptic camera devices shown in this figure consist of an image sensor 108, a microlens array 104, an infrared filter 612 (an optional element that may not be present), and a main lens 102 composed of four or five elements. of lens, but this could be composed of fewer or more lens elements.

Each configuration shows a different MTz (the mechanical track length in the Z axis corresponding to the thickness Tz of the mobile device), as represented in figure 7. Each figure represents the X, Y and Z axes corresponding to those shown in the figure 7, according to the installation of the plenoptic camera in the mobile device (in figures 6A-6C, the image sensor 108 extends along the Z axis, while, in the embodiment of figure 6D, the sensor image 108 extends along the X axis). In all cases, the introduction of a first reflective element 510 (preferably a prism or mirror) that folds the light path reduces the MTz relative to the original unfolded configuration. As can be seen in Figures 6A-6D, in all cases, MTz < OTTL and, of course, MTz < MTTL (considering the original unfolded configuration to calculate the MTTL).

In the first configuration, shown in Figure 6A, the first reflective element 510, such as a prism or mirror positioned at 45 degrees to the optical axis, reflects the light rays 601a captured by the plenoptic chamber 600a just before passing through any optical surface, that is, before reaching any of the lens elements (620, 622, 624, 626, 628) of the main lens 102. In the example of Figure 6A, light rays 601b reflected from the first element 510 (and forming a certain angle with respect to the captured light rays 601a) reach the main lens 102. It is very clear from Fig. 6A that MTza < OTTLa, which in practical terms means that the thickness Sz of the modulus of camera 400 (figure 4) is smaller and easier to fit within the strict requirements of a mobile phone.

In the second configuration, shown in Figure 6B, the main lens 102 comprises a first set (630, 632) and a second set (634, 636) of lens elements. The plenoptic camera 600b of Figure 6B bends the captured light rays 601a after they have passed through the first set of lens elements (the first two lenses 630 and 632) of the main lens 102 (in this case, an achromatic doublet). ) with the aid of a first reflective element 510, a prism or mirror, positioned at 45 degrees to the optical axes of both sets of lens elements. In this case, MTzb = MTza and, in both cases, is limited by the sensor chip dimensions (Dx in Figures 6A-6C). However, due to packaging reasons and/or due to optical design, it might be better to fold the light before or after passing through various optical surfaces.

The third configuration (FIG. 6C) shows a primary lens 102 made up of five lens elements divided into a first set (640, 642, 644) and a second set (646, 648) of lens elements. The captured light rays 601a are reflected after passing through the first set of lens elements (the first three lens elements 640, 642, 644), obtaining the reflected light rays 601 b that fall on the second set (646, 648 ) of lens elements and image sensor 108. Again, MTzc < OTTL = OTzc OTxc.

Figure 6D shows a fourth configuration in which, in addition to the first reflective element 510, a second reflective element 512 (eg a prism or mirror) is used to reduce the thickness MTz (MTz < OTTL = OTxd OTzd). In this case, the sensor extends along the x dimension and therefore its chip dimension does not limit the MTz. In this embodiment, the main lens 102 is made up of four lens elements divided into a first set (650, 652) and a second set (654, 656) of lens elements. The first reflective element 510 is arranged to receive the captured light rays 601a before they reach the main lens 102, to obtain reflected light rays 601b. The second reflective element 512 is arranged between both sets of lens elements and reflects the reflected light rays 601b to obtain additional reflected light rays 601c which fall on the second set (654, 656) of lens elements and the image sensor. 108.

As explained in Figures 5A-5C and 6A-6D above, folded optics allow the thickness (MTz, or Sz in Figure 4 and Tz in Figure 7) of cameras with large focal lengths to be reduced which would commonly lead to smaller dimensions. Large Sz (high focal lengths can fit into really thin modules with low MTz or Sz, as shown in Figure 6D for example). As already stated, in all cases of Figures 5B-5C and 6A-6D, the thickness of the chamber is drastically reduced from its original thickness (the MTTL in the equivalent deployed configuration), allowing large chambers to fit into portable devices that, were it not for the use of folded optics technology, would never be able to meet the specifications of the telephone industry smart in terms of thickness.

The reduced thickness plenoptic camera proposed by the present invention is suitable for installation in any mobile device with strict thickness restrictions, such as a tablet, PDA or smart phone. Figure 7 shows an example of a plenoptic camera embedded in a smartphone 700 having a configuration similar to those depicted in the embodiments of Figures 6B and 6C. As shown in Fig. 7, the plenoptic camera is preferably installed on the back or rear side 710 of the smartphone 700, capturing images from behind the screen. Alternatively, the plenoptic camera can be installed on the front side of the smartphone 700, next to the screen, to capture frontal images. The smartphone 700 has the following dimensions in the X, Y and Z axes shown in Fig. 7: a width Tx, a length Ty and a thickness Tz, respectively. This example is intended to be illustrative but not limiting. In this example, the main lens 102 of the plenoptic chamber is made up of five lens elements (730, 732, 734, 736, 738), and a first reflective element 510 (a prism or mirror) reflects the light after it has passed. has passed through a first set of lens elements (the first two lens elements 730 and 732) of the main lens 102, as in the embodiment of Figure 6B. The other three lens elements (734, 736, 738), which make up the second set of lens elements, microlens array 104 and image sensor 108, are distributed along the X axis, not contributing to the MTz thickness. of the chamber (the critical size Sz in Figure 4). Instead, these elements could be distributed along the Y axis, or in any arrangement such that normal line 504 of image sensor 108 and the optical axis of the second set of lens elements are parallel to the XY plane.

As a conclusion, this new proposed folded optics technique allows to have, at the same time, superior plenoptic performance (with long focal lengths and small f number) and a small Mtz (or thickness Sz of the camera module 400), being ideal for its integration into portable devices such as smartphones, tablets, laptops, etc.

Figures 8A-8D show four more embodiments of plenoptic chamber devices (800a, 800b, 800c, 800d) with folded optics configurations where full folded plenoptic designs (including first reflective element 510) have been further described for the purpose of to be illustrative but never limiting.

In these four embodiments, a first prismatic lens 802 is used as the first reflective element 510 (and, optionally, an additional (second, third...) prismatic lens 812 is used as the second (third, fourth...) element. reflective 512). In this embodiment, the prismatic lens 802 (and, optionally, any additional second, third, etc., prismatic lens(es) 812 used) is essentially a three-surface optical element in which the middle surface is reflective (for example , a body or a prism in which all three surfaces can be aspheres instead of flat surfaces) and is made of glass or plastic. Two of the surfaces of the prismatic lens 802/812 (a first surface 804a/814a and a third surface 804c/814c) are refractive surfaces, and a second middle surface 804b/814b is a reflective surface. Therefore, the prismatic lens 802/812 is an optical element that integrates a lens element of the main lens (formed by the two refractive surfaces 804a/814a and 804c/814c) together with the reflective element 510/512 (formed by the reflective surface 804b/814b) that folds the light path. Light rays passing through first surface 804a/814a of prismatic lens 802/812 have a different optical axis (usually perpendicular) to rays passing through third surface 804c/814c due to reflection from second surface 804b /814b of the 802/812 prismatic lens. The two refractive surfaces (804a/814a, 804c/814c) can have convex, concave or aspherical shapes, and the reflective surface can be flat or convex/concave, spherical or aspherical.

The use of prismatic lenses allows the light path to be folded, achieving long OTTL optical total path lengths (and therefore long effective focal length EFFL) within small thicknesses. Also, integrating the 802/812 prismatic lens together with the other lens elements of the optical system is easier than using, for example, a single mirror, where alignment tasks are certainly more difficult. The fact of having a prismatic lens 802/812 with its two refractive surfaces (804a/814a, 804c/814c) well defined and, therefore, its optical axes well defined, facilitates the alignment processes.

Various prismatic lens options and implementations have been integrated into the different embodiments of Figures 8A-8D and will be detailed below.

Figure 8A shows a primary lens containing five lens elements (802, 822, 824, 826, 828), an optional infrared filter 612, a microlens array 104, and an image sensor 108. The first lens element is a prismatic lens 802 that integrates, within a single entity, a convex lens (a first 804a and a third 804c surface) and the first reflective element 510 (a second surface 804b). The first surface 804a has a convex shape (but this could also be concave or flat), the second surface 804b is a flat surface (but this could be any other non-flat surface) at 45 degrees (but there could be other angles) in relation to with the optical axis, and this flat surface reflects light towards the third surface 804c, a second convex surface (but this could be concave or flat). The first convex surface 804a of the prismatic lens 802 refracts the light rays 801a captured by the plenoptic camera 800a, then these rays are reflected by the first reflective element 510 (the flat surface 804b) of the prismatic lens 802 for a total of 90 degrees along the optical axis (but this could be different if the first reflective element 510 is not tilted 45 degrees relative to the optical axis). The first optical axis extends along the Z axis and, after the reflective element 510, the light follows a second optical axis (X axis), which is perpendicular to the first, reaching the third surface 804c of the prismatic lens 802. The light then passes through the other lens elements (822, 824, 826, 828), reaching the microlens array 104 and finally the image sensor 108.

The embodiment of Figure 8A has a total optical track length OTTLa (OTza ¹ + OTxa ¹ ) of 11.2 mm (but this could be longer or shorter); however, using the 802 prismatic lens to fold the light path allows the majority of the camera's OTTL to be extended along the X axis, leading to a MTza thickness of about 5.1mm (but this could be even shorter, or longer), which makes this lens with a very large OTTL suitable for integration into a mobile phone thanks to its reduced thickness. In this case, the thickness (Z axis) is limited by the size of the sensor. Note that the extreme field rays depicted in Figures 8A-D refer to the field across the sensor diagonal (although these lie in the XZ plane), leading to MTzs (MTza, MTzb, MTzc, MTzd), which refer to to the diagonal of the image sensor 108. This is common practice in optical design, because lenses are usually rotationally symmetrical, but the sensor is rectangular. This is the reason why the last lens diameter coincides with the height of the last ray passing through that last surface. This folded design allows the focal length to be extended up to 9.3mm (in this example, but this could be longer), dramatically improving depth-sensing performance while still meeting the very thin thickness requirements of mobile phones (MTza de 5.1mm).

The example of the lens in Figure 8A should not be construed as a limiting option, but only as an illustrative example of how a folded design can achieve a long focal length and excellent depth performance with a small thickness. The main lens 102 of the camera may have more or fewer lens elements in order to improve optical performance (in this example, five lens elements, but it could be fewer or more lenses), the non-reflective surfaces of the prismatic lens 802 may be formed by convex surfaces, flat surfaces, concave surfaces, or any aspherical surface that the designer may deem appropriate. Likewise, the reflective element 510 (which is, in this case, a flat surface inclined 45 degrees with respect to the optical axis) could be a convex or a concave reflective surface (or a flat surface inclined any other angle in relation to the optical axis). Prismatic lens 802 can be positioned as a first lens element, or as a rear lens element, after one or more lens elements (eg, as a second lens element).

Figure 8B shows an embodiment in which two prismatic lenses (a first prismatic lens 802 and a second prismatic lens 812) have been used to fold the light path twice. In this case, the size of the image sensor 108 will not be limiting the thickness of the lens because the image sensor 108 extends along the X and Y dimensions (it is no longer a limiting factor in the Z dimension: the sensor rectangle extends along the XY dimensions of Figure 7 and the rectangular sensor chip is no longer imposed to be smaller than Tz in Figure 7, as is imposed by the embodiment in Figure 8A ). The main lens 102 is also made up of five elements (802, 832, 834, 836, 812), an optional infrared filter 612, the microlens array 104, and the image sensor 108. The first four lens elements (802, 832, 834, 836) are similar to the embodiment described in Figure 8A. The fifth lens element is a second prismatic lens 812 with two refractive surfaces (first 814a and third 814c surfaces) that have aspherical shapes, and a second reflective surface (in the example, a flat surface at 45 degrees) that acts as a second. reflective element 512. In this case, the total optical OTTL track length of the lens (OTzb ¹ + OTxb ¹ + OTzb ² ) is approximately 12.9 mm (but this could be longer or shorter). The use of a folded optic makes it possible to have a MTzb thickness of only a few millimeters (approximately 5.0 mm in the embodiment). In addition, the use of a second reflective element 512 allows the EFFL effective focal length to be further increased (up to 13.2mm in the example vs. 9.3mm in the Figure 8A embodiment), drastically improving depth sensing performance. of the plenoptic chamber.

Again, the embodiment of Figure 8B should not be construed as limiting, but just as an example. The lens can be made up of fewer or more elements than the five elements in the example, and fewer or more than two reflective elements (510, 512), which can be either prisms or mirrors, or a combination of prisms and mirrors. . The reflective element (510) and any other additional reflective elements (512) can reflect the incoming light rays at an angle of 90 degrees or at any other arbitrary angle. The reflective elements can be arranged (i.e., tilted) such that the angle of incidence (and therefore the corresponding angle of reflection) of the incident light rays can be any angle in the range (0- 90), and preferably within the interval [22.5 67.5]. In some embodiments, the angle of reflection is preferably 45°, thereby obtaining an optical path that bends 90 degrees.

Figure 8C shows an embodiment in which two prismatic lenses (802, 812) have been integrated into a four-element lens, such that two lens elements of all lens elements (802, 842, 812, 844) of the main lens 102 are prismatic lenses (802, 812). Also, camera 800c comprises an optional infrared filter 612, microlens array 104, and image sensor 108. First prismatic lens 802 is similar to that of Figure 8A. However, the second prismatic lens 812 is formed by a planar-concave lens like the first 814a and third 814c surfaces, and the second reflective surface 814b is a flat surface at 45 degrees in front of the optical axis. The second prismatic lens 812 is located between two regular aspherical lenses (842, 844). In this case, the primary lens has an OTTL total optical track length (OTzc ¹ + OTxc ¹ + OTzc ² ) of 12.0mm with an EFFL effective focal length of 10.4mm, and the MTzc thickness is 5, 7mm. In this case, the MTz thickness is limited by the size of the prismatic lenses (802, 812) and the thickness of the last regular lens element 844. If the priority is to reduce the MTz thickness as much as possible, the best solution is, clearly, the use of prismatic lenses as the first and/or last lens element.

Figure 8D shows another embodiment of a plenoptic chamber with a folded optics configuration, where two prismatic lenses (802, 812) have been used in a main lens 102 composed of five lens elements (802, 852, 854, 856, 812). . In this case, the first prismatic lens 802 is similar to that of Fig. 8A; however, a small concavity has been introduced into the reflective element 510 (so small that it cannot be seen in the schematic diagram of Figure 8D). The second prismatic lens 812 integrates an aspherical lens and a concave reflective surface 814b (instead of flat as in embodiments 8A to 8C). The inclusion of surfaces non-planar reflectors (804b, 814b) complicates design but has manufacturing advantages. The primary lens has an OTTL optical total path length (OTZd ¹ + OTXd ¹ + OTZd ² ) of 14mm, with an EFFL effective focal length of 12.4mm. In this embodiment, the thickness MTzd of the lens is 6.2 mm.

The use of one or more prismatic lenses (802, 812) makes it possible to have a large OTTL optical total track length with small MTzs thicknesses. In all cases, the MTz thickness is less than 6.5 mm and therefore it can be integrated into a modern mobile phone, where the thickness never exceeds, as usual practice, 7.5 mm for the rear cameras. and 5mm for the front cameras.

In addition to the prismatic lens technique or any other folded optics technique as described above to reduce the MTz thickness of the device, other strategies could also be used to reduce the thickness of the device. As explained above, the lenses commonly have rotational symmetry, while the image sensors 108 are not circular (they are rectangular). This means that the lens has to be optimized to exhibit good optical performance across the entire diagonal of the image sensor 108 to ensure good performance across the entire sensor, but some of the optimized field is wasted due to the image sensor 108 shape (photons of light colliding within the dotted circle in Fig. 9 but not within the rectangle of the active area of the sensor are not used, the photons are not converted into electrons). Figure 9 shows an image sensor 108 with side sizes ISz x ISx and its corresponding image circle. The diameter of the image circle is set by the diagonal of the active area of the sensor; however, because the sensor is rectangular, a non-negligible part of the optimized field is wasted and therefore a non-negligible part of the lens area is not being used for useful light colliding with the active area of the sensor. sensor.

Figure 10A shows a plenoptic camera 1000a where, as in any normal camera, the rotational symmetry of the lenses produces an image circle that collides with the microlens array (104) and finally collides with the image sensor (108). which, in Figure 10A, is like the circle in Figure 9 but, in fact, the image sensor 108 and the microlens array 104 are rectangles as in Figures 9 and 10B, and the light colliding inside the circle but outside of the image sensor rectangle 108 plays no useful role. Figure 10A shows a folded plenoptic chamber with five lens elements (1002, 1004, 1006, 1008, 1010), in which four different rays reaching the four corners of the image sensor 108, limiting the field of view (FOV) of the plenoptic camera 1000a (light at the very center of the FOV reaching the center of the image sensor 108 is also shown in Fig. 10A).

The light reaching the surface of the microlens array 104 in Figure 10A outside the rectangle formed by the four FOV bounding points is of no use. Only the active area of the image sensor 108 (where photons are converted to electrons) and the microlens array 104 are shown in Figure 10B. Lenses 1008 and 1010 in Figure 10A have also been truncated in Figure 10B ( truncated lenses 1018 and 1020), removing the part of those lenses that would transmit light into the circle but outside the rectangle in Figure 9 (ie, outside the active area of the sensor).

The net result is that plenoptic chambers 1000a and 1000b are functionally identical but, in chamber 1000a, the MTTL (the thickness MTza) is set by the outer circle of lenses 1008 and 1010 (or by the outer circle in Figure 9 ) while, in camera 1000b, the MTTL (the thickness MTzb) is fixed by lens 1012 (exactly the same as lens 1002 in camera 1000a, larger in z-dimension than truncated lenses 1018 and 1020).

In summary, and as shown in the embodiments of Figures 8A-8D, there are a large number of degrees of freedom for the design of the primary lens. The refractive surfaces (804a, 804c; 814a, 814c) of the prismatic lens may be concave, convex, planar, aspherical, or any combination thereof. The reflective surface (804b; 814b) can be flat or convex/concave, and these can have any degree of inclination (not necessarily 45 degrees in front of the optical axis as shown in most figures). Prismatic lenses can be positioned as the first lens element of the main lens, as the last lens element of the main lens, or between regular element lenses of the main lens, depending on particular design needs. The number of prismatic lenses can also be variable (one, two or more prismatic lenses can be used). And, in addition, the lenses can be cut to reduce the thickness.

The embodiments shown in the figures of this document are only examples that should not be interpreted as a limiting characteristic, the scope of the invention should only be drawn from the claims, since there is an unlimited number of possible embodiments that cannot be covered. with examples but they become obvious to a person skilled in the art after having access to the present invention. For example, at all changes in the direction of light propagation on the reflective surfaces in Figures 5B, 5C, 6A, 6B, 6C, 6D, 7, 8A, 8B, 8C, 8D, 10A and 10B, the incident and reflected rays are perpendicular; however, the practical design with different incident and reflected angles might be desirable for some applications. For example, in some silicon sensors, the photosensitive active area is not perfectly centered within the chip area and, for example, in the plenoptic camera 1000b, it might be desirable to move the silicon sensor 108 a little to the right or to the right. left, that could be done by building the reflective surface on lens 1012 at angles slightly above or below 45 degrees to the optical axis of the first surface of lens 1012 (obviously, in this case, the sensor would not be perfectly parallel or perpendicular to the external structure of the mobile phone, but a miniaturization problem would be solved).

Claims (20)

1.

- A plenoptic camera, characterized in that it comprises a single main lens (102), a single microlens array (104) and a single image sensor (108), wherein the plenoptic camera further comprises a first reflective element (510) configured to reflecting light rays (601a) captured by the plenoptic chamber before reaching the image sensor (108); and wherein the primary lens (102) comprises a plurality of uncemented lens elements. 2.

plenoptic camera of claim 1, comprising a three-sided optical element (802) having two refractive surfaces (804a, 804c) that form a lens element of the primary lens (102) and a reflective surface (804b) that forms the first reflective element (510). 3. - The plenoptic chamber of claim 2, characterized in that the three-sided optical element (802) is made of glass or plastic. 4. - The plenoptic chamber of any of claims 2 to 3, characterized in that the refractive surfaces (804a, 804c) of the three-sided optical element (802) are flat surfaces, convex surfaces, concave surfaces, aspherical surfaces or a combination from the same. 5. - The plenoptic chamber of any of claims 2 to 4, characterized in that the reflective surface (804b) of the three-sided optical element (802) is a flat surface, a convex surface or a concave surface. 6.

plenoptic chamber of claim 1, characterized in that the first reflective element (510) is a prism. 7.

plenoptic chamber of claim 1, characterized in that the first reflective element (510) is a mirror. 8. - The plenoptic chamber of any of claims 1 to 7, characterized in that the first reflective element (510) is arranged to receive the captured light rays (601a) before reaching the main lens (102). 9.

plenoptic camera of any of claims 1 to 7, characterized in that the first reflective element (510) is arranged to receive light rays (601a) after being refracted by at least one lens element of the main lens (102). . 10.

plenoptic camera of any preceding claim, characterized in that the optical axis (502) of the first lens element of the main lens (102) is parallel to the image sensor (108). eleven.

plenoptic camera of any preceding claim, comprising at least one additional reflective element (512) configured to reflect light rays (601b) reflected by the first reflective element (510) before reaching the image sensor (108). 12.

plenoptic camera of claim 11, comprising at least one additional three-sided optical element (812) having two refractive surfaces (814a, 814c) forming a lens element of the main lens (102) and a reflective surface (814b). ) that forms the at least one additional reflective element (512). 13.

plenoptic chamber of claim 12, characterized in that the at least one additional three-sided optical element (812) is made of glass or plastic. The plenoptic chamber of any of claims 12 to 13, characterized in that the refractive surfaces (814a, 814c) of the optical element (812) are planar surfaces, convex surfaces, concave surfaces, aspherical surfaces, or a combination thereof. fifteen.

plenoptic chamber of any of claims 12 to 14, characterized in that the reflective surface (814b) of the optical element (812) is a flat surface, a convex surface or a concave surface. 16.

plenoptic chamber of claim 11, characterized in that the at least one additional reflective element (512) is a prism or a mirror. 17.

plenoptic camera of any previous claim, characterized in that at least one of the lens elements of the main lens (102) is non-rotationally symmetric, such that the field of said at least one lens element is adapted to the dimensions ( ISz, ISx) of the image sensor (108). 18.

plenoptic camera of any one of claims 1 to 16, characterized in that the main lens (102) comprises a first set and a second set of lens elements, each set comprising one or more concentric lens elements; and characterized in that the optical axis of the first set of lens elements is perpendicular to the optical axis of the second set of lens elements and parallel to the image sensor (108). 19.

plenoptic camera of claim 18, characterized in that the first reflective element (510) is arranged between the first and the second set of lens elements. twenty.

plenoptic camera of claim 18, characterized in that the first reflective element (510) is arranged to receive the captured light rays (601a) before reaching the main lens (102); characterized in that the plenoptic chamber comprises a second reflective element (512) disposed between the first set (650, 652) and the second set (654, 656) of lens elements, the second reflective element (512) being configured to reflect the light rays (601b) reflected by the first reflective element (510) and already refracted by the first set (650, 652) of lens elements, before reaching the image sensor (108).