JP2019164144A - Three-dimensional depth mapping using dynamic structured light - Google Patents

Abstract

To provide an improved apparatus for generating a dynamic structured light pattern for optical tracking in a three-dimensional space.SOLUTION: The apparatus comprises an array of lasers, such as a VCSEL laser array, to project light in a pattern into a three-dimensional space; and one or more optical elements arranged in cells. The cells are aligned with subsets of the laser array, and each cell individually applies a modulation, in particular an intensity modulation, to light from one or more lasers in the subset, to provide a distinguishable and separately controllable part of the dynamic structured light pattern. A method of generating a structured light pattern is disclosed, in which light is provided from an array of lasers, and light is individually projected from subsets of the array of lasers to provide differentiated parts of the structured light pattern.SELECTED DRAWING: Figure 1

Description

  The present invention relates to three-dimensional depth mapping using structured light, and more particularly, but not exclusively, to a system that performs tracking to provide input to a device such as a computer.

  Tracking to provide input to a computer or other computing device includes ubiquitous technologies such as a computer mouse. Tracking a stylus and finger within a three-dimensional field in front of a computer is also available, using various tracking techniques such as visual imaging, IR imaging, and ultrasound. The term “tracking” can refer to following the position and movement of an object and includes processing the input received at the tracking computer to calculate the position or movement. Thus, for a computer mouse, tracking can include processing the mouse output to calculate motion. For a visually followed object, the term tracking may include continuous frame image processing that captures the object.

  One method of imaging simply uses a camera to view and process the scene. The camera can follow specific marks placed in the scene, or the imaging system can look for clearly recognizable features such as fingers.

  Such visual imaging difficulties include the requirement that the three-dimensional region is well illuminated. Furthermore, the only features that can be tracked are those that are recognized in advance, and motion tracking combined with feature recognition may not give accurate results.

  To overcome the above problems, tracking with structured light has been introduced. A known pixel pattern is projected onto the space to be tracked. Depending on how the pattern deforms on the impact surface, the visual system can calculate the depth and surface information of the object in the scene. Typical patterns used are grids or horizontal bars. Various devices use structured light patterns to enable gesture recognition and 3D depth mapping. The structured light pattern transmitter includes a laser emitter and a diffractive optical element (DOE).

  Projecting narrowband light onto a three-dimensional surface produces a line of illumination that appears distorted from other perspectives than the line of illumination of the projector, resulting in an accurate geometric reproduction of the surface shape Can be used.

  A faster and more versatile method is to project a pattern of many stripes or arbitrary stripes at once, thereby allowing multiple samples to be acquired simultaneously. From a different viewpoint, the pattern appears to be geometrically distorted depending on the surface shape of the object.

  Although there can be many other variations of structured light projection, parallel stripe patterns are widely used. By changing the position of the stripes, any detail 3D coordinates about the surface of interest can be obtained accurately.

  One known method of generating a stripe pattern is laser interferometry that utilizes two wide planar laser beam fronts. Interference between the beam fronts results in a regular and equidistant line pattern. Various pattern sizes can be obtained by changing the angle between the beams. This method makes it possible to generate a very fine pattern accurately and easily without limitation of depth of field. Disadvantages are inherent in lasers such as high implementation costs, difficulty in providing the ideal geometry of the beam, and possible self-interference with speckle noise and the portion of the beam reflected from the object. Is the action. Furthermore, there is no means to modulate individual stripes, such as by Gray code.

  In particular, the disadvantage of using a single source emitter such as an edge emitting laser diode is that the light pattern produced by the single source emitter can only be controlled as a single building block. That is, the light pattern can be turned on completely, turned off completely, or dimmed, but cannot be changed dynamically.

  The structured light pattern can be constructed using invisible light such as IR light. Alternatively, the high frame rate can make the structured light insensitive to the user or avoid interference with other viewing objects of the computer.

  Surface emitting lasers, or VCSELs, are semiconductor laser diodes in which the emission of the laser beam is perpendicular to the top surface, as opposed to conventional edge emitting semiconductor lasers that emit light from the surface formed by cutting individual chips from the wafer. It is a kind.

  In contrast to edge emitting lasers, there are several advantages of making a VCSEL. The edge emitter cannot be inspected until the manufacturing process is complete. If the edge emitters do not function properly due to poor contact or poor material growth, manufacturing time and processing materials are wasted. A VCSEL can be inspected at several stages throughout the process for material quality and processing problems. For example, if the bias is not completely removed from the dielectric material during etching, a temporary inspection process can be used to determine that the top metal layer is not in contact with the first metal layer. In addition, because the VCSEL emits a beam perpendicular to the active region of the laser, tens of thousands of VCSELs can be processed simultaneously on a 76.2 mm (3 inch) gallium arsenide wafer. Further, the VCSEL manufacturing process requires more labor and materials, but the yield can be managed to be a more predictable result.

  There are significant advantages to using a VCSEL laser array in a structured optical system, because the structured optical transmitter can be made smaller by using such an array. Such a reduction is particularly important for embedding a transmitter in a size-constrained device such as a mobile phone or wearable device.

  However, despite the above advantages, VCSEL arrays are not currently used in structured optical scanning systems for several reasons. Many diffraction patterns require a coherent Gaussian shaped beam to create the dense pattern required for high resolution tracking. VCSEL arrays only provide a plurality of individual Gaussian beams that are located next to each other, usually with overlap between them. Multiple points and the overlap between them degrades detection performance in high density regions within the light pattern and limits the use of various diffractive design techniques that require a predetermined Gaussian beam. Such designs include top hat designs, homogeneous line generators, and other complex high performance structures.

  In fact, the problem with standard diffractive designs is that the entire VCSEL laser array is used as a single light source. Therefore, when using a multi-spot design, an array image is acquired for each spot instead of having a focused Gaussian beam. A diffractive design that requires a Gaussian beam as input does not provide the required output at all. The problem is more severe with high-density light patterns because they require the source beam to be focused on a small spot to separate the features, which cannot be done if the light source is a laser array. It is.

  This embodiment provides a VCSEL laser array in which the lasers of the array are modulated individually or in groups. Individual lasers or groups can be modulated statically or dynamically to provide and change the structured light pattern as needed.

  Thus, each laser in the array or group of lasers moderated together comprises its own optical element, typically a diffractive element. The diffractive elements can be individually controlled so that an overall structured light pattern can be selected for a given situation and / or the region of interest can be traced dynamically.

According to a first aspect of the invention,
A laser array configured to project patterned light into a three-dimensional space;
A plurality of optical elements, wherein each optical element defines a cell, the cell is aligned with an individual subset of the laser array, and the optical element of each cell converts light from the individual subset through the optical element. There is provided an apparatus for generating a structured light pattern comprising a plurality of optical elements that are individually modulated to provide a distinguishable portion of the structured light pattern.

In one embodiment, the light modulation is any of diffractive modulation, refractive modulation, and a combination of diffractive and refractive modulation.
In one embodiment, a subset of the optical elements and laser arrays that make up the individual cells are constructed from a single shaped body.

In one embodiment, the width of the cell is 1 mm or less.
In one embodiment, the width of the optical element is 1 mm or less.
In one embodiment, the cells can be individually controlled to change the diffractive modulation.

  In one embodiment, the cell is dynamically controllable to effect a change in the structured light pattern based on receiving and analyzing at least one capture frame, the frame comprising a plurality of pixels in a two-dimensional layout. Including.

In one embodiment, the cell is further controllable with respect to positioning within the structured light pattern and with respect to the shape applied to the light.
In one embodiment, the dynamic control can be configured to apply a high resolution of the structured light pattern to a part of the scene and apply a low resolution of the structured light pattern to another part of the scene.

In one embodiment, the dynamic change of the structured light pattern includes a change in the orientation of the structured light pattern.
In one embodiment, the dynamic change of the structured light pattern includes a change for the cell.

In one embodiment, any of a change, a change in intensity, a change in polarization, a change in filtering parameters, and a change in focus.
In one embodiment, the subset is either an individual laser, a laser pair, a laser triplet, a combination of lasers of different sizes, and a combination of dynamically changing lasers.

In one embodiment, light projected from individual subsets is tiled or overlapped.
In one embodiment, the laser array includes a VCSEL laser array.

In one embodiment, the laser array includes laser bars.
The device can be incorporated into any of a computer, laptop computer, mobile communication device, tablet device, game console, and motion capture device.

The device can be used to track a 3D scene.
According to a second aspect of the present invention, there is provided a method for generating a structured light pattern for three-dimensional tracking, the method comprising:
Providing light from a laser array;
Projecting light individually from a subset of the laser array to provide differentiated portions of the structured light pattern.

According to a third aspect of the invention,
A laser array configured to project patterned light into a three-dimensional space;
A plurality of optical element cells, wherein the cells are aligned with individual subsets of the laser array, each cell individually applying intensity modulation to light passing through the element from the individual subsets. An apparatus for generating a structured light pattern is provided that includes a plurality of optical element cells that provide differentiable portions.

  Unless defined otherwise, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention relates. Although methods and materials similar or equivalent to those described herein can be used in practicing or testing embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and not necessarily intended to be limiting.

  Implementations of the methods and / or systems of embodiments of the present invention may include performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to the actual apparatus and apparatus of the method and / or system embodiments of the present invention, some selected tasks are performed using the operating system by hardware, software, firmware, or combinations thereof. Can be done.

  For example, the hardware for performing selected tasks according to embodiments of the present invention can be implemented as a chip or a circuit. As software, selected tasks according to embodiments of the present invention can be implemented as a plurality of software instructions that are executed by a computer using any suitable operating system. In an exemplary embodiment of the invention, one or more tasks according to exemplary embodiments of the methods and / or systems described herein are performed by a data processor such as a computing platform for executing multiple instructions. Is done. Optionally, the data processor comprises volatile memory for storing instructions and / or data, and / or non-volatile storage for storing instructions and / or data, such as a magnetic hard disk and / or removable media. Including. Optionally, a network connection is also provided. A display and / or user input device such as a keyboard or mouse is optionally provided.

  Several embodiments of the invention are described herein by way of example only with reference to the accompanying drawings. DETAILED DESCRIPTION Reference will now be made in detail to the drawings, and it should be emphasized that the illustrated details are examples and serve to illustrate embodiments of the invention by way of example. In this regard, it will become apparent to those skilled in the art how to practice the embodiments of the present invention by understanding the description in conjunction with the drawings.

FIG. 2 is a simplified schematic diagram illustrating the basic settings of a structured light generator according to an embodiment of the present invention, where a cell of optical elements having individual patterns corresponds to a particular VCSEL in a VCSEL laser array. FIG. 2 is a simplified schematic diagram of the embodiment of FIG. 1 and illustrating the construction of a light pattern, each cell creating a partial tile of the overall structured light pattern. FIG. 2 is a schematic diagram showing a modification of the embodiment of FIG. 1, wherein each cell consists of a triplet of VCSEL lasers. FIG. 2 is a schematic diagram illustrating a variation of the embodiment of FIG. 1, where various light patterns are created by various cell patterns as described above, but the cells are of various sizes and orientations. FIG. 2 is a schematic diagram illustrating a variation of the embodiment of FIG. 1 in which various light patterns are created by various cell patterns as described above, and individual cells are not necessarily organized into separate tile structures. It bears various light features of the structured pattern. 1 is a schematic diagram illustrating a tracking system having a camera and a processor, in accordance with an embodiment of the present invention. FIG. 6 is a schematic diagram illustrating various modulations that may be applied by an optical element, according to embodiments of the invention. FIG. 6 is a schematic diagram showing beam placement or direction change by an optical element according to an embodiment of the present invention. Fig. 6 is a simplified drawing illustrating the use of the optical element of the present invention to focus a beam. 1 is a simplified diagram illustrating an optical element performing beam shaping according to an embodiment of the present invention. 1 is a simplified diagram illustrating a multi-stage optical element including focusing, beam directing, and beam shaping according to one embodiment of the present invention. a and b show the effect of using a stripe on the index finger and show how the orientation of the stripe may have to be changed if the stripe is parallel to the finger. In this case, according to an embodiment of the invention, a dynamic change of the pattern between the vertical and horizontal lines is shown. Fig. 6 shows a dynamic change of a pattern from a wide field of view to a narrow field of view according to an embodiment of the present invention. FIG. 6 shows a dynamic change of a pattern between a low density line and a high density line according to an embodiment of the present invention. FIG. Fig. 5 shows a dynamic change of a pattern between a solid line and a broken line according to an embodiment of the invention. Fig. 5 shows a dynamic change of a pattern between a low intensity line and a high intensity line according to an embodiment of the invention. Fig. 4 illustrates a change in light intensity as a function of darkness or brightness of an object according to an embodiment of the present invention. 6 is a simplified flowchart illustrating dynamic pattern change performed after frame analysis according to an embodiment of the present invention.

  The present invention, in some embodiments, relates to three-dimensional depth mapping using structured light, and more specifically, but not exclusively, to a system that performs tracking to provide input to a computer.

  The following application to the same assignee is hereby incorporated by reference as if fully set forth herein, which is a US patent application filed on September 19, 2010: No. 13 / 497,589, International Publication No. 2013/088442 filed on September 13, 2012, US Provisional Patent Application No. 61 / 926,476 filed on January 13, 2014 And US Provisional Patent Application No. 62 / 035,442, filed August 10, 2014.

  As described above, various devices use structured light patterns to enable gesture recognition and 3D depth mapping. The structured light pattern transmitter includes a light source, for example, an optical element such as a laser emitter and a diffractive optical element (DOE). Since many diffractive designs require a coherent Gaussian shaped beam to create a dense pattern, it is generally not possible to use VCSEL laser arrays. Optical elements, such as VCSEL arrays, produce multiple Gaussian shaped beams with overlap, which reduces detection performance in high density regions within the light pattern and various diffractive design techniques that require a predetermined Gaussian beam Limit the use of. Such designs include top hat designs, homogeneous line generators, and other complex high performance structures.

  However, there are significant advantages to using a VCSEL laser array to reduce the structured optical transmitter. Such a reduction is particularly important for embedding a transmitter in a size-constrained device such as a mobile phone or wearable device.

  Thus, this embodiment provides a VCSEL laser array in which the lasers of the array are modulated individually or in groups. Individual lasers or groups can be modulated statically or dynamically to provide and change the structured light pattern as needed.

  Each laser or group of lasers moderated together in the array comprises a unique optical element, typically a cell of diffractive elements. As described in more detail below, the diffractive element cell so that the overall structured light pattern can be selected for a given situation and / or the region of interest can be traced dynamically. Can be individually controlled to provide different light patterns in different parts of the array.

A typical structure includes stripes, grids, and dots.
As previously mentioned, the problem of using a single source emitter, such as an edge emitting laser diode, is that the light pattern produced by the single source emitter can only be controlled as a single building block. As a result, the light pattern can be fully turned on, completely turned off, or dimmed, but cannot be changed dynamically. In contrast, each VCSEL laser in the array according to this embodiment is controlled at the cell level of the optical element and can therefore be controlled individually, for example, the proper design of the DOE can be used for various use cases and feedback. May provide a dynamic light pattern that can provide flexible detection for. Hereinafter, the term “cell” relates to a surface operable by a single laser or any group of lasers that work together to provide a particular portion of a pattern. Changing the pattern can change the cell structure dynamically.

Instead of the diffractive optical element, a refractive element or a combination of a diffractive element and a refractive element may be used.
According to one embodiment, the cell and / or the optical elements of the cell may be limited by size, for example, the cell may be less than about 1 mm in size.

  More particularly, this embodiment relates to a general purpose lens / DOE design that allows a VCSEL array to be used to create a dynamic light pattern. The DOE is located on the surface adjacent to the VCSEL array so that the DOE plane is parallel to the array / matrix plane. In the present embodiment, the surface of the DOE is divided into cells. Each cell represents a region located on a single VCSEL laser or sub-group of VCSELs that are intended to be controlled together. For clarity, the lasers in a group or subgroup are controlled together separately from the lasers in other groups or subgroups.

  A unique diffraction pattern that forms part of the required structured light pattern can be designed for each cell. The individual patterns produced by the VCSEL laser according to the diffraction pattern of each cell create a sub-pattern of structured light. The overall pattern is then formed from the pattern of individual cells, for example by tiling, overlapping, or other methods for positioning individual features.

  Each cell design may include two optical functions. The first positioning function determines the position of the light feature in the entire structured light image. For example, such a positioning function may consist of a prism blazed grating that bends the position of the diffracted light to the actual position of the tile in the required pattern. The second light function relates to the shape of the light feature. Examples of such light functions may include line generators, multiple spot patterns, or other features or subfeatures of light patterns.

  By properly aligning the VCSEL laser matrix and the cell-based DOE, any pattern may be feasible because the adjacent Gaussian beam shape of the entire array is avoided as a single light source perspective.

  In another embodiment, a dynamic light pattern is presented. By applying various currents to the DOE at the appropriate location, each cell can be individually controlled in terms of output intensity, and thus various features within the structured light pattern can also be controlled.

  Dynamic control, which means changing the cell pattern during tracking, enables various functions. That is, the optical elements of each cell can be dynamically changed according to the received data, for example, in the order starting with the initial laser configuration. A frame of the scene is captured and the frame is, for example, a two-dimensional array of pixels. Analyze the received frame. A new laser configuration is then reached based on the analyzed frame. As the cycle continues, the new laser configuration becomes the initial configuration for the next stage. An example is shown and described below with respect to FIG. For example, by changing the intensity of possible VCSELs, the source of each visible light feature can be identified. Such confirmation is very useful in the depth detection method based on triangulation. A further function is the ability to change the density of the light pattern. Changing the density allows for sampling at a higher resolution after the event is roughly mapped, and corrects intensity errors caused by other event effects such as camera settings or lighting conditions, reflectivity, etc. Can do. Furthermore, the orientation of the features in the light pattern can be changed. Changing the intensity or orientation provides a way to give the image processing software additional opportunities to process the scene from what is effectively a new perspective as described above and according to the data received from the camera.

  Reference is now made to FIG. 1, which is a simplified schematic diagram illustrating the basic configuration of a three-dimensional tracking apparatus using pattern light according to the present embodiment. In this and other drawings, it should be understood that dotted and broken lines represent conceptual and structural lines given by context and are not necessarily physical lines visible in individual embodiments.

  In the configuration of FIG. 1, a laser array, which can be a light emitting array, for example a VCSEL array 10, is provided in the laser matrix 12.1.1. . . 12 n. n is included. A cell 16.1.1.1 in which the optical element 14 is aligned outside the laser so that the individual cells are positioned to modulate the light from the individual lasers. . . 16. n. n. Each laser has, for example, a separation cell of diffractive optical elements, and each separation cell has a different diffraction pattern. The enlarged display 18 of the cell 16 shows four diffractive elements each having its own diffraction pattern. Thus, the diffractive optical element has a plurality of individual designs, one for each cell, each design diffracting the light of one VCSEL laser.

  This setting can generate a structured light pattern from the laser array that is projected into the three-dimensional space to track objects and a portion of the scene in the three-dimensional space. The structured light pattern can be any suitable pattern that can be parsed to provide depth information to the computer, including patterns that include stripes, grids, and / or dot areas.

The cells are aligned with the subsets of the laser array, each cell individually applying diffractive modulation to the light passing therethrough, whereby each subset provides a distinguishable portion of the structured light pattern.
Cell 16.1.1. . . 16. n. n can be individually controlled to change the diffraction modulation. Thus, different parts of the pattern can be different and different structured light patterns can be used in different situations or in different parts of the scene.

  The cell can further be dynamically controlled to effect a change to the structured light pattern. Thus, the pattern can be varied to increase the resolution in the portion of the scene that is deemed interesting and / or can reduce the resolution in the portion of the scene that is deemed not relevant. Alternatively, a particular part of the pattern can be instantaneously changed to indicate that a particular light source has reached a particular part of the scene, thereby providing further clues for triangulation and depth estimation. Typically changes strength. That is, the change is based on controlling the intensity of the laser array that affects the cell. As an alternative, polarization, filtering parameters, focal length, or any other characteristic of light apparent to those skilled in the art can be varied.

  The intensity can vary over part or all of the pattern. For example, a portion of the scene can be illuminated brightly by incident light, and other portions of the scene can be illuminated darkly. You can aim for strong light on brightly lit parts and aim for weak light on darkly lit parts, thereby saving power.

  Alternatively, as described in more detail below, the density of the pattern can be varied, or the orientation of the pattern can be varied to provide different views of the scene for tracking and depth estimation in general. With respect to orientation, scene features may be more effectively illuminated in a given orientation. For example, an elongated feature can be most effectively illuminated by a stripe perpendicular to its longitudinal direction. As the feature orientation changes over time, the stripe orientation can be updated. The density can also change over time to allow more specific features to be tracked or when fine features enter the view.

  The subset shown in FIG. 1 is an individual laser. However, as discussed with respect to the following figures, alternative subsets include laser pairs, laser triplets, combinations of lasers of different sizes, and dynamically changing laser combinations. The cell can be approximately 1 mm in size.

The projected light can be organized as tiles, overlapping, or any other suitable configuration.
The configuration of tracking based on structured patterns can be incorporated into a computer, which captures the movement of laptop computers, or mobile communication devices such as tablets, pod devices, mobile phones, game machines, actors, etc. Motion capture devices, such as those used by animators, or any type of device where tracking in three dimensions can provide useful input.

  Instead of placing a single diffractive optical element in the cell, multiple diffractive elements may be used, and any reference to a cell in this document is to be interpreted as additionally referring to a separate optical element. Should.

  Thus, the present embodiment allows a structured light pattern to be generated using a VCSEL laser array and a diffractive optical element (DOE) and allows each VCSEL to be individually controlled to dynamically change the structured light pattern. To do.

  Dynamic control can change the intensity of individual features in the light pattern, change the density or orientation of features, actually turn on / off individual features of the light pattern, or feedback from structured light analysis Changing any of the above based on: For example, as described in more detail below, the density can be increased if the analysis reveals that a higher resolution is required. The intensity is increased if the analysis reveals that the external illumination is interfering with the readout information. If light reflection from external illumination is a problem, the direction of the light pattern can be changed. As mentioned above, apart from the problem of external lighting, it is possible to change the orientation of the stripe in order to keep it perpendicular to the object being tracked. Therefore, feedback from the event is used to change the illumination pattern. By adjusting the brightness on the part affected by the pattern, the incident illumination conditions in the scene can be addressed. Thus, controlling individual cells may allow a specific region to be modified rather than the rest of the pattern based on feedback from the scene. Patterns can be dynamically changed between grids, stripes, dots, or any other pattern that can be used, different cells can have different patterns, and cells can be dynamically redefined as needed Is done.

  Reference is now made to FIG. 2, which is a simplified schematic diagram illustrating one way in which the configuration of FIG. 1 may be used. In FIG. 2, the laser array is indicated by numeral 20 and each laser 22.1.1. . . 22. n. n corresponds to a single cell. Those cells have different tiles 24.1.1. . . 24. n. Each n is illuminated with a forward projection 26. Within the projection 26, each tile has a different pattern and all the tiles combine to form a complete light pattern.

  Reference is now made to FIG. 3, which is a simplified schematic diagram illustrating a variation of the configuration of FIG. 1, which accommodates two or more, and in the illustrated example, three VCSEL lasers. In FIG. 3, the laser array is indicated by numeral 30 and the individual lasers are combined into three laser cells 32.1 to 32.3 by optical elements. Those cells are a triple of different tiles 34.1. . . 34. Each n is illuminated with a forward projection 36. Within the projection 36, all tiles in the same triplet share a pattern, but each triplet has a different pattern in the overall projected light pattern 36.

  Reference is now made to FIG. 4, which is a simplified schematic diagram illustrating a variation of the configuration of FIG. 1, with different cells having different designs and different orientations. In FIG. 4, the laser array is indicated by numeral 40 and the individual lasers 42.1.1. . . 42. n. n forms one and two laser cells. Those cells are different tiles 44.1.1. . . 44. n. Each n is illuminated with a forward projection 46. Within the projection 46, the size of the tiles can be freely changed, and there is one cell of two sizes that share a single pattern.

  Reference is now made to FIG. 5, which is a simplified schematic diagram illustrating a further variation of the configuration of FIG. 1, in which each cell is a pattern of patterns not necessarily organized into separate tile structures. Responsible for creating various light features. In FIG. 5, the laser array is indicated by numeral 50 and the individual lasers 52.1.1. . . 52. n. n forms one individual laser cell. Laser 52.1.1 illuminates tile triple 54.1 with forward projection 56, resulting in a horizontal line across the tile triple.

  Reference is now made to FIG. 6, which is a simplified schematic diagram illustrating an exemplary system for tracking according to an embodiment of the present invention. The VCSEL array 60 produces laser light that is modulated by the optical element 62 to form a pattern that is illuminated within the 3D volume 64. A camera 66 monitors the 3D volume and produces an image frame that is analyzed by the processor 68. As described in more detail below, the processor 68 both tracks and modifies the light modulation to improve tracking. The processor can be an external processor, such as a processor of a mobile device. The camera can be a digital camera or any image sensor that can be based on, for example, a complementary metal oxide silicon (CMOS) array. The processor can be connected to both the camera and the laser so that when the frame is analyzed, the processor controls the laser accordingly.

  Reference is now made to FIG. 7, which shows a laser from a single optical cell or sub-optical cell 70 or several cells such as 18 shown in FIG. 1 that use the laser array 72 to form a projection pattern. FIG. 6 is a simplified schematic diagram illustrating various operations that are various optical functions of an optical element that can be used to modulate light. Certain cells can be allowed to modify the position of the beam (76), or the phase of the beam can be modified (78), the focus can be modified (80), The shape of the beam can be modified (82), the intensity of the beam can be modified (84), or the polarization of the beam can be modified (86). The above is not an exhaustive list and other modifications will be apparent to those skilled in the art. In addition, the optical function may be used by a single optical element or by a plurality of optical elements as shown in FIG. 11, and multiple functions such as focus and shape are incorporated into the cell element.

  Reference is now made to FIG. 8, which shows an optical element 90 that changes the direction of the light beam. A feature 92 emerging from the device body causes a change in the direction of the light beam, which may be obtained by refraction, diffraction, or a combination of refraction and diffraction, depending on the structure. In the illustrated example, a saw-tooth configuration with a tooth-like shape having an upper surface that slopes downward and a horizontal lower surface bends light downward.

  Reference is now made to FIG. 9, which shows a structure 94 for focusing the light beam. Features 96 are arranged in a serrated configuration, similar to FIG. 8, but the orientation of the tooth-like features is reversed in the lower half of the structure, with the upper and lower halves of the beam at the focal point 98. It fits.

  Reference is now made to FIG. 10, which is a simplified diagram illustrating an optical element modification 100 for shaping a beam. In order to shape the beam in a direction that distinguishes it from the other beams, the surface 102 of the optical element is defined using a preset random function.

  Reference is now made to FIG. 11, which is a simplified drawing illustrating an optical element 110 that can relate to a cell or subcell and combines the optical elements of the previous three figures. The first portion 112 focuses the beam produced by the VCSEL array 113. Portion 114 bends the beam downward in this case, and portion 116 shapes the beam.

  Reference is now made to FIG. 12A, which shows the hand 120 being tracked by a pattern 122 of horizontal stripes. It is clear that this pattern happens to coincide with the length of the finger, so the information / data given about the finger is limited and it is difficult to reveal the shape of the object. Nevertheless, the finger can be a very important point in many cases because it gives a gesture that the system uses as a command. Thus, the system can automatically change the orientation of the stripe, for example, to the orientation shown in FIG. 12B so that the stripe crosses the finger and gives more information as a result. This change is made based on the feedback of the image analysis included in the flowchart.

  Figures 13 through 17 show various changes that can be made to the light pattern to improve tracking. FIG. 13 shows changing the orientation of the stripe from horizontal to vertical as in the previous drawing. FIG. 14 shows narrowing the field of view. Narrowing can be useful, for example, when the feature of interest is a finger. When a finger is found, the field of view can be narrowed to concentrate on the finger.

  FIG. 15 shows increasing the density of the stripes. FIG. 16 shows changing the shape from a pure stripe to a broken line. FIG. 17 shows changing the intensity. A low intensity line in the first image changes to a high intensity line in the second image, and vice versa. Proper selection of low density lines and high density lines can be used to track objects in the depth dimension.

  Reference is now made to FIG. 18, which shows how bright and dark objects can be tracked using the embodiment of FIG. The same parts as those in FIG. 2 are given the same reference numerals, and are not described again unless necessary for understanding the present embodiment. In FIG. 18, the bright color object 180 is in the cell region 24.11. The stripe strength is reduced. In contrast, a dark object 182 is in the cell region 24.2.3. The intensity is increased for dark objects. Different cells can operate independently, and each region responds appropriately to the object being tracked.

  Reference is now made to FIG. 19, which is a simplified flowchart illustrating a procedure for modifying a pattern in a cell according to one embodiment of the present invention. A particular cell begins a cycle with an initial configuration (190). The frame is captured (192) and analyzed (194). From the analysis, a new laser configuration is determined based on predefined analysis guidelines 198 (196). That is, for a particular situation, there is a defined pattern change, which can provide more information about the object in the frame. Thereafter, the new laser configuration becomes the initial laser configuration for the next frame.

The terms “including”, “including”, “including”, “including”, “having”, and their equivalents mean “including but not limited to”.
The term “consisting of” means “including and limited to”.

  Certain features of the invention described in discussing separate embodiments for clarity may be provided in combination in a single embodiment, and the above description clearly illustrates the combination. It will be understood that it should be interpreted as if it were written. On the contrary, the various features of the invention described in discussing a single embodiment for the sake of brevity are also considered separately from any other described embodiments of any suitable part. The above description should be construed as if the separate embodiments are clearly written. Certain features described in discussing various embodiments should not be construed as essential features in those embodiments, unless the embodiment is unable to function without such elements.

  While the invention has been described with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

  All publications, patents, and patent applications mentioned in this specification are specifically and individually indicated as if each publication, patent, or patent application was incorporated herein by reference. To the same extent, are incorporated herein by reference in their entirety. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. As long as paragraph headings are used, they should not necessarily be construed as limiting.

Claims (20)

A device,
A plurality of optical elements, each of the plurality of optical elements including a cell, the cell being aligned with a subset of the laser array, wherein the cell is dynamically modulated with respect to light passing through the cell; The plurality of optical elements that are individually controllable to form a distinguishable portion of the structured light pattern in addition to
A processor configured to provide one or more instructions to the cell to dynamically adjust the modulation of light passing through the cell to affect changes in corresponding portions of the structured light pattern A device comprising: The apparatus of claim 1, wherein the modulation is selected from the group consisting of diffractive modulation, refractive modulation, and a combination of diffractive and refractive modulation. The apparatus of claim 1, wherein the plurality of optical elements are constructed from a single molding element. The apparatus according to claim 1, wherein the width of the cell is 1 mm or less. The apparatus according to claim 1, wherein the width of the optical element is 1 mm or less. The apparatus of claim 1, wherein the cell is controllable to affect a change in a position of the structured light pattern and a shape of the structured light pattern. The dynamically applied modulation applies a high resolution of the structured light pattern to a part of a scene and applies a low resolution of the structured light pattern to another part of the scene. apparatus. The apparatus of claim 1, wherein the dynamically applied modulation to the structured light pattern includes a cell change to the orientation of the structured light pattern. The apparatus of claim 1, wherein the dynamically applied modulation is selected from the group comprising a change in intensity, a change in polarization, a change in filtering parameters, and a change in focus. The subset of the laser array is selected from the group consisting of individual lasers, laser pairs, laser triplets, combinations of lasers of different sizes, and combinations of dynamically changing lasers. The device described. The apparatus of claim 1, wherein light projected from the subset of the laser array is tiled. The apparatus of claim 1, wherein the laser array is configured to project the structured light onto a scene. The apparatus of claim 1, wherein the laser array comprises a source selected from the group consisting of a laser bar and a vertical cavity surface emitting laser (VCSEL) array. The apparatus of claim 1, wherein the apparatus is incorporated into any element of the group consisting of a computer, a laptop computer, a mobile communication device, a tablet device, a game console, and a motion capture device. A method,
Projecting light from a subset of the laser array, and the projected light passes through a plurality of optical elements including cells aligned with the subset of the laser array, the cells being individually controllable Yes,
Dynamically modulating the projected light passing through the cell to form a distinguishable portion of the structured light pattern. Dynamically modulating includes modulating the projected light, said modulating diffracting the light, polarizing the light, changing the wavelength of the light, changing the light filtering parameters. The method of claim 15, wherein the method is selected from the group comprising changing, changing light intensity. Further comprising dynamically changing the structured light pattern, wherein changing the structured light dynamically changes the modulation applied to the projected light from the subset of the laser array. The method of claim 15 comprising: The method of claim 17, wherein dynamically changing the structured light pattern includes increasing a resolution level in a portion of a scene and decreasing a resolution level in a different portion of the scene. The method of claim 17, wherein dynamically changing the structured light pattern comprises changing a patterning density within the structured light pattern. A device,
A plurality of cells aligned with individual subsets of the laser array, each cell being individually controllable and dynamic with respect to light passing through the cells from the individual subset of the laser array The plurality of cells configured to provide a distinct portion of the structured light pattern by applying modulation to the plurality of cells, wherein the modulation applied to the plurality of cells represents an initial modulation;
A processor configured to determine a second modulation configuration based on analyzing at least one captured frame, wherein the second modulation configuration describes a modulation different from the initial modulation;
The apparatus is configured to dynamically adjust a modulation of light passing through the cell to apply a second modulation to the cell.