JP7244013B2 - Methods for epipolar time-of-flight imaging - Google Patents

Description

This application claims the benefit of US Provisional Patent Application No. 62/499,193, filed January 20, 2017.

GOVERNMENT RIGHTS This invention was made with government support under N000141512358 awarded by ONR, 11S1317749 awarded by NSF, HR00111620021 awarded by DARPA, and grants NNX16AD98G and NNX14AM53H awarded by NASA. The Government has certain rights in this invention.

Time-of-flight (ToF) depth sensors have become the technology of choice in applications as diverse as automotive and aviation, robotics, gaming and consumer electronics. These sensors have two main characteristics: LIDAR-based systems, which rely on extremely short pulses of light to sense depth, and continuous wave (CW) systems, which emit modulated light signals over much longer durations. - wave) system. LIDAR-based systems can acquire centimeter-accurate depth maps up to 1 kilometer away in broad daylight, but have low measurement rates. Moreover, the cost per pixel is several orders of magnitude higher than CW systems, and its range, outdoor operation and robustness are severely limited. Continuous wave time-of-flight (CW-ToF) sensors, despite their shortcomings, have found their way into consumer electronics and low-cost robotics because low cost, large-scale production, and high measurement rates often outweigh other considerations. continue to dominate the place of In addition, consumer-grade time-of-flight depth cameras such as Kinect and PMD are inexpensive, compact, and produce video-rate depth maps for short-range applications.

The present invention significantly reduces the shortcomings of CW-ToF using energy efficient epipolar imaging. In one embodiment, a continuously modulated sheet of laser light is projected along a carefully selected sequence of epipolar planes that collectively scan the field of view. On each projected sheet, only strips of CW-ToF pixels corresponding to each epipolar plane are exposed. As shown in FIG. 2, a prototype implementation of the invention couples a specially crafted projection system to a CW-ToF sensor with a controllable region of interest. In some implementations, off-the-shelf CW-ToF sensors can be used. An off-the-shelf sensor can output 320x240 3D video live at 7.5 frames per second, and the frame rate is only limited by the API of the sensor.

Epipolar imaging was initially proposed for live acquisition of direct-only or global-only video using conventional (non-ToF) video sensors. This approach has been extended to the ToF field, but its energy efficiency is very low and it involves capturing over 500 images to compute a single 'direct-only' ToF image. Significant improvements in energy efficiency and robustness are realized with 2D scanning laser projectors and rolling shutter cameras in the context of triangulation-based 3D imaging. The present invention extends this idea to the ToF field. Thus, it inherits all the advantages of non-ToF energy-efficient epipolar imaging, but also addresses the challenges unique to CW-ToF.

The main difficulty is that the range of CW-ToF sensors is severely limited by power consumption and eye safety considerations. Although most CW-ToF sensors electronically subtract the DC component of the incident light, photon noise from strong ambient sources such as sunlight can reduce the noise at typical frame rates outdoors at distances greater than several meters. It can easily overwhelm the CW-ToF signal. By concentrating the energy of the light source onto a single sheet, the epipolar ToF increases this range to 10m and acquires a useful depth signal beyond 15m outdoors, albeit noisier.

A second difficulty is that the depth accuracy of CW-ToF sensors is strongly affected by global illumination effects such as interreflection and global illumination transport. These effects produce longer optical paths and appear as sources of structured additive noise. These effects cannot be canceled after the fact without imposing strong assumptions on the scene geometry and reflection properties, and are quite common indoors (e.g., corners between walls, table and floor glossy surfaces, mirrors, etc.). The present invention demonstrates remarkable robustness to all forms of global transport and especially to specular interreflections, a form of global illumination transport that has never been possible to handle with live CW-ToF. do.

As devices equipped with CW-ToF depth sensors become more common indoors and outdoors, they must be able to operate without interfering with each other. Although decoupling between devices of a given make and model can be achieved by varying the modulation frequency across them, robustness to a wider ecosystem of CW-ToF sensors is desirable. The present invention demonstrates that epipolar ToF enables interference-free live 3D imaging even for devices with exactly the same modulation frequency and source wavelength.

Finally, a CW-ToF sensor must acquire two or more frames with different phases of emitted light to compute a single depth map. This makes them very sensitive to camera shake, unlike conventional cameras where vibrations simply blur the image, and camera shake in CW-ToF violates the static scene assumption and is blurred by motion artifacts. This results in a depth map that is both distorted and corrupted. Epipolar ToF allows us to address both problems: motion blur is minimized by relying on very short exposures for each epipolar plane, and multiple Motion artifacts and depth errors are minimized by acquiring phase measurements of , and the sequentiality of the epipolar plane ToF is improved by scheduling the sequence of epipolar planes such that post-acquisition distortion correction is easier. The rolling shutter-like distortion caused by this is reduced.

Figures (a) to (d) are diagrams to show a comparison of various scenes scanned using the epipolar ToF system of the present invention versus a normal (ToF) system; 1 is a schematic diagram of a system for performing epipolar ToF imaging; FIG. (a) to (f) illustrate several possible epipolar plane sampling schemes and row exposures in ToF imaging. 1 is a depiction of the prototype of the present application; Fig. 2 shows a laser light source component; FIG. 4 is a timing diagram of camera exposure, readout and mirror positions for a particular sequence of rows; (a) to (d) show the results of imaging a white planar target at a range of distances from the sensor under cloudy weather and bright sunlight. (a) is a graph showing the standard deviation in depth measurements obtained using conventional and epipolar ToF imaging. (b) is a graph showing the operating range of the same simulated camera at different levels of acceptable range accuracy; Note that the simulated camera parameters are different from the prototype. FIG. 10 shows that epipolar ToF imaging provides accurate depth return from their surfaces even when the bulb is turned on. FIG. 5 compares depth maps from epipolar and conventional ToF imaging in the presence of global light transport.

The term microcontroller, as used herein, can mean a dedicated hardware device, circuit, ASIC, FPGA, microprocessor executing software, or any other means known in the art. It is further understood that the microcontroller includes connections to both the sensor and the laser light projector for sending control signals and for receiving data. The present invention is not intended to be limited to one method of implementing the functionality of the controller.

The terms camera and sensor are used interchangeably herein.

であり、ここで、ｈ_ｘ（ｔ）はアクティブな光源に対する画素の過渡応答を表し、Ａ_ｘは周囲光に起因する受け取った光、およびアクティブな光源のＤＣ成分である。Ａ_ｘは積分から落ちるが、実際には、ｇ_ω，φ（ｔ）が正または負であるかどうかにより入射光を２つの異なる格納場所（タップと呼ばれる）に積分し、次いで格納された値の間の差を取ることにより、Ｉ_ω，φ（ｘ）は測定され、したがって、周囲光は依然として測定ショット雑音に加わる。 Continuous Wave Time-of-Flight CW-ToF cameras use time-modulated light sources and sensors whose exposure is also modulated during integration. If the illumination modulation function is f _ω t=cos(ωt) and the sensor modulation function is g _ω,φ (t)=cos(ωt+φ), where ω is the modulation frequency in rad/s and φ is the phase offset between the source and sensor modulation function, but the measurement at pixel x is:

where h _x (t) represents the transient response of the pixel to the active light source and A _x is the received light due to ambient light and the DC component of the active light source. A _x falls out of the integration, but in reality it integrates the incident light into two different storage locations (called taps) depending on whether g _ω,φ (t) is positive or negative, and then the stored value I _ω,φ (x) is measured by taking the difference between , so ambient light still adds to the measurement shot noise.

であり、ここでｃは光の速度であり、ｌ（ｘ）は光源から、ｘに対応するシーン点、そして、センサに帰るまでの経路の長さである。 If there is no indirect optical path between the light source and sensor pixel x,

where c is the speed of light and l(x) is the path length from the source to the scene point corresponding to x and back to the sensor.

画素深度ｚ（ｘ）は、光源およびセンサの幾何学的較正パラメータを使用してｌ（ｘ）から計算することができる。 Assuming the scene is static, the path length l(x) can be recovered by capturing pairs of images at the same frequency but at two different modulation phases φ = 0 and φ = ^π / ₂ . can:

Pixel depth z(x) can be calculated from l(x) using the geometric calibration parameters of the light source and sensor.

Epipolar Time-of-Flight FIG. 2 is a schematic diagram of a system for performing epipolar ToF imaging. A projector that produces a movable sheet of modulated laser light is combined with a ToF sensor that can expose rows one at a time. The projector and sensor are arranged in a rectified stereo configuration such that the light sheet is always in the epipolar plane between the projector and camera. At any given time, only rows of camera pixels in the epipolar plane are exposed to light.

To achieve the geometry of FIG. 2, a line laser source with a 1D scanning mirror is used to project a movable light sheet onto the scene, as shown in FIG. 4(b). Current CW-ToF sensors do not provide controllable exposure coding across the 2D pixel array. Given the available off-the-shelf hardware, there are three ways to limit the exposure to pixels on the epipolar plane:
1. use a digital micro-mirror device (DMD) to mask all other pixels;
2. 2. Select the epipolar plane to be imaged using a 1D sensor and controllable mirrors; A 2D sensor with a controllable region of interest (ROI) is used.

In the preferred embodiment, the third option is chosen because it is more light efficient than using a DMD mask, which leads to a simpler design. The ROI is set to be one row high to meet the epipolar ToF requirements.

Epipolar Plane Sampling CW-ToF requires at least two images to recover depth. To cover the entire scene using epipolar ToF, the active epipolar plane has to be swept over the field of view. This provides flexibility in choosing the order in which the epipolar planes are sampled.

FIG. 3 shows several such ordering schemes. FIG. 3(a) shows a conventional prior art ToF, where all epipolar planes are illuminated simultaneously and all camera rows are exposed simultaneously. This requires long exposures and introduces severe artifacts due to motion, ambient light, global light transport and interference between devices. Fig. 3(b) shows that sending very short high-intensity pulses of light to CW-ToF provides resistance to ambient light, but it is still prone to artifacts due to global light transport and motion. indicates that

FIG. 3(c) shows an epipolar ToF plane ordering that produces an effect similar to a rolling shutter camera, where one complete image is acquired for each modulation phase. This results in robustness to ambient light, global illumination and motion blur. However, sensitivity to motion still remains due to the significant delay between the multiple phase measurements acquired for each row. This scheme is undesirable because if the scene or camera moves while these images are being acquired, the retrieved depth map will contain errors that are difficult to correct.

Another embodiment in FIG. 3(d) shows that interleaving the measurements by plane minimizes such artifacts. The ordering strategy shown in FIG. 3(d) loops through the set of modulation phases one epipolar plane at a time. Since the exposure time for each row is very short, all phases required for a single row can be acquired quickly enough to minimize depth and motion blurring artifacts due to camera/scene motion.

Using this strategy, each row is captured at a slightly different time. This induces a rolling shutter-like effect in the acquired depth map, but the individual depth values are free of blurring and artifacts and can be combined into a coherent model by post-processing.

To further facilitate such post-processing while adhering to the kinematic constraints of the mirror's actuator, the epipolar planes are ordered in a sawtooth pattern, as shown in FIG. 3(e). In this scheme, the entire field of view is scanned twice within the same total exposure time, resulting in a higher temporal sampling of the scene and making consistent merging of individual depth map rows easier. This essentially doubles the frame rate but at half the vertical resolution to provide a full depth-of-field map, making depth correction easier for fast camera shake and/or scene motion.

More generally, Figure 3(f) shows that in some applications it may be beneficial to scan different portions of the field of view using different temporal sampling rates. An example of a non-uniform sampling scheme is shown where epipolar planes corresponding to lower image rows are sampled more frequently. This type of sampling can be useful, for example, on vehicles, where the lower portion of the field of view typically moves closer and faster, requiring acquisition at a faster sampling rate.

During operation, the projector generates a sheet of modulated laser light that continuously illuminates the epipolar plane defined between the laser projector and the sensor. The planes can be illuminated in any order, but in a preferred embodiment they are illuminated from top to bottom and then from bottom to top. The actual order in which the planes are illuminated may depend on the particular environment in which the platform is being used or the application for which the depth map is being created. Also, any number of planes can be defined within the field of view, limited only by the laser and sensor capacities and the desired frame rate. In the preferred embodiment, there are 240 planes defined in the field of view, each plane being 320x240 pixels.

The region of interest of the sensor can be set to any part of the field of view and, in operation, is microscopic so that the ROI of the sensor is set to sense a row of pixels in the currently illuminated epipolar plane. A controller synchronizes the laser projector and the sensor. Phase is estimated using two images. In general, the sensor uses four measurements (angles 0, 90, 180, 270) to correlate the incident signal with the shifted input signal. Either two or four of these images can be used for phase estimation, but using four images gives more accuracy but takes longer to capture and Reduce frame rate. If phase unwrapping is required, the phase estimation process needs to be performed at different modulation frequencies, so four images are required for phase unwrapping instead of two images. In some embodiments of the invention, an inertial measurement unit (IMU) can be attached to the sensor and used to compensate for platform motion.

Epipolar ToF Prototype A prototype device for epipolar ToF imaging, shown in FIG. built with FIG. 4 is a depiction of the prototype. A DME660 camera with fast ROI control to capture arbitrary rows of pixels and a custom-made movable light sheet projector as the light source were used in the prototype. The prototype described herein is merely representative of one particular embodiment of the invention, and that other embodiments utilizing different equipment and operating parameters are included within the scope of the invention. should be understood by those skilled in the art.

The ToF sensor used is an EPC660 (from Espros Photonics) with a resolution of 320x240 and the pixels implement ambient desaturation. The sensor is equipped with an 8 mm, F1.6 low-distortion lens and an optical bandpass filter (650 nm center frequency, 20 nm bandwidth). The sensor allows the ROI to be changed for each sensor readout and this feature is used to select different rows to image. A sensor development kit (DME660) from the manufacturer is utilized to read data from the sensor. It should be understood that any ToF sensor may be used, although the invention is not limited to the use of the ToF sensors described.

The line projector utilized in the prototype uses a 638 nm laser diode with a peak power of 700 mW as its light source. Light from the diode is collimated and passed through a Powell lens that stretches the beam cross-section into a broad, nearly uniformly illuminated straight line with a fanout angle of 45 degrees. The laser light is directed to a 1D scanning galvo mirror that can be rotated to deflect the sheet. The rotation range of the mirror gives the projector a 40 degree vertical field of view. The effective center of projection of the projector moves as the mirror rotates, but this effect is negligible because the distance between the fanout point and the galvo mirror is very small compared to the depth in the scene.

A microcontroller is used to synchronize the sensor and the light source. A microcontroller can communicate with the sensor over the I2C bus to set the exposure time, modulation frequency/phase, area of interest, row, and to trigger each capture. The microcontroller can also operate the galvo mirrors of the projector. Additionally, the microcontroller can read the rotational speed of the camera using MEMs inertial magnetic units (IMUs) attached to the sensor. A frequency generator circuit allows selection of the modulation frequency (between 11 MHz and 24 MHz in 1 MHz steps).

The projector and camera are aligned side-by-side in a collimated stereo configuration as required for epipolar imaging. When properly aligned, the projected light sheet illuminates a single row of pixels in the camera, which row is independent of depth. Mirror calibration is performed to determine the mapping between galvo mirror angles and illuminated camera rows.

との間の関係は、各画素における射影補正Η_ω（ｘ）を使用してモデル化される。

Sensor Calibration In practice, measurements read out from sensors do not match their expected values when observed. There are several reasons for this discrepancy, including fixed pattern noise, uneven sensitivity, and crosstalk between the taps and the phase variation of the actual exposure modulation function at each pixel. Expected sensor measurements Ι _ω (x) and observed measurements

is modeled using the projection correction _Ηω (x) at each pixel.

を計算することができる。センサ測定値Ι_{ω，ｋ，ｓ}（ｘ）を最もよく説明するΗ_ω（ｘ）は、補正された測定値と期待される位相との間の最小二乗誤差を最小にする補正Η_ω（ｘ）を見いだすことによって計算することができる。 To find _Ηω (x), the sensor traverses a set of known distances _zk , k=1, . . . , K, parallel to the forehead. For each position in the plane, sensor measurements are collected at different aperture settings (s=1,...,S) to simulate the effect of varying scene albedo. For each planar position k, the path length can be calculated in pixels, l _k (x), and then the expected phase

can be calculated. The _Ηω (x) that best describes the sensor measurements _Ιω,k,s (x) is the correction _Ηω (x ) can be calculated by finding

These calibration parameters are dependent on both modulation frequency and exposure time, so the process is repeated for all frequencies and exposure times. The modulation signal passed to the sensor and light source driver was a square wave, but the harmonics were greatly suppressed at modulation frequencies above 20 MHz, so the modulation function was well approximated by a sine wave.

The time (and frame rate for that matter) required to image a row using the timing prototype is n, the number of readouts per row, the exposure time t _exp , the readout time for the row t _read , and t _mirror , a function of the time it takes for the galvo mirror to move to the next row position in the sampling sequence.

For a two-tap sensor such as that used in the prototype, at least n=2 readings are required to measure depth using a single modulation frequency. FIG. 5 shows a timing diagram of camera exposure, readout and mirror positions for a particular sequence of rows. First, the scanning mirror is moved to the new active row and takes _{t_mirror} time to settle into place. The camera is triggered when the previous row readout is complete (which takes t - - _read time) and the mirror is in place. In this example, t _mirror >t _read , so the speed of the mirror is the bottleneck for the capture rate. Each exposure lasts for time t _exp and at the end of each exposure a row is read. FIG. 5 shows a timing example. _{t_row} is 175 μs and _{t_exp} is set to 100 μs. In the row sampling sequence, the mirror rotates through two rows per step (approximately 0.33°) and the settling time t _mirror for this step size is approximately 100 μs. In total, t _row amounts to 550 μs when n=2, resulting in a frame rate of 7.5 fps (3.8 fps when n=4).

Limitations Currently, the main bottleneck for frame rate is readout time. Although embodiments of the present invention require data from only one row of the sensor for each readout, the minimum area of interest that the EPC660 sensor supports is four rows high, and in reality only one row. is used, it forces a read of 4 rows. Additionally, the development kit limits the sensor's data bus to 20 MHz, but the sensor itself supports bus rates up to 80 MHz. The minimum value of t _exp depends on the peak power of the light source and the desired range. The prototype of the invention described has a source with a peak power of 700 mW, but most other experimental time-of-flight systems have peak source powers in the range of 3W to 10W. Shorter exposure times can be used without loss of range when using brighter light sources. Finally, low cost galvo mirrors can be replaced with faster 1D MEMs mirrors. With these improvements, a system based on the described prototype should operate at video frame rates.

The sensor used in the prototype described only supports a maximum modulation frequency of 24 MHz, but most other time-of-flight sensors can operate in the range of 50 MHz to 100 MHz. This limits the prototype's ability to accurately scan smaller objects or to be used for transient imaging. The EPC660 datasheet specifies that the sensor ADC returns 12-bit values, but the version of the sensor used only returns 10-bits, which works for range and makes the output depth map noisier. become.

Results To run the sensor in normal imaging mode to compare performance under ambient and global illumination, the entire sensor was exposed at once, instead of using a small ROI, and the sensor was exposed to the sheet projector's field of view. remains exposed until the end of the sweep over . In normal ToF imaging in multi-device interference and camera motion experiments, the sheet projector can be replaced with a diffuse source.

Ambient Light The benefit of applying epipolar imaging to time-of-flight in brightly lit environments was simulated and the results are shown in FIG. For a given source power, typical imaging degrades depth accuracy rapidly as the ambient light level increases from 0 lx (complete darkness) to 100 klx (direct sunlight). For epipolar imaging the degradation is much more gradual.

FIG. 7(a) shows simulated results of the standard deviation in depth measurements obtained using normal and epipolar ToF imaging (modulation frequency 15 MHz) for a target 10 m from the camera depending on the ambient light level. . For both cases, the peak source power is 2 W and the total exposure time is the same (7.2 ms per image), but the epipolar ToF concentrates the source power and uses a short exposure for each row. (30 μs) is used, so it is more robust to ambient light.

FIG. 7(b) shows the motion range of the same simulated camera at different levels of acceptable range accuracy. Note that the simulated camera parameters are different from the prototype.

Figure 6 quantitatively compares the described prototypes in normal ToF and epipolar ToF modes in cloudy and clear weather conditions. Normal ToF mode fails in bright sunlight, but epipolar ToF is considerably more robust. FIG. 1(a) shows a raw 3D CW-ToF imaging in daylight (70 klx) with a range of 15 m (a person walking on stairs and phase wrapping around a distant building); FIG. 8 shows an example scene with both strong ambient lighting and global lighting effects. Reflections from the table service cause errors in normal ToF, but are suppressed in epipolar imaging.

FIG. 6(d) shows the standard deviation of depth measurements versus distance to target (slower rising curve is better). Our prototype has a depth error of approximately 3% at 10m in bright sunlight.

Global Illumination FIG. 9 illustrates the ability of epipolar imaging to suppress the effects of global illumination in a few common indoor environments. These results are generated using a single modulation frequency (24 MHz). In the corners of the room, the diffuse interreflection between the walls and the ceiling overestimates the depth and causes corner rounding in normal imaging.

The conference table in the second row of FIG. 9, also shown in FIG. 1(b), appears specular at the grazing angle. In a bathroom scene, wall ghosting due to reflections from mirrors is suppressed by epipolar imaging. Drinking fountains are of particular interest because the direct return from their metal surfaces is very weak, but the surface reflects a lot of indirect light back to the sensor. Epipolar imaging combines three exposures to attempt to recover a usable direct signal. Longer exposures are not useful for normal imaging because interreflections saturate the sensor.

With epipolar imaging, the walls appear straight and meet at sharp right angles, corner interreflections, glossy intermediate projection from the projection screen onto the glossy conference table, from the mirror in the bathroom and with the wall. Diffuse the reflections between the drinking fountain with the Epipolar ToF removes most of the global light transport, resulting in depth maps that are significantly more accurate than regular ToF.

Multi-Camera Interference With epipolar CW-ToF imaging, two cameras moving at the same modulation frequency can typically only interfere with each other in a sparse set of pixels in each image. Each camera illuminates and images a single line in the scene at a time; It can only interfere with the first camera at the point of intersection. Worse cases occur when the light source of one camera forms a collimated stereo pair with the sensors of a second camera, and both cameras happen to be synchronized, but this can be considered a rare occurrence. can.

If there are more than two cameras, each pair of cameras has a sparse set of points where they interfere with each other. When a set of epipolar ToF cameras are moving at different modulation frequencies, each camera's contribution to shot noise in the other camera is greatly reduced. FIG. 1(c) shows the results of operating two CW-ToF cameras simultaneously at the same frequency with normal and epipolar imaging. Epipolar imaging shows no interference between ToF devices operating at the same frequency. Regular ToF has observable errors (ie walls and chairs).

Camera Motion When using a rotating camera with a known rotational trajectory (obtained from a MEMS gyroscope), normal imaging causes each captured ToF measurement to have a motion It has strong artifacts in blurring and depth discontinuities. Theoretically, these can be collected using spatially varying deconvolution, but this is computationally expensive and poor at recovering high frequency components. When using epipolar ToF imaging, motion blur basically has no effect and a depth map is obtained with an effect similar to a rolling shutter. This can be corrected using a simple image warp computed from the rotation. Fig. 1(d) shows an example from a rapidly panning camera, where an undistorted depth map can be obtained even in the presence of severe camera shake during the scene exposure (which should be removed in normal ToF). difficult ghost errors can be observed). Additionally, as previously mentioned, sensors can be equipped with I am you, which are used to compensate for platform promotions.

Epipolar imaging for time-of-flight depth cameras overcomes many of the problems commonly encountered by depth cameras, such as poor performance in brightly lit conditions, systematic errors due to global illumination, inter-device interference and errors due to camera motion. Reduce. Compared to depth cameras, systems such as scanning LIDAR that illuminate and image a single point at a time are very robust against all these effects, but have a low measurement rate. Epipolar imaging can be considered a compromise between these two extremes of full-field capture and point-by-point capture. Since epipolar imaging illuminates and captures a single line at a time, it allows depth cameras to have most of the robustness of point scanning while still having a high measurement rate.

Repeating the pattern row-by-row, as done here for ToF, is also directly applicable to structured light. It should allow applying multi-image structured light methods to generate high-quality depth maps to dynamic scenes where currently only single-shot methods can be used.

In the described prototype, the scanning mirror follows a sawtooth pattern and captures rows in an orderly sequence. However, when using faster scanning mirrors, a pseudo-random row sampling strategy can be implemented, which allows epipolar imaging to be used in conjunction with compressive sensing or similar techniques, allowing fast moving scenes recover the temporally super-resolved depth map of . Although embodiments of the invention have been described herein using specific identified components, the invention is not intended to be limited thereby. The scope of the claimed invention is defined by the set of claims presented below.

Claims (31)

a system,
a modulated light source;
a sensor, wherein the modulated light source and the sensor are in a collimated stereo configuration such that the light sheet projected by the modulated light source is in an epipolar plane between the projector and the sensor;
a microcontroller for synchronizing and configuring the modulated light source and sensor;
the modulated light source is configured to project a sheet of modulated light along a series of epipolar planes, the series of epipolar planes defining a field of view;
The sensor is configured to image a row of illuminated pixels in the field of view from each epipolar plane in the series of epipolar planes, the imaging moving to the next epipolar plane in the series of epipolar planes. Before, capture all modulation phases in the set of modulation phases for each epipolar plane,
A system in which the epipolar planes in the field of view are illuminated and sensed in varying order . A modulated light source
a. a laser light source;
b. an optical element configured to generate a light sheet from the collimated output of the laser light source;
c. and means for directing the sheet of laser light along a series of epipolar planes between the projector and the sensor. 3. The system of claim 2, wherein the means for directing the sheet of laser light is selected from the group comprising rotatable galvanomirrors and MEMS mirrors. 3. The system of claim 2, wherein the sensor is a continuous wave time-of-flight camera with a controllable region of interest. The projected light sheet illuminates a single row of pixels on the sensor, and the controllable target areas of the sensor are all set to sense the single row of illuminated pixels. Item 5. The system according to item 4. 6. The system of claim 5, wherein the laser light projector and sensor are synchronized such that the target area of the sensor is set to sense a row of pixels in the currently illuminated epipolar plane. 7. The system of claim 6, wherein the sensor captures at least two images from each illuminated epipolar surface. 8. The method of claim 7, wherein the projected light sheet is modulated as a repeating wave and further, by the phase of the returned reflection, the sensed depth can be calculated for each sensed pixel. system. 9. The system of claim 8, wherein a depth map of the entire field of view is created based on depths calculated for each sensed pixel from each illuminated epipolar plane in the field of view. 10. The system of claim 9, wherein the microcontroller reads data for a previously illuminated epipolar plane from the sensor while the galvanomirror is rotated to a position to illuminate the next epipolar plane in the series of epipolar planes. . a method,
Projecting a sheet of modulated light along a series of epipolar planes defined by modulated light sources and sensors arranged in a collimated stereo configuration, the series of epipolar planes defining a field of view. do, step and
imaging rows of illuminated pixels in the field of view from each epipolar plane in the series of epipolar planes, the imaging being one for each epipolar plane before moving to the next epipolar plane in the series of epipolar planes; capturing all modulation phases in the set of modulation phases;
determining the depth of each pixel in a single row of illuminated pixels based on two or more of the modulation phases in the set of modulation phases ;
A method in which epipolar planes in the field of view are illuminated and sensed in varying order . 12. The method of claim 11, wherein determining the depth of each pixel comprises calculating the depth of each pixel based on the phase of reflected light from each pixel. 13. The method of claim 12, wherein calculating the depth of each pixel in the illuminated epipolar surface further comprises determining a phase difference of reflected light contained in two or more separate images of the illuminated epipolar surface. described method. 13. The method of claim 12, further comprising creating a depth map based on the depth of each pixel in each illuminated epipolar plane within the defined field of view. 12. The method of claim 11, further comprising synchronizing the modulated light source and the sensor such that the target area of the sensor corresponds to the currently illuminated epipolar surface. A modulated light source
a laser light source;
an optical element configured to generate a light sheet from the collimated output of the laser light source;
12. The method of claim 11, comprising means for directing the sheet of laser light along a series of epipolar planes between the projector and the sensor. A non-transitory computer-readable medium that, when executed,
projecting a sheet of modulated light along a series of epipolar planes defined by modulated light sources and sensors arranged in a collimated stereo configuration, the series of epipolar planes defining a field of view;
imaging rows of illuminated pixels in the field of view from each epipolar plane in the series of epipolar planes, the imaging being one for each epipolar plane before moving to the next epipolar plane in the series of epipolar planes; capturing all modulation phases in the set of modulation phases;
determining the depth of each pixel in a single row of illuminated pixels based on two or more of the modulation phases in the set of modulation phases;
comprising software performing the function of creating a depth map based on the depth of each pixel in each illuminated epipolar plane within the defined field of view;
A non-transitory computer readable medium in which epipolar surfaces in a field of view are illuminated and sensed in varying order . Determining the depth of each pixel includes calculating the depth of each pixel based on the phase of reflected light from each pixel, and calculating the depth of each pixel in the illuminated epipolar plane is illuminated. 18. The non-transitory computer readable medium of claim 17 , further comprising determining a phase difference of reflected light contained in two or more separate images of the epipolar surface. 18. The non-transitory computer readable medium of claim 17 , wherein the software performs a further function of synchronizing the modulated light source and the sensor such that the sensor's region of interest corresponds to a currently illuminated epipolar plane. A modulated light source
a laser light source;
an optical element configured to generate a light sheet from the collimated output of the laser light source;
12. The method of claim 11, comprising means for directing the sheet of laser light along a series of epipolar planes between the projector and the sensor. 12. The method of claim 11, wherein only a subset of the epipolar plane is imaged. 12. The method of claim 11, wherein the epipolar plane is illuminated and sensed in an interleaved or sawtooth pattern. 12. The method of claim 11, wherein one epipolar surface is illuminated and sensed multiple times before sampling another epipolar surface. 12. The method of claim 11, wherein the field of view includes multiple portions, the portions being illuminated and sensed at different temporal scanning rates. 18. The medium of claim 17 , wherein only a subset of the epipolar surface is illuminated and sensed. 18. The medium of claim 17 , wherein the epipolar surface is illuminated and sensed in an interleaved or sawtooth pattern. 18. The medium of claim 17 , wherein one epipolar surface is illuminated and sensed multiple times before sampling another epipolar surface. 18. The medium of claim 17 , wherein the field of view includes multiple portions, the portions being illuminated and sensed at different temporal scanning rates. 3. The system of claim 1, wherein the epipolar plane is illuminated and sensed in an interleaved or sawtooth pattern. 2. The system of claim 1, wherein one epipolar surface is illuminated and sensed multiple times before sampling another epipolar surface. 2. The system of claim 1, wherein the field of view includes multiple portions, the portions being illuminated and sensed at different temporal scanning rates.

Images