JP7403860B2 - Agile depth sensing using triangulation light curtains - Google Patents

Description

This application is a continuation-in-part of U.S. patent application Ser. No. 16/470,885, a national phase application under 35 U.S.C. Claims benefit and priority of Application No. PCT/US2019/021569, which claims benefit of U.S. Provisional Patent Application No. 62/761,479, filed March 23, 2018. Additionally, U.S. Patent Application No. 16/470,885 is a continuation-in-part of U.S. Patent Application No. 15/545,391, which is a national phase application under 35 U.S.C. Claims benefit and priority of International Patent Application No. PCT/US2016/017942, filed February 15, which is a patent application of U.S. Provisional Patent Application No. 62/176,352, filed February 13, 2015. claim benefits. Additionally, this application also claims the benefit of U.S. Provisional Patent Application No. 62/920,178, filed April 17, 2019. The entire contents of these applications are incorporated herein by reference in their entirety.

GOVERNMENT BENEFIT This invention was made with government support under contract HR0011-16-C-0025 awarded by the Defense Advanced Research Projects Agency (DARPA) and contract CNS-1446601 awarded by the National Science Foundation (NSF). It was done. The Government has certain rights in this invention.

3D sensors play a key role in the deployment of many autonomous systems, including field robots and self-driving cars. However, there are many tasks that may not necessarily use a full-fledged 3D scanner. As an example, self-guided vehicles on the road and robots in the field do not need full-fledged 3D depth sensors to detect potential collisions or monitor their blind spots. . Instead, what is needed is for the vehicle to be able to detect whether an object is within a predefined boundary of the vehicle so that collision avoidance can occur. This is a much simpler task than full depth scanning and object identification.

Depth sensors such as LIDAR and Microsoft Kinect(R) use a fixed depth acquisition strategy that is independent of the scene of interest. Due to the low spatial and temporal resolution of these sensors, this strategy may undersample important parts of the scene, e.g. small and/or fast moving objects, or There is a possibility of oversampling areas that are not present, e.g. fixed plane walls.

A light curtain is a safety device that detects nearby obstacles (e.g., humans) and stops machine operation. They are used, for example, in garage doors and/or elevators to prevent doors from closing when an outstretched hand blocks the door, or on factory floors around dangerous machinery. be able to. Light curtains operate on the simple principle that an obstruction is detected if it blocks the line of sight between the source and the sensor. Although light curtains are simple, light curtain systems must be specifically customized for each machine and task, hindering their widespread use in vision and robotics.

Recently, this principle has been extended to use line sensors and sources to generate light curtains (i.e., woven surfaces) of arbitrary shapes. Here, an obstacle is detected when it intersects both the illumination plane generated from the line light source and the imaging plane captured by the line sensor. In some examples, the illumination plane and the imaging plane are rotated using steerable mirrors at different speeds to sweep out an arbitrary line curtain surface in 3D. Such triangulation light curtains can be highly flexible and can be useful in obstacle detection and avoidance for autonomous navigation. Such light curtains are described in related US patent application Ser. No. 16/470,885, entitled "Programmable Light Curtains," of which this application is a continuation-in-part.

Disclosed herein are methods and systems for dynamically adaptively sampling the depth of a scene using triangulation light curtain principles. This method directly detects the presence or absence of obstacles (or scene points) in a specified 3D line. These 3D lines may be sparsely, non-uniformly, or densely sampled in a specified area. Depth sampling can be changed in real time, allowing quick discovery of objects or detailed exploration of areas of interest.

These results were achieved using a novel prototype light curtain system based on a 2D rolling shutter camera with higher light efficiency, working range, and faster adaptation than prior art achievements, making it suitable for autonomous navigation and exploration. and make it widely useful.

A general framework for agile depth sensing for vision and robotics is developed herein. In some examples, the triangulation light curtain intersects the scene along 3D lines. Therefore, instead of capturing 3D data of a scene through a volume of interest (e.g., using LIDAR or Kinect), this framework provides flexibility over time of depth along 3D lines within a scene. Enables accurate sampling. Depth sampling detects the presence or absence of obstacles (or scene points) at these locations in real time without additional calculations.

Depth sampling can be sparse, non-uniform (including random), or dense in a given 3D surface. Sparse sampling can be used to adaptively increase the spatial density of depth in an application-specified region of interest. In some embodiments, sparse sampling may be used to adaptively increase the spatial density of depth in only the region of interest. Alternatively, objects in the scene can be quickly discovered by initial random sampling followed by adaptive sampling of depth. Depth sampling can be changed quickly over time depending on the task at hand.

This sensing framework has several advantages over depth sensing that uses scene-independent fixed acquisition strategies. First, it can capture small, thin, and fast-moving objects that are typically difficult to detect with low frequency, uniform angular resolution sensors like LIDAR and low spatial resolution sensors like Kinect®. Examples of such objects include thin wires, meshes, or balls thrown at high speed.

Fast and dense 3D capture of objects far away from the sensor (e.g. pedestrians) is possible with our system even when the objects are barely visible in the LIDAR point cloud, and the Enables better detection, recognition and tracking. This framework allows the robot to explore regions of interest based on an initial sparse depth estimate. By continuously and sparsely sampling the scene as the robot moves, it is possible to simultaneously detect obstacles and map the 3D scene.

To achieve these results, a novel design of a triangulation light curtain using 2D cameras and laser lines is disclosed. A rolling shutter of a 2D camera and a rotating laser line triangulate in a set of 3D lines in the scene forming a light curtain. By controlling the pixel clock and steering the source mirror, arbitrary textured surfaces can be generated as in US Pat. No. 16/470,885. However, the use of rapid 2D rolling shutter sensors offers significant advantages over line sensor designs. In some embodiments, the light curtain has a refresh rate of 60 fps, a 10x improvement over prior art designs, allowing the curtain to be quickly and adaptively changed. In other aspects, the described system is more light efficient and achieves similar range with less collected light because the optics in front of the 2D sensor is not limited by the size of the steering mirror. In yet other aspects, the described system has fewer moving parts and is more reliable. The described triangulation light curtain and depth sampling system operates at a range of up to 20-30 m outdoors and up to 50 m indoors. As an aspect of the described embodiments, a method of suppressing ambient lighting (including sunlight) is also proposed.

The described system enables real-time adaptation for agile depth-sensing tasks ranging from human-robot interaction to robot manipulation, path planning, and navigation. This system can be reconfigurable, which can have a major impact on robotics and manufacturing applications.

FIG. 3 is a top view schematically showing the planar parallel geometry of a triangulated light curtain using a 2D camera; FIG. 2 shows a design of a light curtain with a 2D camera. FIG. 4 is a diagram showing the results of a method of ambient light suppression. FIG. 3 illustrates that a captured light curtain can be imaged and modified up to 60 times per second to quickly scan different curtains through a volume. FIG. 2 shows the described device for capturing depth of small and fast objects. FIG. 3 illustrates adaptive depth imaging of a table and chair scene. FIG. 3 illustrates depth imaging of a scene using flat curtains, random curtains, and adaptive curtains. FIG. 3 is a diagram showing the result of driving a mobile robot through a high bay scene.

The light curtain can be imaged by steering a line imager equipped with a galvo mirror, but the nature of its design can limit the frame rate to 5.6 fps, and the galvo mirror rotates the field of view of the line sensor. may require the use of a small lens to fit. This may reduce the efficiency and range of the light. Both of these problems can be improved by using a 2D camera and a larger diameter lens to image the light curtain.

と所望のカーテン面の交差部分にある。そのポイントをイメージ化するために必要な投影された光平面

と角度θ_ｐ，ｉは、Ｘ_ｉとカメラ平面の回転軸上にある２つのポイント（例えば、（０，０，０）および（０，１，０））から平面を作成することにより、単純なジオメトリを通して見出され得る。カメラ平面が変化すると（細い線で示されているように）、必要な光平面が新しいポイントごとに算出される。必要なレーザ平面は、最初にＸ_ｉをプロジェクタのフレームに投影して：

を得ることによって同様の方法で見出されることができ、ここで、

は、カメラフレーム内のポイントを較正を通して見出されたプロジェクタフレームに変換する変換行列である。 Instead of continuous scanning of a single line sensor plane, a 2D camera can have a discrete set of imaging planes defined by optics and pixel arrays. In at least some designs of light curtains, intersections of pixels on these planes with the light curtain surface may be found. For design simplicity, in some embodiments, the pixel rays of a given camera column are coplanar, and the light sheet projector emits a true plane of light, with its axis of rotation aligned with the camera column. can be assumed to be parallel to These assumptions can generally be enforced by using low distortion lenses, careful design of light sheet projector optics, and accurate alignment of the projector axis to the camera column. With these assumptions, the light curtain can be designed in two dimensions by looking at the light rays on the XZ plane, as shown in FIG. Thereby, the 3D point X _i is the camera plane

and the desired curtain surface. the projected light plane needed to image that point

and the angle θ _p,i can be calculated simply by creating a plane from X _i and two points on the axis of rotation of the camera plane (e.g., (0,0,0) and (0,1,0)). can be found through the geometry of When the camera plane changes (as shown by the thin line), the required light plane is calculated for each new point. The required laser plane is created by first projecting X _i onto the projector frame:

can be found in a similar way by obtaining, where,

is a transformation matrix that transforms points in the camera frame to the projector frame found through calibration.

は、プロジェクタの回転軸上の２つのポイントとともに使用され、プロジェクタのフレーム内にある投影された光平面

を見出す。次に、カメラフレーム内のこの平面は、式（２）：

によって見出される。 Then the point

is used with two points on the projector's axis of rotation to define the projected light plane within the projector's frame.

Find out. This plane within the camera frame is then defined by equation (2):

found by.

を使用して算出される。 To image a curtain contour, the desired contour is first discretized into a series of m points uniformly distributed along the curtain contour, where m is approximately the number of camera columns. It is. Line segments are then formed between adjacent points. The rays representing each column are then checked for intersections with line segments to create a series of points X _i... X _n on the column rays that lie on the light curtain contour, as shown in Figure 2. produce. Some of the camera rays may not intersect the desired curtain contour, or the design points may be outside the field of view of the light sheet projector or camera. Any point outside the field of view of either module may be marked invalid and not used for light curtain imaging. If the design point is valid, it is converted to the light sheet projector frame using equation (1), and the galvo mirror angle required to create a light sheet passing through the design point is equation (3):

Calculated using

For an ideal system with true planes of illumination and imaging, a 3D point on the surface can be defined by the intersection of the two planes. However, in real systems, the lens may have a small amount of distortion, and there may be small alignment errors between the camera and the light sheet projector. In this way, the points of true intersection of pixels in a given column with the lightsheet plane may not be coplanar, and using known calculation methods of rayplane intersection, each It can be found by calculating the intersection of the pixel ray with the light sheet plane. This means that the actual 3D surface of the light curtain can vary from the designed contour across the camera's field of view, as shown in Figure 2. View (A) shows a light curtain designed with a 2D camera by finding the points at the intersection of the camera's rays with the desired curtain contour. This creates a set of unevenly spaced points on the curtain surface. Due to small optical distortions and alignment errors, the actual 3D contour of the curtain is not a perfect extrusion into 3D of the designed curtain, as shown by the difference in the shape of the curtain along each axis in view (B) . For example, note the varying depth of the curtain front in the z-axis. However, depending on the severity of distortion and misalignment, the curtain contour may be fairly consistent towards the center of the field and only vary significantly towards the edges of the field.

Fast Curtain Imaging with a Rolling Shutter Camera Light curtain imaging may require changing the imaging plane to intersect the light curtain plane as needed. For 2D cameras, this can be done by capturing the entire frame and using only given columns (slower) or by imaging only selected regions of interest on the imager (faster but still slow). It can be done. Although easy to implement, all of these methods may be too slow to enable agile light curtain imaging.

An even faster method of imaging a curtain may involve using a rolling shutter of a 2D CMOS imager to move the imaging plane. A rolling shutter can rapidly change the imaging plane at a uniform rate specified by the imager's pixel clock. This property enables full frame rate light curtain imaging of rolling shutter cameras. Imaging the curtain is as simple as commanding the light sheet projector to the required angle to project the light sheet at the point defined by the intersection of the 2D camera's active row and the light curtain contour. It can be. By synchronizing the movement of the rolling shutter with the movement of the light sheet, it is possible to visualize a light curtain that forms an arbitrary line-woven surface.

For each frame of the rolling shutter camera, an image of the captured curtain can be produced. While imaging the light curtain, the camera captures both laser light and ambient light. If the ambient light is low enough (ie, indoor imaging), the image from the camera can be directly thresholded to create a mask indicating the detected points. However, in many cases the captured ambient light is much louder than the captured laser light and the curtain cannot be detected (i.e. in outdoor sunlight). Narrowband filters significantly reduce captured ambient light, but for maximum performance it is desirable to detect the laser signal in as few bits as possible.

Ambient Subtraction To increase performance in the presence of ambient light, ambient light can be subtracted by capturing both the ambient image and the combined laser+ambient image at each light curtain position. In this example, the surrounding image is subtracted from the combined image to obtain an image of only the curtain lights. This allows for much greater peripheral performance, but requires imaging the same camera plane twice. This can be done with a 2D camera if the entire image or a selectable region of interest is imaged, but in some cases may not be done with a single rolling shutter frame capture. To solve this, the method of the present invention, in some embodiments, uses adjacent columns of the captured image to perform perimeter subtraction.

This method may sacrifice the resolution of the curtain, and captures the raw image where even columns are captured with the laser on and odd columns are captured with the laser off, as shown in Figure 3, to capture the ambient-only image and A combined image can be obtained. Separating and upsampling these columns forms a combined image with a full resolution surround-only image. Subtracting the ambient image from the combined image produces a laser-only image, which can then be thresholded to find the light curtain detection and provide a mask indicating the imaged detection points. can. Filtering this mask with a thin horizontal contraction/expansion filter provides improved robustness to large intensity gradients and upsampling artifacts.

In areas with high ambient light, this technique can still create errors at locations of high intensity gradients, as shown by the fuzzy edges of the laser image in FIG. 3 and the vertical thin lines of the thresholded mask image.

Depending on how well the rolling shutter movement is synchronized with the light sheet projector movement and timing, there may be a slight bleed-through of the laser light into neighboring columns that may appear in the surrounding image. Since this light can appear in the ambient image, it can be subtracted from the combined image and the measurement signal from the laser light can be reduced, thereby reducing the performance of the device. With precise synchronization, this amount of light can be limited to a few bits or removed entirely without significantly impacting system performance.

Limitations In some embodiments, the uniform sampling of the light curtain contour that dual galvo mirrors offer prior art designs trades off the imaging speed and enhanced light efficiency of using a rolling shutter camera. can be abandoned due to the discrete nature of camera pixels. This can lead to situations where there may be gaps in the curtain contour that the light curtain device cannot image.

This can occur when the camera rays are similar in the direction of the curtain plane. This effect is shown in the upper right part of the curtain profile in FIG. Another disadvantage of using a rolling shutter camera may be that in some examples each plane of the camera can only be imaged once in a given frame. If the camera plane intersects many curtain segments (e.g. in a zigzag), one of the segments can be chosen to be the imaged curtain, and sequential curtains can be imaged to points can be captured. One of the limitations of galvo mirrors is that if they are commanded to move very fast (over a 100Hz step function), there can be a lag in the commanded position, and the intersection of the planes may be different from the design. can be different points, which in turn causes an error in the location of the detected points. This may constrain which curtains can be imaged; in practice, random adaptive curtains need to be smoothed with splines before being imaged. This error can be addressed by using the angle measured from the mirror's closed-loop feedback to calculate the position of the detected point, rather than relying on open-loop control.

Hardware Prototype The hardware prototype of the system consists of a light sheet projector and a rolling shutter camera. The systems described are merely exemplary and the embodiments of the systems covered by the claims are not meant to be limited to the components described. The light sheet projector may include a custom designed line laser module using a 1W 830nm laser diode, which is then collimated with a 45° Powell lens and shaped into a line. This laser line is then projected onto and steered by the galvo mirror. The line laser module is mounted in alignment with the galvo mirror in the mount, allowing precise collinear alignment of the laser line with the rotation axis of the galvo mirror. The galvo mirror can have dimensions of 14.5 mm x 8.5 mm and can have an optical scan angle of 50°.

A rolling shutter camera can be fitted with a low distortion C-mount lens with a field of view of 70° (h) x 60° (v). This camera can provide 10 bits of precision and have a native resolution of 1280 x 1024 pixels with 5.3μ square m pixels. The camera can be operated in 2x binning mode at a resolution of 640x512 to increase the signal of the received light and reduce noise. A low distortion lens can be used to ensure that the intersection of the light sheet and camera beam along a given column shape is as close to a line as possible. A 12 nm bandpass filter centered at 830 nm can be placed between the lens and the image sensor to reduce the amount of ambient light collected. The camera can be aligned with the galvo mirror so that the row of the camera is parallel to the axis of rotation of the galvo mirror. The rotated camera can then be placed at a fixed baseline 200 mm from the axis of rotation of the galvo mirror. A microcontroller can be used to synchronize the camera, laser, and galvo mirror. By projecting the light curtain onto its view, a color 2D helper camera can be used to visualize the light curtain and detect results in the scene.

Alternative embodiments include CMOS (Complementary Metal Oxide Semiconductor), InGaAs (Indium Gallium Arsenide), SWIR (Short Wave Infrared), DVS (Dynamic Vision Sensor), ToF (Time of Flight) and CW-ToF (Continuous Wave Different types of sensors can be used, including but not limited to time-of-flight) type sensors.

Calibration Light curtain performance depends on accurate calibration of the camera and light sheet projector. First, the camera intrinsics can be determined. The external calibration of the light sheet projector and camera can then be determined by imaging a set of light planes projected onto a flat wall by the light sheet projector. A checkerboard target of known dimensions can be mounted on a wall and imaged with a calibrated camera to obtain known 3D coordinates of points on each imaged laser line. This can be repeated using the same set of planes at several depths to completely define each plane of light. The best-fitting plane equation can then be found for each set of points using weighted least squares, where the weights are the normalized intensity values. Then, given the equations for the planes, the location of the galvo mirror axis relative to the camera can be found by a least squares fit of the line to the intersection of all planes. A continuous function can be fitted to the relationship between the plane angle (relative to the galvo mirror axis) and the commanded light sheet projector angle, which can then be used to calculate the specified angle for any given design point. Determine the commanded galvo mirror position required.

Capture Process In prior art light curtain designs, the light sheet and camera plane angles were dictated by the position of the galvo mirror. However, for a 2D camera, the plane angle of the camera is defined by the optical system and the active line of pixels on the imager. For rolling shutter cameras, the active line of pixels is defined by the speed of the rolling shutter and the time from the start of the frame. The speed of the rolling shutter is determined by the pixel readout rate, known as the pixel clock. The maximum time a given line is active can be found by dividing the number of pixels on the line by the pixel clock. At the maximum 60 fps capture rate of the camera, the maximum active exposure time of a line is ≈15 microseconds.

To image the light curtain, the software causes the processor to calculate the required galvo mirror positions and send them to the microcontroller that controls the galvo mirrors. At the same time, the camera is triggered to start capturing frames. The software then sequentially commands the galvo mirror position and laser power in lockstep with the timed progression of the rolling shutter. This process is then repeated for each successive light curtain. Calculating the position of the galvo mirror on a frame-by-frame basis allows imaging a different light curtain on a frame-by-frame basis at the camera's full frame rate.

Working Range At some frame rates, the described device has a working range of 50 m while imaging a whiteboard in approximately 50 klx ambient light, and a working range of 20 m indoors. This has a reduced exposure time. When the device is configured for 100 microsecond exposures, it can image over 40 meters outdoors under similar conditions.

Agile and Dynamic Light Curtains The described device is capable of imaging 60 different light curtains per second. This speed and flexibility enables the use of agile and dynamic light curtains that can be used to intelligently and adaptively sample scenes. This functionality has applications in many areas, including path planning, high-resolution safety curtains, and depth sensing. Figure 4 shows the results of imaging various types of light curtains both indoors and outdoors, and is just a sample of the different types of curtains that can be imaged with the described device. The image shown in Figure 4 is the light curtain surface and detection projected onto the helper camera's view for scene visualization. The image shown in Figure 4 is from the 2D helper camera view, with the light curtain surface rendered in blue and the detection rendered in green. The agile nature of these curtains allows for quick checking of planned routes, as shown in view (A), as well as checking the boundaries of and areas within the safety zone, as shown in view (B). This rapid alternation enables many applications, such as detecting obstacles entering or within a safe zone.

High resolution, high speed light curtain:
The rapid capture rate and resolution of the described device allows for the imaging of small and fast objects as they pass through the light curtain, as shown in the views of FIG. 5 (A to C). Several composite images show a small ball with a diameter of 70 mm in view (A), a ball with a diameter of 30 mm thrown through a flat curtain 5 m away in view (B), and a view and detection of a Frisbee with a thickness of 30 mm and a diameter of 265 mm. It shows the location where it was done. Object detection is displayed in view (C). The resolution of the light curtain provides increased detail compared to scanning LIDAR devices and can enable enhanced object recognition and marginal detection of small objects (wires, branches, etc.). This is particularly noticeable when imaging thin structures or objects at a distance, as shown in the views (D to I) of FIG. 5. Views (D to F) show a planar curtain swept through a volume containing a thin wire fence to create the dense 3D point cloud in view (E). The light curtain (green points) in view (F) reconstructs the fence mesh with much higher resolution than the 16-beam Velodyne VLP-16 scanning LIDAR (white points) at a distance of 1.5 m. By imaging a plane curtain, the range and resolution of the light curtain makes it possible to create a high-resolution height map of objects within a 15m range outdoors, as shown in the views (G to I). do. At this range, the static VLP-16 scanning LIDAR only senses objects as a few points shown in white.

Adaptive Depth Imaging The ability to specify depth of interest at high rates enables intelligent depth imaging of the environment based on current knowledge of the scene. For example, when a device first enters a scene, it has no knowledge of the environment, but by quickly and randomly scanning volumes of interest with a light curtain, it can make a rough estimate of the location of objects in the scene. Values can be generated and one or more new curtains can be designed to image around these points of interest. This process is then rapidly repeated to form a precise map of the objects in the scene. Figure 6 shows adaptive depth imaging of a scene containing a table and chairs. Imaging is performed by first randomly sampling the volume of interest to find objects. We then use the detected points in front of these objects (indicated by red dots in the plot and white dots in the bottom row) to design a set of light curtains, which we image and refine to create more Discover many scenes. Over time, the curtain eventually discovers and images the wall behind the scene. Figure 6, view (A) shows a random curtain used to initialize the curtain by randomly sampling a volume of interest within 3 m of the light curtain. These curtains detected some objects and the device used them as design points to fit new curtains in front of the objects. A set of 10 curtains, a scaled version of this spline, was designed to cover the area immediately in front and behind the detected scene points in less than a few milliseconds. These new curtains were then imaged and used to refine and discover more scenes. By interleaving some random curtains with adaptive curtains, the device can keep checking for any changes in the scene and sample the rest of the scene at a lower resolution. The design process for the experiment in Figure 6 was to project all detected points onto the XZ plane and use the closest point within the angular domain as the design point for that domain. A set of design points was determined by dividing the entire plane into these uniformly angled angular regions.

Depth Imaging Comparison A comparison of depth imaging methods using different light curtain types was performed by capturing the same scene with each curtain type and comparing the coverage of the scene at a given time. Methods included planar sweeping, random sampling, and adaptive depth imaging. For plane sweeping, the plane was designed to fully image a 4.25 m scene at 1.0 with a depth resolution of 0.2 m. The adaptive curtain was tuned to sense the front side of the detected object in high resolution and the rest of the scene in low resolution. Figure 7 shows the results of this comparison. When initialized with 0.25 seconds of random detection, the nature of adaptive curtain discovery allows it to sense the proximity of detected objects, wasting time sensing empty areas in front of the scene. There's nothing to do. As a result, we were able to quickly cover interesting parts of the scene in less than 0.25 seconds. Given time, planar sweep curtains can provide complete coverage with high resolution, but with limited time, random and adaptive curtains can image more scenes in less time. For example, a planar sweep curtain could be configured to image the entire scene in 0.5 seconds, but the depth resolution would be 0.4 m, much lower than other methods. Figure 7 shows that the adaptive curtain intelligently covers more scenes in less time than sweeping a plane or random sampling of scenes. After initializing with random sampling, the adaptive curtain found interesting parts of the scene in 0.25 seconds and then continued to refine the map.

Discovery and Mapping Using Adaptive Light Curtains Adaptive light curtains can be used to discover and map scenes from a moving platform. As shown in Figure 8, a small robot with onboard localization was used to move a light curtain device through a cluttered high bay scene. As the robot progressed through the scene, an adaptive light curtain was continuously configured to discover the scene structure and image newly detected objects from each frame. Rather than using a fixed depth sensing strategy, the light curtain intelligently sampled the area around the detected object at high resolution, and sampled the rest of the scene with a random curtain at lower resolution. In the mapping experiment, a set of 5 uniquely random curtains were interleaved with each of the 10 adaptive curtains. Figure 8 shows some examples of curtains that fit objects in a scene. Note that the curtain, shown in blue in view (A), fits tightly around the objects in the scene. As we moved through the environment, the detected points on the curtain, shown in green in view (B), also mapped the scene. The white points on the 3D map indicate the design points of the set of adaptive curtains.

Many modifications and adaptations of the invention will suggest themselves to those skilled in the art to which the invention pertains. The example methods and systems disclosed herein should be construed as illustrative of the invention, rather than as a limitation thereof. The intended scope of the invention is defined by the following claims.

Claims (18)

a light source for creating a light sheet plane;
a line sensor having an adjustable region of interest for sensing a line in a sensing plane, the sensed line being defined by the intersection of a light sheet plane and a sensing plane;
a processor;
Software executed by a processor,
the ability to sample the scene along multiple non-uniform sensed lines within the scene;
the ability to locate one or more objects in a scene on one or more of a plurality of sensed lines;
increasing the density of the sensed lines in an area of the scene containing the one or more discovered objects to create one or more adaptive curtains of the sensed lines; , a system,
A system in which multiple non-uniform sensed lines are randomly distributed in the scene. The light source is
laser diode and
collimation lens,
Powell type lens,
including a steerable galvo mirror;
2. The system of claim 1, wherein the light emitted by the laser diode is formed into a line of light by a Powell type lens before impinging on the steerable galvo mirror. The system of claim 1, wherein the line sensor includes a camera with a rolling 2D shutter. 2. The system of claim 1, wherein locating one or more objects includes detecting variations in light detected by a line sensor that indicate the presence of an object intersecting a sensed line. The discovery of one or more objects
5. The system of claim 4, further comprising determining the position and depth of each of the one or more discovered objects using triangulation based on the positions of the light sheet plane and the sensing plane. The system of claim 1, wherein the plurality of non-uniform sensed lines are sparsely distributed within the scene. 3. The system of claim 2, wherein the camera has a pixel array that includes a plurality of columns of pixels, and further, the rotational axis of the galvo mirror is parallel to the plurality of columns of pixels. The sensed lines of increased density in the area of the scene containing the one or more discovered objects define a light curtain contour, the light curtain contour comprising:
discretizing the light curtain contour by defining a plurality of uniformly distributed points along the contour;
for each point, calculating the angle of the galvo mirror to intersect the light sheet plane with the point;
8. The system of claim 7, imaged by capturing light reflected from an object intersecting a line sensed at each defined point. 9. The system of claim 8, wherein each defined point corresponds to a single column within the camera's pixel array. Capturing the light reflected from the line sensed at each point
10. The method of claim 9, further comprising calculating, for each pixel of a single column of pixels corresponding to a defined point currently illuminated by the light sheet plane, the intersection of the pixel ray with the light sheet plane. system. For each line sensed,
capturing a first image containing only ambient light;
capturing a second image that includes both ambient light and illumination from the light source;
9. The system of claim 8, further comprising subtracting the first image from the second image. The software also tells the processor
Calculate the required galvo mirror position and send it to the microcontroller that controls the galvo mirror,
Trigger the camera to start frame capture,
Sequentially commands the position of the galvo mirror in lockstep with the timed progression of the camera's rolling shutter;
The system according to claim 8. The software also tells the processor
13. The system of claim 12, wherein the system controls a laser photodiode with a desired laser power setting. The system of claim 1, wherein one or more adaptive light curtains may be created to track one or more objects in a scene while sampling the scene with non-uniform and sparse sensed lines. . sampling the scene along a plurality of non-uniform sensed lines in the scene, each sensed line being defined as the intersection of a light sheet plane and a sensing plane;
locating one or more objects in the scene at one or more of the plurality of sensed lines;
increasing the density of sensed lines in an area of the scene containing one or more discovered objects to create one or more adaptive curtains of sensed lines, each adaptive curtain comprising: A computer-implemented method comprising: being configured to sense one or more of the discovered objects, the method comprising :
A method in which multiple non-uniform sensed lines are sparsely and randomly distributed in the scene. The discovery of one or more objects
16. The method of claim 15, comprising determining the position and depth of each of the one or more discovered objects using triangulation based on the positions of the light sheet plane and the sensing plane. For each line sensed,
capturing a first image containing only ambient light;
capturing a second image that includes both ambient light and illumination from the light source;
16. The method of claim 15, further comprising subtracting the first image from the second image. 16. The method of claim 15, wherein one or more adaptive light curtains may be created to track one or more objects in a scene while sampling the scene with non-uniform and sparse sensed lines. .

Images