US20200150273A1 - Estimation of motion using lidar - Google Patents
Estimation of motion using lidar Download PDFInfo
- Publication number
- US20200150273A1 US20200150273A1 US16/741,475 US202016741475A US2020150273A1 US 20200150273 A1 US20200150273 A1 US 20200150273A1 US 202016741475 A US202016741475 A US 202016741475A US 2020150273 A1 US2020150273 A1 US 2020150273A1
- Authority
- US
- United States
- Prior art keywords
- lidar system
- scan
- velocity
- time
- beams
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/50—Systems of measurement based on relative movement of target
- G01S17/58—Velocity or trajectory determination systems; Sense-of-movement determination systems
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/42—Simultaneous measurement of distance and other co-ordinates
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/66—Tracking systems using electromagnetic waves other than radio waves
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4808—Evaluating distance, position or velocity data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/481—Constructional features, e.g. arrangements of optical elements
- G01S7/4814—Constructional features, e.g. arrangements of optical elements of transmitters alone
- G01S7/4815—Constructional features, e.g. arrangements of optical elements of transmitters alone using multiple transmitters
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/481—Constructional features, e.g. arrangements of optical elements
- G01S7/4817—Constructional features, e.g. arrangements of optical elements relating to scanning
Definitions
- This description relates to systems and methods for estimation of motion using Light Detection And Ranging (LIDAR).
- LIDAR Light Detection And Ranging
- objects may be tracked using a laser Light Detection And Ranging (LIDAR) system in conjunction with a video system.
- LIDAR Laser Light Detection And Ranging
- Some such known systems may be complex and difficult to use. Additionally, in some such known systems, the video system may require light in order to detect the object to be tracked.
- FIG. 1 is a block diagram illustrating an example LIDAR system within an electronic environment in which improved techniques described herein may be performed.
- FIG. 2A is a diagram illustrating an example object being tracked within the electronic environment illustrated in FIG. 1 .
- FIG. 2B is a diagram illustrating the example object as tracked within the electronic environment illustrated in FIG. 1 .
- FIG. 2C is a diagram illustrating another example object being tracked within the electronic environment illustrated in FIG. 1 .
- FIG. 2D is a diagram illustrating the other example object as tracked within the electronic environment illustrated in FIG. 1 .
- FIG. 2E is a diagram illustrating the other example object as further tracked within the electronic environment illustrated in FIG. 1 .
- FIG. 3 is a flowchart illustrating an example method for calculating object velocities and rotational velocity using a single set of scans.
- FIG. 4 is a diagram illustrating an example scan of an object over time within the electronic environment illustrated in FIG. 1 .
- FIG. 5 is a diagram illustrating an example sampling of object displacements from the scan illustrated in FIG. 4 .
- FIG. 6 is a diagram illustrating an example interpolation of the object displacements from the sampling illustrated in FIG. 5 .
- FIG. 7 is a flowchart illustrating an example method of determining initial estimates of object velocity and rotational velocity within the electronic environment illustrated in FIG. 1 .
- FIG. 8 is a flowchart illustrating an example method of refining the initial estimates of object velocity and rotational velocity determined by the process illustrated in FIG. 7 .
- FIG. 1 is a diagram that illustrates an example electronic environment 100 in which improved techniques of tracking an object's motion are performed.
- the electronic environment 100 includes a LIDAR system 120 that is configured to track an object 110 over a period of time in which the object 110 is moving using a single scanning motion.
- LIDAR system 120 tracking of the object 110 can be performed while eliminating visible imaging hardware (e.g., video camera hardware). Accordingly, the LIDAR system 120 can be configured to operate in total darkness, into the sun, etc.
- the LIDAR system 120 can be less susceptible to motion of the object 110 than conventional systems. Accordingly, the motion of the object 110 can be determined in some implementations solely from LIDAR measurements, without, for example, video.
- the object 110 is assumed herein to be a rigid body of some unknown shape.
- the object 110 may be a human face.
- the object 110 is assumed to be in motion, both linear and rotational, about an arbitrary axis.
- the time-dependent linear velocity v and time-dependent rotational velocity ⁇ at a point r on the object 110 are related as follows:
- x denotes a cross-product
- v ⁇ is the velocity of a reference frame that contains the origin.
- the reference frame is taken to be the inertial frame of the LIDAR system 120 although this is by no means a requirement.
- the LIDAR system 120 is a single, integrated unit that includes processing circuitry 124 , memory 126 , an illumination system 150 , and a receiver system 160 .
- the LIDAR system 120 takes the form of a handheld unit that may be pointed at the object 110 .
- the components of the LIDAR system 120 may be distributed among different units (e.g., the processing circuitry 124 and memory 126 might be in a computing device separate from a handheld device that includes the illumination system 150 and the receiver system 160 ).
- the processing circuitry 124 includes one or more processing chips and/or assemblies.
- the memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid state drives, and the like.
- volatile memory e.g., RAM
- non-volatile memory such as one or more ROMs, disk drives, solid state drives, and the like.
- the set of processing units 124 and the memory 126 together form control circuitry, which is configured and arranged to carry out various methods and functions as described herein.
- one or more of the components of the LIDAR system 120 can be, or can include, processors configured to process instructions stored in the memory 126 .
- processors configured to process instructions stored in the memory 126 .
- an analysis manager 130 (and/or a portion thereof), shown as being included within the memory 126 in FIG. 1 , can be a combination of a processor and a memory configured to execute instructions related to a process to implement one or more functions.
- the analysis manager 130 is configured and arranged to produce estimates of the linear velocity components v x , v y and rotational velocity ⁇ z of the object 110 over the period of time in which the object 110 is tracked.
- the analysis manager 130 is configured to perform such an estimate in two stages. First, at a given time, the analysis manager 130 , via an initial movement analyzer 132 , performs an initial estimate of the lateral motion of the object 110 using a pair of displacement datasets, e.g., 140 ( 1 ) and 140 ( 2 ) centered about that time. Then the analysis manager 130 , via a movement refinement manager 134 , refines the initial estimate using actual data collection times and, if necessary, subsequent displacement datasets, e.g., 140 ( 3 ) and so on.
- the initial movement analyzer 132 is configured to produce initial velocity data 136 , i.e., v x , v y , ⁇ z from displacement datasets 140 ( 1 ) and 140 ( 2 ). However, it should be appreciated that these initial data 136 are first estimates and may be refined using actual data collection times and, if necessary, subsequent displacement datasets, e.g., 140 ( 3 ) and so on.
- the movement refinement manager 134 is configured to compute refinements 138 , i.e., ⁇ v x , ⁇ v y , ⁇ z from the actual data collection times and, in some cases, the subsequent displacement datasets. Further, the movement refinement manager 134 is configured to determine when sufficient convergence has been achieved so that no more refinement computations are necessary.
- Each of the displacement datasets includes an array of displacement values in the z-direction as a function of the Cartesian coordinates x and y at a respective time.
- the object 110 as shown in FIG. 1 has a flat surface facing the LIDAR system 120
- the difference between the displacement profiles from one time to another is a combination of translations ⁇ x, ⁇ y and rotations ⁇ z (i.e., no deformations).
- the analysis manager 130 finds the velocities and rotational velocities from these translations and rotations.
- the illumination system 150 is configured and arranged to produce the illumination that is reflected from the object 110 in the act of generating the displacement datasets 140 ( 1 ), . . . , 140 (T). As shown in FIG. 1 , this illumination takes the form of multiple beams 190 ( 1 ), . . . , 190 (N) of radiation directed along the z-axis. It should be appreciated that, although the beams 190 ( 1 ), . . . , 190 (N) as shown in FIG. 1 are directed at an angle with respect to the z-axis, in reality the beams travel substantially along the z-axis and the angles in the figure are exaggerated.
- the illumination system 150 includes a scanning mechanism 152 , which includes a laser array 154 , and an aperture 170 .
- the scanning mechanism 152 is configured and arranged to move the laser array 154 in a scanning motion. As shown in FIG. 1 , the scanning mechanism 152 is configured to move each laser in the laser array 154 back and forth along they direction, i.e., orthogonally to the direction of the beams 190 ( 1 ), . . . , 190 (N). However, in some arrangements the scanning may be performed along the x-direction instead or in addition. The scanning mechanism 152 moves the laser array 154 altogether, so that all scans are performed in one scanning motion. Further, it should be appreciated that each individual scan in one direction is performed at a sufficient speed so as to be considered as having taken place in a single time instant.
- a first scan of the laser array 154 in the positive y-direction might be considered to have taken place at an instant of time T 0 , even though data from the first point in the scan and the past point in the scan are not received simultaneously at the LIDAR system 120 .
- the laser array 154 is configured and arranged to produce an array of beams (e.g., beams 190 ( 1 ), . . . , 190 (N)) of laser radiation, i.e., substantially coherent, quasi-monochromatic light.
- the laser array 154 includes a rectangular array of lasers, each producing laser radiation at some wavelength. Each laser in the rectangular array corresponds to a sample point on the object 110 where the beam produced by that laser reflects off the object 110 .
- the array of beams 190 ( 1 ), . . . , 190 (N) is generated using a single laser (not shown) that is fiber coupled and split into multiple fibers (not shown) which are then arranged into a fiber-emitter array.
- the wavelength of the light in each beam 190 ( 1 ), . . . , 190 (N) produced by the laser array 154 is 1550 nm. This wavelength has the advantage of being suited to objects that are, for example, human faces. Nevertheless, other wavelengths (e.g., 1064 nm, 532 nm) may be used as well.
- the aperture 170 is configured to pass the beams 190 ( 1 ), . . . , 190 (N) generated by the laser array 154 out of the LIDAR system 120 to the object 110 .
- the aperture 170 may simply be an opening to let the beams 190 ( 1 ), . . . , 190 (N) pass through.
- the aperture 170 may include optics (not shown) for satisfying certain specifications (e.g., spot size on object).
- the receiver system 160 is configured and arranged to receive the beams reflected from the object 110 and generate the displacement datasets 140 ( 1 ), . . . , 140 (T) from the received beams.
- the receiver system 160 may generate the displacement datasets 140 ( 1 ), . . . , 140 (T) using any number of known techniques (e.g., heterodyne detection) and will not be discussed further.
- the receiver system includes a detector 180 that is configured and arranged to convert the received beams into electrical signals from which the receiver system 160 may generate the displacement datasets 140 ( 1 ), . . . , 140 (T).
- the detector 180 includes a photomultiplier tube (PMT) or an array of charge-coupled devices (CCDs).
- FIGS. 2A and 2B illustrate an example object 210 that may be observed by (e.g., targeted by) the LIDAR system 120 .
- the object 210 may have any shape, but is represented in FIGS. 2A and 2B as a circle.
- FIG. 2A at time T 1 a point 220 on the object 210 is being observed by the LIDAR system 120 .
- the point 220 is located at (3,3) in the (x,y) plane.
- the point 220 is located at (4,3) in the (x,y) plane.
- the movement of the point may be the result of different types of movements of the object 80 .
- the object 220 may have moved from one location to another (translational movement) or the object 220 may have rotated (for example, about an axis parallel to the y axis of the x-y plane).
- a head or face 290 of an individual may be tracked or observed by the LIDAR system 120 .
- a point or location 292 of the head or face 290 may be observed.
- the point 292 is located at (3,2) in the (x,y) plane.
- the point 292 may be observed to be at (4,2).
- the movement of the point may be the result of different types of motion.
- the person or individual may have rotated their head (for example, about an axis parallel to they axis), as illustrated in FIG. 2D .
- the person or individual may have moved their head (without any rotation), as illustrated in FIG. 2E .
- FIG. 3 illustrates an example method 300 of performing the improved technique described herein.
- the method 300 may be performed by constructs described in connection with FIG. 1 , which can reside in memory 126 of the LIDAR system 120 and can be executed by the processing circuitry 124 .
- a LIDAR system defines a first dataset based on a first scan of an object using beams of electromagnetic radiation emitted from a scanning mechanism. For example, the LIDAR system 120 , via scanning mechanism 152 , scans beams of light 190 ( 1 ), . . . , 190 (N) over the object 110 at a point in time. The LIDAR system 120 , via receiver system 160 , receives the beams 190 ( 1 ), . . . , 190 (N) and generates the displacement dataset 140 ( 1 ) from the received beams.
- the LIDAR system defines a second dataset based on a second scan of the object using the beams of electromagnetic radiation emitted from the scanning mechanism.
- the LIDAR system 120 via scanning mechanism 152 , scans beams of light 190 ( 1 ), . . . , 190 (N) over the object 110 at a second point in time.
- the LIDAR system 120 via receiver system 160 , receives the beams 190 ( 1 ), . . . , 190 (N) and generates the displacement dataset 140 ( 2 ) from the received beams.
- the scan performed by the LIDAR system 120 at the second point in time may be the next scan, i.e., the reverse scan to the first scan.
- the scan performed at the second point in time is the next scan in the same direction as the first scan.
- such a situation may be preferable because there may be a better correlation between the displacement datasets when the first and second scans are performed in the same direction.
- the LIDAR system generates initial values of a first velocity (e.g., v x ), a second velocity (e.g., v y ), and a rotational velocity (e.g., ⁇ z ) from the first dataset and the second dataset.
- a first velocity e.g., v x
- a second velocity e.g., v y
- a rotational velocity e.g., ⁇ z
- the LIDAR system defines a third dataset based on a third scan of the object using the beams of electromagnetic radiation emitted from the scanning mechanism.
- the LIDAR system 120 via scanning mechanism 152 , scans beams of light 190 ( 1 ), . . . , 190 (N) over the object 110 at a third point in time.
- the LIDAR system 120 via receiver system 160 , receives the beams 190 ( 1 ), . . . , 190 (N) and generates the displacement dataset 140 ( 3 ) from the received beams.
- the third scan may either be the scan following the second scan or the next scan in the same direction as the second scan but is preferred to be in the same direction.
- the LIDAR system produces refined values of the first velocity, second velocity, and rotational velocity based on the initial values of the first velocity, second velocity, and rotational velocity and actual data collection times and, if necessary, the third dataset. Again, the refinements are performed for the velocities and rotational velocity at the time in between the first scan and second scan. Details of how the LIDAR system 120 generates these refined values are discussed in connection with at least FIG. 8 .
- FIG. 4 illustrates a top-down view of an example scan geometry using the LIDAR system 120 resulting from an example laser array 154 .
- the laser array 154 contains 16 lasers arranged in a two-by-eight array.
- This configuration of the laser array 154 as shown in FIG. 4 is only an example and many configurations using any number of lasers (e.g., 2X4, 2X6, 3X4, 2X6, 2X8, 3X8, 4X4, and so on) may be used.
- the scanning mechanism 152 is configured to move the laser array 154 as a whole during a scan. Each laser in the laser array 154 thus scans the object 110 synchronously and in the same direction.
- the scan path traced out by the scanning mechanism 152 may be any pattern but for simplicity is taken to be along a straight line along one axis (the y-axis as shown in FIG. 4 ).
- Such a linear scan pattern performed by the scanning mechanism 152 produces the scan lines 420 ( 1 ) over the object 110 shown in FIG. 4 . It should be appreciated that, as a first approximation, the scans producing scan lines 420 ( 1 ) are considered to have taken place at a single time instant T 0 even though the scan time across the object 110 is finite. In refining the first approximation, a finite scan time may be taken into account.
- each scan line 420 ( 1 ) is actually two overlapping scan lines. In some implementations, this overlap occurs because the extent of a scan performed by the scanning mechanism 152 is longer than the distance between the lasers in each column of the laser array 154 .
- the scan lines of the next scan appear on a different location on the object 110 .
- the scan lines 420 ( 2 ) are produced from scans at a time T 2 , i.e., the time of the next scan in the same direction.
- the scan lines 420 ( 2 ) are shown in FIG. 4 as dashed to distinguish over the scan lines 420 ( 1 ). Again, each scan line 420 ( 2 ) is actually two overlapping scan lines.
- FIG. 5 illustrates a data sampling 500 based on the scanlines shown in FIG. 4 .
- the data sampling 500 represents what is stored in the memory 126 after the beams 190 ( 1 ), . . . , 190 (N) have been received at the detector and before the displacement datasets 140 ( 1 ), . . . , 140 (T) have been generated.
- the sample points depicted in FIG. 5 are in the reference frame of the object and not that of the LIDAR system 120 .
- Each small circle represents a displacement of the surface of the object 110 in the z-direction at the time T 0 , i.e., during the first scan in the positive y-direction.
- Each small square represents a displacement of the surface of the object 110 in the z-direction at the time T 2 , i.e., during the second scan in the positive y-direction.
- the filling of each circle or square represents the beam produced by the first or second laser in each column of the laser array 154 . It should be appreciated that there is overlap between the two beams along the center of each scanline and this is represented by cross-hatching in FIG. 5 .
- the extents of the first scan and the second scans differ.
- the first scanlines may occupy a range between 10 cm and 200 cm along the y-direction
- the second scanlines may occupy a range between 0 cm and 190 cm along the y-direction.
- the reference frame used to define the origin of the coordinate system is the inertial frame of the LIDAR system 120 .
- the sampling along the x-direction is typically very sparse. While the number of samples taken along the y-direction may be as large or small as necessary, the number of samples along the x-direction is limited by the number of lasers in a row of the laser array 154 . In the case shown in FIG. 5 , while there are a few dozen samples and there may be hundreds of such samples along the scan direction, there may be only eight samples normal to the scan direction.
- the gathering of the data represented in FIG. 5 can be used to compute an initial estimate of the lateral linear velocities v x , v y and the lateral rotational velocity ⁇ z . This is achieved by computing an estimate of the translations ⁇ x, ⁇ y and rotation ⁇ z between the dataset at time T 0 and the dataset at time T 2 . It should be appreciated that, although only lateral motion is considered here, there still may be some axial displacement between time T 0 and time T 2 . However, this displacement may be removed, or compensated for, using the LIDAR range and velocity data from T 0 , T 1 and T 2 .
- the raw data are too sparsely sampled to provide a meaningful covering of the object.
- FIG. 6 illustrates an example of such an interpolation scheme 610 .
- FIG. 6 shows two columns of a scanline from FIG. 5 with a triangular grid drawn between the sample points of the columns. Thus, each sample point is a vertex of two triangles in the grid. Within each triangular area, one may define a respective linear interpolation function using, e.g., barycentric or rectangular coordinates.
- the interpolation scheme 610 can be used to define interpolated object displacement as a continuous function of coordinates within a triangular region. In this way, finding the translations and rotation can be achieved. For example, one may determine a correlation between the two datasets at T 0 and T 2 as a function of translation and/or rotation. The initial estimated translations and rotation may then be those values that maximize the correlation between the datasets.
- maximizing the correlation between the datasets in one implementation means minimizing a sum of squared differences between interpolated object displacement values over translation and/or rotation. Further details are discussed with reference to FIG. 7 .
- FIG. 7 illustrates further detail of element 306 of method 300 shown in FIG. 3 , in which the LIDAR system 120 generates initial values of a first velocity (e.g., v x ), a second velocity (e.g., v y ), and a rotational velocity (e.g., ⁇ z ) from the first dataset and the second dataset.
- a first velocity e.g., v x
- a second velocity e.g., v y
- a rotational velocity e.g., ⁇ z
- a LIDAR system 120 acquires a dense sampling of object displacements at time T 0 via interpolation.
- a dense sampling is a sampling of object displacements a factor of two or more dense than that of the original datasets, e.g., if a raw data set has 200 ⁇ 8 data points, then the dense sampling acquired via interpolation may be 800 ⁇ 800 or higher.
- the LIDAR system 120 shown in FIG. 1 performs a linear interpolation as described with reference to FIG. using a manhattan grid on displacement dataset 140 ( 1 ).
- a result of this interpolation is a dense sampling of object displacements over a rectangular grid for the dataset at time T 0 . It has been found that the interpolation over the manhattan grid as depicted in FIG. 6 results in faster computation time than schemes that use a nearest neighbor approximation.
- the LIDAR system 120 then forms a dense sampling of object displacements at time T 2 (e.g., displacement dataset 140 ( 2 )) to create an interpolated dataset that is a function of object displacements ⁇ x, ⁇ y.
- the LIDAR system 120 uses the interpolation function defined by the second dataset, the LIDAR system 120 computes interpolated object displacements corresponding to each interpolated sample point at time T 0 but shifted by translations ⁇ x, ⁇ y in each respective direction. In some arrangements, the shifting may be done in only one direction.
- the LIDAR system 120 forms the function ⁇ 2 (x i + ⁇ x,y j + ⁇ y), which is considered a function of ⁇ x and ⁇ y.
- the LIDAR system 120 determines values of ⁇ x and ⁇ y that optimize a correlation between the interpolated dataset at T 0 and the interpolated dataset at T 2 .
- optimizing a correlation involves minimizing, or substantially reducing, a sum over squared differences between the displacements at TO and the translated displacements at T 2 .
- the LIDAR system 120 may find the values of ⁇ x and ⁇ y that minimize the following quantity:
- K and L are the number of interpolated sample points in the x and y-direction, respectively. It should be appreciated that the interpolated sample points x i and y j are not necessarily on a rectangular grid.
- the LIDAR system 120 may use any method known in the art to perform the minimization of E trans ( ⁇ X, ⁇ y). The values that minimize this function are denoted ( ⁇ x 0 , ⁇ y 0 ) and represent the initial estimates of the translation of the object at time T 1 between time T 0 and time T 2 .
- the LIDAR system 120 forms another interpolation function from the dataset at time T 2 based on rotation of a sample point through an angle ⁇ z . It should be appreciated that the initial estimate of the rotation is determined independently from the initial estimate of the translation.
- the LIDAR system 120 determines the value of the rotation ⁇ z that maximizes the correlation between the interpolated dataset at T 0 and the interpolated dataset at T 2 .
- maximizing a correlation involves minimizing a sum over squared differences between the displacements at T 0 and the translated displacements at T 2 .
- the LIDAR system 120 may find the value of ⁇ z that minimizes the following quantity:
- E rot ( ⁇ z ) is evaluated at discreet values of ⁇ z close to zero.
- the LIDAR system 120 may use any method known in the art to perform the minimization of E rot ( ⁇ z ).
- the value that minimizes this function is denoted as ⁇ Z0 and represents the initial estimates of the rotation of the object at time T 1 between time T 0 and time T 2 .
- the LIDAR system 120 determines the initial estimates at the time T 1 of the velocities v x , v y and rotational velocity ⁇ z based on the values ⁇ x 0 ⁇ y 0 , and ⁇ z0 .
- FIG. 8 illustrates example detail of element 310 of method 300 shown in FIG. 3 , in which the LIDAR system 120 produces refined values of the first velocity, second velocity, and rotational velocity at time T 1 based on initial estimates of the first velocity, second velocity, and rotational velocity.
- the LIDAR system 120 also uses subsequent datasets and their respective velocity estimates to further refine these values of the first velocity, second velocity, and rotational velocity.
- the LIDAR system 120 determines the actual positions of the sample points of the scan at T 0 and T 2 based on a scan speed.
- the LIDAR system 120 performs the correction of the position of the sample points in the scan according to the time of the scan and based on the initial estimates of the velocities v x , v y and rotational velocity ⁇ z at time T 1 .
- This latter position is the refined position of that sample point. This process of finding the refined positions of sample points continues for all sample points in the T 0 and T 2 scans.
- the LIDAR system 120 repeats 702 , 704 , and 706 from FIG. 7 .
- the LIDAR system 120 finds translations and a rotation that maximizes a correlation between the interpolated object displacements at time T 0 and time T 2 .
- the LIDAR system 120 minimizes or at least substantially reduces a sum of squared differences between the object displacements at various sample points in the xy-plane.
- the result of the minimization of the sum, i.e., the optimization of the correlation is a specification of new translations ⁇ x 1 , ⁇ y 1 , and rotation ⁇ z1 .
- the LIDAR system 120 determines refinements ⁇ v x , ⁇ v y , and ⁇ z to the linear and rotational velocities at time T 1 from the new translations and rotation.
- the LIDAR system 120 determines whether these refinements to the initial estimates of the velocities v x , v y , and rotational velocity ⁇ z are smaller than a threshold.
- the process 310 has converged and the motion of the object 110 at time T 0 has been determined.
- the LIDAR system 120 may determine the motion of the object 110 at subsequent (odd-numbered) times by using subsequent datasets taken at even times (e.g., T 4 , T 6 , etc.).
- subsequent datasets taken at even times e.g., T 4 , T 6 , etc.
- the LIDAR system 120 has generated refined velocities v x , v y , and rotational velocity ⁇ z at time T 3 based on refined positions of object sample points at times T 2 and T 4 . Then the velocity may be considered to be a function of time.
- This time-dependent model for each velocity will produce new adjusted values of the velocities at T 0 and T 2 . From these new adjusted values of the velocities, the LIDAR system 120 may determine further refined positions of the sample points, from which new refinements to the velocities at T 1 may be determined.
- the LIDAR system 120 uses scan data at T 6 to determine velocities at T 5 , from which a quadratic velocity model is generated, and so on until the refinements to the velocity are less than the threshold. It has been found that the above process converges rapidly.
- the time-dependent velocities used for further refinement need not be a simple linear interpolation of the velocities in time.
- any more accurate interpolation of the velocity may be used, such as cubic spline, when more than two velocity estimates are known.
- the components (e.g., modules, processors (e.g., a processor defined within a substrate such as a silicon substrate)) of the LIDAR system 120 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, and/or so forth.
- the components of the LIDAR system 120 can be configured to operate within a cluster of devices (e.g., a server farm).
- the second scan referred to in FIG. 3 may be performed in the opposite direction of the first scan.
- the velocities and rotational velocity so determined are evaluated, as before, at a time halfway between the first scan and second scan.
- one or more portions of the components shown in the LIDAR system 120 in FIG. 1 can be, or can include, a hardware-based module (e.g., a digital signal processor (DSP), a field programmable gate array (FPGA), a memory), a firmware module, and/or a software-based module (e.g., a module of computer code, a set of computer-readable instructions that can be executed at a computer).
- a hardware-based module e.g., a digital signal processor (DSP), a field programmable gate array (FPGA), a memory
- firmware module e.g., a firmware module
- a software-based module e.g., a module of computer code, a set of computer-readable instructions that can be executed at a computer.
- a software-based module e.g., a module of computer code, a set of computer-readable instructions that can be executed at a computer.
- one or more portions of the LIDAR system 120 can be, or can include
- one or more of the components of the LIDAR system 120 can be, or can include, processors configured to process instructions stored in a memory.
- the analysis manager 130 (and/or a portion thereof) can be a combination of a processor and a memory configured to execute instructions related to a process to implement one or more functions.
- the components of the LIDAR system 120 can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server/host devices, and/or so forth.
- the components of the LIDAR system 120 can be configured to operate within a network.
- the LIDAR system 120 can be configured to function within various types of network environments that can include one or more devices and/or one or more server devices.
- the network can be, or can include, a local area network (LAN), a wide area network (WAN), and/or so forth.
- the network can be, or can include, a wireless network and/or wireless network implemented using, for example, gateway devices, bridges, switches, and/or so forth.
- the network can include one or more segments and/or can have portions based on various protocols such as Internet Protocol (IP) and/or a proprietary protocol.
- IP Internet Protocol
- the network can include at least a portion of the Internet.
- the LIDAR system 120 may include a memory.
- the memory can be any type of memory such as a random-access memory, a disk drive memory, flash memory, and/or so forth.
- the memory can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with the components of the LIDAR system 120 .
- a LIDAR system includes a laser system that includes lasers or laser beams that are configured to move in a pattern or patterns with respect to the object that is being tracked.
- the scanning mechanism 152 of the LIDAR system 120 includes a plurality of lasers or beams that are configured to move in a pattern or patterns with respect to the object being tracked.
- the LIDAR system 120 may have one mode in which the laser beams are fixed or stationary and a second mode in which the laser beams move in a pattern or patterns such as a shape.
- a second mode in which the laser beams move in a pattern or patterns such as a shape.
- two or more of the laser beams move in a pattern or patterns when the LIDAR system 120 is in the second mode.
- different laser beams may move independently in different patterns.
- the LIDAR system 120 includes some lasers or produces some laser beams that are stationary and some that are configured to move in a pattern (or patterns) or shape.
- the lasers or beams can move in any pattern or shape.
- the lasers or beams are configured to move in elliptical shape.
- the lasers or beams are configured to move in a line, circle, square, rectangle, triangle, or any other shape.
- the shape or pattern that the lasers or beams move in are dictated or determined by the object that is being tracked.
- the pattern or shape of the laser movement may be similar to the shape of the object that is being tracked.
- an elliptical shape or pattern may be used when tracking a face of an individual as the face of an individual is generally elliptical in shape.
- Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (computer-readable medium, a non-transitory computer-readable storage medium, a tangible computer-readable storage medium) or in a propagated signal, for processing by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.
- a computer program product i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (computer-readable medium, a non-transitory computer-readable storage medium, a tangible computer-readable storage medium) or in a propagated signal, for processing by, or to control the operation of, data processing apparatus, e.g., a
- a computer program such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
- a computer program can be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
- Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- FPGA field programmable gate array
- ASIC application-specific integrated circuit
- processors suitable for the processing of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
- a processor will receive instructions and data from a read-only memory or a random access memory or both.
- Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data.
- a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
- Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
- semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
- magnetic disks e.g., internal hard disks or removable disks
- magneto-optical disks e.g., CD-ROM and DVD-ROM disks.
- the processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.
- implementations may be implemented on a computer having a display device, e.g., a liquid crystal display (LCD or LED) monitor, a touchscreen display, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
- a display device e.g., a liquid crystal display (LCD or LED) monitor, a touchscreen display
- a keyboard and a pointing device e.g., a mouse or a trackball
- Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end, middleware, or front-end components.
- Components may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
- LAN local area network
- WAN wide area network
- the LIDAR system 120 may achieve millimeter range accuracy performance off moving faces of a subject or individual.
- solid object velocity estimation requires processing of multiples samples in order to remove significant velocity components from speech and other biological components.
- the amplitude of the vibration is an insignificant range change for the process of tracking faces of a subject or individual, the instantaneous velocity may be significant and the vibrational velocity may be removed.
- removing vibrational velocity may require processing a velocity data sample significantly longer than the periods of the vibrations to be removed and care to avoid noise or bias.
- noise in the velocity for example, velocity in the z direction
- the vibration or velocity noise is relatively small and can be averaged out to remove its effects.
Abstract
Techniques of tracking an object's motion involve using a LIDAR system that is configured to track an object over a period of time in which the object is moving using a single scanning motion. Using the LIDAR system, tracking of the object can be performed while eliminating visible imaging hardware (e.g., video camera hardware). Accordingly, the LIDAR system can be configured to operate in total darkness, into the sun, etc. The LIDAR system can be less susceptible to motion of the object than conventional systems. Accordingly, the motion of the object 110 be determined in some implementations solely from LIDAR measurements, without, for example, video.
Description
- This application claims priority to U.S. patent application Ser. No. 15/613,974, filed Jun. 5, 2017 and entitled “Estimation of Motion Using LIDAR,” which claims priority to U.S. Provisional Patent Application No. 62/346,866, filed Jun. 7, 2016, entitled “Estimation of Motion Using LIDAR,” each of which is incorporated herein by reference in its entirety.
- This description relates to systems and methods for estimation of motion using Light Detection And Ranging (LIDAR).
- In some known systems, objects may be tracked using a laser Light Detection And Ranging (LIDAR) system in conjunction with a video system. Some such known systems may be complex and difficult to use. Additionally, in some such known systems, the video system may require light in order to detect the object to be tracked. Thus, a need exists for systems, methods, and apparatus to address the shortfalls of present technology and to provide other new and innovative features.
-
FIG. 1 is a block diagram illustrating an example LIDAR system within an electronic environment in which improved techniques described herein may be performed. -
FIG. 2A is a diagram illustrating an example object being tracked within the electronic environment illustrated inFIG. 1 . -
FIG. 2B is a diagram illustrating the example object as tracked within the electronic environment illustrated inFIG. 1 . -
FIG. 2C is a diagram illustrating another example object being tracked within the electronic environment illustrated inFIG. 1 . -
FIG. 2D is a diagram illustrating the other example object as tracked within the electronic environment illustrated inFIG. 1 . -
FIG. 2E is a diagram illustrating the other example object as further tracked within the electronic environment illustrated inFIG. 1 . -
FIG. 3 is a flowchart illustrating an example method for calculating object velocities and rotational velocity using a single set of scans. -
FIG. 4 is a diagram illustrating an example scan of an object over time within the electronic environment illustrated inFIG. 1 . -
FIG. 5 is a diagram illustrating an example sampling of object displacements from the scan illustrated inFIG. 4 . -
FIG. 6 is a diagram illustrating an example interpolation of the object displacements from the sampling illustrated inFIG. 5 . -
FIG. 7 is a flowchart illustrating an example method of determining initial estimates of object velocity and rotational velocity within the electronic environment illustrated inFIG. 1 . -
FIG. 8 is a flowchart illustrating an example method of refining the initial estimates of object velocity and rotational velocity determined by the process illustrated inFIG. 7 . -
FIG. 1 is a diagram that illustrates an exampleelectronic environment 100 in which improved techniques of tracking an object's motion are performed. Theelectronic environment 100 includes a LIDARsystem 120 that is configured to track anobject 110 over a period of time in which theobject 110 is moving using a single scanning motion. Using the LIDARsystem 120 tracking of theobject 110 can be performed while eliminating visible imaging hardware (e.g., video camera hardware). Accordingly, the LIDARsystem 120 can be configured to operate in total darkness, into the sun, etc. The LIDARsystem 120 can be less susceptible to motion of theobject 110 than conventional systems. Accordingly, the motion of theobject 110 can be determined in some implementations solely from LIDAR measurements, without, for example, video. - The
object 110 is assumed herein to be a rigid body of some unknown shape. For example, theobject 110 may be a human face. Theobject 110 is assumed to be in motion, both linear and rotational, about an arbitrary axis. For a rigid body, the time-dependent linear velocity v and time-dependent rotational velocity ω at a point r on theobject 110 are related as follows: -
v=ω×r+v ƒ (1) - where x denotes a cross-product and vƒ is the velocity of a reference frame that contains the origin. In the examples provided herein, the reference frame is taken to be the inertial frame of the
LIDAR system 120 although this is by no means a requirement. - It should be understood that in the electronic environment shown in
FIG. 1 , there is a natural axis of symmetry that is seen to be substantially along the beams emitted by theLIDAR system 120. In this case and in light of Eq. (1), we may decompose the motion of theobject 110 into lateral and axial components. That is, given an axis of symmetry in the z direction, we may define the lateral motion using the components vx, vy, ωz and the axial motion using the components vz, ωx, ωy. In many situations, the lateral motion and axial motion are approximately independent. It is the primary objective of the improved techniques described herein to determine the lateral components of the object's motion. Axial motion of the object may be removed, or compensated for, directly from the LIDAR range and velocity measurements. - As shown in
FIG. 1 , the LIDARsystem 120 is a single, integrated unit that includesprocessing circuitry 124,memory 126, anillumination system 150, and areceiver system 160. In some arrangements, the LIDARsystem 120 takes the form of a handheld unit that may be pointed at theobject 110. However, in other arrangements the components of the LIDARsystem 120 may be distributed among different units (e.g., theprocessing circuitry 124 andmemory 126 might be in a computing device separate from a handheld device that includes theillumination system 150 and the receiver system 160). - The
processing circuitry 124 includes one or more processing chips and/or assemblies. Thememory 126 includes both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid state drives, and the like. The set ofprocessing units 124 and thememory 126 together form control circuitry, which is configured and arranged to carry out various methods and functions as described herein. - In some arrangements, one or more of the components of the LIDAR
system 120 can be, or can include, processors configured to process instructions stored in thememory 126. For example, an analysis manager 130 (and/or a portion thereof), shown as being included within thememory 126 inFIG. 1 , can be a combination of a processor and a memory configured to execute instructions related to a process to implement one or more functions. - The
analysis manager 130 is configured and arranged to produce estimates of the linear velocity components vx, vy and rotational velocityωz of theobject 110 over the period of time in which theobject 110 is tracked. Theanalysis manager 130 is configured to perform such an estimate in two stages. First, at a given time, theanalysis manager 130, via an initial movement analyzer 132, performs an initial estimate of the lateral motion of theobject 110 using a pair of displacement datasets, e.g., 140(1) and 140(2) centered about that time. Then theanalysis manager 130, via amovement refinement manager 134, refines the initial estimate using actual data collection times and, if necessary, subsequent displacement datasets, e.g., 140(3) and so on. - The initial movement analyzer 132 is configured to produce
initial velocity data 136, i.e., vx, vy, ωz from displacement datasets 140(1) and 140(2). However, it should be appreciated that theseinitial data 136 are first estimates and may be refined using actual data collection times and, if necessary, subsequent displacement datasets, e.g., 140(3) and so on. In this capacity, themovement refinement manager 134 is configured to computerefinements 138, i.e., δvx, δvy, δωz from the actual data collection times and, in some cases, the subsequent displacement datasets. Further, themovement refinement manager 134 is configured to determine when sufficient convergence has been achieved so that no more refinement computations are necessary. - Each of the displacement datasets, e.g., displacement dataset 140(1) includes an array of displacement values in the z-direction as a function of the Cartesian coordinates x and y at a respective time. It should be appreciated that although the
object 110 as shown inFIG. 1 has a flat surface facing theLIDAR system 120, in reality theobject 110 may include, e.g., a human face, a portion of a body, an inanimate object, etc., and as such has a complicated profile z=ƒ(x, y) at any time t. Because theobject 110 is assumed to be a rigid body, then the difference between the displacement profiles from one time to another is a combination of translations Δx, Δy and rotations θz (i.e., no deformations). Theanalysis manager 130 then finds the velocities and rotational velocities from these translations and rotations. - The
illumination system 150 is configured and arranged to produce the illumination that is reflected from theobject 110 in the act of generating the displacement datasets 140(1), . . . , 140(T). As shown inFIG. 1 , this illumination takes the form of multiple beams 190(1), . . . , 190(N) of radiation directed along the z-axis. It should be appreciated that, although the beams 190(1), . . . , 190(N) as shown inFIG. 1 are directed at an angle with respect to the z-axis, in reality the beams travel substantially along the z-axis and the angles in the figure are exaggerated. Theillumination system 150 includes ascanning mechanism 152, which includes alaser array 154, and anaperture 170. - The
scanning mechanism 152 is configured and arranged to move thelaser array 154 in a scanning motion. As shown inFIG. 1 , thescanning mechanism 152 is configured to move each laser in thelaser array 154 back and forth along they direction, i.e., orthogonally to the direction of the beams 190(1), . . . , 190(N). However, in some arrangements the scanning may be performed along the x-direction instead or in addition. Thescanning mechanism 152 moves thelaser array 154 altogether, so that all scans are performed in one scanning motion. Further, it should be appreciated that each individual scan in one direction is performed at a sufficient speed so as to be considered as having taken place in a single time instant. Along these lines, a first scan of thelaser array 154 in the positive y-direction might be considered to have taken place at an instant of time T0, even though data from the first point in the scan and the past point in the scan are not received simultaneously at theLIDAR system 120. - The
laser array 154 is configured and arranged to produce an array of beams (e.g., beams 190(1), . . . , 190(N)) of laser radiation, i.e., substantially coherent, quasi-monochromatic light. In many arrangements, thelaser array 154 includes a rectangular array of lasers, each producing laser radiation at some wavelength. Each laser in the rectangular array corresponds to a sample point on theobject 110 where the beam produced by that laser reflects off theobject 110. In other arrangements, the array of beams 190(1), . . . , 190(N) is generated using a single laser (not shown) that is fiber coupled and split into multiple fibers (not shown) which are then arranged into a fiber-emitter array. - In some arrangements, the wavelength of the light in each beam 190(1), . . . , 190(N) produced by the
laser array 154 is 1550 nm. This wavelength has the advantage of being suited to objects that are, for example, human faces. Nevertheless, other wavelengths (e.g., 1064 nm, 532 nm) may be used as well. - The
aperture 170 is configured to pass the beams 190(1), . . . , 190(N) generated by thelaser array 154 out of theLIDAR system 120 to theobject 110. In some arrangements theaperture 170 may simply be an opening to let the beams 190(1), . . . , 190(N) pass through. In other arrangements, theaperture 170 may include optics (not shown) for satisfying certain specifications (e.g., spot size on object). - The
receiver system 160 is configured and arranged to receive the beams reflected from theobject 110 and generate the displacement datasets 140(1), . . . , 140(T) from the received beams. Thereceiver system 160 may generate the displacement datasets 140(1), . . . , 140(T) using any number of known techniques (e.g., heterodyne detection) and will not be discussed further. The receiver system includes adetector 180 that is configured and arranged to convert the received beams into electrical signals from which thereceiver system 160 may generate the displacement datasets 140(1), . . . , 140(T). In some arrangements, thedetector 180 includes a photomultiplier tube (PMT) or an array of charge-coupled devices (CCDs). -
FIGS. 2A and 2B illustrate anexample object 210 that may be observed by (e.g., targeted by) theLIDAR system 120. Theobject 210 may have any shape, but is represented inFIGS. 2A and 2B as a circle. InFIG. 2A , at time T1 apoint 220 on theobject 210 is being observed by theLIDAR system 120. At time T1 thepoint 220 is located at (3,3) in the (x,y) plane. As illustrated inFIG. 2B , at time T2 thepoint 220 is located at (4,3) in the (x,y) plane. The movement of the point may be the result of different types of movements of the object 80. For example, theobject 220 may have moved from one location to another (translational movement) or theobject 220 may have rotated (for example, about an axis parallel to the y axis of the x-y plane). - As illustrated in
FIGS. 2C, 2D, and 2E a head or face 290 of an individual may be tracked or observed by theLIDAR system 120. Specifically, a point orlocation 292 of the head orface 290 may be observed. As illustrated inFIG. 2C , at time T1 thepoint 292 is located at (3,2) in the (x,y) plane. At time T2 thepoint 292 may be observed to be at (4,2). The movement of the point may be the result of different types of motion. For example, the person or individual may have rotated their head (for example, about an axis parallel to they axis), as illustrated inFIG. 2D . Alternatively, the person or individual may have moved their head (without any rotation), as illustrated inFIG. 2E . -
FIG. 3 illustrates anexample method 300 of performing the improved technique described herein. Themethod 300 may be performed by constructs described in connection withFIG. 1 , which can reside inmemory 126 of theLIDAR system 120 and can be executed by theprocessing circuitry 124. - At 302, a LIDAR system defines a first dataset based on a first scan of an object using beams of electromagnetic radiation emitted from a scanning mechanism. For example, the
LIDAR system 120, viascanning mechanism 152, scans beams of light 190(1), . . . , 190(N) over theobject 110 at a point in time. TheLIDAR system 120, viareceiver system 160, receives the beams 190(1), . . . , 190(N) and generates the displacement dataset 140(1) from the received beams. - At 304, the LIDAR system defines a second dataset based on a second scan of the object using the beams of electromagnetic radiation emitted from the scanning mechanism. For example, the
LIDAR system 120, viascanning mechanism 152, scans beams of light 190(1), . . . , 190(N) over theobject 110 at a second point in time. TheLIDAR system 120, viareceiver system 160, receives the beams 190(1), . . . , 190(N) and generates the displacement dataset 140(2) from the received beams. - It should be understood that the scan performed by the
LIDAR system 120 at the second point in time may be the next scan, i.e., the reverse scan to the first scan. However, in other arrangements the scan performed at the second point in time is the next scan in the same direction as the first scan. In some implementations, such a situation may be preferable because there may be a better correlation between the displacement datasets when the first and second scans are performed in the same direction. However, in other situations, it may be preferable to consider a scan in the reverse direction as the second scan. - At 306, the LIDAR system generates initial values of a first velocity (e.g., vx), a second velocity (e.g., vy), and a rotational velocity (e.g., ωz) from the first dataset and the second dataset. When the first and second scans are performed in the same direction, these velocities and rotational velocity are evaluated at the time of the scan in between the first scan and second scan (i.e., the one in the opposite direction). Details of how the
LIDAR system 120 generates these values are discussed in reference toFIG. 7 . - At 308, the LIDAR system defines a third dataset based on a third scan of the object using the beams of electromagnetic radiation emitted from the scanning mechanism. For example, the
LIDAR system 120, viascanning mechanism 152, scans beams of light 190(1), . . . , 190(N) over theobject 110 at a third point in time. TheLIDAR system 120, viareceiver system 160, receives the beams 190(1), . . . , 190(N) and generates the displacement dataset 140(3) from the received beams. Again, the third scan may either be the scan following the second scan or the next scan in the same direction as the second scan but is preferred to be in the same direction. - At 310, the LIDAR system produces refined values of the first velocity, second velocity, and rotational velocity based on the initial values of the first velocity, second velocity, and rotational velocity and actual data collection times and, if necessary, the third dataset. Again, the refinements are performed for the velocities and rotational velocity at the time in between the first scan and second scan. Details of how the
LIDAR system 120 generates these refined values are discussed in connection with at leastFIG. 8 . -
FIG. 4 illustrates a top-down view of an example scan geometry using theLIDAR system 120 resulting from anexample laser array 154. In this case, thelaser array 154 contains 16 lasers arranged in a two-by-eight array. This configuration of thelaser array 154 as shown inFIG. 4 is only an example and many configurations using any number of lasers (e.g., 2X4, 2X6, 3X4, 2X6, 2X8, 3X8, 4X4, and so on) may be used. - As stated previously with reference to the example environment of
FIG. 1 , thescanning mechanism 152 is configured to move thelaser array 154 as a whole during a scan. Each laser in thelaser array 154 thus scans theobject 110 synchronously and in the same direction. The scan path traced out by thescanning mechanism 152 may be any pattern but for simplicity is taken to be along a straight line along one axis (the y-axis as shown inFIG. 4 ). - Such a linear scan pattern performed by the
scanning mechanism 152 produces the scan lines 420(1) over theobject 110 shown inFIG. 4 . It should be appreciated that, as a first approximation, the scans producing scan lines 420(1) are considered to have taken place at a single time instant T0 even though the scan time across theobject 110 is finite. In refining the first approximation, a finite scan time may be taken into account. - It should also be appreciated that scan lines of beams produced by lasers of the
laser array 154 in the same column may overlap. This is shown inFIG. 4 as each scan line 420(1) is actually two overlapping scan lines. In some implementations, this overlap occurs because the extent of a scan performed by thescanning mechanism 152 is longer than the distance between the lasers in each column of thelaser array 154. - Because the
object 110 is in motion, the scan lines of the next scan appear on a different location on theobject 110. InFIG. 4 , the scan lines 420(2) are produced from scans at a time T2, i.e., the time of the next scan in the same direction. The scan lines 420(2) are shown inFIG. 4 as dashed to distinguish over the scan lines 420(1). Again, each scan line 420(2) is actually two overlapping scan lines. -
FIG. 5 illustrates adata sampling 500 based on the scanlines shown inFIG. 4 . Note that the axes as depicted inFIG. 5 are rotated with respect to the axes depicted inFIG. 4 , and the scan direction inFIG. 5 is horizontal. Thedata sampling 500 represents what is stored in thememory 126 after the beams 190(1), . . . , 190(N) have been received at the detector and before the displacement datasets 140(1), . . . , 140(T) have been generated. Note that the sample points depicted inFIG. 5 are in the reference frame of the object and not that of theLIDAR system 120. Each small circle represents a displacement of the surface of theobject 110 in the z-direction at the time T0, i.e., during the first scan in the positive y-direction. Each small square represents a displacement of the surface of theobject 110 in the z-direction at the time T2, i.e., during the second scan in the positive y-direction. The filling of each circle or square represents the beam produced by the first or second laser in each column of thelaser array 154. It should be appreciated that there is overlap between the two beams along the center of each scanline and this is represented by cross-hatching inFIG. 5 . - It should also be appreciated that, due to lateral movement of the
object 110, the extents of the first scan and the second scans differ. For example, while the first scanlines may occupy a range between 10 cm and 200 cm along the y-direction, the second scanlines may occupy a range between 0 cm and 190 cm along the y-direction. Again, the reference frame used to define the origin of the coordinate system is the inertial frame of theLIDAR system 120. - It should further be appreciated that the sampling along the x-direction is typically very sparse. While the number of samples taken along the y-direction may be as large or small as necessary, the number of samples along the x-direction is limited by the number of lasers in a row of the
laser array 154. In the case shown inFIG. 5 , while there are a few dozen samples and there may be hundreds of such samples along the scan direction, there may be only eight samples normal to the scan direction. - The gathering of the data represented in
FIG. 5 can be used to compute an initial estimate of the lateral linear velocities vx, vy and the lateral rotational velocity ωz. This is achieved by computing an estimate of the translations Δx, Δy and rotation θz between the dataset at time T0 and the dataset at time T2. It should be appreciated that, although only lateral motion is considered here, there still may be some axial displacement between time T0 and time T2. However, this displacement may be removed, or compensated for, using the LIDAR range and velocity data from T0, T1 and T2. - In some implementations, before proceeding with the estimates, it may be observed that the raw data are too sparsely sampled to provide a meaningful covering of the object. In order to provide the flexibility needed to compute small translations and rotations, in some implementations, it may be advantageous to resample the datasets. For example, by introducing an interpolation scheme that produces a dense grid, the initial estimates of the translations and rotation may be found with relative ease.
-
FIG. 6 illustrates an example of such aninterpolation scheme 610.FIG. 6 shows two columns of a scanline fromFIG. 5 with a triangular grid drawn between the sample points of the columns. Thus, each sample point is a vertex of two triangles in the grid. Within each triangular area, one may define a respective linear interpolation function using, e.g., barycentric or rectangular coordinates. - However, it should be appreciated that the
interpolation scheme 610 can be used to define interpolated object displacement as a continuous function of coordinates within a triangular region. In this way, finding the translations and rotation can be achieved. For example, one may determine a correlation between the two datasets at T0 and T2 as a function of translation and/or rotation. The initial estimated translations and rotation may then be those values that maximize the correlation between the datasets. Along these lines, maximizing the correlation between the datasets in one implementation means minimizing a sum of squared differences between interpolated object displacement values over translation and/or rotation. Further details are discussed with reference toFIG. 7 . -
FIG. 7 illustrates further detail ofelement 306 ofmethod 300 shown inFIG. 3 , in which theLIDAR system 120 generates initial values of a first velocity (e.g., vx), a second velocity (e.g., vy), and a rotational velocity (e.g., ωz) from the first dataset and the second dataset. - At 702, a
LIDAR system 120 acquires a dense sampling of object displacements at time T0 via interpolation. A dense sampling is a sampling of object displacements a factor of two or more dense than that of the original datasets, e.g., if a raw data set has 200×8 data points, then the dense sampling acquired via interpolation may be 800×800 or higher. For example, in one implementation theLIDAR system 120 shown inFIG. 1 performs a linear interpolation as described with reference to FIG. using a manhattan grid on displacement dataset 140(1). A result of this interpolation is a dense sampling of object displacements over a rectangular grid for the dataset at time T0. It has been found that the interpolation over the manhattan grid as depicted inFIG. 6 results in faster computation time than schemes that use a nearest neighbor approximation. - At 704, the
LIDAR system 120 then forms a dense sampling of object displacements at time T2 (e.g., displacement dataset 140(2)) to create an interpolated dataset that is a function of object displacements Δx, Δy. In one implementation, theLIDAR system 120 considers (Δx, Δy)=(0,0) as the interpolated sample points at T2 having at the same coordinates as the interpolated sample points at T0. Using the interpolation function defined by the second dataset, theLIDAR system 120 computes interpolated object displacements corresponding to each interpolated sample point at time T0 but shifted by translations Δx, Δy in each respective direction. In some arrangements, the shifting may be done in only one direction. - If ƒ0(x, y) represents the object displacements at time T0 and ƒ2(x, y) represents the object displacements at time T2, then for each sample point (xi, yj) in the dense (interpolated) grid, the
LIDAR system 120 forms the function ƒ2(xi+Δx,yj+Δy), which is considered a function of Δx and Δy. - At 706, the
LIDAR system 120 determines values of Δx and Δy that optimize a correlation between the interpolated dataset at T0 and the interpolated dataset at T2. In some implementations, optimizing a correlation involves minimizing, or substantially reducing, a sum over squared differences between the displacements at TO and the translated displacements at T2. In other words, theLIDAR system 120 may find the values of Δx and Δy that minimize the following quantity: -
- where K and L are the number of interpolated sample points in the x and y-direction, respectively. It should be appreciated that the interpolated sample points xi and yj are not necessarily on a rectangular grid. The
LIDAR system 120 may use any method known in the art to perform the minimization of Etrans(ΔX, Δy). The values that minimize this function are denoted (Δx0, Δy0) and represent the initial estimates of the translation of the object at time T1 between time T0 and time T2. - At 708, the
LIDAR system 120 forms another interpolation function from the dataset at time T2 based on rotation of a sample point through an angle θz. It should be appreciated that the initial estimate of the rotation is determined independently from the initial estimate of the translation. - At 710, the
LIDAR system 120 determines the value of the rotation θz that maximizes the correlation between the interpolated dataset at T0 and the interpolated dataset at T2. In some implementations, maximizing a correlation involves minimizing a sum over squared differences between the displacements at T0 and the translated displacements at T2. In other words, theLIDAR system 120 may find the value of θz that minimizes the following quantity: -
- where Erot(θz) is evaluated at discreet values of θz close to zero. The
LIDAR system 120 may use any method known in the art to perform the minimization of Erot(θz). The value that minimizes this function is denoted as θZ0 and represents the initial estimates of the rotation of the object at time T1 between time T0 and time T2. - At 712, the
LIDAR system 120 determines the initial estimates at the time T1 of the velocities vx, vy and rotational velocity ωz based on the values Δx0Δy0, and θz0. -
FIG. 8 illustrates example detail ofelement 310 ofmethod 300 shown inFIG. 3 , in which theLIDAR system 120 produces refined values of the first velocity, second velocity, and rotational velocity at time T1 based on initial estimates of the first velocity, second velocity, and rotational velocity. In some implementations, theLIDAR system 120 also uses subsequent datasets and their respective velocity estimates to further refine these values of the first velocity, second velocity, and rotational velocity. - At 802, the
LIDAR system 120 determines the actual positions of the sample points of the scan at T0 and T2 based on a scan speed. Along these lines, as discussed previously with respect to the initial estimate, it was assumed in making that estimate of the velocities that the sample points of a single scan were all taken at a single point in time. That is not true as the scan has a finite speed. Thus, theLIDAR system 120 performs the correction of the position of the sample points in the scan according to the time of the scan and based on the initial estimates of the velocities vx, vy and rotational velocity ωz at time T1. - As an example, suppose that the first scan corresponding to time T0 takes 10 milliseconds to complete. For the purposes of the initial estimate, the scan was considered to have taken place at T0=5 msec. Further suppose that a sample point (xi, yj) of the scan is known to have been generated at T=1 msec. (For example, the sample points may have been generated at a uniform rate.) Then the linearly corrected (without rotation) sample point has a position equal to (xi′, yj′)=(xi+vxΔT, yj+vyΔT), where ΔT=T0−T. The corrected sample point taking rotation into account has a position equal to (xi″, yj″)=(xi ′cosθz0−yj, sin θzo, xi ′sin θz0+yj′cos θz0). This latter position is the refined position of that sample point. This process of finding the refined positions of sample points continues for all sample points in the T0 and T2 scans.
- At 804, once the refined positions of the sample points at T0 and T2 have been found, the
LIDAR system 120 repeats 702, 704, and 706 fromFIG. 7 . Along these lines, theLIDAR system 120 finds translations and a rotation that maximizes a correlation between the interpolated object displacements at time T0 and time T2. In some implementations, theLIDAR system 120 minimizes or at least substantially reduces a sum of squared differences between the object displacements at various sample points in the xy-plane. The result of the minimization of the sum, i.e., the optimization of the correlation is a specification of new translations Δx1, Δy1, and rotation θz1. - At 806, the
LIDAR system 120 determines refinements δvx, δvy, and δωz to the linear and rotational velocities at time T1 from the new translations and rotation. - At 808, the
LIDAR system 120 determines whether these refinements to the initial estimates of the velocities vx, vy, and rotational velocity ωz are smaller than a threshold. - At 810, if the refinements are smaller than the threshold, then the
process 310 has converged and the motion of theobject 110 at time T0 has been determined. - At 812, if the refinements are not smaller than the threshold, then the
process 310 has not converged and there are more refinements to the linear and rotational velocities. Along these lines, theLIDAR system 120 may determine the motion of theobject 110 at subsequent (odd-numbered) times by using subsequent datasets taken at even times (e.g., T4, T6, etc.). In one implementation, suppose that theLIDAR system 120 has generated refined velocities vx, vy, and rotational velocity ωz at time T3 based on refined positions of object sample points at times T2 and T4. Then the velocity may be considered to be a function of time. With two data points (velocity at T1 and T3), each velocity may be modeled using a linear profile: v(T)=v(T1)+α(T−T1), where the coefficient α is determined from the velocity at T1 and T3. This time-dependent model for each velocity will produce new adjusted values of the velocities at T0 and T2. From these new adjusted values of the velocities, theLIDAR system 120 may determine further refined positions of the sample points, from which new refinements to the velocities at T1 may be determined. If these refinements are not less than the threshold, then theLIDAR system 120 uses scan data at T6 to determine velocities at T5, from which a quadratic velocity model is generated, and so on until the refinements to the velocity are less than the threshold. It has been found that the above process converges rapidly. - The time-dependent velocities used for further refinement need not be a simple linear interpolation of the velocities in time. In some implementations, any more accurate interpolation of the velocity may be used, such as cubic spline, when more than two velocity estimates are known.
- The components (e.g., modules, processors (e.g., a processor defined within a substrate such as a silicon substrate)) of the LIDAR system 120 (e.g., the analysis manager 130) can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, and/or so forth. In some implementations, the components of the
LIDAR system 120 can be configured to operate within a cluster of devices (e.g., a server farm). - In some implementations, the second scan referred to in
FIG. 3 may be performed in the opposite direction of the first scan. The velocities and rotational velocity so determined are evaluated, as before, at a time halfway between the first scan and second scan. - In some implementations, one or more portions of the components shown in the
LIDAR system 120 inFIG. 1 can be, or can include, a hardware-based module (e.g., a digital signal processor (DSP), a field programmable gate array (FPGA), a memory), a firmware module, and/or a software-based module (e.g., a module of computer code, a set of computer-readable instructions that can be executed at a computer). For example, in some implementations, one or more portions of theLIDAR system 120 can be, or can include, a software module configured for execution by at least one processor (not shown). In some implementations, the functionality of the components can be included in different modules and/or different components than those shown inFIG. 1 . - In some implementations, one or more of the components of the
LIDAR system 120 can be, or can include, processors configured to process instructions stored in a memory. For example, the analysis manager 130 (and/or a portion thereof) can be a combination of a processor and a memory configured to execute instructions related to a process to implement one or more functions. - Although not shown, in some implementations, the components of the LIDAR system 120 (or portions thereof) can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server/host devices, and/or so forth. In some implementations, the components of the LIDAR system 120 (or portions thereof) can be configured to operate within a network. Thus, the LIDAR system 120 (or portions thereof) can be configured to function within various types of network environments that can include one or more devices and/or one or more server devices. For example, the network can be, or can include, a local area network (LAN), a wide area network (WAN), and/or so forth. The network can be, or can include, a wireless network and/or wireless network implemented using, for example, gateway devices, bridges, switches, and/or so forth. The network can include one or more segments and/or can have portions based on various protocols such as Internet Protocol (IP) and/or a proprietary protocol. The network can include at least a portion of the Internet.
- In some implementations, the
LIDAR system 120 may include a memory. The memory can be any type of memory such as a random-access memory, a disk drive memory, flash memory, and/or so forth. In some implementations, the memory can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with the components of theLIDAR system 120. - In some implementations, a LIDAR system includes a laser system that includes lasers or laser beams that are configured to move in a pattern or patterns with respect to the object that is being tracked. For example, in some implementations, the
scanning mechanism 152 of theLIDAR system 120 includes a plurality of lasers or beams that are configured to move in a pattern or patterns with respect to the object being tracked. - For example, in some implementations, the
LIDAR system 120 may have one mode in which the laser beams are fixed or stationary and a second mode in which the laser beams move in a pattern or patterns such as a shape. In some implementations, two or more of the laser beams move in a pattern or patterns when theLIDAR system 120 is in the second mode. In some implementations, different laser beams may move independently in different patterns. - In other implementations, the
LIDAR system 120 includes some lasers or produces some laser beams that are stationary and some that are configured to move in a pattern (or patterns) or shape. - The lasers or beams can move in any pattern or shape. For example, in some implementations, the lasers or beams are configured to move in elliptical shape. In other implementations, the lasers or beams are configured to move in a line, circle, square, rectangle, triangle, or any other shape. In some implementations, the shape or pattern that the lasers or beams move in are dictated or determined by the object that is being tracked. For example, in some implementations, the pattern or shape of the laser movement may be similar to the shape of the object that is being tracked. For example, an elliptical shape or pattern may be used when tracking a face of an individual as the face of an individual is generally elliptical in shape.
- Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (computer-readable medium, a non-transitory computer-readable storage medium, a tangible computer-readable storage medium) or in a propagated signal, for processing by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
- Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- Processors suitable for the processing of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.
- To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a liquid crystal display (LCD or LED) monitor, a touchscreen display, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end, middleware, or front-end components. Components may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
- In some implementations, the
LIDAR system 120 may achieve millimeter range accuracy performance off moving faces of a subject or individual. However, in some implementations, solid object velocity estimation requires processing of multiples samples in order to remove significant velocity components from speech and other biological components. A 500 Hz vibration with an amplitude of 0.05 mm (50 microns) will have a maximum velocity of (2*π*500*5E−5=0.157 m/sec) about 16 cm/sec. Even though the amplitude of the vibration is an insignificant range change for the process of tracking faces of a subject or individual, the instantaneous velocity may be significant and the vibrational velocity may be removed. In some implementations, removing vibrational velocity may require processing a velocity data sample significantly longer than the periods of the vibrations to be removed and care to avoid noise or bias. For example, noise in the velocity (for example, velocity in the z direction) can affect or degrade the ability to detect or determine the rotation of the object or the z velocity of the object. In some implementations, the vibration or velocity noise is relatively small and can be averaged out to remove its effects. - While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different implementations described.
Claims (1)
- A method, comprising:defining a first dataset based on a performance of a first scan at a first time of an object using a plurality of beams of electromagnetic radiation emitted from a scanning mechanism;defining a second dataset based on a performance of a second scan at a second time of the object using the plurality of beams of electromagnetic radiation emitted from the scanning mechanism, each of the first scan and the second scan being performed in a direction substantially normal to a direction of the plurality of beams;generating, based on the first dataset and the second dataset, (i) an initial value of a first velocity of the object along a first axis orthogonal to the direction of the plurality of beams, (ii) an initial value of a second velocity of the object along a second axis orthogonal to the direction of the plurality of beams, and (iii) an initial value of a rotational velocity of the object about an axis parallel to the direction of the plurality of beams.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/741,475 US20200150273A1 (en) | 2016-06-07 | 2020-01-13 | Estimation of motion using lidar |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662346866P | 2016-06-07 | 2016-06-07 | |
US15/613,974 US10557942B2 (en) | 2016-06-07 | 2017-06-05 | Estimation of motion using LIDAR |
US16/741,475 US20200150273A1 (en) | 2016-06-07 | 2020-01-13 | Estimation of motion using lidar |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/613,974 Continuation US10557942B2 (en) | 2016-06-07 | 2017-06-05 | Estimation of motion using LIDAR |
Publications (1)
Publication Number | Publication Date |
---|---|
US20200150273A1 true US20200150273A1 (en) | 2020-05-14 |
Family
ID=60578124
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/613,974 Active 2038-01-30 US10557942B2 (en) | 2016-06-07 | 2017-06-05 | Estimation of motion using LIDAR |
US16/741,475 Abandoned US20200150273A1 (en) | 2016-06-07 | 2020-01-13 | Estimation of motion using lidar |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/613,974 Active 2038-01-30 US10557942B2 (en) | 2016-06-07 | 2017-06-05 | Estimation of motion using LIDAR |
Country Status (8)
Country | Link |
---|---|
US (2) | US10557942B2 (en) |
EP (1) | EP3465259A4 (en) |
JP (1) | JP7057289B2 (en) |
CN (2) | CN109416398B (en) |
AU (1) | AU2017279528B2 (en) |
CA (1) | CA3026638A1 (en) |
TW (1) | TWI781106B (en) |
WO (1) | WO2017214144A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11029399B2 (en) | 2018-02-01 | 2021-06-08 | Beijing Voyager Technology Co., Ltd. | System and method for calibrating light intensity |
US20190383942A1 (en) * | 2018-06-18 | 2019-12-19 | DSCG Solutions, Inc. | Lidar-only lock-on tracking system |
US11556000B1 (en) | 2019-08-22 | 2023-01-17 | Red Creamery Llc | Distally-actuated scanning mirror |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150338518A1 (en) * | 2014-05-21 | 2015-11-26 | DSCG Solutions, Inc. | Devices, systems, and methods for real time tracking of an object |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2292605B (en) * | 1994-08-24 | 1998-04-08 | Guy Richard John Fowler | Scanning arrangement and method |
US5988862A (en) | 1996-04-24 | 1999-11-23 | Cyra Technologies, Inc. | Integrated system for quickly and accurately imaging and modeling three dimensional objects |
US9402601B1 (en) * | 1999-06-22 | 2016-08-02 | Teratech Corporation | Methods for controlling an ultrasound imaging procedure and providing ultrasound images to an external non-ultrasound application via a network |
US6937774B1 (en) | 2000-10-24 | 2005-08-30 | Lockheed Martin Corporation | Apparatus and method for efficiently increasing the spatial resolution of images |
JP4106582B2 (en) * | 2001-01-04 | 2008-06-25 | オムロン株式会社 | Vehicle detection device |
US7302174B2 (en) * | 2003-12-31 | 2007-11-27 | Symbol Technologies, Inc. | Method and apparatus for capturing images using a color laser projection display |
US7689321B2 (en) | 2004-02-13 | 2010-03-30 | Evolution Robotics, Inc. | Robust sensor fusion for mapping and localization in a simultaneous localization and mapping (SLAM) system |
JP2006030147A (en) * | 2004-07-22 | 2006-02-02 | Hitachi Ltd | Environment recognizing system and moving mechanism |
GB0701869D0 (en) | 2007-01-31 | 2007-03-14 | Cambridge Consultants | Adaptive radar |
US8855848B2 (en) | 2007-06-05 | 2014-10-07 | GM Global Technology Operations LLC | Radar, lidar and camera enhanced methods for vehicle dynamics estimation |
US7999923B2 (en) | 2009-02-19 | 2011-08-16 | Northrop Grumman Systems Corporation | Systems and methods for detecting and analyzing objects |
WO2011070927A1 (en) * | 2009-12-11 | 2011-06-16 | 株式会社トプコン | Point group data processing device, point group data processing method, and point group data processing program |
US20130332112A1 (en) | 2011-03-01 | 2013-12-12 | Toyota Jidosha Kabushiki Kaisha | State estimation device |
US11284846B2 (en) | 2011-05-12 | 2022-03-29 | The John Hopkins University | Method for localization and identification of structures in projection images |
CN102305932B (en) * | 2011-07-26 | 2013-10-30 | 中国科学院上海光学精密机械研究所 | Moving target imaging method for Fresnel telescope imaging laser radar |
US9134402B2 (en) | 2012-08-13 | 2015-09-15 | Digital Signal Corporation | System and method for calibrating video and lidar subsystems |
US9188676B2 (en) | 2012-08-15 | 2015-11-17 | Digital Signal Corporation | System and method for detecting a face contour using a three-dimensional measurement system |
US8948497B2 (en) * | 2012-09-04 | 2015-02-03 | Digital Signal Corporation | System and method for increasing resolution of images obtained from a three-dimensional measurement system |
US9523772B2 (en) | 2013-06-14 | 2016-12-20 | Microsoft Technology Licensing, Llc | Object removal using lidar-based classification |
US10564285B2 (en) | 2015-11-19 | 2020-02-18 | DSCG Solutions, Inc. | Estimation of motion in six degrees of freedom (6DOF) using LIDAR |
-
2017
- 2017-06-05 US US15/613,974 patent/US10557942B2/en active Active
- 2017-06-06 WO PCT/US2017/036152 patent/WO2017214144A1/en unknown
- 2017-06-06 CN CN201780039761.XA patent/CN109416398B/en active Active
- 2017-06-06 JP JP2018563814A patent/JP7057289B2/en active Active
- 2017-06-06 CN CN202311556031.0A patent/CN117761657A/en active Pending
- 2017-06-06 CA CA3026638A patent/CA3026638A1/en not_active Abandoned
- 2017-06-06 EP EP17810857.7A patent/EP3465259A4/en active Pending
- 2017-06-06 AU AU2017279528A patent/AU2017279528B2/en active Active
- 2017-06-07 TW TW106118829A patent/TWI781106B/en active
-
2020
- 2020-01-13 US US16/741,475 patent/US20200150273A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150338518A1 (en) * | 2014-05-21 | 2015-11-26 | DSCG Solutions, Inc. | Devices, systems, and methods for real time tracking of an object |
Also Published As
Publication number | Publication date |
---|---|
JP2019518957A (en) | 2019-07-04 |
US10557942B2 (en) | 2020-02-11 |
EP3465259A1 (en) | 2019-04-10 |
AU2017279528A1 (en) | 2019-01-03 |
CN117761657A (en) | 2024-03-26 |
TWI781106B (en) | 2022-10-21 |
WO2017214144A1 (en) | 2017-12-14 |
AU2017279528B2 (en) | 2022-03-24 |
TW201743076A (en) | 2017-12-16 |
CN109416398B (en) | 2023-12-08 |
JP7057289B2 (en) | 2022-04-19 |
US20180348372A1 (en) | 2018-12-06 |
CA3026638A1 (en) | 2017-12-14 |
CN109416398A (en) | 2019-03-01 |
EP3465259A4 (en) | 2020-01-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20200150274A1 (en) | Estimation of motion in six degrees of freedom (6dof) using lidar | |
US10571573B2 (en) | Devices, systems, and methods for real time tracking of an object | |
US20200150273A1 (en) | Estimation of motion using lidar | |
JP2021535406A (en) | Systems and methods to reduce optical crosstalk in photodetection and ranging systems | |
US11262454B2 (en) | Dwell-angle-independent tracking and monitoring objects using LIDAR | |
US11513229B2 (en) | Multi-beam processing of lidar vibration signals | |
US20190383942A1 (en) | Lidar-only lock-on tracking system | |
US20190377066A1 (en) | Tracking objects using video images and lidar | |
Sie et al. | Radarize: Large-Scale Radar SLAM for Indoor Environments |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: DSCG SOLUTIONS, INC., VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BELSLEY, KENDALL;SEBASTIAN, RICHARD;REEL/FRAME:057796/0389 Effective date: 20191223 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |