EP4352536A1 - Polarimetric multifunctional lidar sensor for target recognition - Google Patents
Polarimetric multifunctional lidar sensor for target recognitionInfo
- Publication number
- EP4352536A1 EP4352536A1 EP21748959.0A EP21748959A EP4352536A1 EP 4352536 A1 EP4352536 A1 EP 4352536A1 EP 21748959 A EP21748959 A EP 21748959A EP 4352536 A1 EP4352536 A1 EP 4352536A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- polarimetric
- targets
- multifunctional
- data
- lidar
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 claims abstract description 52
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000013473 artificial intelligence Methods 0.000 claims abstract description 27
- 239000011159 matrix material Substances 0.000 claims abstract description 27
- 230000003287 optical effect Effects 0.000 claims description 66
- 238000000034 method Methods 0.000 claims description 48
- 230000010287 polarization Effects 0.000 claims description 43
- 230000008569 process Effects 0.000 claims description 9
- 230000004907 flux Effects 0.000 claims description 5
- 238000005259 measurement Methods 0.000 abstract description 19
- 238000000711 polarimetry Methods 0.000 abstract description 9
- 239000013598 vector Substances 0.000 description 13
- 239000000463 material Substances 0.000 description 12
- 101100175317 Danio rerio gdf6a gene Proteins 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000012512 characterization method Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 239000000443 aerosol Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 239000007788 liquid Substances 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 239000002245 particle Substances 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241000565356 Fraxinus pennsylvanica Species 0.000 description 1
- 235000008331 Pinus X rigitaeda Nutrition 0.000 description 1
- 241000018646 Pinus brutia Species 0.000 description 1
- 235000011613 Pinus brutia Nutrition 0.000 description 1
- 241001106462 Ulmus Species 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 239000005427 atmospheric aerosol Substances 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000028161 membrane depolarization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012883 sequential measurement Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/499—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using polarisation effects
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
- G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Electromagnetism (AREA)
- Optical Radar Systems And Details Thereof (AREA)
Abstract
The present invention describes a polarimetric multifunctional LiDAR system (100) adapted to perform ranging and polarimetric measurements for target recognition in autonomous driving applications. The disclosed system (100) comprises in a single device the capacity of light ranging (LiDAR) and active polarimetry, by measuring partially or totally the Mueller matrix of targets and/or Stokes parameters of light return from targets. By combining polarimetry with Artificial Intelligence, the polarimetric multifunctional LiDAR system (100) is able to add a new functionality that allows for distinguishing targets. In addition, the herein disclosed system is able to determine the location and/or velocity of targets. The herein disclosed system comprises an emission system (11) and a detection system (12), both controlled by a control and data acquisition system (21), and with a data processing system (22) for computing range information and target recognition.
Description
DESCRIPTION
"Polarimetric Multifunctional LiDAR Sensor for target recognition"
Technical Field
The present application describes a polarimetric multifunctional LiDAR system and method for detecting and recognizing targets in the surroundings, for autonomous driving applications.
Background art
A LiDAR (Light Detection and Ranging) is a remote sensing technology used to measure ranges (distances) to targets and/or their velocities. By illuminating the scene and by detecting the light returned from objects in the environment where the scan is performed, a LiDAR sensor is able to determine obstacles, and how far away they are. This information allows for building real-time distance maps of surrounding objects to navigate in unknown environments. LiDARs are essential for self-driving vehicles safety. In this sense, one of the main open challenges is not only to detect, but also to recognize different kinds of static or moving objects, such as walls, pedestrians, cyclists or cars, in an attempt to reduce the risk of vehicles' accidents. Most of the proposals of self driving vehicles rely on a combination of several sensors, such as cameras, LiDARs, SONARs or RADARs. The data obtained by these sensors combined with Artificial Intelligence (AI) techniques are used for obstacle recognition and detection.
This information, once made available in real time, is critical for the realization of active security controls in vehicles. There are several working principles of LiDARs for ranging. The most common method is based on measuring the time of flight (ToF), i.e., the time it takes a light pulse to travel to the target and return. The distance d is then calculated as
where
is the time of flight and c is the speed of light. Usually, to measure the time of flight, the generated light signal is split into two signals. One of the signals (the so-called reference signal) is used as zero reference, while the other (the measurement signal) is the one that travels towards the target, which is located at a distance d, and returns. By scanning the scene and by repeating the time of flight process, LiDARs build real-time distance maps of surrounding objects, named point clouds.
When an optical beam interacts with matter, its polarization state can be changed as a function of the material's properties, the target surface, and the beam characteristics (wavelength, polarization, etc.). Under this scenario, different objects can reflect differently the same polarized incident beam, and consequently, polarization can enhance discriminative power between materials, when compared to unpolarized light. Thus, polarization can be used as an additional degree of freedom to discriminate between materials or to remotely obtain further information about targets.
Polarimetry is the measurement and interpretation of polarization. One important method used to analyze
polarization changes produced by different materials is based on the measurement of the Mueller matrix. This matrix consists of 4x4 real elements that describe the polarimetric properties of materials, enabling us to analyze the interaction of a totally or partially polarized or depolarized wave reflected/transmitted/scattered by a sample. The Stokes-Mueller formalism is the most appropriate representation of polarization for radiometric measurements when considering not fully polarized light. In the Stokes-Mueller formalism, the Stokes vector, S, (composed by the Stokes parameters So, i, S2 and S3) describes the state of polarization of light and it can be written as
where Es and Ep are the parallel and perpendicular, with respect to the plane of incidence, components of the electric field vector. Partially polarized light or depolarized light can be conveniently described based on a Stokes vector.
Additionally, the transformation of the polarization state due to the interaction of light with matter can be described by the Mueller matrix, M. This matrix relates the incident Stokes vector SInput to the existing Stokes vector
SOutput 3S
A Mueller matrix codifies the polarimetric content of its respective sample, which can be synthetized by properly arranging the information in the Mueller matrix elements. In reference (D.H. Goldstein. "Polarized light." 2nd Ed. CRC Press (2011)), Stokes vectors and Mueller matrices are defined and extensively discussed.
As different materials can have different polarimetric responses, the measurement of Stokes vectors of return light or Mueller matrices of targets can be used for material identification, even if they are not fully characterized. With the previous discussion of Stokes vectors and Mueller matrices as background, it is possible to discuss LiDARs that allow to perform polarization measurements. One main application of this kind of LiDARs is cloud research, i.e. to observe cloud characteristics, remotely identifying characteristics of atmospheric aerosol particles or urban aerosols. These devices usually measure the ratio of cross-polarized to parallel-polarized signals, calculating the linear depolarization ratio (references: Shane Mayor, and Scott Spuler, "Polarization Ildar for the remote detection of aerosol particle shape ", Patent Number: US7580127B1, (2006), The Polarization Lidar Technique for Cloud Research: A Review and Current Assessment in: Bulletin of the American Meteorological Society Volume 72 Issue 12 (1991)). Lately, some LiDARs allow for generating or measuring other polarizations. Inventors in reference (Savyasachee Liptarag Mathur, Yunhui Zheng, and Edward L.
Leventhal, "Polarization switching lidar device and method", Patent Number: US20120026497A1, (2010)) propose a polarization switching LiDAR for remote detection and characterization of airborne aggregations of particulates by using linearly polarized and circularly polarized light.
Inventors in patent ( "Lidar sensor for detecting an object", US20180106901A1, (2016)) propose a LiDAR sensor able to perform polarization measurements with a beam splitter for splitting the light to two detectors for detecting an object. This device is able to locate the object, nevertheless, it cannot be recognized. In this sense, this LiDAR sensor does not perform measurements of the Mueller matrix or Stokes vector, and it does not propose the use of Artificial Intelligence to achieve this recognition. Regarding to material discrimination to identify targets, authors in reference (Erandi Wijerathna, Charles D. Creusere, David Voelz, and Juan Castorena, "Polarimetric LIDAR with FRI sampling for target characterization", Proc. SPIE 10407, Polarization Science and Remote Sensing VIII, 104070R, (2017)) propose a polarimetric LIDAR for target characterization, using linearly polarized light with finite rate of innovations. One limitation of this method is that they do not use elliptical or circular polarizations. Inventors in reference (Annemarie I. Holleczek, Andre Albuquerque, Alexandre Correia, Pedro M. Caldelas, Angela R. Rodrigues, and Eduardo Pereira, "Method for material discrimination and respective implementation system", W02020021306, (2020)) propose to combine information on the backscattered light parameters and image processing techniques for identification of obstacles. This method results in a 6D analysis where a 2D location in an image is combined with
information regarding range, reflectivity, velocity and polarization of light in order to provide material discrimination and, consequently, target classification. This method requires a system working with different types of sensors at the same time.
Note that all the previously described devices do not propose the use of Artificial Intelligence, nor to measure partially or fully the Stokes vector or the Mueller matrix. There are other studies where both methods were used, combined with LiDAR sensors. This is the case of authors in reference (Jarrod P. Brown, Rodney G. Roberts, Darrell C. Card, Christian L. Saludez, and Christian K. Keyser "Hybrid passive polarimetric imager and lidar combination for material classification," Optical Engineering 59(7), 073106 (2020)). They combine an imaging Stokes polarimeter that analyzes the linear content of light with a LiDAR sensor. The combination of both sensors and machine learning allows for classification of hidden objects. This method partially characterizes the Stokes vector. However, as for the previous device, this method requires a system working with different types of sensors at the same time. Authors in reference (Ali Haider and Songxin Tan, "Improvement of LiDAR data classification algorithm using the machine learning technique", Proc. SPIE 11132, Polarization Science and Remote Sensing IX, 1113215, (2019)) compare the benefits of polarized light and different classifiers to classify randomly selected trees; such as pine, elm and green ash; as test samples. Following this proposal, our device is able to locate and/or obtain the velocity of relevant targets for autonomous driving applications (such as pedestrians, vehicles, traffic signs, etc.), and to
classify them, by using a polarimetric LiDAR and Artificial Intelligence.
Therefore, the present invention aims to overcome state of the art limitations with regard to the development of a remote sensor system capable of acquiring information of target distances and/or velocities and recognizing these targets.
Summary
The present invention describes a polarimetric multifunctional LiDAR system for target recognition in autonomous driving applications, comprising: an emission system, configured to emit polarized signals; a detection system, configured to detect return signals from targets; a control and data acquisition system, configured to control both the emission system and detection system and to acquire and digitize the return signals; and a data processing system, connected to the control and data acquisition system; wherein the data processing system is configured to determine Stokes parameters of the return signals and/or Mueller matrix elements of the targets and combine the determined Stokes parameters and/or Mueller matrix elements with Artificial Intelligence to determine information about surrounding environment, providing targets recognition, along with targets positions and/or velocity.
In a proposed embodiment of present invention, the emission system comprises at least one light source and at least one optical emission unit.
Yet in another proposed embodiment of present invention, the detection system comprises at least one detector and at least one optical detection unit.
Yet in another proposed embodiment of present invention, the at least one optical emission unit and the at least one optical detection unit are comprised of at least one of an active and/or passive non-polarizing optics and active and/or passive polarizing optics, configured to manipulate, polarize and steer light signals.
Yet in another proposed embodiment of present invention, the control and data acquisition system comprises a collection of software and hardware, configured to control the emission system and the detection systems; acquire return signals from targets; and digitize the return signals.
Yet in another proposed embodiment of present invention, the data processing system is configured to receive data from the control and data acquisition system; calculate Stokes parameters and/or Mueller matrix elements; calculate the distance to targets and/or targets' velocities; and recognize targets supported by Artificial Intelligence.
Yet in another proposed embodiment of present invention, the at least one light source of the emission system comprises at least one emitted polarized light signal and/or at least one transmitted wavelength.
Yet in another proposed embodiment of present invention, the at least one optical emission unit of the emission system comprises at least one polarization generator
configured to modify the polarization of the emitted polarized light signals.
Yet in another proposed embodiment of present invention, the at least one optical detection unit of the detection system comprises at least one polarization analyzer configured to analyze the polarization of the return signals from targets.
Yet in another proposed embodiment of present invention, the data processing system can be integrated in the described polarimetric multifunctional LiDAR system, or the central computer of the vehicle where the LiDAR sensor is installed, or even in a remote system.
Yet in another proposed embodiment of present invention, the Polarimetric multifunctional LiDAR system comprises an optical reference unit configured to steer and/or manipulate a reference signal defined as a portion of the emitted polarized light signals.
Yet in another proposed embodiment of present invention, the control and data acquisition system is configured to control the optical reference unit and acquire the reference signal.
Yet in another proposed embodiment of present invention, the polarimetric multifunctional LiDAR system comprises a non-coaxial configuration or a coaxial configuration.
Yet in another proposed embodiment of present invention, the polarimetric multifunctional LiDAR system comprises an optical coaxial unit configured to steer and/or manipulate
the transmitted polarized light signal and the return signal in the same optical axis.
Yet in another proposed embodiment of present invention, the control and data acquisition system is configured to control the optical coaxial unit.
Yet in another proposed embodiment of present invention, the optical emission unit and/or the optical detection unit comprises at least one of a lens, mirror, prism, filter, attenuator, diffraction grating, beam splitter, optical modulator, etc., positioned in a path of the emitted signal and the return signal configured to manipulate and steer light signals.
Yet in another proposed embodiment of present invention, the optical emission unit and optical detection unit comprises at least one of a linear polarizer, circular polarizer, retarder, variable retarder, Pockels cell, photo-elastic modulator, electro-optic modulator, etc., positioned in a path of the emitted signal and/or the return signal.
The present invention further describes a method for target recognition in autonomous driving applications based on the described Polarimetric multifunctional LiDAR system, comprising an emission system, emitting polarized light signals toward surrounding environment; a detection system, detecting the return signals that exhibit polarization changes which results in radiometric flux variations; a control and data acquisition system, controlling the polarimetric multifunctional LiDAR system, acquiring return signals from targets and digitizing the collected data; and
a data processing system, processing the data to: calculate Stokes parameters and/or Mueller matrix elements, calculate the distance to targets and/or targets' velocities, and process the data to recognize targets.
In a proposed embodiment of the disclosed method, the data computed by the data processing system is structured as a data point cloud, including the plurality of points indicating the position and/or velocity of each of the one or more objects in the environment with respect to the LiDAR system (100) and the Stokes parameters and/or elements of the Mueller matrix associated to each point.
Yet in another proposed embodiment of the method, the polarimetric data comprises Stokes parameters of the return light and/or elements of the Mueller matrix of targets in the data processing system, where the polarimetric data is processed by Artificial Intelligence to recognize targets.
Yet in another proposed embodiment of the method, the data processed by Artificial Intelligence further comprises locations and/ or velocities from the point cloud data to determine targets' shapes that are also used for target recognition.
Yet in another proposed embodiment of the method, the data processed by Artificial Intelligence to recognize targets is combined with other data from the polarimetric multifunctional LiDAR system, comprising other point cloud data, illumination angle, used wavelength, generated polarizations, etc.
Yet in another proposed embodiment of the method, the data processing system is further configured to process data associated with the reference signal and data associated with the return signal, to calculate distances to targets and/or targets' velocities.
Yet in another proposed embodiment of the method, the data processed by Artificial Intelligence for target recognition is combined with data from other sensors from the vehicle where the system is installed or data remotely received from other devices.
General Description
The proposed system discloses a polarimetric multifunctional LiDAR system that joins in a single device the capability of light ranging (LiDAR) and active polarimetry (emission and detection of polarized light, even if the return light is partially polarized or depolarized) with integrated system for control, data acquisition and data processing. The polarimetric multifunctional LiDAR sensor is able to measure distances and/or velocities to objects and perform polarimetric measurements that, assisted by Artificial Intelligence, allow the proposed system to recognize targets. The disclosed device is particularly developed for advanced driver assistance systems and autonomous driving.
As previously approached, current autonomous vehicles, which resort to the use of LiDAR systems, only have the ability to measure distances to the surrounding objects and fail to correctly distinguish target objects other than by their shape, which can result in incorrect classification
of objects. To try to overcome some of these issues, the use of LiDAR sensors is combined with other types of sensors, such as video cameras, SONARs and RADARs, to complement object information (the so-called sensor fusion). However, they could still be insufficient to distinguish similar shape objects of different materials, and the distance evaluation of said objects can result in some errors due to misinterpretation of highly reflective surfaces and foreign light signals that can induce ghost targets. It is an object of the present invention to provide an alternative to the prior state of the art, which covers the gaps found therein.
Therefore, the herein disclosed LiDAR system aims to solve the above-mentioned flaws, particularly ensuring all the capacities and specifications of conventional LiDARs, with regard to wide range detection, distance measurement, resolution and eye-safety. The introduction of polarimetry in the herein disclosed LiDAR system, by polarizing the emitted light, will allow obtaining additional information, which was not possible to get by measuring just intensity. The polarimetric information increases the capacity of the sensor to correctly distinguish targets, as well as to enhance the contrast of objects illuminated by the sensor.
The developed polarimetric multifunctional LiDAR system joins in a single device, the ability to light ranging (LiDAR) and active polarimetry (emission and detection of polarized light, even if the light scattered by the target is partially polarized or depolarized). The proposed system simultaneously measures the distance to the target and performs polarimetric measurements, by measuring partially or totally the Mueller matrix of the sample or partially or
totally the Stokes vector of the return light. Moreover, the herein disclosed device could measure the targets' velocity. In addition, the objects located around this polarimetric multifunctional LiDAR are recognized by processing the measured data and by using at least one Artificial Intelligence, introducing a faster and precise discrimination of targets, allowing to minimize decisioning times.
The disclosed system allows the targets classification even when their shapes are difficult to detect (i.e. when they are partially obstructed from view, or the resolution of the LiDAR is low). Nevertheless, the shape of the objects can be used as additional information combined with polarimetric measurements and Artificial Intelligence to enhance the performance of the disclosed sensor.
The proposed polarimetric multifunctional LiDAR system combined with Artificial Intelligences can promote a great impact on autonomous vehicles enhancing the recognition of targets, improving the classification accuracy of targets, being possible to apply this technology to different kinds of autonomous vehicles (cars, trucks, buses and agricultural vehicles, among others).
Brief description of the drawings
For better understanding of the present application, figures representing preferred embodiments are herein attached which, however, are not intended to limit the technique disclosed herein.
Fig. 1 - illustrates one of the proposed embodiments for the Polarimetric multifunctional LiDAR system, in a non coaxial configuration, wherein the reference numbers refer to:
100 - Polarimetric multifunctional LiDAR system;
11 - emission system;
12 - detection system;
13 - light source;
14 - detector;
15 - optical emission unit;
16 - optical detection unit;
17 - optical reference unit;
21 - control and data acquisition system;
22 - data processing system;
30 - reference signal;
31 - transmitted signal;
32 - return signal.
In the proposed embodiment of the system (100), depicted in Figure 1, both emission (11) and detection (12) systems are controlled by a control and data acquisition system (21), and a data processing system (22) is adapted for ranging and target recognition.
Fig. 2 - illustrates the basic operating principle of the proposed Polarimetric multifunctional LiDAR system, wherein the reference numbers refer to:
100 - Polarimetric multifunctional LiDAR system;
11 - emission system;
12 - detection system;
13 - light source;
14 - detector;
15 - optical emission unit;
16 - optical detection unit;
17 - optical reference unit;
21 - control and data acquisition system;
22 - data processing system;
30 - reference signal;
31 - transmitted signal;
32 - return signal;
51 - object/target.
Fig. 3 - illustrates one of the proposed embodiments for the Polarimetric multifunctional LiDAR system, in a coaxial configuration, wherein the reference numbers refer to:
100 - Polarimetric multifunctional LiDAR system;
11 - emission system;
12 - detection system;
13 - light source;
14 - detector;
15 - optical emission unit;
16 - optical detection unit;
17 - optical reference unit;
18 - optical coaxial unit;
21 - control and data acquisition system;
22 - data processing system;
30 - reference signal;
31 - transmitted signal;
32 - return signal.
In the proposed embodiment of the system (100), depicted in Figure 3, both emission (11) and detection (12) systems are controlled by a control and data acquisition system (21), and a data processing system (22) is adapted for ranging and target recognition.
Fig. 4 - illustrates one of the proposed embodiments for the Polarimetric multifunctional LiDAR system, wherein both
emission (11) and detection (12) systems comprise at least one light source (13) and at least one detector (14), respectively controlled by a control and data acquisition system (21), and a data processing system (22) adapted for ranging and target recognition. The reference numbers refer to:
100 - Polarimetric multifunctional LiDAR system;
11 - emission system;
12 - detection system;
13 - light source;
14 - detector;
15 - optical emission unit;
16 - optical detection unit;
17 - optical reference unit;
21 - control and data acquisition system;
22 - data processing system;
30 - reference signal;
31 - transmitted signal;
32 - return signal.
Fig. 5 - illustrates a polarimetric multifunctional LiDAR system adapted to a vehicle for driver-assistance system, wherein the reference numbers refer to:
100 - Polarimetric multifunctional LiDAR system;
31 - transmitted signal;
32 - return signal;
40 - vehicle;
51 - object/target;
52 - obstacle.
Description of Embodiments
With reference to the figures, some embodiments are now described in more detail, which are however not intended to limit the scope of the present application.
The following description is not to be taken in a limiting sense but is made merely for the purpose of illustrating the general principles of the invention. As used in this application and in the claims, the singular forms "a," "an," and "the" include the plural forms unless the context clearly dictates otherwise.
The systems, apparatus, and methods described herein should not be construed as limiting in any way. The disclosed systems, methods, and apparatus are not limited to any specific aspect or feature or combinations thereof, nor do the disclosed systems, methods, and apparatus require that any one or more specific advantages be present, or problems be solved. Any theories of operation are to facilitate explanation, but the disclosed systems, methods, and apparatus are not limited to such theories of operation.
Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed
systems, methods, and apparatus can be used in conjunction with other systems, methods, and apparatus.
On the one hand, the disclosed polarimetric multifunctional LiDAR system (100) measures the target's (51) distance and/or velocity. On the other hand, the disclosed embodiment requires flux measurements (or intensity measurements) to perform polarization analysis. In this sense, two or more polarization generators (that could be generated by the optical emission unit (15)) or analyzers (by the optical detection unit (16)) should be used. The data processing system (22) receives the measured information from the control and data acquisition system (21) and processes data to calculate target's (51) distance and/or velocity, and one or more Stokes parameters, or one or more elements of the Mueller matrix. The main advantage of Stokes-Mueller polarimetry is that, although the return light (32) is not fully polarized, it can be studied. The disclosed polarimetric multifunctional LiDAR system is configured to collect points to produce a point cloud of distances and/or velocities and polarimetric information. The polarimetric information is then used in the recognition procedure. For this, Artificial Intelligence uses the previously processed information to discriminate relevant targets for autonomous driving applications (for example, but not to be taken in a limiting sense, cars, pedestrians, traffic signs, etc.). Moreover, additional information from the point cloud can also be used by Artificial Intelligence, particularly shape, orientation or velocity determination of the targets. Additional information can be used complementary. The information obtained from the disclosed polarimetric multifunctional
LiDAR system (100) can be used by vehicles for dynamic obstacle detection and tracking to improve road safety.
Figures 1 through 4 illustrate several possible embodiments of the proposed polarimetric multifunctional LiDAR system (100), which is comprised of an emission system (11) and a detection system (12), both controlled by a control and data acquisition system (21), and a data processing system (22) configured to perform ranging and target recognition. The polarimetric multifunctional LiDAR system (100) can also include additional units for ranging and for optical coaxial or non-coaxial configurations.
In Figure 1, it is illustrated a non-coaxial polarimetric multifunctional LiDAR system (100), where the emission system (11) is composed of a light source (13) and an optical emission unit (15). The light source (13) can include at least one of a LASER, diode LASER, single frequency LASER diode, tunable LASER, LED, etc. The detection system (12) is composed of a detector (14) and optical detection unit (16). The detector (14) can include at least one of a photodiode, phototransistor, avalanche photodiode, photocounter, array of detectors, focal plane sensor, etc. The optical emission unit (15) and the optical detection unit (16) can be composed of active and/or passive non-polarizing optics and active and/or passive polarizing optics. The non-polarizing optics can include at least one of a lens, diffractive lens, variable focus liquid lens, mirror, micromirror device, prism, filter, attenuator, diffraction grating, beam splitter, optical modulator, etc. The polarizing optics can include at least one of a linear polarizer, circular polarizer, retarder, variable retarder, Pockels cell, photo-elastic modulator,
electro-optic modulator, etc. The control and data acquisition system (21), comprising a collection of software and hardware, is configured to control the polarimetric multifunctional LiDAR system (100) and also to digitize the acquired data. Thus, the control and data acquisition system (21) can be responsible for steering, polarizing and manipulating the signal, by controlling the active elements, acquiring and digitizing the detected signals, the emission electronics, the timing electronics, communication electronics, among others. With the aim of recognizing targets and their locations, the data processing system (22) is responsible for processing the measured data from the polarimetric multifunctional LiDAR (100). In this context, the data processing system (22) computes the Stokes parameters and/or the Mueller matrix elements as well as the position and/or velocity of targets. The plurality of points indicating the position and/or velocity of each of the one or more objects in the environment with respect to the LiDAR system (100) and the Stokes parameters and/or elements of the Mueller matrix associated to each point are structured as a data point cloud. Then, by using Artificial Intelligence, the data processing system (22) processes the data from the point cloud to recognize targets. Nonetheless, the final recognition process and other computation processes could be performed in the data processing system (22), as well as in the central computer of the vehicle where the LiDAR sensor (100) is installed, or even remotely.
As also illustrated in Figure 2, the light source (13) is configured to emit a light signal (31) that passes through an optical emission unit (15) adapted to generate an emitted polarization state and steering and/or manipulating
the emitted signal (31). The outgoing polarized transmitted signal (31) interacts with surrounding targets (51), being scattered and/or reflected back to the LiDAR system (100). As previously described, this interaction of the transmitted light (31) with the target (51) can modify its state of polarization, mainly reducing the degree of polarization. Before reaching the detector (14), the return signal (32) resulting from the target (51) reflection passes through the optical detection unit (16), where it can be steered and filtered; for example, to reduce background light; and its polarization state is analyzed. The control and data acquisition system (21) receives the light flux measurements from the detector (14) and digitizes it.
In order to measure some (or all) Stokes parameters and/or elements of the Mueller matrix, the embodiment described in Figure 1 can work in temporal mode, being the optical emission unit (15) able to generate multiple states of polarization sequentially in time or modulate the polarization of the transmitted signal (31). The states of polarization generated by the optical emission unit (15) resort to the use of, for example, polarizers, photoelastic modulators, electro-optical modulators, variable retarders, magneto-optical modulators, etc. Using the same approach, the optical detection unit (16) could therefore analyze the return signal (32) resorting to the use of several polarization analyzers, taking sequential measurements in time or modulating the return signal (32).
The time of flight principle can be used to calculate the target (51) distance to the LiDAR system (100). In this sense, an additional signal, the so-called reference signal
(30), is needed for reference zero value. The disclosed polarimetric multifunctional LiDAR (100) is configured to send the reference signal (30) from the light source (13) to the detector (14), passing through the optical reference unit (17). The optical reference unit (17) is responsible for steering and/or manipulating the reference signal (30) in order to reach the detector (14). The optical reference unit (17) can be composed of active and/or passive non polarizing optics and active and/or passive polarizing optics. Other design variations for reference zero value can comprise the use of a light source (13) with an integrated detector or an independent detector dedicated for time reference. Moreover, the reference signal (30) could also pass through some elements of the optical emission unit (15) and optical detection unit (16). Other methods to measure the target (51) distance and/or velocity to the LiDAR system (100) are not excluded, such as frequency-modulated continuous-wave (FMCW).
The digitized data from the control and data acquisition system (21) is computed by the data processing system (22), where the Stokes parameters and/or Mueller matrix elements are calculated. In addition, the distances to targets (51) and/or their velocities can be also computed in the data processing system (22). Then, the point cloud with the polarimetric information from the scene is used by an Artificial Intelligence, duly trained for recognizing the desired target (51) categories. Other information; such as, for example, the shape of the surrounding objects, illumination angle, used wavelength, etc.; can also be used by the Artificial Intelligence. Moreover, in order to improve the accuracy of target (51) recognition, data collected by other vehicle sensors, for example, RADARs,
cameras, etc., can also be included and used by the Artificial Intelligence. Note that part of (or all) the data processing can instead be performed in the central data computer system of the vehicle where the LiDAR sensor (100) is installed or remotely.
The basic principle of the polarimetric multifunctional LiDAR system (100) operation is illustrated in Figure 2, where the polarized transmitted light (31) interacts with the target (51). The return signal (32) is reflected back by the target (51) and detected by the polarimetric multifunctional LiDAR (100). The distance to the object (51), as well as the target recognition procedure, are obtained through the described calculations.
On this basis, Figure 3 discloses an alternative embodiment of the one depicted system in Figure 1, mainly differing on the use of an optical coaxial configuration. In this sense, an optical coaxial unit (18) is used to steer and/or manipulate the transmitted signal (31) and the return signal (32) in order to have the same optical axis at the output of the polarimetric multifunctional LiDAR system (100). This optical coaxial unit (18) can be composed of at least one of an active or passive optical element including lenses, diffractive lenses, variable focus liquid lenses, mirrors, micro-mirrors, prisms, filters, attenuators, diffraction gratings, beam splitters, optical modulators, etc. As in the first embodiment disclosed in Figure 1, in this solution, the data processing system (22) computes and interprets the measurements of the collected return signal (32). In an equivalent way, the control and data acquisition system (21) is responsible for controlling the
polarimetric multifunctional LiDAR system (100) and acquiring the light flux information.
In Figure 4, it is disclosed another embodiment of the disclosed polarimetric multifunctional LiDAR system (100), where the emission system (11) can comprise multiple light sources (13) and/or optical emission units (15) and/or the detection system (12) can have multiple detectors (14) and/or optical detection units (16). This configuration can be used to obtain multiple emissions with the same or different generated states of polarization and with the same or different wavelengths. Moreover, the polarimetric multifunctional LiDAR system (100) can use multiple detectors (14) and/or multiple optical detection units (16). In this sense, several optical detection units (16), each one working as a different polarization analyzer, with its own associated detector (14), allow the disclosed system (100) to be capable of measuring the Stokes vector of the return signal (32) instantaneously. Furthermore, the data processing system (22) is configured to compute and interpret the measurements of the received return signal (32). In an equivalent way, the control and data acquisition system (21) is responsible for controlling the polarimetric multifunctional LiDAR system (100), acquiring data and digitizing the measured information.
Figure 5 provides an illustration of a vehicle (40) comprising the disclosed polarimetric multifunctional LiDAR system (100). As previously referred, the point cloud produced by conventional LiDARs can be used to determine location and/or velocity of targets (51), but also identify said targets through their shapes. By using sensor fusion, i.e., combining data from multiple sensors installed in the
vehicle (40), such as video cameras, RADARs, conventional LiDARs, etc., the accuracy in the target (51) recognition process can be improved. However, an object partially obstructed / hidden can still be difficult to classify. Polarimetry together with a trained Artificial Intelligence can be used to recognize targets at each data point of the point cloud. In contrast to conventional LiDARs which need multiple points to identify objects by their shape, each measurement of the point cloud of the disclosed polarimetric multifunctional LiDAR system (100) could be recognized as a particular target. The method associated with the disclosed polarimetric multifunctional LiDAR system (100) improves the target recognition performance in cases where the object (51) is partially blocked by obstacles (52) from the point of view of the vehicle (40) where the proposed LiDAR system (100) is installed. Therefore, the disclosed polarimetric multifunctional LiDAR system (100) combines conventional methods for object recognition and point clouds with polarimetry, offering a new approach for improving object recognition.
While the present invention has been described with respect to the preferred embodiments thereof, it will be apparent to those skilled in the art that the disclosed invention may be modified in numerous ways and may assume many embodiments other than those specifically set out and described above. Accordingly, it is intended by the appended claims to cover all modifications of the present invention that fall within the true spirit and scope of the invention.
Claims
1. Polarimetric multifunctional LiDAR system (100) for target recognition in autonomous driving applications, comprising: an emission system (11), configured to emit polarized signals (31); a detection system (12), configured to detect return signals (32) from targets (51); a control and data acquisition system (21), configured to control both the emission system (11) and detection system (12) and to acquire and digitize the return signals (32); and a data processing system (22), connected to the control and data acquisition system (21); wherein the data processing system (22) is configured to determine Stokes parameters of the return signals (32) and/or Mueller matrix elements of the targets (51) and combine the determined Stokes parameters and/or Mueller matrix elements with Artificial Intelligence to determine information about surrounding environment, providing targets (51) recognition, along with targets (51) positions and/or velocity.
2. Polarimetric multifunctional LiDAR system (100) according to the previous claim, wherein the emission system (11) comprises at least one light source (13) and at least one optical emission unit (15).
3. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the detection system (12) comprises at least one detector (14) and at least one optical detection unit (16).
4. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the at least one optical emission unit (15) and the at least one optical detection unit (16) are comprised of at least one of an active and/or passive non-polarizing optics and active and/or passive polarizing optics, configured to manipulate, polarize and steer light signals.
5. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the control and data acquisition system (21) comprises a collection of software and hardware, configured to control the emission system (11) and the detection system (12); acquire return signals (32) from targets (51); and digitize the return signals (32).
6. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the data processing system (22) is configured to receive data from the control and data acquisition system (21); calculate Stokes parameters and/or Mueller matrix elements; calculate the distance to targets (51) and/or targets (51) velocities; and recognize targets (51) supported by Artificial Intelligence.
7. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the at least one light source (13) of the emission system (11) comprises at least one emitted polarized light signal (31) and/or at least one transmitted wavelength.
8. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the at least one optical emission unit (15) of the emission system (11) comprises at least one polarization generator configured to modify the polarization of the emitted polarized light signals (31).
9. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, wherein the at least one optical detection unit (16) of the detection system (12) comprises at least one polarization analyzer configured to analyze the polarization of the return signals (32) from targets (51).
10. Polarimetric multifunctional LiDAR system (100) according to any of the previous claims, comprising an optical reference unit (17) configured to steer and/or manipulate a reference signal (30) defined as a portion of the emitted polarized signals (31).
11. Method for target recognition in autonomous driving applications according to the Polarimetric multifunctional LiDAR system (100) described in any of the previous claims 1 to 10, comprising: an emission system (11), emitting polarized light signals (31) toward surrounding environment; a detection system (12), detecting the return signals (32) that exhibit polarization changes which results in radiometric flux variations; a control and data acquisition system (21), controlling the polarimetric multifunctional LiDAR system (100), acquiring return signals (32) from targets (51) and digitizing the collected data; and
a data processing system (22), processing the data to: calculate Stokes parameters and/or Mueller matrix elements, calculate the distance to targets (51) and/or targets' velocities, and process the data to recognize targets (51).
12. Method according to the previous claim 11, wherein the data computed by the data processing system (22) is structured as a data point cloud, including the plurality of points indicating the position and/or velocity of each of the one or more objects in the environment with respect to the LiDAR system (100) and the Stokes parameters and/or elements of the Mueller matrix associated to each point.
13. Method according to any of the previous claims 11 and 12, wherein the polarimetric data comprises Stokes parameters of the return signal (32) and/or elements of the Mueller matrix of targets (51) in the data processing system (22), where the polarimetric data is processed by Artificial Intelligence to recognize targets (51).
14. Method according to any of the previous claims 11, 12 and 13, wherein the data processed by Artificial Intelligence further comprises locations and/ or velocities from the point cloud data to determine targets' shapes for target recognition.
15. Method according to any of the previous claims 11, 12, 13 and 14, wherein the data processed by Artificial Intelligence for target recognition is combined with other data from other sensors from the vehicle (40) where the system (100) is installed; and/or data remotely received from other devices.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PT11735721 | 2021-07-21 | ||
| PCT/IB2021/056652 WO2023002237A1 (en) | 2021-07-21 | 2021-07-22 | Polarimetric multifunctional lidar sensor for target recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4352536A1 true EP4352536A1 (en) | 2024-04-17 |
Family
ID=77155826
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP21748959.0A Pending EP4352536A1 (en) | 2021-07-21 | 2021-07-22 | Polarimetric multifunctional lidar sensor for target recognition |
Country Status (2)
| Country | Link |
|---|---|
| EP (1) | EP4352536A1 (en) |
| WO (1) | WO2023002237A1 (en) |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7580127B1 (en) | 2006-07-21 | 2009-08-25 | University Corporation For Atmospheric Research | Polarization lidar for the remote detection of aerosol particle shape |
| US8054464B2 (en) | 2010-01-25 | 2011-11-08 | Sigma Space Corp. | Polarization switching lidar device and method |
| DE102016220468A1 (en) | 2016-10-19 | 2018-04-19 | Robert Bosch Gmbh | Lidar sensor for detecting an object |
| US11561084B2 (en) * | 2017-04-19 | 2023-01-24 | Arizona Board Of Regents On Behalf Of The University Of Arizona | Polarization sensitive devices, methods and applications |
| WO2020021306A1 (en) | 2018-07-23 | 2020-01-30 | Bosch Car Multimedia Portugal, S.A. | Method for material discrimination and respective implementation system |
| CN114026458A (en) * | 2019-04-17 | 2022-02-08 | 密歇根大学董事会 | Multi-dimensional material sensing system and method |
-
2021
- 2021-07-22 WO PCT/IB2021/056652 patent/WO2023002237A1/en active Application Filing
- 2021-07-22 EP EP21748959.0A patent/EP4352536A1/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023002237A1 (en) | 2023-01-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Khader et al. | An introduction to automotive lidar | |
| US8264536B2 (en) | Depth-sensitive imaging via polarization-state mapping | |
| US7339670B2 (en) | Wavelength normalized depolarization ratio lidar | |
| US7161664B2 (en) | Apparatus and method for optical determination of intermediate distances | |
| US8078410B2 (en) | Sensing using polarization diversity and wavelength dependent backscatter | |
| US11561084B2 (en) | Polarization sensitive devices, methods and applications | |
| EP2705350B1 (en) | Remote measurement of shallow depths in semi-transparent media | |
| JP4405155B2 (en) | Image analysis system | |
| CN103499818B (en) | A kind of infrared and laser compound detection system | |
| CN106338342B (en) | Space-time hybrid modulation infrared interference spectrum imaging system quasi real time and methods and applications | |
| US11619722B2 (en) | Vehicle lidar polarization | |
| CN107843888A (en) | Coherent lidar system for automated vehicle | |
| US11681033B2 (en) | Enhanced polarized light collection in coaxial LiDAR architecture | |
| WO2020021311A1 (en) | Terrestrial vehicle range finder device and operation method thereof | |
| EP3271749B1 (en) | Multiple-beam triangulation-based range finder and method | |
| US20220229162A1 (en) | Scanning lidar with optical switching | |
| WO2023002237A1 (en) | Polarimetric multifunctional lidar sensor for target recognition | |
| WO2020021306A1 (en) | Method for material discrimination and respective implementation system | |
| CN111965663A (en) | Optical system, in particular lidar system, and vehicle | |
| WO2023047175A1 (en) | Road monitoring with polarimetry by multifunctional lidar | |
| US11002666B2 (en) | Bistatic electro-optical device for substance-on-surface chemical recognizer | |
| JP2023547877A (en) | Peak association technology for multiple targets in coherent LIDAR systems | |
| EP3608653B1 (en) | Apparatus and method for measuring particle size using backscattered light | |
| Giakos et al. | Superresolution multispectral imaging polarimetric space surveillance LADAR sensor design architectures | |
| JP3096795B2 (en) | Tracking ranging system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20240112 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |