WO2022093303A1

WO2022093303A1 - Unisource multi-modal perception in autonomous vehicles

Info

Publication number: WO2022093303A1
Application number: PCT/US2021/018306
Authority: WO
Inventors: Jian Li; Han SU
Original assignee: Futurewei Technologies, Inc.
Priority date: 2021-02-17
Filing date: 2021-02-17
Publication date: 2022-05-05

Abstract

Techniques are presented to improve the operation of autonomously driving automobiles and other transportation or robotic equipment with varying degrees of autonomous operation. This includes systems and methods that combine, for example, both frequency, such as color for a camera-based system, and polarization vision in the perception of the sensor systems of autonomous driving systems. Multi-modality can be used by one or more type of sensor device system, such as lidar, radar, ultrasound (sonar), of electromagnetic waves with polarization, multiple frequencies and wavelengths. The sensor systems can also use a hierarchical modality between single (yet multiple sensing data type) and multiple sensors.

Description

UNISOURCE MULTI-MODAL PERCEPTION IN AUTONOMOUS VEHICLES

FIELD

[0001] The following is related generally to the field of autonomously operating systems and, more specifically, to autonomous driving vehicles.

BACKGROUND

[0002] Automobiles and other vehicles are becoming more autonomous, as both fully autonomous vehicles (AVs) and in systems of varying degrees of autonomous operation to assist a driver or operation. These systems rely on inputs from sensors, such as cameras receptive to light in the visible spectrum or lidar based sensors, for example. Processors on the vehicle use the inputs from these sensor systems to determine control inputs for control systems of the vehicle, such as for steering and braking. If these sensors systems incorrectly sense the environment within which the vehicle is operating, whether through misperception or by being intentionally spoofed, incorrect control inputs may be determined, and the control systems can operate the vehicle in error with possibly catastrophic results. As the numbers of such AVs, and the number of autonomous systems within even vehicles that are not fully autonomous, increases, it is important to improve the safety and performance of such systems.

SUMMARY

[0003] According to one aspect of the present disclosure, an autonomous vehicle includes: an electro-mechanical control system configured to receive control inputs and control operation of the autonomous vehicle in response thereto; a sensor system configured to sense a plurality of different modalities of an electromagnetic sensor signal over a period of time; and one or more processing circuits connected to the electro-mechanical control system and the sensor system. The one or more processing circuits are configured to: receive, from the sensor system, the plurality of modalities of the electromagnetic sensor signal as sensed over the period of time; generate, from the plurality of modalities of the electromagnetic sensor signal as sensed over the period of time, an intermediate output for each of the modalities of the electromagnetic sensor signal as sensed over the period of time; perform a comparison of the intermediate outputs generated from the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time; and, based on results of the comparison, generate and provide the control inputs to the electro-mechanical control system.

[0004] Optionally, in the preceding aspect, in comparing the intermediate outputs generated from the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time, the one or more processing circuits are configured to perform majority voting operations between the intermediate outputs.

[0005] Optionally, in any of the preceding aspects, the one or more processing circuits are further configured to: determine whether to operate in a multi-modality mode or single-modality mode, where the intermediate outputs are generated and compared in response determining to operate in a multi-modality mode; and, in response to determining to operate in single-modality mode, generate and provide the control inputs to the electro-mechanical control system based on a single modality of the electromagnetic sensor signal over the period of time.

[0006] Optionally, in any of the preceding aspects, the sensor system includes a receiver configured to sense the plurality of modalities of the electromagnetic sensor signal.

[0007] Optionally, in the preceding aspect, the receiver includes a plurality of polarization filters, the plurality of different modalities corresponding to the electromagnetic sensor signal as filtered by the plurality of polarization filters. [0008] Optionally, in any of the preceding two aspects, the receiver includes a plurality of frequency filters, the plurality of different modalities corresponding to the electromagnetic sensor signal as filtered by the plurality of frequency filters.

[0009] Optionally, in any of the preceding aspects, the sensor system includes a transmitter configured to emit the plurality of modalities of the electromagnetic sensor signal.

[0010] Optionally, in any of the preceding aspects, the sensor system includes a lidar system.

[0011] Optionally, in any of the preceding aspects, the sensor system includes a radar system.

[0012] Optionally, in any of the preceding aspects, the sensor system includes a visual spectrum camera system.

[0013] Optionally, in any of the preceding aspects, the sensor system further includes a sonar system.

[0014] Optionally, in any of the preceding aspects, the sensor system is further configured to sense multiple modalities of a sonar signal.

[0015] Optionally, in the preceding aspect, the multiple modalities of the sonar signal include different frequencies.

[0016] Optionally, in any of the preceding aspects, the electro-mechanical control system includes a steering system.

[0017] Optionally, in any of the preceding aspects, the electro-mechanical control system includes a speed control system.

[0018] According to an additional aspect of the present disclosure, there is provided a method of controlling an autonomous system that includes: sensing by a sensor system of a plurality of different modalities of an electromagnetic sensor signal over a period of time; receiving, at one or more processing circuits from the sensor system, the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time; and generating, by the one or more processing circuits from the plurality of modalities of the electromagnetic sensor signal as sensed over the period of time, an intermediate output for each of the modalities of the electromagnetic sensor signal as sensed over the period of time. The method also includes: performing, by the one or more processing circuits for each of the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time, a comparison of the intermediate outputs generated from the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time; based on the comparison, generating by the one or more processing circuits of control inputs for an electro-mechanical control system of the autonomous system; providing the control inputs to the electro-mechanical control system; and controlling of the autonomous system by the electro-mechanical control system in response to the control inputs.

[0019] Optionally, in the preceding aspect of a method, performing the comparison of the intermediate outputs includes majority voting between the intermediate outputs.

[0020] Optionally, in any of the preceding aspects of a method, the method further includes: determining whether to operate in a multi-modality mode or single-modality mode, wherein generating the intermediate outputs is in response determining to operate in a multi-modality mode; and, in response to determining to operate in singlemodality mode, generating and providing the control inputs to the electro-mechanical control system based on a single modality of the electromagnetic sensor signal over the period of time.

[0021] Optionally, in any of the preceding aspects of a method, the method further includes emitting by the sensor system of the plurality of different modalities of the electromagnetic sensor signal over the period of time during which the autonomous system is in operation.

[0022] Optionally, in any of the preceding aspects of a method, the method further includes filtering the plurality of different modalities of the electromagnetic sensor signal by a plurality of different polarization filters, wherein the plurality of different modalities of the corresponding sensor signal correspond to different polarizations of the electromagnetic sensor signal. [0023] Optionally, in any of the preceding aspects of a method, the method further includes filtering the plurality of different modalities of the electromagnetic sensor signal by a plurality of different polarization filters, wherein the plurality of different modalities of the corresponding sensor signal correspond to different frequencies of the electromagnetic sensor signal.

[0024] Optionally, in any of the preceding aspects of a method, the sensor system includes a lidar system.

[0025] Optionally, in any of the preceding aspects of a method, the method further includes a radar system.

[0026] Optionally, in any of the preceding aspects of a method, the method further includes a visual spectrum camera system.

[0027] According to other aspects, a control system for autonomously operable equipment includes one or processing circuits configured to: receive, from a sensor system, a corresponding sensor signal as sensed in a plurality of different modalities over a period of time; perform a majority voting between the plurality of different modalities of the corresponding sensor signal as sensed over the period of time; and, based on results of the majority voting, generate and provide control inputs for an electro-mechanical control system for the autonomously operable equipment.

[0028] In the preceding aspect for a control system for autonomously operable equipment, the control system can further include the electro-mechanical control system, wherein the electro-mechanical control system are configured to receive the control inputs and to control the operation of the autonomously operable equipment in response thereto.

[0029] In either of the preceding two aspects for a control system for autonomously operable equipment, the control system can further include the sensor system, wherein the sensor system includes a receiver configured to sense the corresponding sensor signal in the plurality of different modalities.

[0030] In the preceding aspect for a control system for autonomously operable equipment, the receiver includes a plurality of polarization filters, the plurality of different modalities corresponding to the sensor signal as filtered by the plurality of polarization filters.

[0031] In either of the preceding two aspects for a control system for autonomously operable equipment, the receiver includes a plurality of frequency filters, the plurality of different modalities corresponding to the sensor signal as filtered by the plurality of frequency filters.

[0032] In any of the preceding aspects for a control system for autonomously operable equipment, the sensor system includes a transmitter configured to emit the corresponding sensor signal.

[0033] In any of the preceding aspects for a control system for autonomously operable equipment, the sensor system includes a lidar system.

[0034] In any of the preceding aspects for a control system for autonomously operable equipment, the sensor system includes a radar system.

[0035] In any of the preceding aspects for a control system for autonomously operable equipment, the sensor system includes a visual spectrum camera system.

[0036] In any of the preceding aspects for a control system for autonomously operable equipment, the sensor system includes a sonar system.

[0037] In any of the preceding aspects for a control system for autonomously operable equipment, the autonomously operable equipment is an autonomous vehicle.

[0038] In any of the preceding aspects for a control system for autonomously operable equipment, the autonomously operable equipment is robotic equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying figures for which like references indicate elements.

[0040] FIG. 1 is a block diagram illustrating some of the elements that can be incorporated into a generic autonomous vehicle system.

[0041] FIG. 2 is a schematic representation of a lidar sensor system for an autonomous vehicle.

[0042] FIG. 3 is a schematic representation of a lidar sensor system for an autonomous vehicle that uses multiple polarizations.

[0043] FIG. 4 is a flowchart for an embodiment for the multi-modal operation of a sensor system of an autonomous vehicle or other autonomously operating system.

[0044] FIG. 5 is a more detailed flow of an embodiment for multi-modal signal processing.

[0045] FIG. 6 illustrates triple modular redundant architecture, in which three CPUs are run in parallel in a lockstep manner and the resultant outputs are compared.

[0046] FIG. 7 is a high-level block diagram of a more general computing system that can be used to implement various embodiments described in the preceding figures.

DETAILED DESCRIPTION

[0047] The following presents techniques to improve the operation for autonomously driving automobiles and other transportation or robotic equipment with varying degrees of autonomous operation by introduction of unisource multi-modal perception. This includes systems and methods that combine, for example, both frequency, such as color for a camera-based system, and polarization vision in the perception of the sensor systems of autonomous driving systems. Multi-modality can be used by one or more type of sensor device system, such as lidar, radar, ultrasound (sonar), of electromagnetic waves with polarization, multiple frequencies and wavelengths. The sensor systems can also use a hierarchical modality between single (yet multiple sensing data type) and multiple sensors.

[0048] It is understood that the present embodiments of the disclosure may be implemented in many different forms and that claims scopes should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the inventive embodiment concepts to those skilled in the art. Indeed, the disclosure is intended to cover alternatives, modifications and equivalents of these embodiments, which are included within the scope and spirit of the disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present embodiments of the disclosure, numerous specific details are set forth in order to provide a thorough understanding. However, it will be clear to those of ordinary skill in the art that the present embodiments of the disclosure may be practiced without such specific details.

[0049] FIG. 1 is a block diagram illustrating some of the elements that can be incorporated into a generic autonomous vehicle system. Depending on the particular embodiment, the system may not contain all of the elements shown in FIG. 1 and may also include additional elements not shown in FIG. 1. The following discussion will mainly be presented in the context of an autonomously operating automobile, but can also apply to other vehicles or robotic systems and to varying degrees of autonomy. For example, many current non-autonomous vehicles employ driver assist systems that can apply many of the techniques described here, such as a radar or lidar based system that monitors the distance to another car and provides warnings to the driver.

[0050] The autonomous vehicle of FIG. 1 includes a set of sensors 101 that the vehicle uses to perceive the environment through which it moves or operates in. These sensors receive physical signals, typically in an analog form, that can then be converted into a digital form through analog to digital (A/D) converters before supplying their output to the processing circuitry of the in-vehicle computer 121. One of the sensors can be a camera system 103 that can sense light in or near (e.g., infrared) the visible spectrum. Depending on the embodiment, the camera system 103 can be a single camera or multiple camera, such as located to have differing fields of view or being sensitive to different frequencies of the visible or near visible portions of the electromagnetic spectrum. The camera system 103 will sense light present in the environment, but also light from the AV itself, such as from headlights or, in some cases, light emitted specifically for the use of the camera system 103 itself.

[0051] The sensors 101 can also have other systems that make use of the electromagnetic spectrum, such as a radar system 105 or lidar system 107. The radar system 105 can include one or more transmitters producing electromagnetic waves in the radio or microwaves domain, one or more transmitting antennas, and one or more receiving antennas, where the same antenna can be used for both transmitting and receiving in some embodiments. The lidar system 107 can be used to measure distances (ranging) by use of transmitting laser light and measuring the reflections. Differences in laser return times and wavelengths can then be used to determine a three dimensional representation of the autonomous vehicle’s environment. A sonar system 109 can use sound waves to provide information on the autonomous vehicle’s environment. The radar system 105, lidar system 107, and sonar system 109 will typically emit signals as well as monitor received signals.

[0052] The sensors 101 can also include a GPS system 111 that receives signals from global positioning satellites (GPS) or, more generally, global navigation satellite systems (GNSS) that provide geolocation and time information. The sensors can also include inertial measurement units (IMU) 113, such as accelerometers, that can be used to detect movement of the autonomous vehicle.

[0053] The outputs from the sub-systems of the sensors 101 are then provided to the in-vehicle computer systems 121 over a bus structure 119 for the autonomous vehicle. The in-vehicle computer systems 121 can include a number of digital processors (CPUs, GPUs, etc.) that then process the inputs from the sensors 101 for planning the operation of the autonomous vehicle, which are translated into the control inputs for the electrical-mechanical systems used to control the autonomous vehicle’s operation. In this schematic representation, the one or more processing units of the in-vehicle computer systems 121 include a block 123 for major processing of the inputs from the sensors 101 , including deep neural networks (DNNs) for the driving operations, including: obstacle perception, for determining obstacles in the AV’s environment; path perception, for determining the vehicle’s path; wait perception, for determining the rate of progression along the path; and data fusion, that assembles and collates the various perception results. A mapping and path planning block 125 is configured to take the inputs from the DNN block 123 and determine and map the autonomous vehicle’s path, which is then used in the control block 127 to generate the control signal inputs provided to the electro-mechanical systems used to operate the autonomous vehicle or system. Although broken down into the blocks 123, 125, and 127 in FIG. 1 , the one or more processors corresponding to these blocks can perform functions across multiple ones of the blocks.

[0054] The control inputs from the in-vehicle computer 121 provides control inputs to the electro-mechanical systems used to control the operation of the autonomous vehicle. Each of these electro-mechanical systems receives a digital input from the in-vehicle computer 121 , which is typically converted by each of the systems to an analog signal by a digital to analog (D/A) conversion to generate an analog signal used for actuators, servos, or other mechanisms to control the vehicles operation. The control systems can include steering 131 ; braking 133; speed control 135; acceleration control 137; and engine monitoring 139.

[0055] As noted above, many of the systems for the sensors 101 are both signal generators and signal sensors. This is true of the radar system 105, the lidar system 107, and the sonar system 109. This can also be true of the camera system 103, where this can be used to receive light present in the environment, but the system can also be a generator of signals in the visible or near visible electromagnetic spectrum, such as by emitting infra-red light signals or even through the headlights when operating at night or low light situations.

[0056] The sensors 101 , such as the camera system 103 implemented as a conventional RGB (red/green/blue), pixel-based computer vision system, are often not reliable due to environmental situations such as rain, fog, dust, outliers, and other sources of error, making them easy to spoof. This can result in operational error and even accidents. The use of multiple types of sensor systems can improve the situation, but if one or more of the sensors provides erroneous inputs to the in-vehicle computer 121 , this can result reduced reliability for the control inputs to the electromechanical system operating the autonomous system. To further improve the perception abilities of electric vehicles, autonomous driving systems, and other such systems, the following introduces the use of multi-modal perception and presents embodiments that provide this capability with the introduction of additional major components. The multi-modal perception for a given sensor system can include the use of different frequencies in the perception of sensor signals (such as for the sound signals of the sonar system 109 and the electromagnetic waves of the camera system 103, radar system 105, or lidar system 107) or in the perception of different polarizations (such as for the transverse electromagnetic waves of the camera system 103, radar system 105, or lidar system 107). The following discussion mainly focusses on the example of the use of multiple polarization modes for the lidar system 107, but more generally extends to other multi-modalities of other sensor systems.

[0057] Transverse waves, such as an electromagnetic wave, exhibit polarization. When travelling in the +z direction, the polarization of an electromagnetic wave is determined by the relative amplitude and phase of the x and y components of the wave. For example, when an electric field E is polarized at 45° relative to the x-axis, the x and y components (Ex, E_y) are of equal amplitude and in phase, where other relative amplitudes between the x and y components will result with a different polarization angle. For a circular polarization travelling in the +z-direction, the x and y components are again of equal amplitude, but the E_y component is a quarter wavelength behind the Ex component, resulting in a left circular polarization. Polarization is a well-understood physical phenomenon and embodiments presented below present sensor systems that detect electromagnetic wave of multiple polarities, such as through use of polarization filters with axes aligned at different angles, to improve the detection of depth, texture, and shape, manage reflections, and suppress glare from the surface of lakes or other reflective surfaces.

[0058] Polarization vision (such as found with some invertebrate animals) is complementary to color vision (as with vertebrate animals) and applies to light and other electromagnetic waves, making possible better perception. As described below, a single sensor type can offer multiple wavelengths (or frequencies) in conjunction with polarization for multi-modal perception, increasing the accuracy and reliability of autonomous vehicles and other sensor-based autonomous systems.

[0059] The phenomenon of polarization is familiar from the glare that often results from light scattered off of a reflective surface, such as a water surface. The effect of a polarization filter on glare can be to significantly increase the ability to see what lies behind the reflective surface. For example, when a polarizer has its axis at one angle the level of glare is such that what is below the surface cannot be detected; but when the polarization filter is rotated by 90° it is possible to see through the glare and perceive features under the surface. By sensing multiple polarizations by a sensor system, more information can be used to produce more reliable inputs for the control systems of autonomous systems.

[0060] FIG. 2 is a schematic representation of a lidar sensor system for an autonomous vehicle, such as the lidar system 107 of FIG. 1 , without the incorporation of multi-modal operation. Lidar is light detection and ranging, or laser imaging, detection, and ranging system is a combination of laser scanning and three dimensional scanning that can be used to generate a 3-D image of the environment about the autonomous vehicle. A signal processing block 201 , such as can be formed of one or more processors, can control a laser transmitter 203 that provides a laser to scan optics 207 and transmit the lidar signal to the environment about the autonomous vehicle. This is commonly a rotating transmitter (as indicated by the arrow) mounted on the autonomous vehicle. The same scan optics 207 can also be the receiver or, alternately or additionally, one or more additional receivers can be mounted on the autonomous vehicle.

[0061] The beam transmitted from the scan optics will reflect off of objects, such as target 209, in the vicinity of the autonomous vehicle as a reflected beam. The reflected beam is then received at the scan optics 207 and/or other lidar sensor, with the result then supplied to the receiver 205, which also receives input from the laser transmitter 203. Based on comparing the transmitted and received signals supplied to the receiver 205, the result is supplied to the signal processing 201. This data can then be used to generate an image, or 3-D point cloud, of the obstacles in the vicinity of the autonomous vehicle by the DNNs 123 of the in-vehicle computer 121. The neural networks of 123 can then generate a three dimensional point cloud 211 of objects in the environment that can then be used by the mapping, path planning block 125.

[0062] The embodiments presented in the following incorporate additional polarizer lens/films for sensing at varied frequencies, polarization angles, or both. Additional digital signal processing features can include features to realize learning-to-learn policies and hierarchical modalities. Multiple frequency/wavelength and polarization perception can be provided from a source of a single sensor device rather, rather than or in addition to multiple sensor devices. The system can combine polarization perception and frequency perception (such as RGB/pixel perception for the camera system 103) from one source of the signal emitter with the system support of the in- vehicle computer 121. Policy adjustment can be implemented via learning-to-learn between sensor hierarchical perception modality based on external environmental changes, operator preference, or other trusted sources. The following discussion is mainly presented in the context of the lidar system 107 and the emitted/received lidar signal pairs, but can also apply to the sonar system 109, radar system 105, and camera system 103. The lidar system 107, sonar system 109, and radar system typically emit their own sensor signals, but this can also be the case of the camera system 103 that can emit light in the visual and near visual (such as infrared) spectrum: for example, the headlights of a automobile serve as emitters for the camera system 103. In the case of the sonar system, sound waves are longitudinal and do not display polarization, but multi-modal operation can be implemented through use of different frequencies. FIG. 3 illustrates the multi-modality approach as applied to a lidar system and multiple polarizations.

[0063] FIG. 3 is a schematic representation of a lidar sensor system for an autonomous vehicle that uses multiple polarizations. FIG. 3 is arranged similarly to FIG. 2 and uses similar numbering, but now incorporates the use of multiple modalities and majority voting or other comparisons. A laser transmitter 303 again provides the laser light to the scan optics 307 that emits the transmitted beam. Relative to FIG. 2, the transmitted beam now is transmitted with multiple polarizations. Alternately or additionally, the laser can emit multiple frequencies. In some embodiments, the polarizing elements can be a “plug-n-play” arrangement, such as a rigid glass or flexible film of different polarization angles mounted around existing scan optics of lidar system to achieve single-source multi-modality. To avoid an ambiguous result (i.e., a tie), the polarizing element can have an odd number of sides, i = [1 , N] where N is an odd number, and hence an odd number of polarization options to improve the robustness of majority voting. Although FIG. 3 shows a lidar example, similar arrangements could be applied to transmitted/received signal pairs for radar, sonar, and a camera plus headlight arrangement, for example. [0064] FIG. 3 illustrates two examples of form factors that can be used for the polarizer of the scan optics. The embodiment of polarizer 371 show a top view of a pentagon shape polarizer of 3 angles (which may include a non-polarization one) around the central scan optics for transmitting and receiving the laser beam. The different facets can correspond to different polarization filters. For embodiments using different frequency modalities, each facet could have a different frequency filter, for example. Another possible form factor is illustrated by the triangular form factor 373. The number of different polarizations received is a design decision, as more polarization angles provide more information, but increase complexity and, as the differences in polarization angles are smaller, the additional information gained incrementally decreases.

[0065] The beam as transmitted by the scan optics will then scatter off of objects in the vicinity of the autonomous vehicle, such as represented by target 309, and the beams reflected back are then sensed by the scan optics 307 or other receivers of the system. The multiple sensed modalities can then be supplied to the receiver 305 and passed on for signal processing 301 , where this can be as described above with respect to single modality case of Figure 2, except being done for each of the modalities. The final 3-D point cloud 311 is then generated by the majority vote block 321 based on the intermediate outputs. Depending on the embodiment, different amounts of processing can be done to generate the intermediate output results. For example, this could be computing the full 3-D point cloud 311 for each of the modalities and comparing these, or the comparison could be performed at an earlier stage, where this decision is a trade-off between a fully data set to compare and increased computational complexity.

[0066] Relative to the lidar system of FIG. 2, the embodiment of FIG. 3 illustrates a number of aspects that can improve the perceptional abilities of the autonomous systems. The system of FIG. 3 senses multi-frequency/wavelength polarization perception at a single sensor device; combines polarization perception and frequency perception from one source of the signal emitter with corresponding system support; and policy adjustment can be implemented via learning-to-learn between sensor hierarchical perception modality based on external environmental changes, operator preference, or other trusted sources. This differs from previous arrangements, such as a conventional RGB/pixel-based computer vision methods that achieve multimodality by multiple types of sensors. Deep learned based computer vision alone is often not reliable due to environmental situations such as rain, fog, dust, outliers, and so on, and easy to spoof.

[0067] FIG. 4 is a flow chart for an embodiment for the multi-modal operation of a sensor system of an autonomous vehicle or other autonomously operating system as described with reference to FIG. 3. FIG. 5 provides additional detail relative to the higher level flow of FIG. 4. Staring at 401 , a sensor system senses a corresponding signal (as electromagnetic signal in the case of the lidar system 107, the sonar system 109, radar system 105, or camera system 103, or a sound signal in the sonar system109 example) over a period of time. Referring to the example of FIG. 3, this is sensing of the reflected beams as sensed by the single source of the sensor device at the scan optics 307, where the multiple different modalities can be separated out by the different polarization and/or frequency filters, such as on the different facets of embodiments 371 and 373.

[0068] At 403 the different modalities of the sensor signal, such as different polarizations, different frequencies, or both, as reflected of objects (e.g., target 309) in the vicinity of the autonomous vehicle are received at processing circuits of the system, where, with respect to FIG. 3, this can correspond to receiver 305, the signal processing block 301 , or elements of both. Relative to FIG. 1 , the receiver 305 of FIG. 3 can be part of the (in the lidar example) lidar system 107 and the signal processing 301 of FIG. 3 can correspond to part of the drive DNNs 123 or, in some cases, some or all of the signal processing can be incorporated into the lidar sensor system 107. The intermediate outputs are generated from the multi-modality sensor signals by the processing circuits at 405, where this processing can be performed within the particular sensing system, but will commonly be done in the in-vehicle computer 121. The amount of processing done to generate the intermediate results can vary depending on the embodiment, such as generating a 3-D point cloud for each modality or at some earlier stage of processing.

[0069] The intermediate outputs of the different modalities are then compared at 407 by the processing circuits for each of the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time to generate intermediate outputs. For the main embodiments presented here, this comparison is a majority voting, where this can be done in the comparison/majority voting block 321 , which can be part of the Drive DNNs 123 of the in-vehicle computer 121. Although represented as separate block in FIG. 3 based on functional grounds and for explanatory purpose, the generation of the intermediate outputs and comparison/majority voting can be executed on different processors or the same processor, where this is again a design choice.

[0070] Based on the results of 407, control inputs are generated for the electromechanical systems used for the operation of the autonomous vehicle or other autonomous system at 409. As represented in FIG. 1 , the control inputs can be generated in the one or more processing circuits involved in mapping, path planning 125 and the control block 127. The control inputs are then provided to the electron mechanical systems (steering 131 , braking 133, speed control 135, acceleration control 137, engine monitoring 139) at 411 , which are then used at 413 to control the autonomous system. Although shown as a single sequence, it will be understood that, while the autonomous system is in operation, this will be an ongoing process with the steps cycling over the period of operation.

[0071] FIG. 5 is a more detailed flow of an embodiment for multi-modal signal processing. The flow of FIG. 5 starts at 501 and relates to the operation of the control logic operating on the one or more processors of the in-vehicle computer 121 and the control circuitry of, in this example, the lidar system 107 as represented in more detail in FIG. 3. In this embodiment, the initial input at 503 is the calibrated time-sharing polarization and non-polarization digital signals from the receiver 305 are calibrated and the corresponding data labelled.

[0072] At 505 the one or more processors of the in-vehicle computer determines whether multiple modalities of the sensor signal with non-zero accuracy weights are available in order to perform time share signal processing for the different modalities, such as for each of the for i_th polarizations. If only a single modality is available, the flow goes to the output of 515. If multiple modalities (polarization, frequencies, or both) are available, at 507 a point cloud is generated for each i_th modality. In the representation of FIG. 3, this is generated by the signal processing circuitry 301 that can be part of the drive DNNs 123. These point clouds can also be used to train and update the learning-to-learn policy of these neural networks with meta-data for the point clouds, such as accuracy weights for the different modalities’ point clouds. Given variations in the external environment, in some embodiments not all modalities may be equally weighted. A hierarchical modality is one way to reduce signal noise such as by, for example, relying more on the lidar system 107 than the camera system 103 in a foggy environment. The learning-to-learn policy within the driven DNNs can be hierarchical too.

[0073] The embodiment of FIG. 5 introduces a hierarchical modal operation at 509, 511 , and 513. At 509 the systems decides whether or not to generate the output in a single source mode, where the decision can be based factors such as the learn policy of the neural networks and the accuracy weights. In single source mode, at 511 the output is generated by the comparison in the compa re/majority voting block 321 within the drive DNNs 123 as an intermediate fused modality perception for next level modality process, labelling the corresponding learning-to-learn policy for the neural networks. In this case, the result is not a majority voting, but the fusing of the intermediate results. When the flow instead goes to 513, at 513 a finalized modality perception is generated, such as by a majority vote between the different modalities, with the corresponding learn-to-learn being labelled.

[0074] The output is provided at 515, with the operation log being flushed and the determined data structures, such as key values (KVs), stored as signatures to the local storage for the processor or processors for future verification and reuse, after which the flow ends at 517.

[0075] To place the sort of multi-modal majority voting described above in context, it should be noted that, when incorporated into an autonomous vehicles or other autonomous systems, this may be one of a number of other redundancies using comparisons, such as majority voting, between results. For example, the comparisons/majority voting described above looks at such comparisons for individual sensor systems (such as the lidar system 107), but the in-vehicle computer 121 will also compare the results of the different sensor systems of the sensors of 101. Additionally, for systems that require a high degree of reliability, such as autonomous vehicles or other autonomous systems, processor redundancy can be used. FIG. 6 can be used to described ways in which the multi-modal operation described above can integrated into a system incorporating processor redundancy.

[0076] FIG. 6 illustrates a triple modular redundant architecture in which three CPUs are run in parallel in a lockstep manner and the resultant outputs are compared. This redundancy can provide error detection and correction, as the output from the parallel operations can be compared to determine whether there has been a fault. Each of CPU-A 601 , CPU-B 603, and CPU-C 605 are connected to the debug unit 611 and over the bus structure 617 to RAM 615 and to the flash memory 613 or other storage memory for the system, where these components can largely operate in a typical manner. The debug unit 611 can be included to test and debug programs running on the CPUs and allow a programmer to track its operations and monitor changes in resources, such as target programs and the operating system.

[0077] CPU-A 601 , CPU-B 603, and CPU-C 605 are operated in parallel, running the same programs in a lockstep manner under control of the internal control 607. Each of CPU-A 601 , CPU-B 603, and CPU-C 605 can be operated on more or less the same footing and are treated with equal priority. The outputs of the three CPUs go to a majority voter block 609, where the logic circuitry within majority voter 609 compares the outputs. In this way, if the output from one of the CPUs disagrees with the other two, the majority result is provided as the system output from the majority voter 609. Although shown as three CPUs operating in parallel, more generally these can of other processor types, such as graphical processing units GPUs, or parallel multi-processor such systems, such as a set of three CPU-GPU pairs operated in parallel.

[0078] It is important to note that the multi-modal majority voting described above is different, and independent of, the multi-processor lockstep majority voting described with respect to FIG. 6. In FIG. 6, multiple processor paths use the same input and operate in parallel with their outputs then being compared in majority voter 609. The process described with respect to FIGs. 1-5 uses multiple different inputs (different polarizations or other modalities) to determine intermediate output, which are then compared as in the majority voting process. The multi-modal process can be done in a single processor (or processor system). For example, referring to FIG. 6, in one embodiment each of CPU-A 601 , CPU-B 603, and CPU-C 605 could independently perform the process described with respect to FIGs. 1-5, where the output from each parallel path then also subjected to the comparison of majority voter 609. In alternate embodiments, the different modalities could be spread across different ones of CPU- A 601, CPU-B 603, and CPU-C 605, where part or all of the majority voting between the modalities could be part of the operation of majority voter 609.

[0079] FIG. 7 is a high-level block diagram of one embodiment of a more general computing system 700 that can be used to implement various embodiments of the processing systems described above. In one example, computing system 700 is a network system 700. Specific devices may utilize all of the components shown, or only a subset of the components, and levels of integration may vary from device to device. Furthermore, a device may contain multiple instances of a component, such as multiple processing units, processors, memories, transmitters, receivers, etc.

[0080] The network system may comprise a computing system 701 equipped with one or more input/output devices, such as network interfaces, storage interfaces, and the like. The computing system 701 may include a central processing unit (CPU) 710 or other microprocessor, a memory 720, a mass storage device 730, and an I/O interface 760 connected to a bus 770. The computing system 701 is configured to connect to various input and output devices (keyboards, displays, etc.) through the I/O interface 760. The bus 770 may be one or more of any type of several bus architectures including a memory bus or memory controller, a peripheral bus or the like. The CPU 710 may comprise any type of electronic data processor. The CPU 710 may be configured to implement any of the schemes described herein with respect to the accuracy of autonomous vehicles and other autonomous systems of Figures 1-6, using any one or combination of elements described in the embodiments. The memory 720 may comprise any type of system memory such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, the memory 720 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs.

[0081] The mass storage device 730 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus 770. The mass storage device 730 may comprise, for example, one or more of a solid-state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like.

[0082] The computing system 701 also includes one or more network interfaces 750, which may comprise wired links, such as an Ethernet cable or the like, and/or wireless links to access nodes or one or more networks 780. The network interface 750 allows the computing system 701 to communicate with remote units via the network 780. For example, the network interface 750 may provide wireless communication via one or more transmitters/transmit antennas and one or more receivers/receive antennas. In an embodiment, the computing system 701 is coupled to a local-area network or a wide-area network for data processing and communications with remote devices, such as other processing units, the Internet, remote storage facilities, or the like. In one embodiment, the network interface 750 may be used to receive and/or transmit interest packets and/or data packets in an ICN. Herein, the term “network interface” will be understood to include a port.

[0083] The components depicted in the computing system of FIG. 7 are those typically found in computing systems suitable for use with the technology described herein, and are intended to represent a broad category of such computer components that are well known in the art. Many different bus configurations, network platforms, and operating systems can be used.

[0084] The technology described herein can be implemented using hardware, firmware, software, or a combination of these. Depending on the embodiment, these elements of the embodiments described above can include hardware only or a combination of hardware and software (including firmware). For example, logic elements programmed by firmware to perform the functions described herein is one example of elements of the described lockstep systems. A CPU and GPU can include a processor, FGA, ASIC, integrated circuit or other type of circuit. The software used is stored on one or more of the processor readable storage devices described above to program one or more of the processors to perform the functions described herein. The processor readable storage devices can include computer readable media such as volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer readable storage media and communication media. Computer readable storage media may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. A computer readable medium or media does (do) not include propagated, modulated or transitory signals.

[0085] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a propagated, modulated or transitory data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

[0086] In alternative embodiments, some or all of the software can be replaced by dedicated hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Applicationspecific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), special purpose computers, etc. For example, some of the elements used to execute the instructions, such as an arithmetic and logic unit (ALU), can use specific hardware elements. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more computer readable media/ storage devices, peripherals and/or communication interfaces.

[0087] It is understood that the present subject matter may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this subject matter will be thorough and complete and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications and equivalents of these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the following detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be clear to those of ordinary skill in the art that the present subject matter may be practiced without such specific details.

[0088] Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0089] The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The aspects of the disclosure herein were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure with various modifications as are suited to the particular use contemplated.

[0090] For purposes of this document, each process associated with the disclosed technology may be performed continuously and by one or more computing devices. Each step in a process may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device.

[0091] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMS What is claimed is:

1. An autonomous vehicle, comprising: an electro-mechanical control system configured to receive control inputs and control operation of the autonomous vehicle in response thereto; a sensor system configured to sense a plurality of different modalities of an electromagnetic sensor signal over a period of time; and one or more processing circuits connected to the electro-mechanical control system and the sensor system and configured to: receive, from the sensor system, the plurality of modalities of the electromagnetic sensor signal as sensed over the period of time; generate, from the plurality of modalities of the electromagnetic sensor signal as sensed over the period of time, an intermediate output for each of the modalities of the electromagnetic sensor signal as sensed over the period of time; perform a comparison of the intermediate outputs generated from the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time; and based on results of the comparison, generate and provide the control inputs to the electro-mechanical control system.

2. The autonomous vehicle of claim 1 , wherein, in comparing the intermediate outputs generated from the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time, the one or more processing circuits are configured to perform majority voting operations between the intermediate outputs.

3. The autonomous vehicle of either of claims 1 or 2, wherein the one or more processing circuits are further configured to: determine whether to operate in a multi-modality mode or singlemodality mode, where the intermediate outputs are generated and compared in response determining to operate in a multi-modality mode; and

24 in response to determining to operate in single-modality mode, generate and provide the control inputs to the electro-mechanical control system based on a single modality of the electromagnetic sensor signal over the period of time.

4. The autonomous vehicle of any of claims 1-3, wherein the sensor system includes a receiver configured to sense the plurality of modalities of the electromagnetic sensor signal.

5. The autonomous vehicle of claim 4, wherein the receiver includes a plurality of polarization filters, the plurality of different modalities corresponding to the electromagnetic sensor signal as filtered by the plurality of polarization filters.

6. The autonomous vehicle of either of claim 4 or 5, wherein the receiver includes a plurality of frequency filters, the plurality of different modalities corresponding to the electromagnetic sensor signal as filtered by the plurality of frequency filters.

7. The autonomous vehicle of any of claims 1-6, wherein the sensor system includes a transmitter configured to emit the plurality of modalities of the electromagnetic sensor signal.

8. The autonomous vehicle of any of claims 1-7, wherein the sensor system includes a lidar system.

9. The autonomous vehicle of any of claims 1-8, wherein the sensor system includes a radar system.

10. The autonomous vehicle of any of claims 1-9, wherein the sensor system includes a visual spectrum camera system.

11. The autonomous vehicle of any of claims 1 -10, wherein the sensor system further includes a sonar system.

12. The autonomous vehicle of any of claims 1-11 , wherein the sensor system is further configured to sense multiple modalities of a sonar signal.

13. The autonomous vehicle of claim 12, wherein the multiple modalities of the sonar signal include different frequencies.

14. The autonomous vehicle of any of claims 1-13, wherein the electromechanical control system includes a steering system.

15. The autonomous vehicle of any of claims 1-14, wherein the electromechanical control system includes a speed control system.

16. A method of controlling an autonomous system, comprising: sensing by a sensor system of a plurality of different modalities of an electromagnetic sensor signal over a period of time; receiving, at one or more processing circuits from the sensor system, the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time; generating, by the one or more processing circuits from the plurality of modalities of the electromagnetic sensor signal as sensed over the period of time, an intermediate output for each of the modalities of the electromagnetic sensor signal as sensed over the period of time; performing, by the one or more processing circuits for each of the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time, a comparison of the intermediate outputs generated from the plurality of different modalities of the electromagnetic sensor signal as sensed over the period of time; based on the comparison, generating by the one or more processing circuits of control inputs for an electro-mechanical control system of the autonomous system; providing the control inputs to the electro-mechanical control system; and controlling of the autonomous system by the electro-mechanical control system in response to the control inputs.

17. The method of claim 16, wherein performing the comparison of the intermediate outputs includes majority voting between the intermediate outputs.

18. The method of any of claims 16-17, further comprising: determining whether to operate in a multi-modality mode or single-modality mode, wherein generating the intermediate outputs is in response determining to operate in a multi-modality mode; and in response to determining to operate in single-modality mode, generating and providing the control inputs to the electro-mechanical control system based on a single modality of the electromagnetic sensor signal over the period of time.

19. The method of any of claims 16-18, further comprising: emitting by the sensor system of the plurality of different modalities of the electromagnetic sensor signal over the period of time during which the autonomous system is in operation.

20. The method of any of claims 16-19, further comprising: filtering the plurality of different modalities of the electromagnetic sensor signal by a plurality of different polarization filters, wherein the plurality of different modalities of the corresponding sensor signal correspond to different polarizations of the electromagnetic sensor signal.

21. The method of any of claims 16-20, further comprising: filtering the plurality of different modalities of the electromagnetic sensor signal by a plurality of different polarization filters, wherein the plurality of different modalities of the corresponding sensor signal correspond to different frequencies of the electromagnetic sensor signal.

22. The method of any of claims 16-21 , wherein the sensor system includes a lidar system.

23. The method of any of claims 16-22, wherein the sensor system includes a radar system.

27

24. The method of any of claims 16-23, wherein the sensor system includes a visual spectrum camera system.

25. A control system for autonomously operable equipment, comprising: one or processing circuits configured to: receive, from a sensor system, a corresponding sensor signal as sensed in a plurality of different modalities over a period of time; perform a majority voting between the plurality of different modalities of the corresponding sensor signal as sensed over the period of time; and based on results of the majority voting, generate and provide control inputs for an electro-mechanical control system for the autonomously operable equipment.

26. The control system of claim 25, further comprising: the electro-mechanical control system, wherein the electro-mechanical control system are configured to receive the control inputs and to control the operation of the autonomously operable equipment in response thereto.

27. The control system of any of claims 25-26, further comprising: the sensor system, wherein the sensor system includes a receiver configured to sense the corresponding sensor signal in the plurality of different modalities.

28. The control system of claim 27, wherein the receiver includes a plurality of polarization filters, the plurality of different modalities corresponding to the sensor signal as filtered by the plurality of polarization filters.

29. The control system of claim 28, wherein the receiver includes a plurality of frequency filters, the plurality of different modalities corresponding to the sensor signal as filtered by the plurality of frequency filters.

30. The control system of any of claims 25-29, wherein the sensor system includes a transmitter configured to emit the corresponding sensor signal.

28

31. The control system of any of claims 25-30, wherein the sensor system includes a lidar system.

32. The control system of any of claims 25-31 , wherein the sensor system includes a radar system.

33. The control system of any of claims 25-32, wherein the sensor system includes a visual spectrum camera system.

34. The control system of any of claims 25-33, wherein the sensor system includes a sonar system.

35. The control system of any of claims 25-34, wherein the autonomously operable equipment is an autonomous vehicle.

36. The control system of any of claims 25-35, wherein the autonomously operable equipment is robotic equipment.